How to tell if an AI pilot deserves to survive

The question after a promising pilot is not whether the team made AI work. It is whether the organization can afford to keep it working when the pilot team leaves.

A pilot review often begins with a demonstration. The assistant produces an answer in seconds. The workflow looks smoother than it did six weeks ago. People who volunteered for the trial are enthusiastic, and the sponsor can already imagine the announcement: a successful AI pilot is moving into production.

Then someone from operations asks a less cinematic question. Who will fix it when the document structure changes? Who will resolve the cases it cannot handle? What happens when the person who built the prompts is on holiday? Which existing step will disappear, and which team is responsible when the answer is wrong?

These questions can sound like resistance to innovation. They are actually the questions that determine whether the innovation exists outside the conditions that produced it.

A pilot can demonstrate capability without establishing a viable operating arrangement. Its result may be real, valuable and still insufficient to justify the next investment.

I have argued that a pilot is a decision instrument, and that kill criteria must be agreed before evidence arrives. This essay starts at a different moment: the experiment has ended, some evidence is in, and the organization must decide whether to inherit the thing it built. The decision now is not simply whether the pilot passed its own test. It is whether the next version of the work deserves to exist.

The burden of proof changes after the demo

During a pilot, the team is permitted to be exceptional. It can borrow a subject-matter expert, clean an awkward dataset by hand, watch every output and hold daily conversations with the vendor. Those are sensible ways to learn quickly. They become dangerous only when the business case silently assumes they will remain free and available forever.

Consider a hypothetical AI assistant for handling supplier disputes. During a controlled trial, it drafts plausible responses and retrieves relevant contract clauses. A specialist checks every answer. The team reports a 60% reduction in drafting time.

That number is not false. It is simply answering a smaller question than the investment committee needs answered.

The committee needs to know the time to reach an accepted resolution, including review, correction, escalation and rework; the rate and severity of incorrect recommendations; whether ordinary employees can reproduce the result; and the cost of running the service when the experts return to their normal jobs. A faster draft is an intermediate output, not yet an economic outcome.

The Gartner 2027 CIO and Technology Executive Agenda survey, released on October 19, 2026, provides a timely reason to be demanding. In its survey of 2,002 technology executives, 13% reported significant or transformational AI value. Organizations with guidelines for deciding when pilots should not advance were 2.3 times more likely to fall into Gartner's high-performing 'vanguard' group; only 31% reported having those guidelines. This is a vendor-reported survey association, not proof that adopting a checklist causes superior results. Its practical implication is narrower: stopping and advancing are management decisions that deserve explicit criteria.

A survival review is my proposed way to make that decision inspectable.

Five exhibits, one decision

I would ask the pilot team to bring five exhibits to a review chaired by the future operating owner, not solely by the project sponsor. None requires a hundred-slide deck. Together they make the hidden assumptions visible.

1. The outcome, measured against something real

Name the unit of work and the alternative. Not 'AI produced 1,000 answers', but 'accepted supplier disputes resolved per paid hour, at the required quality'. The comparison might be the existing process, a simpler rules-based improvement or a human team with better search. The strongest alternative is not always the one the AI vendor chose for the demo.

If a baseline is all that is available, describe how the case mix, seasonality and staff experience might have changed. If a randomized or staggered comparison is feasible, use it. If not, acknowledge that the observed difference may not be causal. An enthusiastic pilot cohort is not a neutral counterfactual.

A useful exhibit contains the numerator, denominator, case selection, excluded cases and uncertainty. It also records what the intervention displaced. If it merely makes an unwanted report cheaper, the company has improved an activity it might have been better off ending.

2. The transfer, not just the original team's result

A pilot's first team knows the system's weaknesses by heart. That knowledge is an asset during discovery and a liability when it becomes an invisible dependency.

Ask a second competent team to perform the same workflow with the documented process, ordinary permissions and a normal support arrangement. Measure the outcomes and the amount of expert rescue required. The question is not whether the second team matches the first immediately. It is whether the result is reproducible and the exceptional support falls as the system becomes a normal capability.

This is the second-team test. It changes the evidence from 'our team can make this work' to 'another team can make this work under conditions we can afford'. If the result collapses without the original builders, production funding may be premature even when the prototype is excellent.

3. The complete operating bill

A model invoice is not the cost of the service. Add integration maintenance, evaluation, data preparation, security review, incident response, monitoring, human approval, correction, exception handling and vendor changes. Include the opportunity cost of employees whose work is interrupted to rescue a result.

Separate necessary human judgment from preventable repair. A regulated approval can be a feature of the design; repeated manual correction of an unreliable output is a defect in the operating economics. Both consume time, but they imply different decisions.

The hidden metric in AI automation is supervision. A pilot that saves six minutes of drafting while adding seven minutes of checking has not yet earned a claim of end-to-end efficiency. Those figures are illustrative, not empirical findings. The point is to trace the complete path to a dependable outcome.

The October 20, 2026 Gartner CEO survey release reports that 44% of surveyed CEOs struggle to demonstrate AI-driven financial returns. That is self-reported executive experience, not a measured failure rate for AI projects. It reinforces why the full operating bill belongs in a production decision rather than in an appendix.

4. The owner who can actually take it over

An AI system without a permanent operating owner is still a project, even if users have begun depending on it. The owner must be able to change the workflow, allocate support capacity, enforce quality boundaries and stop the service. Naming a steering committee is not the same as granting those decision rights.

I would require two signatures: the business owner who accepts responsibility for the outcome and the operational or risk owner who accepts the conditions under which the service may run. They may disagree. That disagreement is useful evidence about readiness, not a meeting to smooth over.

The NIST AI Risk Management Framework and its voluntary Playbook distinguish governance, context mapping, measurement and risk management. They do not certify a particular pilot or prescribe a universal production threshold. They support a more basic proposition: performance evidence without assigned responsibility and ongoing management is incomplete assurance.

5. The way out

A production decision creates a future obligation. The team should be able to explain what happens when model behavior changes, a vendor raises prices, a critical integration fails or an unacceptable error occurs. Which operations can fall back to the previous process? Which records must be retained? Who can suspend automation without waiting for the original sponsor?

The UK's portfolio of AI assurance techniques illustrates different real-world approaches to assessing AI systems; inclusion is not government endorsement. The point for a survival review is to choose assurance and recovery methods appropriate to the actual workflow and its potential harm, not to collect badges.

If the organization cannot describe a safe fallback, it may be buying a dependency before it understands the cost of leaving.

The decision is not pass or fail

There are three defensible outcomes, and the middle one needs the most discipline.

Fund a bounded production transition when comparative value is credible, the second-team result is reproducible enough, operating costs and harms fit the agreed limits, and a permanent owner accepts the obligation. This is permission to transition under monitored conditions, not permission for unlimited rollout. The production review should specify the next cohort, budget ceiling, operating thresholds and next decision date.

Redesign one unproven assumption when the evidence points to a specific fixable constraint. Perhaps a narrower case type works, but the broad workflow does not. Perhaps the model is sound but the handoff process is not. A redesign is not an indefinite extension. It needs a new question, a limited budget, an owner and evidence that would change the decision. If the team cannot name what the next test will resolve, it is asking for more time rather than more information.

End the pilot and preserve the learning when the expected value does not clear the alternative, the necessary conditions are too expensive or risky, or the organization has no credible path to ownership. Ending a pilot is not the same as declaring the idea permanently wrong. Record the binding constraint, the evidence and the condition that could justify revisiting it. A new model, regulation or workflow may later change that constraint. The original budget does not have to remain alive while everyone waits.

A worked survival hearing

Imagine a procurement team testing an AI tool that reads supplier onboarding documents. The pilot reports 85% correct extraction on a curated set and enthusiastic user feedback. The figures below are hypothetical and should not be treated as benchmark targets.

The operating owner asks for a second sample from ordinary incoming cases. Accuracy is lower for handwritten attachments and documents in unfamiliar formats. The pilot team has been manually cleaning files before the tool sees them. When that cleaning and the review queue are included, the time saved per accepted onboarding case is modest and uncertain.

A different team can run the workflow, but it requires repeated intervention from the original engineer. The owner is willing to support a narrow document class, not the full promised scope. Security approves the data handling for that narrow class, with a manual fallback.

The right decision is neither 'AI works, scale it' nor 'AI failed, kill it'. It is a bounded redesign: restrict the eligible document class, measure accepted cases against the existing process, budget for normal support and rerun the second-team test. The broad pilot claim ends. A narrower hypothesis gets one more chance because it is specific and falsifiable.

Now imagine that the same second trial shows no credible improvement after all review and support costs. The responsible decision is to stop, record which document types caused the problem, and redirect the budget. The organization has learned something useful without pretending the pilot must become a product.

This is the distinction between celebrating technical capability and allocating organizational capacity.

What I would put in front of the investment committee

The review should fit on one decision sheet, supported by the underlying evidence rather than replacing it.

No single numerical score should be allowed to cancel a critical failure in another column. A high return does not neutralize a privacy breach. A reliable model does not create an owner. An enthusiastic team does not make the cost of exception handling disappear.

Nor should this be treated as a universal five-item compliance standard. High-risk uses need more demanding assurance and evidence; low-risk internal tools may justify lighter evaluation. The discipline is proportionality with visible trade-offs.

The most expensive outcome is an orphaned success

Organizations rarely struggle to find people who will champion a successful demonstration. They struggle to find someone willing to inherit its permanent consequences.

That is why I would move the decisive meeting out of the pilot team's room. Let the builders explain what they learned, but let the people who must operate, fund, govern and eventually retire the service decide whether it deserves to continue. The sponsor should be able to advocate. The operating owner should be able to refuse.

A pilot earns its survival when the next team can say, with evidence: we know what this changes, we know what it costs, we know how it fails, and we are willing and able to own it.

The test of success is not whether the pilot survived its review. It is whether the review produced a better decision than enthusiasm alone would have produced.

Sources and evidence notes

• Gartner, 2027 CIO and Technology Executive Agenda survey release, October 19, 2026. Survey of 2,002 CIOs/technology executives; reported association, not causal evidence of exit criteria effectiveness.
• Gartner, CEO Survey release, October 20, 2026. Survey of 203 executives; perceptions and priorities, not measured pilot outcomes.
• NIST AI Risk Management Framework and voluntary Playbook. Governance and assurance guidance, not a certification or an ROI methodology.
• UK Government, Portfolio of AI Assurance Techniques. Illustrative assurance methods, not endorsements.
• The five-exhibit survival hearing, decision sheet and supplier-onboarding scenario are Berk Bayri's proposed decision framework and illustrative reasoning, not externally validated intervention effects.