A pilot with no kill criteria is not an experiment
A pilot is not an experiment if every result can be used to justify continuing. The missing design element is the authority to stop.
The most revealing question in an AI pilot review may not be whether the prototype worked. It may be: what result would have made you stop?
There is a familiar moment in an enterprise AI programme. A pilot has not produced the improvement it promised, but it has produced something else: a credible demo, a handful of enthusiastic users, a list of integration problems and a sponsor who can explain why the next phase will be different. The team asks for a few more weeks. It is a perfectly plausible request. Sometimes another iteration is exactly right.
The problem is that the same request can be made after almost any result. If the model performs well, continue. If it performs badly, continue so it can improve. If adoption is high, continue because demand is proven. If adoption is low, continue because training is needed. When all observations support the same decision, the project is no longer learning in a way that can change its own future.
This is not an argument for impatiently killing work. It is an argument for giving a pilot a real right to fail before the organisation becomes invested in its survival.
I have written elsewhere that a pilot is a decision instrument. That essay asks what a pilot is for. This one asks a narrower, more uncomfortable question: how do you stop a pilot from changing its own rules whenever the evidence arrives?
A pilot can be technically successful and still deserve to end
Suppose a team tests an AI assistant that drafts responses to customer disputes. The assistant produces fluent drafts and staff prefer starting from them. These are useful observations. But the team still needs to know whether the accepted resolution rate improves, whether difficult cases are escalated correctly, how much checking is required and what the workflow costs when operated by an ordinary team.
Imagine that draft production falls from six minutes to two, while human verification rises from two minutes to seven. The numbers are illustrative, not an observed study. The visible task has become faster; the whole task has become slower. The assistant may still have value for certain case types, but the original efficiency claim is not supported.
Without a stopping rule, the project can celebrate draft speed and quietly move the verification burden out of the frame. With a stopping rule, the team must either end that claim or formally propose a different experiment. That distinction is a governance mechanism, not a punishment.
The UK's 2026 Test and Learn guidance makes a related point in the public-sector context: define outcomes and critical assumptions early, test them proportionately, and use evidence to decide whether to continue, adapt or stop. It also warns against allowing invested effort to become the reason for continuing. That is guidance for government evaluation, not a validated corporate AI kill-criteria template. The operating rule below is my application of the principle.
Three different reasons to stop need three different rules
An executive review often reduces the outcome to a traffic light. Red, amber or green. That compresses decisions which have very different consequences.
A safety stop is not an economic stop. If the system exposes sensitive information, makes an unauthorized external action or crosses an agreed severity threshold, the correct response may be to suspend the trial immediately. You do not wait for the next quarterly value review. The NIST AI Risk Management Framework's Manage guidance includes monitoring, override, incident response, recovery and decommissioning as parts of operational AI risk management. NIST does not prescribe the numerical limits for your particular pilot. Those must be established in context.
An economic stop means the value proposition no longer clears its own hurdle. If the best plausible end-to-end result, including integration, correction, supervision and support, cannot beat the agreed alternative by enough to justify the cost and risk, the original business case is exhausted. A team may propose a new, smaller scope; it should not be allowed to keep the old benefit claim alive through increasingly selective measurement.
A learning stop means the pilot can no longer answer its question. Perhaps the participating cases are no longer representative, the workflow changes halfway through, or the team has collected too little valid data to distinguish the options. In that case, the result may be inconclusive rather than negative. Continuing the same test under the same design may only create more noise. Pause, redesign the evaluation or end it. Do not call an inconclusive result success.
These are different kinds of stop, with different owners. Safety may require immediate operational authority; economics belongs with the budget owner; evaluation validity belongs with someone who can challenge the method. A sponsor can be consulted on all three without being the only person empowered to exercise them.
Write the decision before you know the answer
A credible kill criterion is not "stop if ROI is poor." It is a small contract linking an observable condition to an authorized action. I would record six fields before releasing the pilot budget: the business claim, the comparison, the evidence window, the threshold or boundary, the decision owner, and the next action.
For a hypothetical dispute-handling assistant, that contract might say:
An illustrative pre-commitment
The test covers 200 eligible disputes across agreed case types, with a comparison group or a documented baseline. A serious confidentiality incident suspends live use immediately. At the planned review, expansion requires an improvement in accepted resolutions per paid hour without exceeding the pre-agreed quality and escalation limits. If the evidence is too imprecise to decide, the evaluation owner may authorize one bounded redesign; additional rollout is not the default.
Those quantities are examples, not statistically validated thresholds. In practice, sample size, precision and review timing should be designed with an evaluator rather than selected because a round number fits the slide. The important discipline is that the conditions and decision rights exist before the sponsor knows whether they are convenient.
The UK guidance on evaluating AI interventions distinguishes impact on outcomes from technical model assessment and encourages evaluation design before implementation. A benchmark score may support a technical decision. It does not, on its own, establish that a new service model creates value.
There is another trap: reviewing noisy numbers every day until one happens to cross a favourable threshold. Repeated looks at the same outcome, without a suitable statistical design, can make a conventional significance claim unreliable. A stopping rule should specify planned reviews, minimum evidence and how interim safety checks differ from the final outcome test. "Stop immediately if people may be harmed" and "scale once the experiment demonstrates value" are not the same statistical problem.
The person who can stop the pilot should not need permission from the person who sold it
Most companies can write a criterion. Fewer can make it operational when a senior sponsor wants to continue.
The failure mode is structural. The sponsor has approved the budget, defended the initiative and attached their credibility to it. The team has learned the tool, negotiated with the vendor and built internal dependencies. None of that proves the next investment is justified. Yet it makes stopping feel like losing work that has already been done.
Research on escalation of commitment in information-systems projects describes this problem through multiple theoretical lenses. It is not evidence that any particular AI pilot will fail, nor does it provide a universal kill threshold. It does support treating continuation decisions as vulnerable to pressures beyond the current expected value of the project.
My preferred design is to separate advocacy, evaluation and continuation authority. The team building the pilot should be able to explain why it might work. The evaluator should be able to say what the evidence can and cannot support. The budget owner should have to authorize the next tranche rather than letting funding roll forward automatically. For high-risk cases, the safety owner must have the power to suspend use without waiting for a business-case meeting.
A stop rule with no empowered owner is a slide, not a control.
Changing the hypothesis is allowed. Changing it invisibly is not.
Rigid criteria can be bad management. New information sometimes reveals that the original hypothesis was wrong but an adjacent opportunity is real. A pilot for drafting disputes might fail to save time but show a promising improvement in consistency for one narrow case category. It would be foolish to forbid further investigation simply because the first plan was imperfect.
The answer is not to freeze every assumption. It is to version the decision. Close the first test against its original criteria. Record what was learned, including the result that did not meet the threshold. Then write a new claim, a new comparison and a new spending limit for the narrower opportunity. Treat it as a redesign with its own approval, not as proof that the first pilot succeeded after all.
This also protects the people doing the work. When a failed assumption can become a legitimate learning result, teams have less reason to conceal it. A cancelled trial can leave behind reusable evaluation data, failure cases and an improved understanding of the workflow. Those are assets, but they should not be priced as realised business benefits.
Do not turn stopping into another vanity metric
There is a fashionable answer to endless pilots: celebrate how many you kill. That can be as misleading as celebrating how many you launch. A team rewarded for termination can rush weak evidence just as a team rewarded for launches can inflate activity.
The useful unit is a decision made with enough evidence for its consequences. Sometimes that means stop. Sometimes it means one more bounded test. Sometimes it means scale within a limited set of conditions and continue monitoring. The evidential burden should rise with the irreversibility, cost and risk of the next commitment.
This is why the executive dashboard problem matters. A count of pilots is operational telemetry. A record of what the organisation learned, what it decided and which resources it redirected is management evidence. The two should not be confused.
The 2026 UK Test and Learn annex is useful here precisely because it does not equate iteration with indefinite continuation. It describes a progression from exploring assumptions to testing and, only when justified, growing. That is a public-policy method, not a guarantee that a corporate AI programme using it will succeed. Its transferable idea is that adaptation needs governance, not just enthusiasm.
A better question for the next steering committee
Before approving the next AI pilot, ask the sponsor to complete this sentence:
"If we observe ___ by ___, then ___ has the authority to stop ___, and the next tranche of funding will not be released unless ___."
If the sentence cannot be completed, the team may have a useful prototype plan. It may have a persuasive sales case. It does not yet have a controlled investment experiment.
The point of kill criteria is not to make organisations less ambitious. It is to make ambition less dependent on its own momentum. A pilot earns its continuation by changing what the company knows. It should not earn it merely by continuing to exist.
Sources and evidence boundaries
- HM Treasury / Evaluation Task Force, Test and Learn (2026). Public-sector evaluation guidance updated May 2026; supports pre-agreed outcomes, decision points and adaptation. The six-field corporate contract is Berk's synthesis.
- UK Evaluation Task Force, Guidance on the Impact Evaluation of AI Interventions (2024, current HTML companion). Guidance distinguishes impact evaluation from model assurance. Does not prescribe a universal corporate threshold.
- NIST AI RMF Playbook, Manage function. Voluntary risk-management guidance on monitoring, override, incident response and decommissioning; not a pilot ROI method.
- Desai and Chulkov, Escalation of Commitment in MIS Projects: A Meta-Analysis. Earlier research synthesis on commitment dynamics in information-systems projects; not a contemporary AI-pilot prevalence estimate.
- All numerical examples in the essay are illustrative scenarios, not observed corporate outcomes. The decision framework and distinctions between stop types are Berk's editorial synthesis.