The most expensive AI mistake may happen after the pilot succeeds
A successful pilot rarely fails at the moment of success. The costly error is the chain of hard-to-reverse commitments the announcement sets off before production evidence exists.
Most talk about AI failure looks at the trial. The more expensive failure can begin on the day the trial is declared a success. On that day a number leaves the project team. It might be a time saving, a deflection rate or a cost per case. Within weeks it is converted into a volume commitment in a vendor contract, a smaller hiring plan, a lower budget baseline, a target in a board update. Each conversion is reasonable alone. Together they form a cascade sized to the pilot's headline figure, not to what production will deliver.
This essay assumes the pilot already passed its review. The question here comes later: what is that yes allowed to commit the company to, and in what order?
Success is a bundle, not one decision
"Approve production" sounds like one choice. It releases several commitments at once, and they do not cost the same to undo.
- A pilot can be undone in days. Switching off a feature flag is cheap.
- A budget line can be undone in a quarter.
- A contract can take a year or more. A minimum-volume or multi-year term turns an experiment into a fixed cost.
- Removed capacity can take longer. Once the team, the old process or the know-how is gone, rebuilding it is slow.
- A public target is hard to recall. Once it has gone to a board, investors or customers, retreating has its own price.
These tiers are my generalization; real exit terms vary by contract and country. The pattern is what matters: the commitments that deliver the business case's savings are usually the hardest to reverse, so they tend to be released first.
What the public evidence supports
I have two sources. Neither tests the mechanism I describe, and I use them only for what they report.
IBM's CEO study, published on 6 May 2025, surveyed 2,000 CEOs between February and April 2025. IBM reported that only 25% of AI initiatives had delivered the expected return over the previous few years, and only 16% had scaled enterprise-wide. These are CEO self-assessments of initiatives of every kind, not a pilot-to-production conversion rate. They do suggest that launching an initiative and earning its promised return at scale are separate achievements.
Klarna is the second example. It was not a classic pilot-then-scale case, but it shows a headline figure travelling into a workforce decision. In February 2024 Klarna said its assistant had handled two-thirds of customer service chats in its first month, equivalent to 700 full-time agents, with an expected $40 million profit improvement that year. The same report noted a hiring freeze outside engineering. In May 2025 CEO Sebastian Siemiatkowski told Bloomberg that cost had been too predominant a factor, that quality had suffered, and that Klarna was recruiting so customers could always reach a human. A Klarna spokesperson said the assistant still handled about two-thirds of inquiries.
This is not a story of AI failing and being abandoned. The assistant stayed. What changed was the plan around it: a headcount decision made on early numbers was partly rebalanced about fifteen months later. The reasons are the CEO's own account, and I cannot say from outside whether a slower sequence would have cost less.
The proposal: a commitment ledger
Before the post-pilot meeting, ask for one page listing every commitment the success would trigger. This is my proposed practice, not an established standard. Each line gets four columns.
- The commitment. For example: sign a three-year volume agreement, cut next year's hiring plan, retire the legacy process, announce a savings target.
- Cost of reversal. Days, a quarter, a year or more, or not recoverable. Include the cost of rebuilding any capacity that is removed.
- The evidence that releases it. A figure observed in production, with ordinary teams and ordinary cases, not the pilot cohort. The second-team test and the supervision cost belong here.
- The owner who can say no. Someone who can hold the commitment back without the sponsor's permission.
Three rules follow.
Order by reversibility. Spend the cheapest-to-undo items first: a larger cohort, a bounded extension, a short contract term. Irreversible items come last, and only against production evidence.
Keep the fallback alive until the new path has earned it. Do not remove the human team or the old process until the production figure has held through a normal period, awkward cases included. Running both costs money; treat it as the price of an option.
Announce the gate, not the number. If a target must go public, state it as conditional on the production result. Otherwise people must defend the pilot figure instead of test it.
A hypothetical, invented for illustration
Imagine an invoice-matching assistant that cut handling time by 40% in a pilot run by finance operations' strongest people. This is not a real company or result.
| Commitment | Cost of reversal | Releases when |
|---|---|---|
| Extend to two more teams for a quarter | Days | Immediately |
| Three-year volume contract | A year or more | Two quarters of production cost per matched invoice, including review and exceptions |
| Remove two planned hires | A quarter, then slow to rebuild | Exception rate holds on ordinary invoices handled by the second and third teams |
| Retire the manual process | Hard to rebuild | Fallback tested, owner named, audit trail accepted by the controller |
| Announce a savings figure | Not recoverable | After the contract and hiring gates, stated as a result, not a forecast |
The 40% is not wrong. It answers a smaller question than the right-hand column.
The counter-argument
Waiting costs money. Every cautious month is a month of unrealized savings, vendors reward commitment with better prices, and a team that sees leadership hedge may lose the energy that made the pilot work.
All true. The ledger does not say go slowly. It puts the price of delay and the price of lock-in on the same page. If a volume commitment saves 20% per unit, set that against being locked in when usage lands at half the forecast. Often the better move is ramp terms, step-down rights or a shorter first term at a higher unit price. That is what buying reversibility costs. Some commitments should be made early. Decide that line by line, on purpose.
What I would do on Monday
Take the last AI pilot your organization called a success. List every commitment it triggered and mark each with its cost of reversal. If the most expensive came first, or none is tied to a production figure, you have found the part of the success most likely to cost you later.
Where this connects
- How to tell if an AI pilot deserves to survive decides whether the pilot continues. This essay starts after that answer is yes.
- A pilot with no kill criteria is not an experiment sets stopping rules before results exist. The ledger sets release rules after them.
- The second team is the real innovation test shows the kind of evidence the third column should demand.
- The hidden metric in AI automation is supervision covers review work that shrinks pilot savings.
Sources and limits
- IBM's findings are CEO self-reports about many kinds of initiatives, gathered February to April 2025 by the IBM Institute for Business Value with Oxford Economics. They are not a pilot failure rate.
- Klarna's 2024 figures are company claims as reported by Fortune and others; I could not open Klarna's own release. The 2025 statements are the CEO's, reported from a Bloomberg interview; I did not read the Bloomberg original. The case does not isolate why Klarna's results changed.
- The commitment ledger, the reversal tiers, the ordering rules and the invoice-matching table are my proposed framework and an invented illustration, not findings from these sources.