Innovation & Capability7 min read·By , Independent AI Advisor & Fractional AI Lead·

The most expensive AI mistake may happen after the pilot succeeds

A successful pilot rarely fails at the moment of success. The costly error is the chain of hard-to-reverse commitments the announcement sets off before production evidence exists.

Most talk about AI failure looks at the trial. The more expensive failure can begin on the day the trial is declared a success. On that day a number leaves the project team. It might be a time saving, a deflection rate or a cost per case. Within weeks it is converted into a volume commitment in a vendor contract, a smaller hiring plan, a lower budget baseline, a target in a board update. Each conversion is reasonable alone. Together they form a cascade sized to the 's headline figure, not to what production will deliver.

This essay assumes the pilot already passed its review. The question here comes later: what is that yes allowed to commit the company to, and in what order?

Success is a bundle, not one decision

"Approve production" sounds like one choice. It releases several commitments at once, and they do not cost the same to undo.

  • A pilot can be undone in days. Switching off a feature flag is cheap.
  • A budget line can be undone in a quarter.
  • A contract can take a year or more. A minimum-volume or multi-year term turns an experiment into a fixed cost.
  • Removed capacity can take longer. Once the team, the old process or the know-how is gone, rebuilding it is slow.
  • A public target is hard to recall. Once it has gone to a board, investors or customers, retreating has its own price.

These tiers are my generalization; real exit terms vary by contract and country. The pattern is what matters: the commitments that deliver the business case's savings are usually the hardest to reverse, so they tend to be released first.

What the public evidence supports

I have two sources. Neither tests the mechanism I describe, and I use them only for what they report.

IBM's CEO study, published on 6 May 2025, surveyed 2,000 CEOs between February and April 2025. IBM reported that only 25% of AI initiatives had delivered the expected return over the previous few years, and only 16% had scaled enterprise-wide. These are CEO self-assessments of initiatives of every kind, not a pilot-to-production conversion rate. They do suggest that launching an initiative and earning its promised return at scale are separate achievements.

IBM Study: CEOs Double Down on AI While Navigating Enterprise Hurdles

IBM Newsroom · 2025-05-06

Klarna is the second example. It was not a classic pilot-then-scale case, but it shows a headline figure travelling into a workforce decision. In February 2024 Klarna said its assistant had handled two-thirds of customer service chats in its first month, equivalent to 700 full-time agents, with an expected $40 million profit improvement that year. The same report noted a hiring freeze outside engineering. In May 2025 CEO Sebastian Siemiatkowski told Bloomberg that cost had been too predominant a factor, that quality had suffered, and that Klarna was recruiting so customers could always reach a human. A Klarna spokesperson said the assistant still handled about two-thirds of inquiries.

This is not a story of AI failing and being abandoned. The assistant stayed. What changed was the plan around it: a headcount decision made on early numbers was partly rebalanced about fifteen months later. The reasons are the CEO's own account, and I cannot say from outside whether a slower sequence would have cost less.

Klarna froze hiring because of AI. Now it says its chatbot does the work of 700 full-time staff

Fortune Europe · 2024-02-28

Klarna reinvests in human talent for customer service as AI chatbot handles two-thirds of inquiries

Customer Experience Dive (reporting Bloomberg interview of 8 May 2025) · 2025-05-09

The proposal: a commitment ledger

Before the post-pilot meeting, ask for one page listing every commitment the success would trigger. This is my proposed practice, not an established standard. Each line gets four columns.

  1. The commitment. For example: sign a three-year volume agreement, cut next year's hiring plan, retire the legacy process, announce a savings target.
  2. Cost of reversal. Days, a quarter, a year or more, or not recoverable. Include the cost of rebuilding any capacity that is removed.
  3. The evidence that releases it. A figure observed in production, with ordinary teams and ordinary cases, not the pilot cohort. The second-team test and the supervision cost belong here.
  4. The owner who can say no. Someone who can hold the commitment back without the sponsor's permission.

Three rules follow.

Order by reversibility. Spend the cheapest-to-undo items first: a larger cohort, a bounded extension, a short contract term. Irreversible items come last, and only against production evidence.

Keep the alive until the new path has earned it. Do not remove the human team or the old process until the production figure has held through a normal period, awkward cases included. Running both costs money; treat it as the price of an option.

Announce the gate, not the number. If a target must go public, state it as conditional on the production result. Otherwise people must defend the pilot figure instead of test it.

A hypothetical, invented for illustration

Imagine an invoice-matching assistant that cut handling time by 40% in a pilot run by finance operations' strongest people. This is not a real company or result.

CommitmentCost of reversalReleases when
Extend to two more teams for a quarterDaysImmediately
Three-year volume contractA year or moreTwo quarters of production cost per matched invoice, including review and exceptions
Remove two planned hiresA quarter, then slow to rebuildException rate holds on ordinary invoices handled by the second and third teams
Retire the manual processHard to rebuildFallback tested, owner named, audit trail accepted by the controller
Announce a savings figureNot recoverableAfter the contract and hiring gates, stated as a result, not a forecast

The 40% is not wrong. It answers a smaller question than the right-hand column.

The counter-argument

Waiting costs money. Every cautious month is a month of unrealized savings, vendors reward commitment with better prices, and a team that sees leadership hedge may lose the energy that made the pilot work.

All true. The ledger does not say go slowly. It puts the price of delay and the price of on the same page. If a volume commitment saves 20% per unit, set that against being locked in when usage lands at half the forecast. Often the better move is ramp terms, step-down rights or a shorter first term at a higher unit price. That is what buying reversibility costs. Some commitments should be made early. Decide that line by line, on purpose.

What I would do on Monday

Take the last AI pilot your organization called a success. List every commitment it triggered and mark each with its cost of reversal. If the most expensive came first, or none is tied to a production figure, you have found the part of the success most likely to cost you later.

Where this connects

Sources and limits

  • IBM's findings are CEO self-reports about many kinds of initiatives, gathered February to April 2025 by the IBM Institute for Business Value with Oxford Economics. They are not a pilot failure rate.
  • Klarna's 2024 figures are company claims as reported by Fortune and others; I could not open Klarna's own release. The 2025 statements are the CEO's, reported from a Bloomberg interview; I did not read the Bloomberg original. The case does not isolate why Klarna's results changed.
  • The commitment ledger, the reversal tiers, the ordering rules and the invoice-matching table are my proposed framework and an invented illustration, not findings from these sources.

Get new essays as they publish.

One email per essay. No noise between.

Berk Bayri

Independent AI Advisor & Fractional AI Lead

I help leadership teams decide where AI belongs, test it before it scales and build the teams that run it, drawing on 25+ years of building digital products, experiences and capabilities.