Decision instrument
A pilot or test designed to reduce uncertainty about a decision, not to show that something works. A good decision instrument can end in no.
A decision instrument is a test built to inform a choice. Its success is measured by how much uncertainty it removes, including when the answer is that the idea should not go ahead. A demonstration shows that something is possible; a decision instrument shows whether it should be funded, scaled or stopped.
This changes how a pilot is designed. It needs a question it can answer, observations that could go against the idea, and an owner who will act on the result. It also needs to expose the human work required to keep the system dependable, rather than hide that work inside the demonstration.
A pilot that cannot stop, or that changes its own rules whenever the evidence arrives, is not functioning as a decision instrument. See AI pilot and kill criteria.
Related terms
AI pilot
A limited trial of an AI use case to test whether it works in practice. A pilot proves one team can succeed; capability starts when others can reproduce the result.
Kill criteria
Conditions agreed before a pilot starts that stop, pause or redesign it. They turn a pilot into an experiment instead of a way to keep spending.
Second-team test
A test of whether a different competent team can reproduce a pilot's useful result without inheriting the first team's exceptional conditions or constant expert help.
Support decay
The fall in exceptional help a transferred capability needs from its original team over time — the real measure of whether an AI pilot has become an organizational capability.
Used in these essays
A pilot with no kill criteria is not an experiment
A pilot is not an experiment if every result can be used to justify continuing. The missing design element is the authority to stop.
The wrong questions about AI right now
Many of the questions that helped us orient ourselves around generative AI are now too blunt to be useful. The harder work is no longer asking what AI is in the abstract, but specifying where it works, where it fails, what authority it should have, and what the whole system costs.
Keep the AI ideas you rejected
A new model release should not restart your AI roadmap. It should reopen only the ideas that were rejected for a constraint the release actually changed.
Make the AI vendor demo fail
A polished AI demo proves that a system can succeed under prepared conditions. A buying decision needs different evidence: what happens when the system is wrong, blocked, uncertain or halfway through an action.
The hidden metric in AI automation is supervision
As AI moves from assisting work to leading it, hours saved stop telling the whole story. The scarce resource shifts to human supervision: approvals, exceptions, context and judgment.
The benchmark needs a runtime address
AI systems are increasingly dynamic at runtime. Enterprise evaluation should qualify the serving route, harness, tools and fallback conditions, not just the model name.
Your AI data agent is creating a second dataset
AI data agents do more than answer business questions. At scale, the pattern of questions, corrections and dead ends becomes a new signal: a map of what the organization is trying to understand, where its definitions are weak, and which decisions deserve better infrastructure.
Your chatbot is borrowing from the next interaction
AI customer service is usually measured one interaction at a time. But a failed automated interaction can change which channel a customer chooses next time. That makes future adoption part of the economics, not a separate trust metric.
The cost of solving the wrong problem well
Good execution cannot rescue a problem that was never worth solving. It usually makes the mistake more expensive.