Benchmark
A standardised test used to compare AI models. Useful as evidence about someone else's system, but only trustworthy for you when it describes the exact serving route, harness and conditions you will run.
An AI benchmark is a fixed set of tasks and a scoring method used to compare models: reasoning problems, coding challenges, question answering and so on. Leaderboards built on benchmarks have become the shorthand for "which model is best".
Benchmarks are real evidence, but about a specific thing. A leaderboard is evidence about someone else's system, under someone else's conditions. A model score without the path that produced it is increasingly hard to trust, because the same model name may be served through different routes and wrapped in different harnesses.
Using benchmarks well
- Ask which exact system was tested: the runtime address.
- Check whether the serving route and harness match what you will deploy.
- Keep failure in the score, not just the average.
- Treat the evidence as expiring when the route changes materially.
"Which model is best?" is the wrong question. The better one is which configured system performs best on your workload at your acceptable quality, latency, privacy, cost and recovery constraints. A benchmark also says little about adaptation beyond what it tests.
Read more in The benchmark needs a runtime address.
Related terms
Runtime address
The full description of the system that was actually evaluated — model version, serving route, harness, tools, precision and operating conditions — not just the model's name.
Serving route
The actual path a request takes to be answered — which model, hardware, precision, routing and fallback rules handled it — as opposed to the model name a vendor advertises.
Harness
The software around a model that manages context, tool use, sub-agents and the execution environment. It shapes real-world capability, so it must be evaluated with the model, not ignored.
Used in these essays
How to build an AI strategy: decisions, not a tool list
An AI strategy is a set of decisions about workflows, ownership, evidence and stopping, and this guide walks through them in the order I would take them.
AI agent governance: what executives must decide first
AI agent governance is a short list of leadership decisions about authority, ownership, evidence and supervision that have to be made before an agent acts.
The question your board should ask about AI is not “what are we doing?”
A list of AI projects tells a board what management chose to launch. It does not show where AI already shapes decisions nobody approved.
AI is making intelligence cheap. Judgment is not getting cheaper.
Better and more accessible AI answers do not settle which questions matter, which evidence counts, or who owns a decision.
Your Synthetic Customer Cannot Tell You What You Forgot to Ask
Synthetic customers can test assumptions, but they cannot reveal what your research never asked.
AI agents are exposing your org chart as fiction
An org chart tells you who manages whom. An agent can reveal who actually authorizes a decision, rescues an exception, and carries the consequences.
How to tell if an AI pilot deserves to survive
A pilot earns another investment only when its result can survive ordinary work, a credible alternative, and an owner who can carry it without the pilot team.
A pilot with no kill criteria is not an experiment
A pilot is not an experiment if every result can be used to justify continuing. The missing design element is the authority to stop.
Five AI metrics I would remove from an executive dashboard
Licenses, prompts, pilots, hours saved and benchmark scores can all rise while the business stands still. The executive dashboard needs a different contract.
Stop asking what AI can do. Ask what your company can absorb.
The gap that matters is between what AI can produce and the changes an organization can reliably absorb into ordinary work.
The 10-minute test for whether an AI use case has a business case
Before calculating AI ROI, trace how an accepted output changes a real operating decision and creates captured value.
Your AI agent can follow the objective and still break the company
AI agents can satisfy a local objective while violating the organization’s broader constraints. The control problem is not only model alignment; it is management design.
OpenAI Decisions API turns probability into policy
The provocative part of OpenAI's Decisions API is not that AI can make choices. It is that a model score can quietly become an action — and an action can quietly become policy.
OpenAI Dots is a test of whether AI can carry a goal, not just complete a task
Dots matters less as another capable assistant than as a test of persistent delegation: can AI keep carrying a goal without giving the user a new system to manage?
Cahit Arf’s 1959 question about thinking machines
In a 1959 text on thinking machines, Cahit Arf moved from clocks and relays to a harder question: what happens when a machine meets a problem that was not anticipated when it was built?
The wrong questions about AI right now
Many of the questions that helped us orient ourselves around generative AI are now too blunt to be useful. The harder work is no longer asking what AI is in the abstract, but specifying where it works, where it fails, what authority it should have, and what the whole system costs.
A cheaper AI model can move the cost instead of removing it
A lower model bill can hide a higher workflow bill. The useful AI TCO question is not only what got cheaper, but where the cost moved.
Keep the AI ideas you rejected
A new model release should not restart your AI roadmap. It should reopen only the ideas that were rejected for a constraint the release actually changed.
Make the AI vendor demo fail
A polished AI demo proves that a system can succeed under prepared conditions. A buying decision needs different evidence: what happens when the system is wrong, blocked, uncertain or halfway through an action.
The first AI incident report should be incomplete
OpenAI's new misalignment disclosure framework exposes a useful enterprise design principle: record anomalous AI behavior before the organization has finished explaining it. Otherwise incident systems quietly become filters for what teams already understand.
The hidden metric in AI automation is supervision
As AI moves from assisting work to leading it, hours saved stop telling the whole story. The scarce resource shifts to human supervision: approvals, exceptions, context and judgment.
The next AI interface may never be seen
Agents are turning software capabilities into an interface of their own. The next enterprise design problem is deciding what should be callable, by whom, and under which boundaries.
The benchmark needs a runtime address
AI systems are increasingly dynamic at runtime. Enterprise evaluation should qualify the serving route, harness, tools and fallback conditions, not just the model name.
Your AI data agent is creating a second dataset
AI data agents do more than answer business questions. At scale, the pattern of questions, corrections and dead ends becomes a new signal: a map of what the organization is trying to understand, where its definitions are weak, and which decisions deserve better infrastructure.
Your chatbot is borrowing from the next interaction
AI customer service is usually measured one interaction at a time. But a failed automated interaction can change which channel a customer chooses next time. That makes future adoption part of the economics, not a separate trust metric.