Evaluation
A structured test of whether an AI system does its job under real conditions, including the tools, context and fallbacks it will actually run with. A benchmark score is one input, not the verdict.
An evaluation asks whether an AI system is fit for a specific job. It should be run against the task, the data, the tools and the fallback conditions the system will really use. Testing only the model name answers a narrower question than the one the business needs answered.
Public benchmarks are useful but unstable. A few points on a leaderboard can disappear when the task, the prompt, the tool harness or the context changes. So a result like "the model passed our evaluation" is a weak statement unless it says which configuration passed.
For consequential decisions, the evaluation should leave versioned evidence that can reconstruct the serving configuration that was tested, and it should say when a material runtime change makes that evidence expire. See benchmark and runtime address.
Related terms
Benchmark
A standardised test used to compare AI models. Useful as evidence about someone else's system, but only trustworthy for you when it describes the exact serving route, harness and conditions you will run.
Runtime address
The full description of the system that was actually evaluated — model version, serving route, harness, tools, precision and operating conditions — not just the model's name.
Serving route
The actual path a request takes to be answered — which model, hardware, precision, routing and fallback rules handled it — as opposed to the model name a vendor advertises.
Reasoning model
A type of AI model built to work through problems in intermediate steps before answering. Like 'foundation model' or 'agent', it names a category, not an explanation of what a system does.
Used in these essays
A pilot with no kill criteria is not an experiment
A pilot is not an experiment if every result can be used to justify continuing. The missing design element is the authority to stop.
Five AI metrics I would remove from an executive dashboard
Licenses, prompts, pilots, hours saved and benchmark scores can all rise while the business stands still. The executive dashboard needs a different contract.
Stop asking what AI can do. Ask what your company can absorb.
The gap that matters is between what AI can produce and the changes an organization can reliably absorb into ordinary work.
The 10-minute test for whether an AI use case has a business case
Before calculating AI ROI, trace how an accepted output changes a real operating decision and creates captured value.
Your AI agent estate is becoming a capital allocation problem
Once agents multiply across workflows, the management problem changes: which agents deserve more capacity, which should be merged or constrained, and which should be retired?
Design systems need an interaction policy layer for AI
Components can keep an AI experience visually on-system while the agent is behaviorally off-policy. Design systems now need reusable rules for what AI may present, decide, execute, defer, expose as uncertain, confirm, reverse and escalate.
Your AI agent can follow the objective and still break the company
AI agents can satisfy a local objective while violating the organization’s broader constraints. The control problem is not only model alignment; it is management design.
Stop Adopting AI
AI adoption is rising faster than enterprise value because companies keep installing new intelligence inside old operating models.
OpenAI Decisions API turns probability into policy
The provocative part of OpenAI's Decisions API is not that AI can make choices. It is that a model score can quietly become an action — and an action can quietly become policy.
Cahit Arf’s 1959 question about thinking machines
In a 1959 text on thinking machines, Cahit Arf moved from clocks and relays to a harder question: what happens when a machine meets a problem that was not anticipated when it was built?
The wrong questions about AI right now
Many of the questions that helped us orient ourselves around generative AI are now too blunt to be useful. The harder work is no longer asking what AI is in the abstract, but specifying where it works, where it fails, what authority it should have, and what the whole system costs.
A cheaper AI model can move the cost instead of removing it
A lower model bill can hide a higher workflow bill. The useful AI TCO question is not only what got cheaper, but where the cost moved.
The second team is the real innovation test
A successful pilot proves that one team could make something work. Organizational capability begins when a different team can reproduce the useful result without inheriting the original team's exceptional conditions.
Keep the AI ideas you rejected
A new model release should not restart your AI roadmap. It should reopen only the ideas that were rejected for a constraint the release actually changed.
Make the AI vendor demo fail
A polished AI demo proves that a system can succeed under prepared conditions. A buying decision needs different evidence: what happens when the system is wrong, blocked, uncertain or halfway through an action.
The first AI incident report should be incomplete
OpenAI's new misalignment disclosure framework exposes a useful enterprise design principle: record anomalous AI behavior before the organization has finished explaining it. Otherwise incident systems quietly become filters for what teams already understand.
The hidden metric in AI automation is supervision
As AI moves from assisting work to leading it, hours saved stop telling the whole story. The scarce resource shifts to human supervision: approvals, exceptions, context and judgment.
The next AI interface may never be seen
Agents are turning software capabilities into an interface of their own. The next enterprise design problem is deciding what should be callable, by whom, and under which boundaries.
The benchmark needs a runtime address
AI systems are increasingly dynamic at runtime. Enterprise evaluation should qualify the serving route, harness, tools and fallback conditions, not just the model name.