Berk Bayri

Benchmark

A standardised test used to compare AI models. Useful as evidence about someone else's system, but only trustworthy for you when it describes the exact serving route, harness and conditions you will run.

An AI benchmark is a fixed set of tasks and a scoring method used to compare models: reasoning problems, coding challenges, question answering and so on. Leaderboards built on benchmarks have become the shorthand for "which model is best".

Benchmarks are real evidence, but about a specific thing. A leaderboard is evidence about someone else's system, under someone else's conditions. A model score without the path that produced it is increasingly hard to trust, because the same model name may be served through different routes and wrapped in different harnesses.

Using benchmarks well

  • Ask which exact system was tested: the runtime address.
  • Check whether the serving route and harness match what you will deploy.
  • Keep failure in the score, not just the average.
  • Treat the evidence as expiring when the route changes materially.

"Which model is best?" is the wrong question. The better one is which configured system performs best on your workload at your acceptable quality, latency, privacy, cost and recovery constraints. A benchmark also says little about adaptation beyond what it tests.

Read more in The benchmark needs a runtime address.

Used in these essays

AI Strategy & Transformation11 min read

How to build an AI strategy: decisions, not a tool list

An AI strategy is a set of decisions about workflows, ownership, evidence and stopping, and this guide walks through them in the order I would take them.

AI Agents & Systems13 min read

AI agent governance: what executives must decide first

AI agent governance is a short list of leadership decisions about authority, ownership, evidence and supervision that have to be made before an agent acts.

AI Strategy & Transformation7 min read

The question your board should ask about AI is not “what are we doing?”

A list of AI projects tells a board what management chose to launch. It does not show where AI already shapes decisions nobody approved.

AI Strategy6 min read

AI is making intelligence cheap. Judgment is not getting cheaper.

Better and more accessible AI answers do not settle which questions matter, which evidence counts, or who owns a decision.

Problem Framing & Decision Design6 min read

Your Synthetic Customer Cannot Tell You What You Forgot to Ask

Synthetic customers can test assumptions, but they cannot reveal what your research never asked.

AI Agents & Systems13 min read

AI agents are exposing your org chart as fiction

An org chart tells you who manages whom. An agent can reveal who actually authorizes a decision, rescues an exception, and carries the consequences.

Problem Framing & Decision Design12 min read

How to tell if an AI pilot deserves to survive

A pilot earns another investment only when its result can survive ordinary work, a credible alternative, and an owner who can carry it without the pilot team.

Problem Framing & Decision Design10 min read

A pilot with no kill criteria is not an experiment

A pilot is not an experiment if every result can be used to justify continuing. The missing design element is the authority to stop.

AI Strategy & Transformation11 min read

Five AI metrics I would remove from an executive dashboard

Licenses, prompts, pilots, hours saved and benchmark scores can all rise while the business stands still. The executive dashboard needs a different contract.

Innovation & Capability11 min read

Stop asking what AI can do. Ask what your company can absorb.

The gap that matters is between what AI can produce and the changes an organization can reliably absorb into ordinary work.

Problem Framing & Decision Design4 min read

The 10-minute test for whether an AI use case has a business case

Before calculating AI ROI, trace how an accepted output changes a real operating decision and creates captured value.

AI Governance & Evaluation13 min read

Your AI agent can follow the objective and still break the company

AI agents can satisfy a local objective while violating the organization’s broader constraints. The control problem is not only model alignment; it is management design.

AI Agents & Systems14 min read

OpenAI Decisions API turns probability into policy

The provocative part of OpenAI's Decisions API is not that AI can make choices. It is that a model score can quietly become an action — and an action can quietly become policy.

AI Agents & Systems14 min read

OpenAI Dots is a test of whether AI can carry a goal, not just complete a task

Dots matters less as another capable assistant than as a test of persistent delegation: can AI keep carrying a goal without giving the user a new system to manage?

Problem Framing & Decision Design10 min read

Cahit Arf’s 1959 question about thinking machines

In a 1959 text on thinking machines, Cahit Arf moved from clocks and relays to a harder question: what happens when a machine meets a problem that was not anticipated when it was built?

Problem Framing & Decision Design14 min read

The wrong questions about AI right now

Many of the questions that helped us orient ourselves around generative AI are now too blunt to be useful. The harder work is no longer asking what AI is in the abstract, but specifying where it works, where it fails, what authority it should have, and what the whole system costs.

AI Strategy & Transformation9 min read

A cheaper AI model can move the cost instead of removing it

A lower model bill can hide a higher workflow bill. The useful AI TCO question is not only what got cheaper, but where the cost moved.

Problem Framing & Decision Design8 min read

Keep the AI ideas you rejected

A new model release should not restart your AI roadmap. It should reopen only the ideas that were rejected for a constraint the release actually changed.

AI Strategy & Transformation7 min read

Make the AI vendor demo fail

A polished AI demo proves that a system can succeed under prepared conditions. A buying decision needs different evidence: what happens when the system is wrong, blocked, uncertain or halfway through an action.

AI Governance & Evaluation8 min read

The first AI incident report should be incomplete

OpenAI's new misalignment disclosure framework exposes a useful enterprise design principle: record anomalous AI behavior before the organization has finished explaining it. Otherwise incident systems quietly become filters for what teams already understand.

Innovation & Capability10 min read

The hidden metric in AI automation is supervision

As AI moves from assisting work to leading it, hours saved stop telling the whole story. The scarce resource shifts to human supervision: approvals, exceptions, context and judgment.

AI Agents & Systems9 min read

The next AI interface may never be seen

Agents are turning software capabilities into an interface of their own. The next enterprise design problem is deciding what should be callable, by whom, and under which boundaries.

AI Governance & Evaluation9 min read

The benchmark needs a runtime address

AI systems are increasingly dynamic at runtime. Enterprise evaluation should qualify the serving route, harness, tools and fallback conditions, not just the model name.

AI Strategy & Transformation9 min read

Your AI data agent is creating a second dataset

AI data agents do more than answer business questions. At scale, the pattern of questions, corrections and dead ends becomes a new signal: a map of what the organization is trying to understand, where its definitions are weak, and which decisions deserve better infrastructure.

AI Strategy & Transformation8 min read

Your chatbot is borrowing from the next interaction

AI customer service is usually measured one interaction at a time. But a failed automated interaction can change which channel a customer chooses next time. That makes future adoption part of the economics, not a separate trust metric.