Berk Bayri

The people who build an AI model know what it can do

A common misconception about AI models and evaluation, tested against the evidence.

The myth
AI companies understand their models well enough to know, before release, what those models can and cannot do.
The reality
Model builders know how they trained a model, not everything it learned. Capabilities and failure modes are discovered by testing after training, and some only surface once the model is in wide use. A vendor's description is a starting point, not a guarantee.

Explanation and evidence

The argument

A bridge engineer can tell you the load a bridge was built to carry. A drug maker has to list known side effects before the product reaches a pharmacy.

It is natural to assume the companies behind large AI models hold the same kind of knowledge about what they ship. They do not, and several of them say so openly.

These models are not written instruction by instruction. They are trained on enormous amounts of data, and what they end up able to do is an outcome of that training, not a specification someone typed in. Builders control the ingredients and the process. They do not get a complete map of the result.

So the knowledge comes afterwards. Teams test a finished model to find out what it does well, where it fails and what it should not be allowed to do. That testing is serious work. It is also, by its nature, a sample.

The builders know how the model was made. What it can do is something they are still finding out, just earlier than you.

Why people believe it

Most technology we buy has been designed to a specification, and vendors speak about AI products in the same confident language: the model "can" summarise contracts, "will" catch fraud, "does not" leak data.

Release notes, model cards and tables add to the impression of a fully characterised product. They are useful documents. They describe what was measured, which is not the same as everything that is there.

What the evidence says

Anthropic's chief executive, Dario Amodei, wrote in April 2025 that people outside the field are often alarmed to learn that "we do not understand how our own AI creations work." In the same essay he described models showing unexpected emergent behaviours that builders struggle to predict or rule out.

Reporting from Axios in September 2026 described the leading labs as still racing to understand their newest systems, with OpenAI's chief executive acknowledging that no one fully understands the consequences of how capable the models are becoming.

Research on capability testing points the same way. are meant to estimate what a model can do, yet abilities that were present but not visible have repeatedly been drawn out of models, sometimes long after release.

None of this means the models are uncontrolled or that vendors are hiding something. It means the honest description of a model is "this is what we found", not "this is all it does".

The better question

Do not ask a vendor only what the model can do. Ask how they found out, what they have not tested, and what they learned about it after release.

Then test it yourself on your own data and your hardest cases. Where a decision cannot be undone, about someone's health, money or job, keep the final decision with a person who can explain it.

Sources

The Urgency of Interpretability

Dario Amodei

Behind the Curtain: AI creators race to understand their creations

Axios · 2026-09-04

The Elicitation Game: Evaluating Capability Elicitation Techniques

arXiv · 2025-02-04

Related reading

Related misconceptions