AI incident reporting
Recording anomalous AI behaviour as evidence at the moment it is observed, before root cause is known, keeping what was seen separate from the later explanation of why.
AI incident reporting is the practice of documenting unexpected or harmful behaviour from an AI system. The central idea in this approach is that an AI incident should not need a finished explanation before it becomes evidence. The first report should be incomplete, and that is a feature.
Observation and explanation are different jobs
Traditional reporting often waits for a root cause. With AI systems the cause can take weeks to understand, if it is ever fully known, and meanwhile the anomalous behaviour is lost or rewritten to fit a story. Separating observation (what happened, as seen) from explanation (why we think it happened) keeps the original record difficult to improve after the fact, while the explanation can be revised as understanding grows.
Principles
- Preserve the strange thing before making it make sense.
- Version the root cause; it will change.
- Treat severity and uncertainty as separate axes: something can be minor and unexplained, or serious and well understood.
- Let the incident system act as a sensor: patterns across reports show where systems drift, where guardrails fail and where calibration is off.
This is not an argument for calling every odd output a crisis; it is an argument for capturing it.
Read more in The first AI incident report should be incomplete.
Related terms
Recovery contract
A short written artifact in an AI vendor evaluation that states how failures are detected, contained, reversed and handed back to people — and who is responsible at each step.
Calibration
How well a model's stated confidence matches reality: if it says 90% on a hundred comparable cases, roughly ninety should be correct. Once probabilities drive actions, it becomes an operating metric.
Guardrails
Controls placed around an AI system — input and output checks, permission limits, review steps and escalation rules — that keep its behaviour within acceptable bounds.
Used in these essays
The wrong questions about AI right now
Many of the questions that helped us orient ourselves around generative AI are now too blunt to be useful. The harder work is no longer asking what AI is in the abstract, but specifying where it works, where it fails, what authority it should have, and what the whole system costs.
The first AI incident report should be incomplete
OpenAI's new misalignment disclosure framework exposes a useful enterprise design principle: record anomalous AI behavior before the organization has finished explaining it. Otherwise incident systems quietly become filters for what teams already understand.