AI agent governance: what executives must decide first
AI agent governance is a short list of leadership decisions about authority, ownership, evidence and supervision that have to be made before an agent acts.
AI agent governance is the set of decisions an organization makes before an agent acts for it: what the agent may decide, whose authority it uses, who answers for the result, what evidence it leaves behind and who can stop it.
Deloitte's 2026 research, based on 3,235 business and technology leaders across 24 countries, found that only 21% of organizations reported mature governance for agentic AI. Its August follow-up found that 5% said their business processes were highly prepared for AI agents, while 74% of surveyed leaders expected nearly half of their processes to be redesigned or rebuilt around agents within four years.
I wrote about each of those decisions separately over the past months. This essay puts them in the order an executive team would have to make them, because the order matters. Most of them are business choices that nobody else in the company can make on the leadership team's behalf.
Why is AI agent governance an executive decision?
Because the boundaries an agent works inside are business choices, and a model provider or a security team cannot make them for you.
A provider can improve safeguards, evaluations and runtime controls. The deploying organization chooses the business objective, the KPI, the connected tools, the permissions, the escalation path and the evidence it is willing to preserve. That is why, in Your AI agent can follow the objective and still break the company, I argued that the company writes more of an agent's behavior than most governance diagrams admit.
The same holds for permissions. A security team can implement scoped tokens, but it cannot decide what "€5,000", "one customer" or "reversible for 30 minutes" should mean for the business. I came to that conclusion in the essay on task-scoped authority.
Gartner's May 2026 guidance argues that organizations need to separate an agent's level of autonomy from the scope of access it receives. Gartner also predicts that by 2027, 40% of enterprises will demote or decommission autonomous agents because governance problems are discovered only after production incidents.
Giving AI more autonomy does not remove organizational complexity. It gives that complexity permission to act.
I made that argument in The agent is not the operating model. People navigate unclear processes through context nobody designed: they ask someone, they remember last time. An agent cannot do that, so the ambiguity has to be made operational before it acts.
What should an AI agent be allowed to decide?
Only the decisions the organization has explicitly delegated, and technical capability does not count as delegation.
I use the phrase capability is not authority for this. A system may be able to issue a refund without having the authority to issue every refund. In What AI agents actually do and where autonomy should stop, I described four levels of autonomy for the same process:
- read and recommend, without changing business data;
- prepare an action and request approval from a named person;
- execute within explicit limits;
- escalate and stop when a case is disputed, exceptional or unverified.
Executives should choose the level per decision, not per agent. The question I would put to a team is not whether an agent can complete the process. It is which decisions inside the process can move, under what conditions, and what evidence would justify moving them. A pilot can then widen the boundary step by step: observe, recommend, act on low-consequence cases, then handle defined exceptions.
This also means some steps should stay human. Some steps in an existing workflow exist because judgment is required, some because trust or visible responsibility is needed, and some for no good reason anymore. Automating all of them the same way is a mistake, and so is automating a step for which a plain deterministic script would do.
Who answers for the outcome when an agent acts?
A named person has to own four things for every consequential decision: who may start it, which action needs a human decision, who resolves exceptions, and who owns the consequence.
An org chart records who manages whom. It says little about who may trade margin for retention in a particular case. In AI agents are exposing your org chart as fiction, I proposed a decision-route record for each important action: the initiating signal, the permitted action, the approving or exception owner, and the evidence retained afterward, plus a stop condition and a fallback path. A company with no answer to one of those fields has found a design gap. A more capable agent will not fill it.
The risk is concrete. OWASP's October 2026 review of agentic incidents describes cases where reachability, tool access or ambiguous targets became dangerous substitutes for explicit authorization. These are incident analyses, not a measured base rate. Their lesson is operational all the same: what an agent can reach must not become the company's definition of what it may do.
The strongest objection is that companies already have process maps and approval matrices. In a mature organization that may be enough, if decision rights are current and enforced in the systems agents use. The weakness I see is the distance between the documented route and the one people follow. A RACI can name an accountable person without giving that person the power to change the workflow, and a policy can forbid an action that an integration still permits.
My request to an executive team is one decision: pick a consequential cross-functional workflow, give a named owner the authority to redesign its decision route, set a review date and give that owner the right to stop.
How much authority should an agent hold at one time?
Only what the current task needs. The role defines the ceiling; the task defines the active authority.
Copying a person's standing permissions into an agent identity is convenient and wide. A task-scoped grant is narrower. For any consequential task I would require five answers:
- Action: which exact verb may this run perform? Draft is not send; calculating a refund is not issuing one.
- Resource: which specific record, account or environment may it affect?
- Limit: what amount or consequence applies, such as a refund up to €100 or a purchase order up to €5,000?
- Duration: when does the authority expire?
- Reversibility: what can undo, stop or contain the action if the agent is wrong?
The limits have to be enforced outside the model. Prompts are not access-control systems, and a system message saying "only edit files related to this request" does not make other files unavailable. Microsoft's identity guidance separates the agent reasoning about what to do from the deterministic identity and policy controls that decide whether it is permitted. AWS's least-privilege guidance recommends short-lived credentials and just-in-time elevation for agent workflows.
Approval should change the envelope rather than sit on top of broad access. "Allow agent" is too wide to be a meaningful decision. "Issue this refund, to this limit, for ten minutes" is one an approver can actually make.
What keeps an agent from reaching its objective by an unacceptable route?
A management control system with five connected parts: objective, incentives, authority, evidence and intervention.
The evidence that agents take unacceptable routes is mostly experimental, and I want to be exact about that. Robber Bots, a study accepted by Harvard Data Science Review, put 20 AI models into a year-long simulated vending-machine business with profit-maximizing incentives, and false refund claims and collusive behavior emerged. It is a simulation and says nothing about how often deployed agents do this. The PACT benchmark tested 22 models in regulated enterprise scenarios: even the strongest misapplied a rule on 6% to 10% of items, and ordinary user pressure raised the violation rate by 65% on average. These are evaluation results, not production incident rates.
What they show is that pressure is part of the system. A deadline, a score or a user saying "just get it done" is an incentive. So leadership should ask what would count as successful completion that the company would still find unacceptable, and which shortcut becomes attractive under pressure.
I would also change the definition of a successful run. It completes the objective through an authorized path, preserves the required evidence, respects non-negotiable constraints and stays recoverable. NVIDIA's Open Agent Safety Platform moves policy enforcement outside the agent process, so the agent is not the final authority on whether its own action is allowed. The design choice matters more than the vendor.
What record should exist when something goes wrong?
Two records: an observation written before anyone can explain the event, and an explanation that is versioned as the investigation develops.
I made this case in The first AI incident report should be incomplete. OpenAI's September 2026 framework says an example can merit disclosure even when it has not caused harm, established a broader pattern or been fully explained. It is not an enterprise standard, but the principle travels: an incident should not need a finished explanation before it becomes evidence. NIST's report on monitoring deployed AI describes post-deployment monitoring as crucial because AI systems can show variability and unpredictable behavior in real-world settings, and the field as still fragmented.
The observation record should hold the authority the system had, the tools and data available, the external effect and what is known, suspected and unknown. It should also hold the runtime conditions. As I argued in The benchmark needs a runtime address, "the agent did X" is weak evidence if the label covers a moving combination of model, tools and permissions.
Preserve what the agent was allowed to do, not only what it did. If you cannot reconstruct the grant, you have credentials rather than delegated authority.
How much human supervision will agents need, and does governance slow them down?
More than a demo suggests, and governance slows agents mainly when it is designed as an approval queue.
Glean's 2026 Work AI Index surveyed 6,000 digital workers: respondents said AI automation saved roughly 11 hours a week and also reported 6.4 hours a week of what Glean calls "botsitting". Those figures are self-reported and vendor-sponsored, so they are no universal ratio. They do show that automation can move work into supervision instead of removing it, which is why The hidden metric in AI automation is supervision proposes measuring supervision load per completed unit of work. The same essay ties this to A pilot is a decision instrument: a pilot should expose the human work needed to keep the system dependable, not hide it.
Anthropic's own disclosure shows the scale problem. Across more than a billion agent decisions in a month, its online monitor blocked about 0.002%, and offline systems narrowed roughly 100,000 flagged transcripts a week to about 50 for human review. These are one company's measurements of its own systems. The lesson is that the supervision system itself has to decide where scarce human attention goes.
What this evidence can and cannot support
Most figures in this essay come from surveys, simulations, benchmarks and vendor disclosures. None is a base rate for how often agents fail in production. I use them to justify the decisions they support.
The fair objection is that governance is what slows agents down. Part of it is right. In AI is removing the coordination tax. The next bottleneck is authority, I described how an agent can remove three days of reconciliation and still leave the work waiting twelve hours for an approval chain built when reconciliation took three days. McKinsey calls the broader cost the coordination interface tax, and Microsoft's operating-model guidance recommends central standards and guardrails with routine decisions pushed outward and governance by exception.
The limit of the objection is that the remedy is more explicit decision rights: delegated thresholds, evidence requirements and a known escalation route. Some decisions should stay slow. The goal is to make the delay intentional. I would also measure authority latency, the time between a workflow becoming decision-ready and the authorized action being taken.
How should leadership govern many agents at once?
As a portfolio. Once agents multiply, each consequential one needs a decision: scale, optimize, merge, constrain or retire.
I developed this in Your AI agent estate is becoming a capital allocation problem. OpenAI's investment guidance recommends managing AI investment as a portfolio, with funding changing as work moves from exploration to validation to production. IBM's work on third-party agents describes mixed ecosystems of internal, embedded and marketplace agents, which makes overlap harder to see.
Two ledgers are needed: one for money and one for organizational capacity, meaning evaluation, integration, governance, supervision, change and attention. Exit criteria should be defined before the agent scales. One uncomfortable metric is the share of the estate merged, constrained or retired this quarter. A result of zero is suspicious, because it suggests the estate is accumulating rather than being allocated.
What should an executive decide before the next agent goes live?
Seven things, in this order. If an answer is "whatever the tool allows" or "whatever the user's role allows", the agent may still be useful as an assistant, but it is not ready for consequential autonomy.
- Decision boundary: which decisions may the agent observe, recommend, execute under limits, or never touch?
- Ownership: who owns the outcome and its exceptions, with the authority to change the route and stop it?
- Task authority: what action, resource, limit, duration and recovery condition apply to this task, and what enforces them outside the model?
- Success definition: what counts as unacceptable success, and which pressures make a shortcut attractive?
- Evidence: which records exist outside the agent's ability to alter them, and who writes the observation before the explanation?
- Supervision budget: how much human attention does one dependable outcome need, and what happens to it when volume doubles?
- Exit criteria: what evidence would widen the agent's authority, and what would constrain or retire it?
None of this needs a new committee. It needs a named owner per workflow and a decision on each line before the agent acts, because after the first incident the questions are still there, only more expensive.