AI Agents & Systems6 min read·By , Independent AI Advisor & Fractional AI Lead·

What AI agents actually do, and where autonomy should stop

An AI agent is a system that can pursue a goal by choosing and using tools. That does not mean it should be allowed to choose its own authority.

Consider a customer who has been charged twice. They contact support, attach two payment confirmations and ask for a refund. A can explain the refund policy. An AI assistant can draft a helpful response. An agent with the right connections might inspect the transactions, check the policy, prepare a refund and update the support ticket. Whether it should actually send the money back is a different question.

That difference is the useful place to begin when asking what are ? They are software systems that use AI to pursue a goal, often by choosing actions and working with other tools over several steps. The interesting part is not that they can produce text. It is that the output of one step can determine what happens next.

An agent may decide to search a document, call an , compare a result with a rule, try a different route when a tool fails and report the outcome. Yet the word agent tells you surprisingly little about how much autonomy a particular product actually has. Some are constrained assistants; others can carry out changes in connected systems. The permissions, environment and approval rules matter more than the label.

A model, a workflow and an agent are not the same thing

A language model generates responses from the context it receives. It may be impressive at reasoning through a problem, but generating a proposal is not the same as changing anything in the world.

A conventional automation follows a prescribed path: when an invoice arrives, extract a field, copy it into an accounting system, notify the approver. The steps can be sophisticated, but the system generally operates according to a sequence a designer has specified.

An AI agent introduces a different kind of flexibility. Given a defined objective, it can choose among available tools and decide which step to attempt next, based on what it has learned so far. It might retrieve a contract before checking an invoice, or ask for missing information rather than proceeding. That flexibility is valuable when the route cannot be fully written down in advance.

It is also why an agent can be the wrong choice for a straightforward task. If a deterministic script can reliably move a record from one system to another, adding a model creates new cost, and failure modes without necessarily improving the result. The first design question is not whether a task can be done by an agent, but what uncertainty requires the system to make a choice?

Google Cloud's overview of AI agents describes goal pursuit, reasoning, and varying degrees of autonomy. These are capabilities, not a guarantee that every agent is reliable or should be trusted with every decision.

The mechanics are less mysterious than the marketing

Most contemporary agents combine a model with a loop and a collection of permitted tools. The model receives the objective and relevant context. It proposes an action, the system executes that action if allowed, and the result is returned as new context. The loop continues until the task is complete, a limit is reached or human input is needed.

Planning can be explicit or lightweight. Memory may be no more than the current task history, or it may involve retrieving stored documents and previous records. Neither elaborate planning nor a permanent memory is compulsory. In fact, keeping the scope small often makes an agent easier to test.

The refund example can be broken into observable decisions. First, retrieve the customer's authorized order history. Then compare the two transactions to establish whether a duplicate payment is plausible. Next, check the current refund policy and whether the case falls within a predefined limit. If the evidence is incomplete, request clarification. If the amount is exceptional, prepare a recommendation for a person with authority to approve it.

Notice what is absent: an instruction to "use your judgment" as a substitute for an agreed business policy. A model's confidence is not a refund authorization.

Why a successful demonstration is weak evidence

A demo usually shows the easiest path through a task: the customer exists, the records are clean, the tools respond, and the policy is unambiguous. Production work includes conflicting records, outdated permissions, service outages, malicious instructions in retrieved material and legitimate that nobody encoded.

An agent can also make a plausible series of individually reasonable moves that add up to a bad business decision. It may find the wrong customer, apply an outdated rule or retry a payment action whose first attempt succeeded but returned no confirmation. The problem is not simply whether the model can answer a question correctly. It is whether the entire system remains safe when the environment stops cooperating.

For that reason, I would test an agent against the cases a polished demonstration avoids: missing evidence, contradictory inputs, duplicate requests, unauthorized changes, interrupted tools and requests that are outside scope. Measure completed outcomes, human and recovery cost, rather than counting only successful model responses.

This is closely related to the hidden supervision cost of AI automation. A task that moves from execution to exception handling has not necessarily disappeared from the organization.

How much autonomy should an AI agent have?

Autonomy is not an on-or-off switch. For the same refund process, an agent could operate at several levels:

  • Read and recommend: inspect records and propose a resolution, without changing business data.
  • Prepare and request approval: assemble the action and evidence so a named person can authorize it.
  • Execute within explicit limits: issue a refund only when the amount, identity checks, policy conditions and rate limits are satisfied.
  • and stop: hand over disputed, exceptional or unverified cases rather than improvising permission.

The last behavior matters at every level. A well-designed agent must know when not to complete its task.

The NIST AI Risk Management Framework offers a useful governance structure for mapping risks, measuring performance and managing AI systems. It does not assign universal permission levels. Those must come from the organization, the task and the consequences of an error.

In an earlier article about task-scoped authority, I argued against giving an agent all the standing permissions of a human role. A support manager's account may allow policy changes, customer updates and refunds. An agent handling one refund needs only the narrow actions required for that refund. Its inability to reach unrelated systems is a feature.

A practical test before you build one

Take one recurring business task and describe the outcome a person actually cares about. Then write down the decisions between the request and that outcome. Some decisions will be simple rules. Some require retrieval. Some depend on missing information. A few genuinely need an adaptable sequence of actions.

Build the smallest version that can navigate that uncertainty. Give it read-only access first, if possible. Record every tool call and the evidence behind consequential recommendations. Introduce state-changing actions only after the team can answer who approved them, what limits apply and how to reverse or contain a mistake.

Compare the agent with the simplest credible alternative, which may be a conventional automation or a better interface for a human. Include the cost of monitoring, troubleshooting and maintaining the integration. If the agent cannot outperform that alternative on the outcome that matters, its novelty is not a reason to deploy it.

The question "what are AI agents?" has a technical answer: AI-enabled systems that can pursue objectives through sequences of decisions and actions. The more important organizational answer is that an agent is a new way for software to act inside boundaries someone must design and own.

The promise is not unlimited autonomy. It is useful delegation, with visible limits and a reliable way back when the task goes wrong.

Get new essays as they publish.

One email per essay. No noise between.

Berk Bayri

Independent AI Advisor & Fractional AI Lead

I help leadership teams decide where AI belongs, test it before it scales and build the teams that run it, drawing on 25+ years of building digital products, experiences and capabilities.