How Many Steps Should an AI Agent Be Allowed to Take Before Human Review?

Geoffrey Chen

After AI Enters the Workflow · Season One: “From Answering Questions to Participating in Work” · Article 8

An AI agent is asked to resolve an unpaid invoice. It checks the customer record, finds the invoice, reads prior correspondence, compares the account with policy, drafts a reminder and schedules it for delivery. Six actions have occurred without a person intervening.

In another workflow, an agent searches public websites, opens twenty pages and prepares a summary. It may take fifty actions, yet none changes an external record.

Which agent has been allowed to go too far? A rule such as “require approval every ten steps” cannot answer. The sixth step in the first workflow may communicate a demand under the organisation’s name; the fiftieth step in the second may remain a reversible reading operation.

As agents gain the ability to select tools and pursue goals over time, organisations need boundaries for autonomous action. But step count is at best a rough operational signal. Control should be designed around changes in permission, consequence, uncertainty and reversibility.

Agents differ from ordinary automation

Conventional automation normally follows a path specified in advance. If a field has a particular value, run a known procedure. The designers can inspect the branches and restrict the system to a narrow state space.

An AI agent may choose which information to seek, which tool to call and how to revise its plan after an unexpected result. Anthropic describes trustworthy agent design in terms of systems that exercise discretion over process and tool use while pursuing a task. Anthropic, “Trustworthy agents in practice”

This discretion is the source of usefulness. A rigid script fails when the document is in an unexpected place; an agent can search. A script stops when a field is missing; an agent can infer what to ask next. But discretion also enlarges the number of paths the system may take, including paths the designer did not explicitly anticipate.

Agent control therefore cannot consist only of testing one successful demonstration. It must constrain what the system can observe, decide, change and communicate across many possible trajectories.

Risk depends on four dimensions, not a single number

The first dimension is permission. Reading a public page differs from reading a confidential personnel file; drafting an email differs from sending it; proposing a database update differs from committing it.

The second is consequence. A mistaken search wastes time. A mistaken transfer, account closure or public statement can cause financial, legal or reputational harm.

The third is uncertainty. Some tasks have objective success criteria. Others require interpretation, disputed facts or normative judgement. An agent should not receive the same autonomy when it cannot tell whether the task has been completed correctly.

The fourth is reversibility. A local draft can be discarded. A message sent to thousands of people cannot be fully recalled. A deleted record may be recoverable from backup; a disclosed secret cannot be made unknown.

Step count can interact with these dimensions, but it does not replace them. Ten low-permission, reversible steps may be safer than one irreversible step with broad authority.

Error can grow faster than the number of steps

Multi-step action creates the possibility of compounding error. If each stage assumes that the previous one was correct, an early mistake changes the context for every later decision.

The agent may retrieve the wrong customer, then correctly analyse the wrong account, generate a perfectly worded notice and send it through the authorised channel. Each local action looks coherent. The trajectory as a whole is wrong.

Longer task horizons also make evaluation more difficult. METR studies AI capabilities partly through the length of tasks systems can complete with specified reliability. The research is not a direct workplace autonomy rule, but it highlights that successful operation over longer horizons is a measurable capability with changing limits. METR, “Research—Time Horizon 1.1”

Organisations should not convert a capability estimate into permission automatically. A system that can complete a long technical task in an evaluation may still lack the organisational knowledge, data quality and authority needed for a real decision. Capability asks what the system can do under test conditions; governance asks what it should be allowed to do here.

Asking at every step is not effective control

The opposite extreme is to interrupt the user constantly: “May I open this file? May I compare these values? May I continue?” Frequent confirmation appears safe but can make oversight meaningless.

People habituate to low-value prompts and click through them. The system transfers attention costs to the user without explaining the significance of the decision. Approval becomes a ritual rather than judgement.

Human oversight should occur where human knowledge or authority can change the outcome. A reviewer needs a concise account of what the agent has done, which evidence it used, what uncertainty remains and what consequence the next action will create. The person must have time, competence and the power to refuse.

Article 14 of the EU AI Act expresses these qualities for high-risk systems: overseers should be able to understand capacities and limitations, remain aware of automation bias, interpret outputs and disregard, override or interrupt the system. EU AI Act, Article 14

An approval box without those conditions does not create effective human control.

A controlled action envelope

Instead of choosing a universal number of steps, define an action envelope for each task. The envelope should state:

  • the objective and permitted sub-tasks;
  • the data sources the agent may access;
  • the tools and operations it may use;
  • spending, volume, time and retry limits;
  • actions that are draft-only;
  • actions that always require approval;
  • conditions that require escalation or termination;
  • the evidence and activity trace returned to the reviewer;
  • how changes can be rolled back.

Within a narrow envelope, the agent may take many steps. Crossing a boundary—moving from read to write, internal to external communication, low to high monetary value, reversible to irreversible action—should trigger a checkpoint regardless of count.

Least privilege is essential. An agent that only needs to draft a refund recommendation should not receive authority to issue refunds. Tool design can enforce this distinction by separating “prepare” from “commit” operations and by requiring approval tokens or independent services for consequential actions.

NIST’s AI Risk Management Framework encourages organisations to map the context and manage risk throughout deployment. Applied to agents, this means that autonomy is a property of the whole socio-technical arrangement—model, tools, data, permissions, monitoring and users—not of the model alone. NIST, “AI Risk Management Framework 1.0”

Human oversight must change as agents improve

Better agents do not simply remove the need for supervision. They change its timing and form.

When a system is unreliable at routine steps, people must intervene frequently. As routine reliability improves, review can move towards boundaries, exceptions and sampled audits. But higher capability may also allow the agent to attempt more consequential work, increasing the importance of permission design and incident response.

Anthropic’s study of agent autonomy in practice distinguishes, among other things, how much systems act without approval and how users configure autonomy. Such evidence is useful because nominal product capability does not tell us what permissions people actually grant. Anthropic, “Measuring AI agent autonomy in practice”

A mature organisation should monitor both sides: what agents are capable of doing and what their deployed configurations permit. Changes to models, tools or data can alter the risk of an existing workflow even if its written policy has not changed.

Conclusion: ask whether the next action crosses a control boundary

There is no generally safe number of autonomous steps. The relevant boundary is not between step nine and step ten. It is between different kinds of action and consequence.

The practical rule is:

Allow an agent to continue while it remains inside a defined, observable and reversible action envelope. Require human judgement before it crosses into broader permission, material consequence, unresolved uncertainty or irreversible action.

Step limits may still be useful as circuit breakers. A large number of retries can reveal confusion, looping or unexpected cost. But they should complement, not replace, semantic boundaries.

An organisation retains control when it can explain the agent’s objective, restrict its tools, observe its path, identify boundary crossings, stop execution and repair the effects. It loses control when the system is allowed to keep acting merely because no fixed step count has yet expired.

Primary sources and further reading

Continue reading: Explore the After AI Enters the Workflow series.

Discover more from Geoffrey Chen

Subscribe to get the latest posts sent to your email.