Short answer
There is no universal dollar threshold. General-purpose AI should not perform the final action when one error could cause physical harm, irreversible transfer of money, loss of legal rights, disclosure of sensitive data, a public commitment or a large correlated impact—and the system cannot reliably intercept the anomaly before execution. Judge consequence, probability, scale of exposure, reversibility and detection time together. An extreme value on any one dimension can justify mandatory human approval.
Do not decide from a low error probability alone
Risk is often simplified to probability multiplied by loss, but automation changes speed and scale. A person might misdirect one email in a day; an agent can misdirect ten thousand in a minute. One faulty formula can repeat across every customer bill. A confidential link indexed or copied outside the organisation may never be truly recalled.
Ask at least five questions: What is the credible worst consequence of one action? How many targets can the same failure affect? Can the harm be fully reversed? How long until detection? Can affected people be notified and made whole? Strong performance on a model benchmark does not answer these operational questions.
The NIST AI Risk Management Framework treats risk in terms of potential impacts on individuals, organisations and society, and calls for governance, mapping, measurement and management in context. That supports use-specific controls, not a universal accuracy threshold. NIST AI Risk Management Framework
Separate a wrong suggestion from a wrong action
An incorrect sentence on a screen and a system sending that sentence to a customer, moving money or modifying production are different risk states. The suggestion stage leaves opportunities for a person and deterministic software to intervene. Execution converts probability into consequence.
The most useful boundary separates generation permission from action permission. A model may draft an email but not select final recipients and send it. It may prepare payment instructions but not submit them. It may propose files for deletion, while deletion requires deterministic rules, preview and approval. The fact that a tool can call an API does not mean it should receive every permission exposed by that API.
Split authority by exact action and apply least privilege, monetary or volume limits, allowlists, sandboxes, delayed queues and reversal mechanisms. For high-cost actions, model output should remain a proposal awaiting authorisation.
General models should not autonomously decide safety or basic rights
Medical treatment, hazardous equipment control, law enforcement, withdrawal of benefits, employment rejection, denial of credit and similar decisions can affect health, liberty, livelihood and equal opportunity. Treat these as high-cost even when the average impact cannot be expressed neatly in dollars.
Such uses require domain validation, applicable regulation and professional standards, an explainable process, effective appeal and an accountable decision-maker. A general generative model may organise evidence or identify questions to investigate, but fluent language does not confer final authority.
The Australian Human Rights Commission's work on human rights and technology explains how AI-assisted decision-making can affect fundamental rights and why transparency, accountability and independent oversight matter. Australian Human Rights Commission: Human Rights and Technology Final Report
Use a lower threshold for harm that cannot be completely repaired
Some actions can be reversed technically while their effects remain. A mistaken dismissal notice can be withdrawn on the same day but still cause personal and professional harm. Published health information may have been copied before deletion. A false alert may continue to affect trust or reputation after correction.
Do not assess reversibility from the presence of an Undo button. Ask whether outside recipients have acted, information has been replicated, a legal deadline has passed, or the affected person can be restored to the previous position. If complete restoration is not credible, add controls before execution rather than relying on repair afterwards.
For actions that are truly irreversible, potentially life-threatening or capable of disclosing data at scale, the reasonable default is to prohibit autonomous execution by a general-purpose model. If automation is necessary, use purpose-validated deterministic systems, redundant safeguards and formal authorisation rather than free-form generated instructions.
Scale turns a small error into a serious incident
Undercharging by one dollar may appear minor; repeated across millions of transactions, it becomes a material financial and trust problem. One inaccurate personalised suggestion may be easy to correct, but a system applying the same mistaken assumption to every user creates correlated exposure.
Set a blast radius: maximum recipients per batch, transactions per hour, files changed per operation, and data accessible to each account. Begin in a small, observable, reversible environment and expand in stages. Expansion itself should require renewed approval because it changes the worst case.
Also inspect chains of individually modest actions. Reading an email, extracting bank details, creating a payment and sending confirmation can each look routine. Combined, they form an end-to-end money transfer.
The later the detection, the less risk you can accept
An error intercepted by deterministic validation before execution causes little or no external loss. An error discovered through customer complaints weeks later can accumulate. Controls should specify detection windows and stopping times.
Use real-time rule checks, anomaly thresholds, reconciliation, receipts and incident alerts. Approve early batches item by item; after stable operation, retain random sampling and automatic stop conditions. A system that cannot reliably observe whether an action succeeded, targeted the correct object or produced an anomalous result lacks a basic prerequisite for autonomy.
“Continuous monitoring” must connect to response: who receives the alert, how quickly they act, who can pause the process and how affected parties are told. A dashboard without an operator and authority is not a control.
Use a loss budget to choose the automation level
Define risk tolerance for a particular task rather than debating whether AI is trustworthy in general. Specify maximum loss per action, cumulative daily exposure, people affected, sensitivity of data and maximum restoration time. You might allow AI to schedule non-critical internal meetings automatically but forbid external invitations; or permit purchase-order drafting up to a limit while requiring two people to approve payment.
A loss budget does not mean pricing harm and accepting it casually. It translates abstract risk into permission boundaries. When consequences cannot be estimated with confidence, treat that uncertainty as additional risk and choose less automation.
Include hidden cost: investigation, notification, legal advice, compensation, staff time, reputation and future compliance duties. Comparing minutes saved only with direct refunds systematically understates an incident.
A practical four-level authority model
At level one, AI supplies a draft or analysis but cannot change external state. This suits uncertain work that a person can judge effectively. At level two, AI prepares a precise action; a person sees its target, content and consequence and approves it individually. At level three, AI executes low-risk actions within explicit rules, strict caps, reversibility and real-time monitoring, with sampled review. At level four, AI autonomously performs high-consequence or irreversible actions.
Level four should generally remain prohibited for general-purpose generative AI. Level three still requires enforceable rules around output. A prompt that asks the model to “be careful” is behavioural guidance, not a permission boundary.
Reassess when a task changes. An internal drafting tool connected to a customer database has a new data risk. Ten automated operations per week expanded to one hundred thousand per day have a new scale risk. A model or tool-chain update may invalidate earlier tests.
Conditions that should create a hard stop
Default to human approval plus additional rule validation for payments, securities trades, legal filings, medical treatment, hazardous machinery, account privileges, bulk deletion, publication of sensitive information, dismissal or denial of an essential service. Do not execute if the final action cannot be previewed, the target's identity cannot be confirmed, the basis cannot be recorded, or the effect cannot be reversed.
Other stop signals include untrusted input, exposure to hostile external instructions, a shared high-privilege account, no test environment, no incident owner, insufficient review capacity, or uncertainty about model and prompt versions. A successful technical demonstration does not cure these governance gaps.
NIST's Generative AI Profile addresses content, information-security, human configuration and value-chain risks, emphasising measurement, documentation and continuing management rather than an assumption that a model will constrain itself. NIST AI 600-1: Generative AI Profile
Do not reduce approval to an uninformed click
A high-risk approval screen should reveal the final action, not merely say, “AI is ready—continue?” The approver needs exact recipients, amount, file changes, data scope, publication state and irreversible effects. Highlight material differences and revalidate sensitive fields.
For a batch, show total scale and anomalies; do not conceal ten thousand actions behind one button. Very high-risk action may require two approvers in different roles or a cooling-off delay that can intercept both mistakes and compromised accounts.
Approvers also need adequate time and competence. If the design makes item-level checking impossible, reduce the batch or remove execution authority.
My assessment: design the incident before granting permission
Before allowing AI to act, write three credible failure scenarios: wrong target, wrong content and hostile input. For each, state maximum impact, detection, stopping procedure, recovery and owner. If the team cannot tell these stories concretely, it does not yet understand how the system fails and should not grant autonomous authority.
The goal of good automation is not to remove every human step. It is to reduce low-value work without breaking the loss budget. One critical approval may take seconds while preserving the boundary between an editable draft and a public act. That is often a worthwhile cost.
Decision checklist
- Could the worst result affect health, safety, rights, livelihood, substantial property or sensitive data?
- How many people, transactions, files or systems can the same error reach at once?
- Can harm be completely reversed, or merely stopped from growing?
- Will an error be found before execution, within minutes or only weeks later?
- Do deterministic rules check targets, values, formats, permissions and anomalies?
- Are generation rights separated from sending, payment, deletion and publication rights?
- Can an approver see the final action and aggregate exposure, not just an AI summary?
- Are there hard caps, a stop control, an incident owner, notification and remedy?
Conclusion
The cost of error is not determined by one dollar figure or a model accuracy score. Safety and rights impacts, irreversibility, large-scale exposure, late detection and poor remedy can all lower the threshold to mandatory approval or prohibition. Constrain permission and blast radius before pursuing speed. AI can propose an action quickly; the final step that makes it real should be governed by controls proportionate to the worst credible consequence.
Related questions
- When Should AI Not Make the Decision for You?
- What Must AI Confirm Before Sending Emails, Making Payments or Editing Files?
Continue reading: All articles in How Far Should You Trust AI?
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.