Short answer
A general-purpose AI system should not hold final authority when a decision materially affects rights, health, safety, livelihood, liberty or significant property and an error would be difficult to detect, explain, challenge or reverse. AI may organise information, identify patterns and propose options. A competent person with the necessary information and genuine discretion must decide. The test is not whether a page contains a “human review” button. It is whether a person can understand the evidence, reject the system, give reasons and correct the outcome in time.
Consequence is not the only issue; correctability matters
Decision risk comes from the size of the consequence and the ability to discover and repair error. If an automatic folder suggestion is wrong, an employee can usually see and move the file. If a benefit application is rejected automatically, the person may lose income for months and not know AI participated or how to appeal.
Assess four dimensions: severity of impact; reversibility; ability of the affected person to understand and challenge; and availability of a responsible person who can decide again. A serious weakness in any dimension should reduce automation.
Australian AI adoption guidance calls for impact understanding, testing and monitoring, and maintenance of human control for each use. General capability does not grant a system authority for every decision. Australian Government: Guidance for AI Adoption—Implementation Practices
Do not give AI final authority over basic rights and opportunity
Hiring, dismissal, lending, insurance, education admission, benefits, migration, policing and housing can determine a person's opportunity and conditions of life. Historical data may carry previous discrimination. Proxy variables can reconstruct protected characteristics. A system may be unable to provide a sufficiently specific reason.
High average accuracy can coexist with a much higher error rate for one group. When an automated result is final, an affected person faces the closed explanation that “the system decided”. A genuine human decision-maker must be able to inspect complete material, consider exceptions, correct data and provide a contestable reason.
The Australian Human Rights Commission has observed that AI-informed decision-making can improve efficiency while creating human-rights harms, and emphasises transparency, accountability and independent oversight. Australian Human Rights Commission: Human Rights and Technology Final Report
This does not mean every use of classification is prohibited. It means a consequential output needs a lawful purpose, tested relevance, monitored effects and meaningful recourse rather than an opaque score treated as destiny.
Do not let general AI make final medical, legal or safety judgments
Symptoms, contracts and safety incidents depend heavily on context. A model may offer education, questions or a document summary while lacking a complete record, jurisdiction, evidentiary rule, equipment state or physical observation. Incorrect advice can arrive in a professional register.
Doctors, lawyers and safety professionals also make mistakes, but they operate under qualifications, records, procedures and duties. They can ask questions, examine facts, explain and adjust when new information appears. A chatbot does not acquire those duties by producing a recommendation.
Validated specialised systems can support professional decisions within a regulated intended use. Support and substitution are different. The final professional needs to understand the operating boundary and must not merely click accept.
Emergency contexts are particularly unsuitable for casual delegation. A person should use established emergency and professional channels rather than wait for a conversational system to infer urgency from incomplete text.
Irreversible action requires approval in advance
Transferring funds, deleting data, publishing, signing a contract, sending mass notices, changing production infrastructure or lodging a government statement converts AI output into an external action. A free-form model should not trigger irreversible or costly actions independently.
The interface should display the exact object, amount, recipient and proposed change before execution and require explicit confirmation. Add transaction limits, least privilege, dual approval or a delay window. “Execute first, notify later” is appropriate only where the action is low risk and fully reversible.
Approval should bind to the final action. If details can change after approval, the control is illusory. A person who approved a draft should not be deemed to approve a later model-edited payment or recipient list.
If the organisation cannot explain the decision, AI should not own it
Explanation does not require disclosure of every model parameter. It requires that an affected person can learn which relevant information was used, which rule or criterion mattered, where data may be wrong and how to obtain review. If the organisation cannot explain how an output entered the decision, it cannot verify accuracy or resolve a dispute effectively.
The OAIC advises that an organisation using AI in decision-making should understand how outputs are produced, have a human responsible for verifying accuracy who can overturn the decision, and be able to provide meaningful explanation. OAIC: Guidance on privacy and commercial AI products
“The model is complex” is not a sufficient explanation to the person affected. It can be a reason not to use that system for that decision.
People must choose contested objectives
AI can optimise a specified metric. It cannot legitimately decide what an institution ought to optimise. Should hiring prioritise experience or potential? Should a public service prioritise speed or equal access? Should school discipline prioritise consistency or individual circumstances? These involve values and rights.
Hiding value choices inside a score makes an ethical or political issue appear technical. Legitimately authorised people should disclose the objective, hear affected stakeholders and own trade-offs. AI can compare possible consequences, not confer legitimacy.
Even a neutral-sounding goal such as “reduce risk” needs definition. Reducing risk to the institution can increase burden on applicants. The identity of the risk bearer belongs in the decision.
Do not let a complete answer conceal incomplete information
Real decisions often require material the model did not see: tone in a conversation, an unscanned annexure, a recent change, individual preference or an unusual physical condition. AI can still generate a complete conclusion, creating the impression that available information was sufficient.
The system needs required fields and stop rules. If essential material is absent, it may list questions to resolve but should not infer the missing content. For high-impact decisions, the ability to abstain is more important than answering every case.
Reviewers should see an explicit inventory of supplied and missing material. “Based on the documents provided” is too vague if no one can see which documents were actually processed.
A human in the loop can be a rubber stamp
If a person approves thousands of items daily, sees only the AI summary, lacks original evidence and is measured on rapid acceptance, the system makes the decision in substance even if a human clicks. Human decision-makers can exhibit automation bias and alter an initially correct judgment after erroneous advice. NIST AI RMF: Human-AI Interaction
Effective oversight requires manageable workload, competence, independent evidence, authority to reject, error feedback and periodic testing. Asking a person to make an initial assessment before seeing AI advice can reduce anchoring in some contexts.
Measure overrides and their outcomes. Zero overrides may signal perfect performance, but more plausibly it can signal fear, interface friction or passive acceptance. Oversight quality is an empirical question.
AI can still participate without deciding
Withholding final authority does not prohibit useful assistance. AI can organise application material, identify missing documents, create comparison tables, retrieve relevant policy, propose options and draft a notice for human editing. Each use still needs privacy, bias and accuracy controls, but decision authority remains human.
AI can also challenge a proposed decision: search for omitted counterexamples, flag conflicting rules, or check whether a notice explains the route to review. Expanding the decision-maker's field of view may be more valuable than producing an unchallengeable score.
Keep assistance modular. The permission to summarise a case should not technically include the permission to send the outcome. Separation limits the harm of both model error and account compromise.
A ladder of decision authority
Level one is advice: AI supplies information and a person decides independently. Level two is conditional recommendation: AI applies disclosed rules and a person inspects evidence and may override. Level three is low-risk automatic execution with sampling and reversal. Level four is high-impact automatic decision. Requirements for validation, monitoring, explanation and remedy rise with each level; level four should generally remain closed to general-purpose generative AI.
Assign the level to a specific action, not a product. The same system may automatically format an application but not reject it. Permissions should keep those capabilities separate.
Review the level when scale changes. A low-impact action performed once may become consequential when repeated across a million people or when combined with another automated system.
My assessment: authority should follow accountability
A decision-maker should understand the issue, access relevant evidence, give reasons, hear challenge and repair error. AI may assist each step but cannot be questioned as a responsible legal or institutional actor, carry a duty or compensate loss. Final authority therefore remains with a person or institution capable of accountability.
If no genuine human decision-maker can be provided, the honest response is to narrow or pause the use. A confirmation button cannot manufacture missing governance capacity.
Decision checklist
- Does this affect rights, health, safety, livelihood, liberty or significant property?
- How quickly can error be detected, and can the effect be fully reversed?
- Are inputs complete, accurate and relevant to this person or context?
- Do objectives and weights contain value choices requiring public justification?
- Can the affected person learn AI's role, correct data and seek review?
- Does the reviewer see original evidence and have time for independent judgment?
- Can the reviewer reject AI without unreasonable performance pressure?
- Are stop, investigation, notification and remedy mechanisms available?
Conclusion
AI should not hold final authority when consequences are serious, reversal is difficult, information is incomplete, values are contested or effective appeal is absent. Retaining a human decision is not ceremonial. It preserves explanation, discretion, accountability and remedy. AI can broaden information and options, but the final step affecting another person's life belongs to someone who can answer questions and take corrective action.
Related questions
- Which Tasks Should You Give to AI, and Which Should You Keep?
- When Is the Cost of Error Too High for AI Automation?
Continue reading: All articles in How Far Should You Trust AI?
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.