After AI Makes a Mistake, Why Is the Chain of Responsibility Harder to Find Than the Error?

Geoffrey Chen

After AI Enters the Workflow · Season One: “From Answering Questions to Participating in Work” · Article 10

An AI-assisted service sends a customer an incorrect decision. The mistake is visible: a relevant document was omitted, the recommendation was wrong and the customer lost access to a benefit for several weeks.

Finding the cause is harder. A vendor supplied the general model. An internal team connected it to the organisation’s records. Another team configured retrieval. A manager approved the workflow. A frontline employee saw the recommendation and clicked “confirm”. The policy owner had not updated one guidance document. Monitoring detected no incident because the final transaction completed successfully.

Everyone touched part of the system. No one appears to have owned the whole decision.

This is why AI accountability cannot be solved by saying either “the AI made the mistake” or “a human was ultimately responsible”. The first attributes agency to software that cannot repair the harm or accept an obligation. The second names a species rather than a role. A usable responsibility chain must show who had authority, knowledge, control and duties at each stage—and who can now provide a remedy.

Cause and responsibility are different questions

Technical investigation asks what produced the error. It may identify an incorrect data field, a retrieval failure, an ambiguous prompt, a model limitation, a software bug or a reviewer’s action.

Responsibility asks a different set of questions. Who was required to prevent this kind of failure? Who selected the data and model? Who set the permissions and review standard? Who had authority to adopt the result? Who had a duty to monitor the system? Who can reverse the decision and compensate the affected person?

One cause can involve several responsible parties, and one responsible party may not have directly caused the error. A manager may be accountable for deploying a workflow without adequate review even though the immediate defect arose in a vendor component. A frontline employee may have clicked approval but lack the knowledge or time required for meaningful judgement.

Blaming the nearest person can therefore leave the system unchanged. A good incident analysis separates causal contribution, role obligation and remedial capacity.

The AI interface creates an illusion of a single actor

A conversational interface speaks in the first person: “I reviewed the file” or “I recommend approval”. This compresses a complex technical and organisational arrangement into one apparent speaker.

Behind that speaker may be a foundation model, orchestration code, system instructions, retrieval services, third-party tools, internal databases, access policies and human approvals. Each component is designed and governed by different actors. The simplicity of the interface hides the distribution of control.

The phrase “the AI decided” is therefore often an incomplete description. It may mean that a model generated a score, that software automatically applied a rule, that an employee relied on a recommendation or that an institution adopted the output as its decision. Accountability depends on distinguishing those events.

European AI regulation similarly allocates obligations among actors such as providers and deployers rather than treating “AI” as one responsible entity. The European Commission’s guidance explains the Act’s structure and role-based obligations, including separate provisions for general-purpose model providers. European Commission, “Navigating the AI Act” and “Guidelines on obligations for General-Purpose AI providers”

The legal details depend on the system and jurisdiction, but the structural lesson is general: responsibility follows roles and control, not the personality projected by an interface.

Why “a human reviewed it” often fails to locate responsibility

Human review is frequently used as a universal answer. Yet the phrase can describe very different arrangements.

Was the reviewer given the original evidence or only the AI recommendation? Did the reviewer understand the model’s limitations? Was there enough time for independent analysis? Could the reviewer reject the result without managerial penalty? Was the task within the reviewer’s professional competence? Was the decision documented?

If the answer is no, the human may function as a rubber stamp. Assigning responsibility to that person after an incident is unfair and ineffective. The organisation designed a process in which the nominal reviewer lacked the conditions for judgement.

Meaningful review requires information, competence, time, authority and a defined duty. It also requires a manageable volume of work. A system that produces hundreds of recommendations per hour cannot claim individual human judgement merely because someone approves batches.

Accountability therefore belongs partly to the designers of the review role: those who decide what evidence is displayed, how uncertainty is expressed, when escalation is possible and how performance is measured.

Without records, there is no recoverable responsibility chain

After an incident, an organisation needs to reconstruct what happened. That requires more than the final output.

Useful records may include the model and system version, relevant instructions, retrieved sources, tool calls, data state, generated recommendation, human edits, approvals, timestamps and the policy in force. The record should be proportionate and respect privacy and security, but it must be sufficient to explain consequential action.

Without a trace, each participant can offer a plausible account while the interaction among components remains invisible. The organisation may fix the visible sentence without discovering that the same retrieval defect affects thousands of cases.

The US Government Accountability Office’s AI Accountability Framework organises oversight around governance, data, performance and monitoring, emphasising documentation and accountability throughout the system lifecycle. GAO, “Artificial Intelligence Accountability Framework”

NIST’s AI RMF Core likewise emphasises roles, documentation, measurement, monitoring and response. NIST AI Resource Center, “AI RMF Core” These frameworks matter because accountability cannot be reconstructed from intentions after the fact; it must be designed into the workflow.

Naming one accountable person is necessary but not sufficient

An organisation should identify an accountable owner for an AI use case. The Australian Government’s Policy for the responsible use of AI in government, Version 2.0, includes governance expectations for accountable officials and agencies using AI. Australian Government, “Policy for the responsible use of AI in government—Version 2.0”

But a name on a register does not create practical control. The owner needs visibility into system changes, authority over deployment, access to risk information, resources for monitoring and the ability to suspend the workflow. Responsibilities must also be allocated to data owners, technical operators, policy owners, reviewers, security teams and incident responders.

A useful responsibility map connects each material function to a role:

  • defining the purpose and prohibited uses;
  • selecting and maintaining data;
  • choosing and configuring models and tools;
  • setting evaluation and release criteria;
  • approving consequential results;
  • monitoring performance and group impacts;
  • receiving complaints and correcting decisions;
  • notifying affected people and regulators where required.

The accountable owner ensures that the map is complete. The owner does not personally perform every function.

Remedy is the test of whether responsibility is real

Accountability becomes concrete when a person is harmed. Can the organisation explain the basis of the decision? Can it pause the effect, reconsider the evidence and provide a human determination? Can it restore access, correct the record and compensate loss where appropriate? Can it identify and repair similar cases?

If nobody has authority to provide these remedies, declarations of responsibility are largely symbolic. A complaints inbox that cannot alter the underlying system does not complete the chain.

Remedy also feeds learning. The incident should update tests, data rules, training, monitoring and workflow design. Otherwise, the organisation treats each complaint as an isolated exception rather than evidence about the system.

This is why responsibility should be mapped backwards from repair. The role that can adopt an AI-supported decision must be connected to a role that can suspend, reverse and explain it.

Do not concentrate responsibility on the ordinary user

Frontline staff are often the most visible human participants and therefore easy to blame. But their control may be narrow. They did not select the model, design the interface, set throughput targets or decide which evidence would be shown.

Users remain responsible for duties genuinely within their role: following procedures, recognising clear anomalies, escalating uncertainty and not misrepresenting output. Organisations remain responsible for creating conditions in which those duties can be performed.

Responsibility should follow effective control and reasonable capacity. It should not be shifted downwards simply because a person supplied the final click.

Conclusion: AI cannot bear responsibility, and “human responsibility” must name actual roles

AI systems can cause or contribute to errors. They can record actions, support diagnosis and help generate corrections. They cannot owe an explanation, restore a right or accept institutional consequences.

The governing principle is:

For every consequential AI-enabled action, an organisation must be able to identify who authorised the purpose, who controlled the inputs and process, who adopted the result, who monitored the system and who can provide a remedy.

These may be different people and institutions. The chain can be distributed without becoming vague. What makes it a chain is the explicit connection among authority, evidence, action and repair.

Finding an erroneous output is a technical achievement. Finding and activating the responsibility chain is an organisational one. If the organisation cannot do the second, it was not ready to delegate the first.

Primary sources and further reading

Continue reading: Explore the After AI Enters the Workflow series.

Discover more from Geoffrey Chen

Subscribe to get the latest posts sent to your email.