On 9 September 2026, Reuters reported a case from the Oklahoma court system in which Stephens County judge Lawrence Wheeler acknowledged using ChatGPT for legal research. At least two cases cited in one of his orders did not exist. The account was based on an 17 August letter from the local district attorney to the state attorney general’s office. The attorney general later concluded that the available evidence did not support a criminal prosecution, but that does not settle questions of judicial ethics or discipline. Oklahoma’s Supreme Court had already dealt with a closely related problem. On 11 May 2026, it ordered a lawyer in another case to show cause after filings contained nonexistent or AI-generated authorities. The case record for 26 June states that the lawyer admitted failing to verify the citations and accepted responsibility.
The philosophically important issue is no longer whether generative AI can hallucinate. That is already well established. The more precise question is what happens to epistemic responsibility when a professional decision-maker delegates research, summarisation or drafting to an AI system.
My judgement is that the responsibility does not automatically move with the task. Research work can be delegated. Epistemic responsibility cannot simply be delegated in the same way. An AI system may function as a highly efficient search tool, generator of candidate answers or drafting assistant, but where the final judgement enters the public world under a human professional role, institutional authority or legal power, responsibility for verification remains attached to that human and the surrounding institution.
Three levels need to be kept separate. The first is information generation. A model can produce case names, statutes, summaries, lines of argument and apparently complete legal analysis. The second is epistemic warrant. A conclusion becomes a justified basis for action not because it sounds plausible, but because its sources can be traced, its citations checked, its reasoning challenged and its errors corrected. The third is institutional responsibility. Judges, lawyers, doctors and researchers do not merely transmit text. Their professional role carries an assurance that the material has passed through the relevant procedures of verification.
These levels are not interchangeable. Improvement at the first level does not automatically give an AI system epistemic authority at the second, and it certainly does not transfer institutional responsibility at the third.
There is a clear connection here with the traditional epistemology of testimony. Social epistemology has long asked why we are justified in relying on what other people tell us, and under what conditions testimony can transmit knowledge. Modern knowledge is necessarily dependent on experts, documents, databases and institutions. No individual can independently verify everything. But dependence has never meant the simple abandonment of judgement. Reliable knowledge practices normally preserve source identification, assessments of trustworthiness, cross-checking and traceable responsibility.
Generative AI makes this problem sharper because its outputs have the linguistic form of testimony without necessarily having the normative structure of a responsible speaker. A model can provide a complete-looking legal citation, but it does not bear the professional consequences when that citation is fictitious. Whether one chooses to call AI output “testimony” is less important here than recognising that the ordinary link between assertion and accountable speaker has been weakened.
For this reason, describing AI as “another research assistant” is only partly correct. Human assistants also make mistakes, but they normally exist within an organisational structure. They have identities, work records, training, assigned duties and a chain of accountability. A generative model produces an output through the interaction of prompts, system constraints, learned parameters and probabilistic generation. It may make a substantial causal contribution to the work, but causal contribution is not the same thing as professional epistemic standing.
This is the difference between instrumental delegation and responsibility delegation. Instrumental delegation means assigning part of the work process to another system: searching cases, summarising material, comparing precedents. Responsibility delegation would mean that another subject takes over the obligation to verify the result and bears the normative consequences of error. The first is already common in AI-assisted work. The second generally has not occurred.
A judge can say, “ChatGPT gave me the citation.” That explains how the error entered the workflow. It does not explain why the citation was entitled to appear in a judicial order. It is a causal explanation, not an epistemic justification.
Sustenesis Theory helps make the distinction more precise. Knowledge is not merely a sentence that remains locally coherent. It is a structure that sustains coherence through constraints, feedback, testing and correction. Difference, in this case, begins with preserving the distinction between model-generated candidate information and verified legal authority. If a workflow collapses that distinction, fabricated material can move directly into formal judgement. Constraint includes primary legal sources, citation rules, professional ethics, adversarial challenge, peer review and appellate mechanisms. These constraints are not obstacles added after knowledge has been produced. They are part of the conditions under which legal knowledge can be maintained. Sustained Coherence is therefore not the fluency of a paragraph but the capacity of the judgement to survive source checking, procedural review and subsequent correction.
Seen in this way, the important change introduced by generative AI is not simply that machines are beginning to “know” legal facts. It is that the generation of plausible propositions has become more separable from the processes that establish their warrant. In older research workflows, producing a case name usually implied some contact with an identifiable source. A language model can now generate a citation with realistic form even where no source exists. The distance between linguistic form and epistemic basis, which has always existed, has been technologically enlarged.
This also shows why the phrase “human in the loop” is insufficient. A person who merely presses the final approval button has not necessarily provided epistemic oversight. Meaningful human supervision requires the ability to inspect sources, understand decisive evidence, identify conflicts and reject the model’s output. If the human role is reduced to copying AI-generated material into an official document, the person remains in the workflow without actually performing the validating function.
None of this implies that AI should be excluded from legal work. That would turn a structural problem into a simple question of being for or against a tool. AI can reduce research costs, help identify relevant material, compare large bodies of text and improve some forms of review. The philosophical boundary is different: increased efficiency must not be confused with a transfer of epistemic authority. A system may help you find an answer without thereby completing the work of establishing why the answer deserves to be believed.
Under current technical and institutional conditions, the better principle is therefore not to prohibit delegation but to preserve the chain of responsibility. Generation can be outsourced. Search can be outsourced. Preliminary comparison can be outsourced. But when a conclusion enters a judicial order, medical judgement, scientific publication or another high-responsibility setting, the person and institution exercising formal authority must still be able to state where the evidence came from, how it was checked, where it might fail, and how errors will be corrected.
This judgement has a clear boundary. If future AI systems acquire stable institutional identities, durable audit histories, persistent memory, explicit scopes of authorisation and some legally recognised capacity to bear responsibility, then partial transfer of responsibility to machines would become a different philosophical and legal question. The Oklahoma case does not establish anything like that. It reveals a more immediate problem: where generative capacity has advanced faster than the structure of responsibility, professionals cannot treat a stronger tool as a reason to surrender their own duty of verification.
References
https://www.reuters.com/legal/litigation/oklahoma-judge-used-ai-in-ruling-that-contained-false-citations-prosecutor-says-2026-09-09/
https://oscn.net/dockets/GetDocument.aspx?bc=1065411959&cn=DF-123818&ct=appellate&fmt=pdf
https://oscn.net/dockets/GetCaseInformation.aspx?cmid=141887&db=appellate&number=DF-123818
https://plato.stanford.edu/entries/epistemology-social/
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.