Judgment in the Age of AI · Article 5
Asking several models can be useful, but majority agreement among AIs is not a vote on reality. Systems may have learned from the same webpages, repeat the same conventional account, use the same search results or share similar generation biases. Five matching answers can be one error copied five times.
A deceptively persuasive example
Three AI services attribute a quotation to the same author and chapter. The original book contains no such sentence because the internet already holds many pages repeating a false attribution. Model agreement reflects the distribution of available text, not necessarily the historical fact.
What agreement can tell you
- Wide disagreement signals ambiguity, insufficient evidence or instability and calls for more checking.
- Agreement shows that a claim is common, but an independent primary source is still needed.
- Agreement becomes stronger evidence only when the systems provide genuinely independent, inspectable evidence paths.
Replace model voting with multi-path verification
Assign different jobs instead of repeating one prompt: one system extracts claims, another searches for counterevidence, and another locates official sources. You then compare the evidence. The useful diversity lies in the paths, not the number of answers.
Agreement among models is an investigative clue; independent support among sources is evidence.
References
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.