Short answer
Sometimes, but it is not a simple two-vote majority. Agreement can increase confidence when two capable, complementary systems are genuinely independent and use different evidence paths. If they share training material, architecture, search results, prompts or a false premise, agreement measures correlation rather than truth. Disagreement between AI systems is a useful alarm. Agreement is at most one signal and cannot replace original evidence.
Why majority voting can work in principle
If several judges each have some competence and their errors are independent, a majority can reduce the effect of an accidental individual error. The critical conditions are not merely number but ability and independence. Five people copying one wrong answer do not create five pieces of evidence. Five thermometers connected to one failed sensor do not become accurate by agreeing.
The intuition becomes difficult when applied to language models. Different products may use overlapping public corpora, benchmarks, fine-tuning methods and search pages. Multiple responses from one model even more clearly share parameters and blind spots. Different names on an interface do not guarantee different sources of error.
A 2025 empirical study of more than 350 language models found substantial error correlation. On one leaderboard dataset, when two models were both wrong, they agreed on the same wrong answer 60 per cent of the time. The researchers also found high error correlation among larger, more accurate models and considered the downstream effect of algorithmic monoculture in settings including hiring. Kim et al.: Correlated Errors in Large Language Models
That figure is not a universal rate for every question. It demonstrates the structural point: independence must be measured, not inferred from different model labels.
Asking one model twice is not consulting two experts
Generation can be stochastic, so repeating a prompt can produce different wording or answers. Sampling several solutions and selecting the most common—often called a self-consistency strategy—can improve performance in some mathematics and structured reasoning tasks. The samples still come from the same model, knowledge and framing.
If the model consistently applies a wrong formula, confuses the same two people, or accepts a false premise in the prompt, ten generations may agree ten times. Self-consistency then measures internal stability, not correspondence with reality. Conversely, different wording need not be factual disagreement; it can reflect alternative expression or calculation paths.
Normalise answers into comparable claims before voting. “I recommend A” and “A has the lowest cost under these conditions” may share a conclusion but not a justification. If one justification depends on a false price, the agreement is fragile.
The prompt itself can create correlated error. If it asks, “Why did policy X begin in 2022?” when the policy began in 2023, every model may accept the embedded date. An independent second opinion should first test the premise.
Different providers may share the same information ecosystem
Models can learn the same popular misconception from the web and retrieve the same highly ranked pages at answer time. Several pages repeating one unsupported claim can make systems converge even when their searches run separately. Computation is separate; evidence ancestry is not.
This occurs in breaking news, rumours, product comparisons and historical anecdotes. Independence requires more than changing a chat window or brand. Ask whether the second system used a different primary source, a different professional method, visible evidence and controls aimed at different failures.
One model summarising legislation and another summarising the same secondary blog are not independent review. A model summary followed by a person opening the official legislation and checking the decisive sections is a new evidence path.
Models may also be related through supply chains invisible to the user. An application branded by one company can call a model or search service from another. Without architecture and provenance information, assume less independence than the logo suggests.
Disagreement is often more informative than agreement
When two systems disagree on a defined fact, at least one is wrong, the question is ambiguous, versions differ, or uncertainty is high. Do not immediately ask a third model to cast a vote. Locate the disagreement: date, definition, jurisdiction, calculation or value criterion. Then find the primary source capable of resolving that point.
Disagreement can allocate human attention. Research has shown that model-ensemble disagreement can act as a useful uncertainty or human-labelling signal in some tasks, but performance depends on the task and ensemble and is not a universal guarantee. Du et al.: Ensemble Disagreement Scores in Large Language Models
Agreement should not create automatic passage. It may influence prioritisation: a low-risk fact reached through genuinely different evidence paths may need less additional effort. A high-impact decision still returns to the record. Disagreement means “investigate”; agreement means only “possibly stable”.
Persistent disagreement is itself an answer sometimes. The evidence may be genuinely contested or the requested precision unavailable. A responsible outcome can state the range and unresolved issue instead of manufacturing consensus.
Compare evidence before comparing conclusions
Final answers alone conceal independence. Ask each system for load-bearing facts, sources, assumptions, unknowns and conditions that would alter its conclusion. Compare whether they cite the same source, use different data, share an unverified premise, or apply the same default value standard.
If conclusions agree and evidence differs, verify whether both paths genuinely converge. If conclusions agree but evidence conflicts, at least one reasoning chain is defective. If conclusions differ while facts agree, the difference may be a value weighting for a person to choose rather than a matter for model voting.
Source lists are not sufficient. Two models may cite different secondary articles that trace back to one press release. Follow each decisive claim to its earliest authoritative record.
A useful question is: what observation would make each system change its answer? If neither can identify one, the responses may be rhetorical positions rather than testable judgments.
When multi-model methods can help
For tasks with defined answers, abundant testing and controlled consequences, multi-model or multi-sample methods can be sound engineering choices: classification, field extraction, code candidates and some quantitative problems. First measure each model, the combination rule and error correlations on representative labelled cases. One observed agreement is not a validation study.
Disagreement can be routed to a person, while agreed results pass deterministic rules and audit sampling. Models can also take different roles: one generates, one searches for counterexamples, and one checks only approved sources. Role prompts do not create independent knowledge, so performance must still be tested.
Open-ended advice, ethical choice, legal judgment and healthcare decisions do not become safe through model volume. Three systems can produce the same popular recommendation that is unsuitable for the individual facts and responsibilities.
An ensemble may also increase cost and complexity. More calls create more data exposure, latency and records to manage. Use it only where measured error reduction exceeds those costs.
Design a more valuable second opinion
Do not show the first answer to the second system, because it can anchor the response. Provide the original question independently and deliberately change the task. One system forms the strongest explanation; another seeks evidence that would disprove it; a third checks specified primary sources. Separating generation, challenge and verification exposes more gaps than asking three systems to “answer”.
Use different evidence types. After a language model proposes an explanation, confirm records in a database, recalculate with a deterministic tool, run the code, or use accountable professional judgment. Real diversity comes from methods, not a model list.
Preserve system versions and inputs. A later model update can alter the combination's behaviour. If every model retrieves through the same search provider, retrieval diversity may be lower than expected.
For consequential workflows, include a control question with a known answer or a set of hidden test cases. If the models start agreeing on known errors, stop the process. Agreement monitoring without ground truth can drift silently.
Use an evidence matrix
Create four columns: claim, AI A evidence, AI B evidence, and independent adjudicating evidence. The first two expose shared sources and assumptions. The final column must come from an external record capable of resolving the issue. If no adjudicating evidence exists, mark the result unresolved instead of taking a vote.
Add a consequence column. For low-impact items, two plausible independent paths may be enough. For high-impact items, even three agreeing systems need formal approval or qualified review. The evidence threshold follows consequence, not model count.
Over time, record which pairs fail together. A combination that appears diverse in general may share a weakness on one language, jurisdiction or document type. Independence is local to the task.
My assessment: agreement is a lead, not evidence
I value disagreement because it identifies a place to investigate. When outputs agree, I look for a common ancestor. If both depend on one webpage, default assumption or training convention, they effectively have one vote.
The strongest combination is often not AI plus AI. It is generative AI plus a different kind of verification: an official record, reproducible calculation, test, original document or professional judgment with responsibility. These methods provide resistance at the point of a shared model blind spot.
Checklist
- Are these different samples, different models, or different evidence paths?
- Did the second system judge independently before seeing the first answer?
- Do they cite the same webpage, database or secondary claim?
- When conclusions agree, are the reasons and load-bearing facts valid?
- Does disagreement concern fact, version, definition or values?
- Is there an original record, calculation or test that can adjudicate?
- Was the ensemble tested on representative real cases?
- Does the consequence allow action based only on model agreement?
Conclusion
Agreement between two AI systems offers a limited stability signal. It materially increases confidence only when the systems are capable, errors are sufficiently independent and evidence paths differ. Real models often share data, methods and misconceptions and can therefore agree on the same wrong answer. Use disagreement to trigger investigation and evidence to resolve it. Do not mistake consensus among similar systems for an independent fact.
Related questions
- How Can You Verify an AI Answer Without Redoing All the Work?
- Why Does Human Review Still Miss AI Errors?
Continue reading: All articles in How Far Should You Trust AI?
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.