Short answer
Fluent confidence and factual correctness come from different mechanisms. A generative model's central task is to produce contextually plausible next content, not to complete a fact check before every sentence. It can learn how people express certainty without acquiring evidence equal to that certainty. The more complete the answer, coherent the explanation and realistic the citation format, the easier it becomes to mistake “shaped like an answer” for “demonstrated to be true”.
A language model predicts expression before it proves a fact
Large language models learn statistical relationships among words, concepts and structures in large bodies of material. Given a question, they generate further content based on the prompt and context. This mechanism can reconstruct substantial real knowledge and support complex reasoning. Its output condition, however, is not the same as a database query: it can produce a pattern-fitting answer without first locating an auditable record.
When training material contains a stable, common fact and the question is clear, the language pattern often points to the right answer. When the subject is rare, ambiguous, newly changed or requires joining similar entities, the most plausible continuation may not match reality. The model still generates, filling gaps with realistic names, dates, institutions and explanations.
NIST calls this “confabulation”: generative systems confidently present erroneous or false content, including contradictions, divergence from input, and invented logic or citations. It describes confabulation as a natural risk of systems generating from statistical distributions, particularly for open-ended long responses and domains requiring specialised context. NIST AI 600-1: Generative AI Profile
“I don't know” is not the default outcome
A conventional database can return no record. A calculator can report an invalid expression. A conversational model is trained and presented to provide a useful response. Unless it is calibrated to abstain, given a refusal mechanism, or explicitly asked to mark insufficient evidence, it may be more likely to complete an answer than preserve a blank.
When the model says “I am not certain”, the phrase is not necessarily a measured probability. When it says “definitely”, the word may be part of the generated register. Confidence words in natural language are not dependable instruments. True confidence calibration requires a defined task, representative distribution and evaluation method. A normal chat interface rarely gives the user such a scale.
This distinction explains why asking a model to rate its own confidence is weak evidence. The rating may still be generated from patterns associated with the answer. Self-critique can reveal problems and is useful as an extra pass, but it is not independent verification because the same system and context may reproduce the same blind spot.
A detailed explanation can explain a wrong answer
People often request step-by-step reasoning to improve reliability. An explanation may expose some logic and make review easier, but it cannot repair a false premise by itself. A model can generate an answer and then create a narrative that fits it. The narrative may be internally coherent while using a wrong fact, omitted condition or invalid rule.
This is hazardous in law, history, technical troubleshooting and health. A nonexistent case may be accompanied by a plausible citation and holding. A nonexistent software setting may arrive with a realistic menu path. An incorrect diagnosis may be surrounded by genuine symptoms that do not establish the conclusion for this person. Detail improves readability and the persuasive power of unverified material.
Verification should therefore begin with the conclusion and decisive premises and return to independent evidence. Do not assess only whether the reasoning sounds elegant. An explanation is an object to verify, not a verification device.
Models can combine nearby truths into a nonexistent whole
Many confabulations are not random. They combine neighbouring facts incorrectly: one author's title with another author's date, features from two related products under a single version, or a real legal principle attached to the wrong jurisdiction. Each element looks familiar, but the combination does not exist.
These errors are harder to detect than nonsense because recognition of several genuine elements encourages trust in the whole. Do not ask only whether the terms sound familiar. Check relationships: did this author write this work; did this feature exist in the named version; was this rule in force in the specified place and time?
Numbers create a similar trap. A reported figure may be real but refer to a different year, denominator, population or definition. Copying the number correctly does not preserve its meaning. Verification includes units, scope and methodology.
Knowledge limits, retrieval and tools change the error, not eliminate it
Without current retrieval, a model cannot reliably know recent prices, officeholders, laws or software versions. Connecting search or company data provides fresher evidence but moves the failure boundary. Retrieval can omit the decisive source, select an unreliable page, mix editions, or find correct material that is then summarised incorrectly.
Tools add permission and execution failures. A model may understand the question yet select the wrong account or file. It may treat instructions embedded in an untrusted webpage as part of the user's task. “It has search” does not mean a fact was verified. “It supplied a citation” does not mean the citation supports the sentence. Check the source, date, scope and inference from source to conclusion.
Grounding is still valuable. Limiting an answer to a defined set of documents can reduce unconstrained invention and make checking practical. It should change the claim from “the model knows” to “the output can be traced to these materials”. The user still needs to know whether the materials are complete and authoritative.
Some questions never had one objective answer
“Who is the best candidate?”, “Is this email polite?” and “Is this project worthwhile?” contain preferences and values. AI can produce a very definite answer, but the certainty may come from criteria it silently supplied rather than a false fact. If the team did not define its objective, the model's default preferences can be mistaken for an objective result.
For these questions, ask the system to state its criteria, show how the result changes under alternative criteria, identify unknown facts and preserve the human value choice. Turning a subjective dispute into a precise score can move disagreement from visible discussion into invisible weighting.
The same applies to advice. A general recommendation can be sensible for an average case yet unsuitable for a person's finances, health, obligations or risk tolerance. Correct general knowledge does not make an individual recommendation correct.
Why fluent answers are unusually persuasive
Fluency reduces the effort of reading. Complete sentences, professional terminology, neat lists and a courteous register signal competence. Immediate response removes the visible delay associated with expert investigation, hiding the evidence work that did not occur. When the answer matches expectation, confirmation bias further weakens the impulse to check.
The interface matters. If it displays one polished answer but not alternative explanations, evidence gaps or historic error rates, the user sees a result without its uncertainty. A small warning that the system “may make mistakes” does not necessarily offset the confident form of every sentence.
The remedy is not permanent distrust. Move trust from style to observable signals: an original source that opens; text that directly supports the claim; a result that can be independently reproduced; explicit missing information; and an answer that survives counterexamples and boundary conditions.
Check the answer in three layers
The first layer is directly verifiable fact: names, dates, numbers, laws, product functions and quotations. Return these to authoritative or primary sources. The second layer is inference from fact: cause, trend, risk and comparison. Ask whether alternative explanations exist and whether evidence supports the strength of the language. The third is advice: whether someone should buy, hire, treat or act. Advice additionally requires goals, cost, personal circumstances and consequence.
An answer may be broadly correct in layer one, overclaim in layer two and give unsuitable guidance in layer three. Sampling facts alone does not validate the complete decision chain.
Within each layer, identify the load-bearing claims. A travel plan can survive a slightly wrong description of a museum but not a false visa requirement. A financial comparison may survive a rounding error but not the wrong tax treatment. Verification effort should follow what can change the decision.
Make boundaries visible, then verify them anyway
Ask AI to separate known facts, assumptions, inferences and unknowns; provide an accessible source and exact supporting location for important claims; identify conditions that would change the answer; present the strongest counterexample; and leave unsupported fields blank. With supplied documents, request page or paragraph references and sample them against the original.
These instructions improve inspectability, not certainty. A model can invent an unknown, page number or source. Stronger systems constrain answers to approved material, use deterministic queries for critical fields, compare citations with surrounding claims, stop when evidence is absent, and evaluate on real past failures.
Repetition is also not independence. Asking the same model the same question in a new chat may produce another sample from similar patterns. A genuinely independent check uses a different evidence path: the original document, an official register, a calculation, a test or a qualified person.
My assessment: tone shows that it can write, not that it knows
I treat an AI response as work from a very fast assistant with inconsistent evidence habits. Its structure, candidate explanations and proposed checks may be useful immediately. Decisive facts need independent support. The more perfectly an answer matches what I hoped to hear, the more important it becomes to ask what is missing.
Trust should not be one global choice. A definition in the same response may be low risk; a current legal provision needs a live official source; personal medical action should not be decided by general output. Claim-level trust is more mature than either worship or blanket rejection.
One-minute checklist
- Is this sentence a fact, inference or recommendation?
- Is the answer recent, rare, specialised or easily confused with something similar?
- Do cited links open and directly support the adjacent claim?
- Could two people, products, rules or versions have been combined?
- Does the detailed explanation depend on an unchecked premise?
- What relevant material did the model not see?
- Would I check more carefully if the answer contradicted my expectation?
- Which decisive fact needs independent confirmation before action?
Conclusion
AI's confident style comes from language generation. Reliability comes from data, task, retrieval, tools and verification. The two can correlate, but they are never equivalent. The most dangerous error is often not absurd text but a well-explained answer assembled from real fragments that confirms expectation. Do not use tone as a reliability measure. Separate facts, inferences and advice, then look for evidence and limits at each layer.
Related questions
- How Can You Verify an AI Answer Without Redoing All the Work?
- How Can You Check Whether AI-Generated Papers, Data and Links Are Real?
Continue reading: All articles in How Far Should You Trust AI?
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.