If AI Has Found the Sources, Why Is the Research Not Finished?

After AI Enters the Workflow · Season One: “From Answering Questions to Participating in Work” · Article 5

A researcher asks an AI system whether a particular intervention improves employee performance. Within seconds, the system produces a confident answer and a list of academic-looking references. Several titles are relevant. Some links open. The task appears almost complete.

Then the researcher reads the papers. One studied students rather than employees. Another measured self-reported confidence rather than performance. A third compared the intervention with no training, not with the existing program. The most impressive statistic comes from a secondary article that misstates the original result. One citation does not exist at all.

The AI did find material. It did not finish the research.

This distinction is easy to lose because search used to be one of the slowest visible parts of research. If a system can produce ten sources and a coherent synthesis in a minute, it feels as if the intellectual distance has also collapsed. In reality, locating material is only the beginning of a chain that includes reading, interpreting, comparing, qualifying and deciding whether the evidence is sufficient for the claim.

The existence of a source does not mean it supports the claim

Research depends on relationships between claims and evidence. A real publication can still be irrelevant to the sentence for which it is cited.

Suppose an article states that “AI improves the productivity of knowledge workers by 30 per cent”. A study may report a similar number, yet apply only to a narrowly defined writing exercise, a particular model version and participants with a specific background. It may measure completion time rather than total organisational productivity. It may exclude the time required for later verification. The citation exists, but the claim has expanded beyond it.

Several questions must therefore be asked before a source can support a sentence:

• Does it study the same population, task and setting?
• Does it measure the outcome named in the claim?
• Is the reported figure an average, a subgroup result or a model estimate?
• What comparison produced the effect?
• What period, model and jurisdiction apply?
• Which qualifications did the authors place on the finding?

AI can help surface these questions. It cannot answer them reliably without access to the complete source and a process that checks its interpretation.

The problem is not limited to fabricated references. A 2026 study of citation constraints and reference hallucinations examined how large language models behave when generating citations under different conditions. Its very subject reflects the continuing need to test whether references exist and whether they have been represented accurately, rather than treating formatted citations as evidence. Davis and Mahmoud, “Citation Constraints and Reference Hallucinations in Large Language Models”

Search, reading, judgement and synthesis are four different jobs

Search asks which documents may be relevant. Reading asks what those documents actually say. Judgement asks how much weight they deserve. Synthesis asks what can responsibly be concluded from them together.

AI can contribute to all four, but success in one does not prove success in the next.

A retrieval system may locate passages with matching terms while missing a governing definition elsewhere. A summary may accurately compress individual sections but lose the paper’s limitations. A comparison table may place studies side by side without noticing that their outcome measures are incompatible. A synthesis may create a smooth consensus from studies that actually disagree.

Research quality therefore cannot be assessed by counting documents or by asking whether an answer contains links. The important unit is the supported claim. Each material statement should be traceable to evidence that genuinely bears the required weight.

This is particularly important in law, medicine, public policy and current technology. A document may be authoritative but superseded; relevant but from another jurisdiction; rigorous but directed at a different question. The research process has to preserve those distinctions.

Retrieval solves where to look, not what to believe

Retrieval-augmented generation can make external knowledge available to a model at generation time. The original RAG work combined a language model with a retrievable corpus, improving performance on knowledge-intensive tasks and making external material easier to update. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”

That architecture addresses an important limitation of relying only on a model’s trained parameters. But retrieval itself introduces choices: how the collection is built, how documents are divided, which query is used, how results are ranked and how much text enters the context.

A relevant document can be missed because the query uses different vocabulary. A passage can be retrieved without the table or footnote that qualifies it. A secondary source can outrank the original because it repeats the keywords more often. When many passages are supplied, the model may not use all of them equally; long-context research has shown position-related variations in the use of relevant information. Liu et al., “Lost in the Middle”

Retrieval should therefore be treated as the construction of a reading list, not as a verdict on truth. Its output needs provenance, date, document status and a route back to the complete source.

AI self-checking helps, but it is not independent verification

One way to improve generated answers is to ask the model to verify its own claims. Research on Chain-of-Verification proposed a process in which a model drafts an answer, creates verification questions, answers them separately and then revises the response. The method reduced hallucinations across several tasks. Dhuliawala et al., “Chain-of-Verification Reduces Hallucination in Large Language Models”

Such methods are valuable. They can force a system to revisit claims and reduce errors that arise from one-pass generation. They do not transform self-review into an independent source of evidence. The same model may carry the same mistaken assumption into the verification step. If the underlying documents are absent or the retrieval process is defective, further reasoning can make the error more elaborate rather than correct it.

Independence means that a claim is checked against something not created by the same generative path: an original document, a reproducible calculation, a database query, a test, a second qualified reviewer or another evidentiary process with different failure modes.

AI can organise and perform parts of verification. It should not be allowed to define the claim, select the evidence, judge the evidence and certify the result without external constraint.

Build a claim–evidence ledger

A practical research workflow can make the relationship between prose and evidence explicit. For each consequential claim, maintain a small ledger containing:

1. the exact claim the draft intends to make;
2. the source that supports it, preferably primary or authoritative;
3. the relevant passage, table or calculation;
4. the source’s population, date, jurisdiction and method;
5. qualifications or conflicting evidence;
6. whether the support is direct evidence, a reasonable inference or background only;
7. the person who reviewed and accepted the claim.

This may sound slower than asking an AI for a finished report. That is precisely the point. Research includes a cost of justification. AI can reduce mechanical work by extracting metadata, comparing methods, flagging missing fields and drafting cautious formulations. It should make the ledger easier to build, not make the ledger disappear.

The ledger also changes editing. If a sentence becomes stronger, its evidence must be reconsidered. If a source is replaced, the affected claims are visible. If new information arrives, the researcher can update the relevant part without treating the whole generated narrative as an indivisible block.

NIST’s Generative AI Profile stresses the importance of documented, measurable and managed risks in the context of use. For research systems, that implies monitoring source quality, grounding, confabulation and human review rather than assuming that citations solve reliability. NIST, “Generative Artificial Intelligence Profile”

More sources do not necessarily mean better research

AI makes it inexpensive to expand a bibliography. That can improve breadth, but it can also create citation theatre: a long reference list that no one has read closely and that does not map to the text’s actual claims.

A strong research article may rely on a small number of well-chosen primary sources, supplemented where necessary by independent analysis and contrary evidence. A weak one may cite fifty documents that repeat the same unverified proposition.

The relevant questions are qualitative. Have the strongest available sources been used? Are important disagreements represented? Can the conclusion survive the removal of a single convenient study? Is uncertainty expressed at the same level as the evidence? Does the reader have a practical way to inspect the basis of the claim?

AI is particularly useful for widening the initial search and revealing unfamiliar terminology. It is less suited to making the final, normative decision that the search has been adequate. That decision depends on the consequences of error, the standards of the field and the judgement the author is willing to defend.

Conclusion: AI can shorten the route to sources, but it cannot decide when evidence is enough

Finding a source is an event in research, not the completion of research. A credible result requires the researcher to establish that the source exists, applies to the question, supports the precise claim, retains its qualifications and deserves the weight assigned to it.

AI can accelerate discovery, extract structured details, compare documents and expose gaps. It can even participate in systematic verification. But the final research product must retain a chain from claim to evidence and an accountable judgement about sufficiency.

The operating principle should be:

A source list shows where the system looked. Research is complete only when the important claims can be justified by what those sources actually establish.

The correct use of AI is not to replace the slow moment of deciding whether evidence is enough. It is to spend less time finding candidate material and more time examining the relationship between evidence and conclusion.

Primary sources and further reading

• Kimberly Davis and Qusay Mahmoud: Citation Constraints and Reference Hallucinations in Large Language Models
• Patrick Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Nelson F. Liu et al.: Lost in the Middle—How Language Models Use Long Contexts
• Shehzaad Dhuliawala et al.: Chain-of-Verification Reduces Hallucination in Large Language Models
• NIST: Artificial Intelligence Risk Management Framework—Generative Artificial Intelligence Profile

Continue reading: Explore the After AI Enters the Workflow series.