RAG hallucination rates: the real numbers, and what to do about the residual

A bare language model answers from patterns in its training data. Ask it for specifics, statutes, figures, references, and it generates plausible text whether or not truth is available. In peer-reviewed testing published in Scientific Reports, GPT-3.5 fabricated 55% of the academic references it produced, and GPT-4 fabricated 18%.

Retrieval-augmented generation changes the input, not the nature of the model. The system first retrieves relevant passages from a defined corpus, then instructs the model to answer from those passages. When retrieval works, the model paraphrases evidence instead of inventing it. That single change is where most of the measured improvement comes from.

Three independent measurements sketch the landscape.

1. Ungrounded references: GPT-3.5 fabricated 55% of generated citations, GPT-4 18% (Scientific Reports, 2023).

2. Grounded legal research tools: Stanford's benchmark of 202 legal queries, hand-scored by experts, found Lexis+ AI hallucinating on 17% of responses and Westlaw AI-Assisted Research on 33%, versus 43% for ungrounded GPT-4 on the same queries.

3. AI web search: Columbia's Tow Center found eight AI search engines collectively misattributed or fabricated sources on more than 60% of news queries; the best, Perplexity, was still incorrect 37% of the time.

The pattern is consistent: grounding cuts the rate by half or more, and the residual stays in double digits for open-ended work. Both vendors in the Stanford study had marketed hallucination-free or near-hallucination-free products; the researchers' verdict was that the claims were overstated.

The residual is not one bug but a family. Retrieval can miss: the relevant passage never reaches the model, which answers from general knowledge anyway. Retrieval can mislead: a passage about a similar but different question anchors a confident wrong answer. And synthesis can drift: the model cites a real source while mischaracterizing what it says, the subtlest failure in the Stanford data, because the citation looks perfect.

Notice what these have in common: none are visible in the answer itself. A fluent paragraph with a real-looking citation reads identically whether the pipeline worked or failed. The error only becomes visible at the source.

Ask three questions of any reliability claim. What was the task? Rates measured on narrow extraction tasks do not transfer to open-ended research. Who scored it? Self-reported accuracy without expert hand-scoring hides mischaracterization errors, precisely the kind RAG is worst at. And what counts as a hallucination? A real case cited for a proposition it does not support is an error even though the citation resolves.

A vendor unwilling to publish task, scoring method, and definition is reporting marketing, not measurement. The Stanford study exists because two confident claims met one independent benchmark.

If the floor for grounded systems is a double-digit error rate on hard queries, the design question is not how to prompt the rate away but how to make each individual answer cheap to check.

That is a citations-architecture question. Answers where every sentence links to the exact passage it came from turn verification into a one-click read: the Stanford failure modes, missed retrieval, wrong anchor, mischaracterization, all become visible the moment the source opens. This is the principle Tatsulok is built on: answers assembled from your own library and curated collections, every claim linked to its passage, and an explicit "your sources do not contain this" instead of improvised support. The residual rate stops being a hidden liability and becomes a visible, checkable property of each answer.