Why AI cannot stop hallucinating, according to the people who build it

There is a quiet assumption behind a lot of AI use: the next model will fix it. The hallucinations are teething problems, the reasoning goes, and one more upgrade will make the checking unnecessary.

OpenAI's researchers published the strongest argument against that hope, about their own technology. In "Why Language Models Hallucinate" (Kalai, Nachum, Vempala, and Zhang, September 2025), they argue mathematically that hallucination is a predictable consequence of how models are trained and evaluated, not a defect awaiting a patch. The paper's example is disarmingly simple: guess a date and you have a 1-in-365 chance of a point; say "I don't know" and you are guaranteed zero. Every benchmark built that way teaches every model the same lesson.

Models are compared on accuracy leaderboards, and most leaderboards give no credit for honest uncertainty. Between two models, one careful and one confident, the confident guesser posts better numbers even when the careful one is more truthful, because its lucky guesses count and its wrong ones cost no more than an abstention would.

Training follows the scoreboard, so the incentive soaks into the models themselves. The authors' point is that this survives perfect data and infinite compute: as long as the test rewards a guess over a gap, the gap will be filled. It is the same reason a student bluffs on an exam that has no penalty for wrong answers. The behavior is rational, given the rules.

Structural means permanent, and permanent changes the strategy. Waiting is not a defense. Prompting alone is not a defense, because you are asking the model to act against its training. The durable defenses restructure the game instead.

Grounding changes what the model draws from: answers assembled from retrieved sources instead of open-ended recall, which cuts the guessing space sharply. Honest-gap architecture changes the reward: a system designed to say "the sources do not contain this" makes abstention a first-class output instead of a scoreboard penalty. And verification changes who catches the residue: your one-click check at the passage, because even grounded systems retain double-digit error rates on hard queries, as Stanford's benchmark measured at 17% to 33%.

The paper reframes what a refusal means. In a world where every incentive pushes toward confident guessing, a tool that tells you "your sources are silent on this" is doing something deliberately engineered against the grain, and that engineering is what you are actually paying for in a trustworthy system.

So when you evaluate AI tools, invert the demo instinct. Do not only ask what impressive answers it produces; ask what it does when the honest answer is nothing. Ask a question you know its sources cannot answer, and watch. The tools that improvise have told you how they will behave on the day it matters.

None of this is a reason to avoid AI, any more than the permanence of human error is a reason to avoid colleagues. It is a reason to work with the property instead of hoping past it: ground consequential questions in defined sources, prefer tools where abstention is designed in, and keep the verification habit for what carries your name.

That is the posture Tatsulok is built for: answers only from your documents and curated collections, every claim linked to its exact passage, and "your sources do not contain this" as an engineered, first-class answer. The guessing incentive is real, so we removed the room to guess.

This week's drill, no signup needed: ask your usual AI tool one question from your field whose answer you know is not publicly available. If it answers anyway, you have witnessed the scoreboard problem live, and you now know exactly what the paper is about.