What is context rot, and why does AI get worse the more you give it?

Context rot is the tendency of a language model to become less reliable as the amount of text you give it grows. Ask a short question about a short document and you get a careful answer. Ask the same question with sixty pages of background attached and the answer gets vaguer, misses things that are plainly in the text, or confidently states something the documents contradict.

The term came out of a Hacker News discussion in June 2025 and was formalised a month later by Chroma, whose technical report evaluated 18 models including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. Their finding was blunt: models do not use their context uniformly, and reliability falls as input length rises even on tasks as simple as retrieving or copying a sentence.

The most important thing to understand is what context rot is not. It is not running out of room. A model with a 200,000-token window can degrade badly at 50,000 tokens, with three quarters of the window still empty. Nothing overflows, nothing is truncated, no error appears. The answer just quietly gets worse.

Enough that it changes how you should work. The NoLiMa benchmark from Adobe Research, published at ICML 2025, tested 13 models that all advertise support for 128,000 tokens or more. At 32,000 tokens, a quarter of the smallest window they claim, 11 of the 13 scored below half of their own short-context baseline. GPT-4o went from 99.3% to 69.7%.

What NoLiMa isolated is why. Standard needle-in-a-haystack tests hide a sentence that shares obvious wording with the question, so a model can succeed by literal matching. Remove that overlap, and force the model to make the connection itself, and long-context performance falls apart. Real work is almost entirely the second kind: the contract does not use the same word as your question, and the answer usually requires joining two facts from different pages.

That is the part worth internalising. Degradation is not uniform across tasks. Simple lookups hold up longest; reasoning that has to travel between distant parts of the input degrades fastest. The harder your question, the earlier context rot arrives.

Because the window is a resource to manage, not a number to maximise. Every token you add competes for the model's attention with every other token, and that attention is not spread evenly.

The pattern was documented before the term existed. The Lost in the Middle study found that models use information best when it sits at the very start or the very end of a long input, and measurably worse when it is buried in the middle. Chroma's report extended this: the position of relevant information, the presence of plausible-looking distractors, and even how coherently the surrounding text reads all change the result. More context is not neutral. Irrelevant material actively costs you accuracy on the material that matters.

So the intuitive response to a disappointing answer, which is to paste in more background, is often the thing making it worse.

The 2026 default is hybrid, and it inverts the instinct. Instead of loading everything and hoping the model finds the relevant part, you find the relevant part first and load only that, then let the model reason over a much smaller, denser input.

In practice that means three habits. Scope the question to the documents that can actually answer it rather than the whole archive. Prefer retrieving passages over pasting files, so a hundred-page contract contributes the two clauses that matter rather than a hundred pages of noise. And keep a working session focused on one line of enquiry, because a long conversation is itself a long context and rots the same way.

This is what retrieval-augmented generation is for, and it is why the choice between long context and retrieval turned out not to be a choice: you want the retrieval step precisely so the reasoning step gets a short, dense input.

Here is the honest part that most write-ups skip. Retrieval reduces the error rate. It does not take it to zero. Stanford's RegLab benchmarked purpose-built, retrieval-grounded legal research tools and still measured hallucination in 17% to 34% of queries. Better context is necessary and it is not sufficient.

Which is exactly why the citation matters more than the confidence. A cited answer gives you something that does not depend on the model having been right: a link to the passage it drew from, which you can open and read in ten seconds. If the passage says what the answer says, you are done. If it does not, you caught it. Either way you did not have to trust anything.

That is the practical defence against context rot, and it is also the difference between a tool that makes you faster and a tool that makes you dependent. You keep the judgment. The machine does the gathering, at whatever scale, and shows its work so you can check it.