Why long chats make AI worse
Why does AI lose track of my early instructions in a long chat?
Every chat sits on AI's desk, and the desk fills up.
Three hours in
You've been working with it all afternoon. The early answers were sharp.
Now it's mixing up details, suggesting things you already rejected and forgetting the rule you set at the start.
AI isn't tired. Its desk is full.
The context window
Everything in the conversation sits on the desk: your messages, its replies, every file, every pasted email.
Context window: how much fits on the desk at once. It's measured in tokens.
Token: a chunk of text, about three-quarters of a word on average. "Unbelievable" might be three tokens: un, believ, able.
Today's desks are big. Hundreds of thousands of tokens is a few novels' worth.
A bigger desk doesn't mean better attention
You'd think a big enough desk would solve this. It doesn't.
In 2025, a company called Chroma tested 18 AI models on simple tasks while making the input longer and longer. The task stayed the same, with more text around it each time.
Every model got worse as the text grew, even on easy tasks. They called it context rot.
What that's like
Say you need to find one sentence in a stack of paper.
In 5 pages it's easy. In 500 pages you'd still manage. Now imagine many of those pages say nearly the same thing in slightly different words. You start mixing them up.
That's what an afternoon of drafts looks like to the model: lots of versions that are almost right.
The middle gets lost
Researchers also found that models pay most attention to the start and the end of what they're given, and less to the middle.
The rule you set in message 3? By message 60, it's in the middle.
Where you'll see it
- You paste a 200-page report and ask about one table.
- You're on your sixth version of the same email in one chat.
- In Claude Code, a session where it has opened dozens of files.
Try this
New task, new chat. Don't keep one giant conversation going for everything.
When a chat gets long, hand it over. Ask: "Summarise what we've decided, what's still open, and any rules I've given you." Paste that into a fresh chat.
Give it the part that matters. The relevant section, not all 200 pages.
Put the important bit near the end. Ask your actual question last, after the material.
Sources
- Chroma, "Context Rot: How increasing input tokens impacts LLM performance", Jul 2025: trychroma.com/research/context-rot
- Liu et al., "Lost in the Middle: How language models use long contexts", 2023: arxiv.org/abs/2307.03172
- Anthropic, "Effective context engineering for AI agents", Sep 2025: anthropic.com/engineering/effective-context-engineering-for-ai-agents