What goes wrong when there's too much on AI's desk?
A full desk isn't the only problem. What's on it matters too.
Drew Breunig, who writes about how AI systems are built, sorted the ways a conversation goes wrong into four types:
You've already met poisoning in "Why AI gets stuck on its first idea". This lesson covers the other three.
Context distraction: the conversation gets so long that the model leans on what's in it, and stops using what it learned in training.
When Google had its Gemini model play Pokémon, it worked fine at first. Once its running notes grew past about 100,000 tokens (a few hundred pages), it started repeating moves it had already made instead of coming up with new plans.
AI had so much of its own history to read that it mostly re-ran it.
You've been working on a slide deck with it for two hours. You ask for a fresh idea for the closing slide.
AI suggests a slight variation of the opening slide, which it already wrote three times today.
The best material on the desk is its own earlier work, so it keeps reaching for it.
Made-up example.
Context confusion: things on the desk that have nothing to do with the task end up shaping the answer anyway.
The model doesn't skip what's irrelevant. If it's on the desk, it gets read, and it gets used.
Researchers at Berkeley run a public leaderboard that tests how well models use tools. Every model did worse when it was given more than one tool, and every model sometimes reached for a tool that had nothing to do with the job.
You paste in last quarter's sales report "just for background" and ask for help writing a job ad.
The job ad mentions revenue targets.
Or you mention once that you're tired, and every answer after that gets a little gentler and shorter.
AI used what you gave it. It just couldn't tell which parts you meant it to use.
Made-up examples.
Context clash: new information arrives that conflicts with something already on the desk, and the model gets tangled between the two.
The most common cause is giving it a task in pieces. You say what you want, then add a detail, then change your mind about a detail.
Every version is still on the desk.
Microsoft and Salesforce researchers took tasks that models handle well when asked in one go, then fed the same information in over several messages instead.
Performance dropped by 39% on average.
OpenAI's o3 model went from 98.1 to 64.1 on one test, just from getting the same request in pieces.
The models often made an early guess at what you wanted, then struggled to let it go when the rest arrived.
Give the whole request in one go. If you change your mind halfway through, start a fresh chat with the final version instead of patching the old one.
Only paste what the task needs. "Background" is still on the desk.
When answers start sounding like earlier answers, clear the desk and start again with a short summary (see "Why long chats make AI worse").