Context rot is why your AI agent gets worse mid-task
By Lindsey J
Your AI agent gets more capable every model release. It can still get noticeably worse in the middle of one long conversation. Researchers studying long-context benchmarks call this "context rot," a term Anthropic's own engineering team uses in its guidance for agent builders. As the number of tokens in a context window grows, a model's ability to accurately recall information from earlier in that context goes down, even when nothing else about the model changes. This is not a prompting mistake. It is a property of how transformer models attend to tokens. It has a design answer.
Every token in a context window has to attend to every other token. Add tokens and the number of pairwise relationships grows faster than the window itself. A model's attention stretches thinner the longer a task runs. Anthropic frames the decline as a gradient rather than a cliff: capability erodes steadily rather than all at once. A model stays highly capable overall. Its precision on information retrieval and long-range reasoning still declines as context grows. Anthropic states this pattern emerges across all models, though some show more gentle degradation than others.
What context rot means for an agent you ship
A one-turn question never meets this problem. A multi-step agent does: a support flow, a design review, a long coding session, anything with more than a handful of back-and-forth turns. The failure is not dramatic. The agent does not crash. It does not refuse. It quietly re-asks for information the user already gave it. It can lose track of a decision made ten turns earlier while sounding exactly as confident as it did on turn one. A companion piece on this site walks through where context sits among the other terms in an agent stack.
The metric most product teams are not tracking yet
Most teams measure an AI agent at the session level, tracking whether the task completed, whether the user escalated, and how long it took. Few break that number down by turn count or conversation length. That breakdown is the diagnostic this problem needs. Plotting task success against turn number is one way to see it. A context rot problem then shows up as a visible decline rather than a flat line with occasional noise. A team that already logs agent transcripts has this data today. Nobody has asked it the question yet.
A decision rule beats a bigger context window
The common fix is a bigger model with a bigger window. Anthropic's own guidance argues the opposite. Treat context as a precious, finite resource. Curate the smallest set of high-signal tokens for each step rather than letting the window fill up by default. Three techniques carry that principle into a shipped product. Compaction summarizes and reinitializes the window while keeping the facts and decisions that still matter. Structured note-taking has the agent write an external memory file instead of relying on everything staying in the window. Sub-agent delegation lets a focused agent return a condensed summary, often 1,000 to 2,000 tokens, instead of handing its full working context back to a coordinator.
The decision rule that follows is simple to state and easy to skip. Set a context budget: the percentage of the window (not the raw token count, which changes across models) at which the agent compacts, writes its note, or hands off. Trigger that step automatically, before the model hits its own limit. Waiting for a visible failure means a user already watched the agent get something wrong. The same discipline that evaluating a prompt change before shipping it already demands applies here too.
A worked example
Picture a support team running an AI agent through a multi-step account migration. The first ten exchanges go well. By exchange thirty the agent re-asks for an account number the customer already gave it twice. The detail is still technically in the window. The model is simply attending more heavily to everything that arrived after it. The fix is not a cleverer prompt. It is a compaction step built into the flow at a fixed point. That step pulls out the facts that still matter, the account id, the migration step completed, the outstanding blocker, and writes them into a short structured note. From there the model's attention budget restarts clean.
Self-assessment: is your agent exposed to context rot
- Do you measure AI agent task success by turn count or conversation length, not only by overall session outcome
- Does your agent have a fixed context budget that triggers compaction or hand-off, or does it run until the model's own window fills
- When your agent hands a task to a human, does the hand-off carry a structured summary or the raw transcript
- Has your longest real agent session, not your demo script, been tested against this failure mode
FAQ
Is context rot the same as running out of memory
No. Running out of memory is a hard limit. The window is full and nothing more fits. Context rot is a soft limit. The agent still has room in the window. Its accuracy on anything far back in that window measurably declines before the window is full.
Does a bigger context window fix this
A bigger window does not fix this on its own. Anthropic states this pattern emerges across all models it describes. Some decline more gently than others. None are exempt. A bigger window changes where the decline starts. It does not change whether the decline happens.
Who owns this decision, product or engineering
Both, by design. Engineering builds the compaction and hand-off mechanism. Product sets the budget threshold and the success metric that proves the mechanism is working. That split is the same one product already owns for a conversion threshold, with engineering owning the instrumentation behind it.
If your product's AI agent handles anything longer than a quick question, its accuracy at turn forty is worth checking as closely as its accuracy at turn one. We help product teams find where context rot already costs them accuracy. Then we design the compaction or hand-off logic that catches it before a user does. Talk to us about an agent reliability review.
