Context Compression for AI Agents: When to Summarise, Prune or Retrieve Again

Forty turns into a support run, the agent quotes a 30-day return window when the published policy now says 45. The old number came from a rolling summary written three compactions earlier. To avoid this, split the agent's own memory from shared knowledge. The agent can summarize or prune its own working notes. Policies, prices, and IDs stay out of summaries and get fetched again when needed. In Sanity Context, a GROQ query against the Content Lake returns the current document by `_id` at request time. A dropped fact comes back as it is published now, not as an old summary described it.

The cost of getting this wrong builds up over a run. Every tool result and every model response is added to the history and sent back on every turn. Long runs and background agents pay for the same tokens again and again while accuracy slips. Chroma's July 2025 "Context Rot" report tested 18 models and found that reliability drops as input length grows, even on simple copying tasks.

This guide treats compression as a sorting problem, not a size problem. Content the agent wrote can be compressed. Content the business owns should be kept as a reference and retrieved again when the agent needs it.

Why does context grow on long agent runs?

Context grows on long agent runs because the harness adds every model response and every tool result to a history array, then sends the whole array back to the model on every turn. That array is the context. Nothing leaves unless the harness removes it. If a run calls a search tool, reads three documents, and checks an order status, all of that raw output is still there at turn ten, turn twenty, and beyond. A common sample loop caps a run at 50 iterations. A background agent working through a queue overnight can get close to that cap on a single task.

The growth shows up in two places. The first is cost. Cost per conversation is the second metric to watch after success rate. You control it through token economics, model routing (a small model for easy turns, a large one for hard ones), and how well retrieval results are cached. A long tool result that stays in history is billed again on every turn that follows it. Agent evaluations make this obvious, because eval runs alone can use several times as much as every other project on a team.

The second is accuracy. A bigger window doesn't fix the problem. Even a million-token window shouldn't be filled, because too much context leads to semantic collapse and misdirection: the model loses track of what it was doing. Old tool output competes for the model's attention with the instruction it is trying to follow now. A price lookup from turn four sits next to a revised price from turn eighteen, and the model has to guess which one is current.

The practical rule follows from this. If a run burns thirty tool calls on one conversation, add a budget to the loop. Context management belongs in the harness, the code that decides what goes into the window and how much. Compression is one of the decisions that code makes. Picking the wrong kind of compression is how an agent ends up confidently wrong at turn forty.

What are the main ways to compress agent context?

There are four main ways to compress agent context: rolling summaries, pruning tool output, keeping references and retrieving again, and keeping memory outside the model. Anthropic's guidance on context engineering treats the window as a finite attention budget and describes versions of all four. The Claude Platform also offers server-side context editing that clears old tool results and thinking blocks or compacts history automatically. Each technique removes tokens in a different way, and each loses something different.

1. Rolling summaries (compaction). The harness replaces older turns with a summary the model writes. This keeps the thread of the task, and it is the right default for conversational state: what the user asked, what was tried, and what failed. The cost is that a summary is a paraphrase, and a summary of a summary is a paraphrase of a paraphrase. Exact numbers and clause wording are the first things to erode.

2. Pruning tool output. The harness clears raw tool results once the agent has acted on them and keeps a stub, such as the tool name and its arguments. This is cheap and safe for bulky output the agent no longer needs, like a full page of search results after it has picked one. It is risky when the agent may need the raw value later and has no way to get it back.

3. Keeping references and retrieving again. The agent keeps a document's ID, or the query that found it, instead of the document. When it needs the fact again, it fetches the current version. Nothing gets paraphrased and nothing goes stale, but this only works if the source supports exact lookup.

4. Keeping memory outside the model. The agent writes notes to an external store, such as Anthropic's memory tool, and reads them back in later sessions. This suits user preferences, progress on a long task, and lessons learned.

You don't pick one technique for everything. You decide which content each technique applies to. Summarize the story of the task. Prune output the agent has already used. Retrieve again for facts the business owns. Write external notes for things only the agent knows.

Which facts must stay word for word, and which can be summarized?

Keep a fact word for word when a paraphrase could be wrong in a way that costs money, breaks trust, or fails an audit. That covers policies, prices, IDs, SKUs, legal and compliance wording, dates, and quantities. Anything that describes the progress of the task can be summarized: the user's goal, the approaches tried, the tools that failed, and what the agent plans to do next.

The reason comes down to how models handle prose. A tool that returns prose forces the model to paraphrase, and paraphrasing is where facts go to die. Teams built agents against the Sanity Context MCP endpoint, and the agents that worked well had tools returning schema-shaped responses the model could pass straight through. The ones that struggled got a wall of text back and re-narrated it badly. A rolling summary takes every structured result and turns it into that wall of text after the fact. "Returns accepted within 45 days, excluding final-sale items" turns into "flexible return policy" by the second compaction, and the final-sale exception is gone.

For each field in context, ask one question: if this were deleted, could the agent get back exactly the same value? If it has an ID or a query that returns the same object, the field can leave the window. If not, keep it word for word, either in context or in an external store. Only summarize content where a close approximation is good enough.

This gives the harness a short set of rules. Keep identifiers exact and never let a summary rewrite them. Drop full documents but keep their IDs in a small reference list. Never summarize a price, a policy clause, or a regulatory statement. Retrieve it again. Summarize reasoning, attempts, and dialogue freely.

There is one exception to watch for. Some facts must stay word for word but belong only to the agent's session, such as a ticket number the user typed. No shared source has them, so they belong in external memory or in a pinned slot that the compaction step is told to keep. If such a fact is treated as summarizable, it will be lost at the first compaction.

What are best practices for agent memory vs a shared knowledge base?

Treat agent memory and a shared knowledge base as different kinds of context with different owners, and never let one stand in for the other. Agent memory is what the agent writes for its own future use: notes, summaries, and condensed playbooks. A shared knowledge base is content people write that the agent retrieves: product data, policies, and documentation. When the two get mixed, a summary the agent wrote at 2 a.m. can end up overriding the published return policy.

A practical taxonomy splits context into four parts. Static instructions are tied to a release and owned by product or brand teams. Per-turn runtime state lasts a single turn and is owned by the application. Retrieved content is stable and owned by the content team. Agent-authored notes carry across sessions and are owned by the agent. Each part needs different governance. A stack that treats all four as one bucket ends up with the wrong people making the wrong decisions. Anthropic has introduced "dreaming", where an agent reflects on a session and writes a playbook for the next one. That makes the boundary between memory and knowledge more important, not less.

Best practices follow from that split. Let the agent compress, rewrite, and expire its own memory freely, because it owns that content. Never let it copy business facts into memory as if they were its own notes. It should store a reference and retrieve the fact from the knowledge base. Give the knowledge base the governance that memory lacks: citations, review, and a way to detect when two sources disagree. When a source changes, the source should win over the agent's stale notes about it.

Memory layers that run alongside the model, such as Mem0, Zep, and LangGraph memory stores, are built for agent memory: preferences and session history across conversations. They aren't designed to be a governed source of truth for policies, prices, or catalog data. That job needs content a team owns and edits, which is what a shared knowledge base is for.

How does Sanity Context let an agent drop facts and fetch them again?

In GROQ mode, Sanity Context queries the dataset at request time. That lets an agent drop a fact from its context window and fetch the current version later. There is no build step and nothing to keep in sync. An agent that keeps only a document's `_id`, or the GROQ query that found it, can run `*[_id == $id]` at turn forty and get what the content team has published. GROQ mode needs a deployed schema (`sanity schema deploy`, Studio v5.1.0 or later). The schema is what tells the agent where to look.

This changes what compression has to keep. The agent's history can hold a short list of IDs and queries instead of full documents. The harness can then prune or compact aggressively, because the facts that matter can be recovered exactly. Results come back as schema-shaped objects the model can pass straight through, so the price in the answer comes from the price field, not from the model's restatement of it.

Sometimes the agent needs to search for something rather than look up a known ID. With dataset embeddings enabled, a GROQ query can combine structured filters with semantic similarity. A hybrid query wraps a BM25 `match()` on the title, weighted with `boost()`, and `text::semanticSimilarity()` together in `score()`. It then orders results by `_score` and returns `_id`, title, price, and stock. One rule applies: `text::semanticSimilarity()` only works inside `score()`, so it ranks results but doesn't filter them. Filter the candidate set first. The embeddings live with the content in the Content Lake, so a price change or a deletion reaches the index without a separate vector pipeline.

This is what Sanity means by the Content Operating System for the AI era. CMSes bolt on AI. Sanity is built for it. In practice, the agent retrieves from the same governed store the business edits, so a fact it fetches again can't drift from the published version. For shared knowledge that isn't in structured datasets, Knowledge Bases provide a compact outline through the `initial_context` tool. The outline is small enough for an agent to keep in context for a whole conversation, and cited entries sit behind it.

How do you test context compression on your own agent runs?

Test context compression the same way you test any other harness change. Use a fixed set of representative conversations, score them with a rubric you wrote, and rerun the set on every model, prompt, and tool change. Twenty conversations is enough to start. Once compression kicks in, it changes what the model sees on every later turn. It can lower the success rate just as easily as a prompt edit can, so don't ship it until the test set still passes.

For compression, the most useful method is replay. Take saved long traces from real runs, ideally ones that went past the point where compaction or pruning would start. Replay each one twice: once with the full history and once with the compression strategy turned on. Then compare the final answers on the facts that must be exact. Does the price in the answer match the price in the source? Did the policy clause keep its exceptions? Did order and document IDs come through unchanged? A summary that keeps the task on track but changes one digit in an ID is a failure, even if a general quality rubric scores it well.

Measure cost alongside accuracy. Record tokens and tool calls per conversation for both versions, because cost per conversation is the second metric after success rate. A strategy that saves tokens but loses exact facts doesn't actually save anything. Also watch for the opposite problem. Aggressive pruning can raise cost when the agent repeats tool calls to recover what it threw away. If a conversation burns thirty tool calls, add a budget to the loop and treat that as a new rule.

Turn every failure the replay finds into a rule. If a summary rewrote a policy, mark that field as never-summarize and replace it with a reference the agent retrieves again. If an agent on Sanity Context loses a price after pruning, check that it kept the document's `_id` so a GROQ lookup can restore the current value. Running these replays on every change is how you know a compression setting works before you rely on it.