Progressive Disclosure for Agent Context: Load Content Only When the Agent Needs It
An agent that loads your whole product catalog into every prompt answers slowly, costs more, and still quotes the wrong spec. In an agentic architecture, the content layer fixes this with progressive disclosure.
An agent that loads your whole product catalog into every prompt answers slowly, costs more, and still quotes the wrong spec. In an agentic architecture, the content layer fixes this with progressive disclosure. The agent holds a small index of summaries for the whole conversation and retrieves full documents or fields only on demand. In Sanity Context, a Knowledge Base outline served by the `initial_context` tool is that index, and GROQ projections fetch the next layer.
Two things are at stake: cost and accuracy. Even a million-token window doesn't mean you should fill it. Too much context can lead to semantic collapse and misdirection, and every extra token is billed again on every turn. Teams that stuff top-k chunks into the prompt pay for that context whether the agent reads it or not.
This guide treats retrieval as a series of small, deliberate loads instead of one big one. It covers the pattern, how it compares with up-front RAG stuffing and context compression, the content structure that makes it work, how to evaluate it, and where Sanity fits.

What is progressive disclosure for agent context?
Progressive disclosure for agent context is a loading pattern. The agent first sees a compact index of what exists, with a name and a one-line description for each item, and pulls in full content only when a task calls for it. Context starts small and grows in deliberate steps, and the question in front of the agent justifies each one.
The clearest reference implementation is Anthropic's Agent Skills format. At startup the agent loads only each skill's name and description from its SKILL.md file. When a task matches a description, the full SKILL.md body loads. Bundled scripts and reference files load only when the task needs them. Anthropic's documentation and the open SKILL.md spec describe this as three stages: discover, activate, and fully load. Anthropic's essay on context engineering generalizes the idea as just-in-time retrieval. The agent keeps lightweight identifiers in context, such as file paths, stored queries, and links, and resolves them at runtime through tools.
Applied to content instead of skills, the same three stages look like this. Discover: the agent holds an outline of topics or documents, each with a short summary and a stable identifier. Activate: the agent fetches the one document, or a projection of it, that the outline suggests holds the answer. Fully load: the agent follows a reference or requests the specific fields it still needs, such as a price, a stock location, or a compatibility table.
Software has used this pattern for a long time. Good documentation sites already work this way: a table of contents, then a page, then an anchor. What changes with agents is that every byte in the window is paid for on every turn, and irrelevant bytes make reasoning worse. Progressive disclosure treats context as a budget the agent spends on evidence, not a bucket the platform fills in advance. A content layer built for agents can answer at all three levels of detail, with the same canonical source behind each one.
Why does stuffing retrieved content into the prompt up front fail?
Stuffing retrieved content into the prompt up front fails because it pays the full context cost on every turn and makes the model worse at using what it receives. The standard RAG loop embeds the user's question, pulls the top-k chunks from a vector index, and pastes all of them into the prompt before the model has decided what it needs.
Three problems compound. The first is cost. Those chunks are billed on every turn whether the agent reads them or not, and a multi-turn conversation sends them again and again. The second is accuracy. Even a million-token window doesn't mean you should fill it, because too much context can lead to semantic collapse and misdirection, where the model loses track of what it is doing. Five near-duplicate chunks covering three product generations are how an agent ends up quoting the wrong spec with full confidence. The third is latency. Larger prompts take longer to process, and your customers feel the delay.
Chunking makes all three worse. A chunk is a slice of text cut by token count, not by meaning, so a price can land in one chunk and the condition that qualifies it in another. The model gets fragments without the structure that told a human reader how they relate.
Tool menus fail the same way, because tool descriptions ride along with every request just like retrieved content. Ten focused tools beat fifty overlapping ones. At around forty tools, the model gets confused and picks the wrong one. The index-first approach applies here too: expose a small set of well-described tools, and have each tool return a narrow, schema-shaped result.
None of this means retrieval is wrong. It means retrieval should be a decision the agent makes for a specific question, not a preprocessing step the platform runs blind. Up-front stuffing aims for recall on the first turn. Progressive disclosure aims for precision across the whole conversation and lets the agent go back for more when its first answer turns out to be incomplete.
When should an agent disclose more, summarize, or retrieve again?
An agent should disclose more when the current layer can't answer the question. It should summarize when its own working history is filling the window. It should retrieve again when the content it needs has a canonical source it can re-fetch. Progressive disclosure and context compression work together, but they act on different kinds of context.
It helps to split context into four kinds, each with its own lifetime and owner. Static instructions, such as the system prompt and brand rules, are scoped to a release and owned by product or brand teams. Retrieved content is pulled on demand by the agent based on the question, and the content team owns it. Per-turn state, such as the user's cart or session, belongs to your application. Agent-authored notes, such as summaries of earlier work, persist across sessions.
Retrieved content is the natural target for progressive disclosure because it already lives somewhere authoritative. If the agent drops a product document from context, it can fetch it again by id with nothing lost. Summarizing that document instead puts a paraphrase between the model and the facts, and paraphrasing is where facts go to die. Tools should return schema-shaped objects, not prose rewrites.
Compaction belongs on the other side. Anthropic's context engineering essay describes compaction as summarizing a conversation near the context limit and restarting from the summary. The essay pairs it with structured note-taking and sub-agents for long-running tasks. Conversation history and agent-authored notes have no source to re-fetch, so compressing them is the only way to keep them.
This gives you a practical rule. If the missing information has a stable identifier in your content backend, disclose the next layer or retrieve again. If the bloat is the agent's own reasoning trail, compact it. If the agent keeps re-retrieving the same document across turns, keep the identifier and a one-line summary in its notes instead of the full body. Then it can fetch the fields again when the conversation comes back to that topic.
What content structure lets an agent load one layer at a time?
Content that supports progressive disclosure has three properties. Each document has a short summary, its typed fields can be requested one at a time, and explicit references let the agent follow relationships one hop at a time. Without these, there is no smaller layer to load first, and the agent is back to reading whole pages.
Summaries give the discover stage something to show. A one- or two-sentence summary field, written for a reader deciding whether to open the document, is the content equivalent of a skill description. Typed fields make the later stages precise. An agent answering a pricing question needs price, currency, and availability, not the marketing body copy. If those values sit in named fields instead of a rich text blob, the agent can ask for just them. References give the agent a path. A product that references its warranty policy, its compatible accessories, and its stock location lets the agent move to the one related document it needs, instead of pulling in everything nearby by similarity.
Schema alone isn't enough, and this part surprises people. Teams that point an agent at a well-typed schema still see it fail in three places. One is counter-intuitive field names, like a field called body that actually holds a slug. Another is second-order reference chains the schema doesn't connect: to find products with a given feature, you chain product to productFeature and match on the feature's id. The third is gaps in data quality. The model needs to know the shape of the data, not just its types.
That makes content modeling a decision about agent architecture. This is the Model your business pillar in practice. The content model should mirror how your business actually reasons about products, policies, and support answers. Descriptions on fields and documents should tell an agent what each one means and where to look next. A summary field and reference follows are ordinary modeling practices, not special features, but every later layer depends on them.
Types are not shape
What does a content layer look like in an agentic architecture built on Sanity Context?
In an agentic architecture built on Sanity Context, the content layer is the Content Lake, exposed to agents through the Sanity Context MCP endpoint. It offers two ways to disclose content progressively. GROQ projections return exactly the fields an agent asks for. Knowledge Bases give the agent a compact outline before it reads anything.
The GROQ path works one layer at a time. A first query filters with predicates that must hold, ranks with hybrid scoring, slices to a handful of results, and projects only a few fields:
*[_type == "product" && category == $category && price < $maxPrice && stockLocation == $warehouse] | score(boost([title] match text::query($queryText), 2), text::semanticSimilarity($queryText)) | order(_score desc) [0...10] { _id, title, price, "stock": stockLocation->{ name, available } }
The `match()` clause is a BM25 keyword match on the title, weighted 2x with `boost()`, and `text::semanticSimilarity()` adds a semantic score across the whole document. A second call fetches full fields for the one `_id` the agent picked, or follows a reference with `->`. Dataset embeddings are tied to the content, so a price change or a deleted record reaches retrieval without a separate pipeline.
The Knowledge Base path, currently in beta, mirrors Agent Skills directly. Its Outline lists every entry path plus a one-line summary. It is small enough for an agent to hold in context for a whole conversation and is served by the `initial_context` tool. Entries are Markdown documents with citations back to the source, and the agent retrieves them on demand. A Purpose statement tags entries `[core]` or `[peripheral]`, so the agent knows what matters most before it opens anything.
This is where Sanity, the Content Operating System for the AI era, differs from CMSes that bolt on AI. Retrieval, freshness, and governance live in one backend. Agent instructions are stored as documents, authored in the Studio, and staged with Content Releases.
One source behind all three layers
When should you use GROQ mode versus Knowledge Base mode in Sanity Context?
In Sanity Context, use GROQ mode when the schema tells the agent where to look, and Knowledge Base mode when finding the answer is the hard part. Both are forms of progressive disclosure, but they keep the index in different places.
GROQ mode queries the dataset at request time. A request like 'size L latex gloves, under $200 a pallet' is a filter over product documents. It stays exact across hundreds of thousands of records, with no build step and nothing to keep in sync. Here the schema itself acts as the discover layer, the projection is the activate layer, and reference follows are the full load. GROQ mode needs a deployed schema, using `sanity schema deploy` with Studio v5.1.0 or later.
Knowledge Base mode serves an index built ahead of time. It suits sources where the answer could be anywhere, such as overlapping PDF manuals, a documentation site, and support material that describe the same topics in different words. The build organizes the outline by what the sources are about, not by folder structure, URL paths, or file names, so each fact has one home. The pre-generated outline gives the agent a strong hypothesis about where the answer lives before it starts reading.
The middle case is a structured catalog where the useful detail sits in prose fields. The answer there is not to mix source types. Enable dataset embeddings and stay in GROQ mode, so one query combines structured filters with semantic ranking. One rule matters for anyone writing these queries: `text::semanticSimilarity()` is valid only as an argument to `score()`. Semantic search ranks, it does not filter. Narrow the candidate set with predicates first, then rank what is left.
A Sanity Context MCP connection serves one source type. If both a dataset and a Knowledge Base are attached, the dataset wins silently, which is easy to miss in testing. Override this per connection with `?mode=groq` or `?mode=knowledge_base` so each agent gets the disclosure path its content needs.
How do you measure accuracy and token cost with and without progressive loading?
To measure progressive loading, run the same frozen eval suite twice, once with up-front retrieval and once with index-first loading, and compare success rate and cost per conversation side by side. With only one of those numbers, a team can easily ship a cheaper agent that answers worse, or a more accurate one nobody can afford to run.
Start with about twenty representative conversations. Pull them from real support logs and sales questions, and include the hard ones: multi-hop questions that need a reference follow, and questions where the obvious document is outdated. Then freeze the set. Score each conversation against a written rubric. Did the agent answer correctly, cite the right source, and avoid inventing values? Rerun the suite on every model, prompt, and tool change, and ship only when the results stay green.
The second metric after success rate is cost per conversation. Log input and output tokens per turn, the number of tool calls, and how many tokens each retrieval returned. Progressive loading should show smaller first-turn prompts and more tool calls, each returning less. Check whether the total goes down, because an agent that discloses badly can make ten round trips where one would do. Track model routing too, since a small model can often handle the outline step while a larger one writes the final answer. Watch retrieval cache behavior as well.
Budget for the evals themselves. Agent evals can use several times as much as every other project on a team, so keep the suite small and representative instead of large and noisy.
There is no universal figure for how much progressive disclosure saves. The savings depend on how much irrelevant context your current pipeline carries, so your own baseline is the benchmark. In Sanity Context, the comparison is concrete. One configuration projects full documents. The other projects `_id`, title, and a summary first and fetches fields on demand. Both run against the same Content Lake and the same twenty conversations.
Score both axes on the same suite
Progressive disclosure support: Sanity Context vs Pinecone, Contentful, and pgvector / Neon
| Feature | Sanity | Pinecone | Contentful | pgvector / Neon |
|---|---|---|---|---|
| Summary-first index the agent holds in context | Knowledge Base Outline (beta): every entry path plus a one-line summary, served by the `initial_context` tool and small enough to stay in context for a whole conversation. | Assembled: no native summary-first index layer. Teams model a summary namespace or metadata field and write their own outline tool. | Assembled: content types can hold a summary field, but an agent-facing outline must be built on App Framework plus an external search service. | Assembled: Postgres stores summary columns well, but you design the id-first outline tool and its query logic yourself. |
| Field-level retrieval, layer by layer | GROQ projections return only the requested fields, such as `{ _id, title, price }`. The first call returns summaries and ids, the second returns full fields for the chosen document. | Returns matches with stored metadata. Full field-level content usually lives in a separate store that your code fetches after the vector query. | Field selection is available through its delivery APIs, so agents can request specific fields. Ranking for retrieval is handled by an external search service. | SQL column selection is native. You choose which columns each agent tool returns and write the tool layer that exposes them safely. |
| Following references one hop at a time | `->` dereferences inside the same query, for example `stockLocation->{ name, available }`, so the agent follows a relationship without calling a second system. | No document relationship model. Relationships are encoded in metadata and resolved by application code against your source of truth. | References between entries resolve through its APIs. Agent tooling that walks them one hop at a time is yours to build and maintain. | Foreign keys and joins are native to Postgres. Exposing them as safe, single-hop agent tools is application work your team owns. |
| Hybrid retrieval inside the content backend | Native: a `match()` BM25 clause and `text::semanticSimilarity()` blended with `score()` and `boost()` in one GROQ query, after predicates filter the candidate set. | Metadata filtering is native. Combining keyword relevance with dense similarity is configured in your index setup and query code, outside the content source. | Relies on external search through App Framework integrations, so hybrid ranking quality depends on the search vendor you connect. | Possible with pgvector plus Postgres full-text search. You write and tune the SQL that blends and weights the two scores. |
| Index freshness when content changes | Dataset embeddings are tied to content, so edits, price changes, and deletions reach retrieval without a separate vector pipeline to maintain. | Re-embedding on change, handling deletions, and running backfills are a pipeline you build and maintain alongside your content source. | Keeping an external search index in sync with content changes is an integration your team owns, monitors, and repairs when it drifts. | Embeddings can sit beside rows, but regenerating them when text changes requires triggers or background jobs you write and operate. |
| How production agents connect | Sanity Context MCP endpoint in GROQ mode or Knowledge Base mode, overridable per connection with `?mode=groq` or `?mode=knowledge_base`. | Agents reach it through your own tool or framework integration, so the retrieval interface and disclosure steps are the ones you design. | Agents typically connect through custom tools built over its APIs and the external search layer, with no built-in retrieval mode to choose. | Agents connect through tools you write over SQL, and access control and query safety become part of that build. |
| Governed agent instructions | Agent instructions stored as documents, authored in the Studio, and staged with Content Releases, so editors preview agent behavior before it ships. | Outside its scope. System prompts and agent instructions live in your application code or a separate configuration tool. | Prompts could be modeled as entries, but staging agent behavior alongside content changes is a pattern your team assembles. | Outside its scope. Prompts live in code or a table you manage, with versioning and review workflows built separately. |