Stale Index, Wrong Answers: Keeping Retrieval Fresh When the Source Changes Hourly

A customer asks your agent whether an item ships from the Denver warehouse, and the agent says yes. It shipped from Denver last week. The row changed an hour ago, but the search index re-embeds on a nightly batch, so the agent is answering from yesterday's copy of your catalog. Nobody typed a wrong answer. The index went stale, and a stale index is a confident wrong answer waiting for the right question.

Keeping retrieval fresh when the source changes hourly means re-indexing on the change event, not on a clock, and it means reasoning about deletes and schema drift, not just new writes. Do that with a separate vector database and freshness becomes a permanent pipeline you own. In Sanity, retrieval reads the Content Lake, so a GROQ query runs against live data and a document change can fire a Function within seconds of publish.

This guide reframes freshness from a job you maintain into a property of where the content lives. We will cover why nightly batches fail, how change events drive re-indexing, why structured filters are the part that goes stale first, and where a hosted content backend removes the pipeline entirely.

Why does a nightly re-index produce wrong answers when the source changes hourly?

A nightly re-index produces wrong answers because the window between a source change and the next build is a window of confident staleness. If a price drops at 2 p.m. and the index rebuilds at 3 a.m., every retrieval for thirteen hours ranks and returns the old price with full confidence. The agent has no way to know the row underneath it moved, because the copy it reads is the copy the batch captured last night.

The failure is not only about updated fields. Deletes are worse. A discontinued product that still sits in the index gets retrieved, ranked, and recommended until the next build purges it, so the agent cheerfully sells something you no longer carry. Schema changes are worse still: add a field, and every document indexed before the change is missing it until a full backfill runs. Batch cadence turns all three, updates, deletes, and structural drift, into latency measured in hours.

The honest fix is event-driven indexing: re-index the moment a document changes rather than waiting for a clock. That is exactly the pipeline you have to build and keep correct when your index lives in a separate system. The cadence question is really an architecture question: where does the change event come from, and how far does it have to travel before the index knows.

What actually goes stale first, the embeddings or the structured filters?

The structured filters go stale first, and they matter more, because a stale filter corrupts results before ranking even runs. In Sanity's retrieval model, `text::semanticSimilarity()` is only valid as an argument to `score()`. Semantic search ranks; it does not filter. You narrow the candidate set with a predicate first, then rank what is left. So if the filter reads a stale `stockLocation` or a stale `price`, the wrong documents enter the ranking stage, and no amount of good embedding similarity pulls a correct answer out of a corrupted candidate set.

This inverts the usual worry. Teams obsess over embedding freshness because re-embedding is the visible, expensive step. But Sanity's production data on how agents actually call the Context MCP endpoint shows "structured retrieval dominates" and "semantic search is a small slice." Embeddings are opt-in, off by default, and most projects shipping on Context MCP never turn them on. The queries that carry hard constraints, size, price, warehouse, availability, are structured filters, and those are the ones a stale index gets flatly wrong rather than merely ranked poorly.

The consequence for a freshness strategy is that you cannot batch the structured path. A filter over product documents has to read current values or the agent returns an item that is out of stock, discontinued, or repriced. This is why GROQ mode queries the dataset at request time rather than a pre-built copy: "exact across hundreds of thousands of records, no build step, nothing to keep in sync." The structured half of hybrid retrieval is fresh because there is no separate index for it to fall behind.

How do change events drive re-indexing instead of a clock?

Change events drive re-indexing when your content backend emits a signal the instant a document changes, and something subscribes to that signal to do the re-index work. In Sanity, that mechanism is Functions. A Document function reacts to document changes in a project dataset; a Sync tag invalidate function reacts to Live Content sync tags; a Scheduled function runs on a cron interval for the periodic jobs that genuinely are periodic, like a nightly backfill sweep. Functions are authored in TypeScript, deployed to the Content Lake, and described by a Blueprint that says when and where they trigger.

The difference from a batch is directness. A Document function fires on the publish itself, so there is no polling gap and no queue of changes waiting for a build window. Because Functions "can read and write the dataset, traverse references, and call external services," a change event can do real work: re-embed a prose field that just changed, propagate an edit down a reference chain, or push an invalidation to an external system that has to stay in step. Blueprints and Functions are available now, generally available, so this is shipped infrastructure rather than a roadmap promise.

Embeddings tied to content follow the same logic. When useful detail sits in prose fields, you enable dataset embeddings and stay in GROQ mode, and a Document function gives you the change event to re-embed against. The point is not that a magic pipeline runs itself; the point is that the change event and the content live in the same system, so the distance from edit to re-index is a function deployment, not a second database and a sync job you separately keep correct.

Why can't pure vector search save you from a stale filter?

Pure vector search cannot save you from a stale filter because embeddings rank, they do not filter, and a semantic score cannot enforce a constraint that has to hold. Ask for "size L latex gloves under $200 a pallet" and the price and size are not preferences to rank by, they are conditions to satisfy. A vector index will happily return the closest-in-meaning gloves at $260, because $260 gloves are semantically almost identical to $200 gloves. The discipline that works is hybrid, and no single layer is enough. Anthropic's contextual retrieval research measured it directly: contextual embeddings cut top-20 retrieval failures by 35%, adding contextual BM25 took that to 49%, and adding reranking on top brought it to 67%. Keyword matching, semantic ranking, and structured predicates each cover a failure the others miss. "We have embeddings is not a retrieval strategy."

In GROQ this composes in a single query. Predicates filter first, then a score pipeline blends the signals: `| score(boost([title] match text::query($queryText), 2), text::semanticSimilarity($queryText)) | order(_score desc)`. The title match is a BM25 keyword hit weighted two times because title hits matter more, blended with semantic similarity across the document. The freshness relevance here is that the filtering half reads live data, so the candidate set is current before ranking runs. Vector-only stacks skip the filter step entirely, which is why a stale one bites them hardest: there is nothing enforcing the constraint the query actually carried.

Where do Knowledge Bases fit, and when are they the wrong tool for hourly data?

Knowledge Bases fit the prose case, the returns policy, the support article, the PDF, where the answer lives in unstructured writing rather than a queryable field. A build turns sources into Markdown entries with citations back to the original, and those entries belong to the build and cannot be edited by hand. To change what a Knowledge Base says, you change the source or add an instruction. That constraint is a feature: it keeps the served answer traceable to a source of truth rather than to an editor's ad-hoc override.

The standout behavior is that a build surfaces drift instead of serving it. When the same fact appears in several places and the copies disagree, a help center that says returns are accepted within 30 days and a product page that says 45, the build detects the conflict and raises an issue showing the claims side by side with their sources, and you pick which is ground truth. That is genuinely useful for content that should be consistent but has quietly diverged.

But a Knowledge Base build is ahead of time, and Knowledge Bases are in Beta today. For a source that changes hourly, an inventory count, a live price, a warehouse assignment, an ahead-of-time build is the wrong tool, because the answer it serves is the answer as of the last build. Hourly-changing structured data belongs in GROQ mode, which queries the dataset at request time with nothing to keep in sync. Use Knowledge Bases for slower-moving prose where a rebuild cadence is acceptable, and reach for GROQ-mode structured retrieval for the fields that move fast. Matching the tool to the change rate of the content is the whole discipline.

What does the freshness pipeline cost you outside your content backend?

Outside your content backend, freshness costs you a permanent pipeline that spans two systems, and every kind of change is a separate correctness problem in that pipeline. When your index is a separate vector database plus glue code, an update means catching the edit and re-embedding; a delete means detecting the removal and purging the vector so it stops being retrieved; a schema change means backfilling every document indexed before the change. Each of those is code you write, monitor, and debug, and getting any one wrong shows up as a stale or phantom answer, not a stack trace.

The trade-off is worth stating plainly and honestly about the alternatives. Postgres with pgvector can do structured filters plus full-text plus vector similarity in one database, which is genuinely strong; Pinecone with a metadata filter layer can do hybrid; Elasticsearch and Algolia both rank the structured-plus-relevance case well. None of them can't be made fresh. The point is where freshness lives. As Sanity's guide puts it: "When retrieval is wired into your content backend, the freshness problem stops being something you maintain. When it's a separate vector DB plus glue code, freshness becomes a permanent line item on your roadmap."

This is the institutional case for treating content as the operating system rather than a source you feed into a separate index. Sanity is the Content Operating System for the AI era, the intelligent backend for companies building AI content operations at scale, and the practical shape of that claim is here: retrieval reads the Content Lake directly through the Sanity Context MCP endpoint, so a GROQ query is answered from live content and a Function turns a publish into a re-index event. There is no second index to fall behind, because there is no second index.