How to Evaluate and Tune Vector Search Without Breaking Production Relevance

Your retrieval quality looked fine in the demo, and then a support agent confidently cited a deprecated refund policy to a customer. Nobody changed the model. Somebody changed a chunk size, or a reranker threshold, or added ten thousand new documents, and relevance quietly degraded in production while your offline eval still showed green. This is the failure mode that makes teams afraid to touch their vector search at all: every tuning change is a bet placed blind, with real customer answers on the line.

Sanity Context (previously Agent Context) exists to close that gap. Sanity is the AI Content Operating System, an intelligent backend designed to keep AI workflows governed, reviewable, and safe inside the editorial loop, so a relevance change is something you stage and inspect rather than ship and pray. The retrieval path that agents query is the same Content Lake your editors already trust.

This guide reframes vector search tuning as a governed engineering discipline rather than a one-off science experiment. We will cover how to build an evaluation set that reflects real queries, how to change embeddings and hybrid weights without silent regressions, and how a content-native retrieval stack removes an entire class of pipeline drift you would otherwise have to babysit.

Why offline relevance scores lie to you in production

The comforting number on your evaluation dashboard measures a frozen moment. Production is not frozen. Content gets added, corrected, and retired every day, embeddings for new documents land through a different code path than your original backfill, and the query distribution your real users generate looks nothing like the fifty hand-picked questions you validated against. A recall@10 of 0.92 on a static set tells you almost nothing about whether the agent found the current refund policy this morning.

The deeper problem is coupling. In a typical stack, the content store, the embedding pipeline, the vector index, and the reranker are four separate systems with four separate deployment cadences. A change to any one of them can move relevance, and because they are decoupled, the change is invisible to the others. You update a product description in the CMS; the vector index does not know until the next sync job runs; the eval set still contains the old text. The result is drift that no single dashboard can see, because no single system owns the whole path.

This is the first thing to fix conceptually. Tuning vector search is not a matter of finding the one true hyperparameter. It is a matter of making the entire retrieval path observable and reproducible, so that when relevance moves you can attribute the movement to a specific, reviewable change. Sanity Context collapses those four systems toward one: embeddings are tied to content in the Content Lake, so when the description changes, the vectors that back retrieval update within minutes, and the thing your eval measures is the thing your agent actually queries.

Building an evaluation set that survives contact with real users

A good evaluation set is not a list of questions you find interesting. It is a sample of the questions your users actually ask, labeled with the passages that should have answered them. Start by mining production logs for real query text, cluster them so you cover the head and the long tail, and for each cluster select representative queries and annotate the golden documents a correct answer must retrieve. This gives you graded relevance rather than a single right answer, which matters because most real questions have several acceptable supporting passages and one or two that are subtly wrong.

The metrics follow from the labels. Recall@k tells you whether the right material made it into the candidate set at all, which is the failure that produces hallucination because the model literally never saw the answer. Mean reciprocal rank and normalized discounted cumulative gain tell you whether the right material ranked high enough to survive the context window and the reranker. Track them separately, because a change that improves recall while hurting ranking is common and easy to misread as pure progress.

The part teams skip is refresh. An evaluation set decays the moment content changes underneath it, so treat the golden labels as content that lives beside the content they describe. Because Sanity Context serves both the agent retrieval path and your editorial content from the same Content Lake, you can query the current state of a document with GROQ at eval time rather than trusting a stale snapshot, and you can catch the case where a golden document was edited or retired since you last labeled it. Your eval measures reality, not a photograph of last quarter.

Tuning hybrid retrieval: weights, thresholds, and what actually moves relevance

Pure semantic search retrieves on meaning and misses exact tokens; a customer typing an error code or a SKU wants the literal match, and cosine similarity will happily rank a thematically related but wrong document above it. Pure keyword search does the opposite, nailing the exact token and missing the paraphrase. Hybrid retrieval blends both, and the tuning question is how to weight them and where to threshold, per query type rather than globally.

The knobs that genuinely move relevance, in rough order of impact: chunk size and overlap, because a chunk that splits a policy mid-sentence can never be retrieved cleanly; the blend weight between semantic and lexical scores; the candidate count you retrieve before reranking; and the reranker threshold that decides what reaches the model. Change one at a time, measure against your graded set, and keep the semantic and lexical contributions legible so you can see which one carried a given result.

In Sanity Context this blending is native inside the Content Lake rather than assembled across a vector database and a separate search engine. A single GROQ query can combine text::semanticSimilarity() for meaning with a BM25 match() for exact tokens, and blend them with score() and boost() so the error code and the paraphrase both surface with weights you control. Because it is one query against one store, the thing you tune is the thing that runs in production, and there is no second system whose weights can silently disagree with the first. You are tuning a query, not reconciling a pipeline.

Changing embeddings without silently regressing every query

Swapping an embedding model is the single most dangerous relevance change you can make, because it invalidates every vector at once. A new model with a better benchmark average can still be worse for your specific corpus, worse for a specific query cluster, or worse in ways that only show up under the long tail. The mistake is to reindex in place and ship, discovering the regression when a customer does. The discipline is to treat an embedding change like a schema migration: build the new index alongside the old, run both against your graded evaluation set, and compare per-cluster, not just in aggregate.

The operational trap is the backfill window. Reembedding a large corpus takes time, and if new content lands through one embedding version while old content sits under another, you get a split index where relevance depends on when a document happened to be written. That inconsistency is exactly the kind of drift that offline evals miss and production surfaces.

Because dataset embeddings in Sanity Context are tied to content rather than maintained as a separate vector pipeline, updates propagate within minutes and there is no long-running external job to fall out of sync with your source of truth. Staging an embedding or instruction change through Content Releases lets you preview the new retrieval behavior against real content and promote it deliberately, the same way your team already stages a website launch. The relevance change becomes a reviewable release with a rollback, not an irreversible reindex you notice after the fact.

Governing the change: staging, review, and rollback for relevance

The organizational reason relevance breaks in production is that retrieval tuning usually lives entirely in engineering config, invisible to the content and support teams who own the correctness of the answers. An engineer bumps a threshold in a config file; nobody who knows the domain reviews it; the regression ships. Treating relevance as code without treating it as content is how confident wrong answers reach customers.

The fix is to give relevance changes the same lifecycle a content change gets: a staging environment that mirrors production, a review step by someone who can judge whether the new behavior is actually better, and a clean rollback when it is not. That means the agent instructions, the retrieval configuration, and the content itself all need to be stageable together, because a change to any one of them can move the answer.

Sanity Context puts that governance in the Studio. Editors and domain owners govern agent instructions and can stage agent behavior through Content Releases the same way they stage the site, so a relevance change is previewed, reviewed by someone accountable for the answer, and promoted or rolled back as a unit. Roles and permissions decide who can change what, and the change is auditable after the fact. This is the pillar the section maps to, automate everything, applied honestly: the automation is safe precisely because a human in the editorial loop can inspect and reverse it before customers ever see it.

Monitoring relevance after you ship, not just before

Evaluation before release is necessary and insufficient. The query distribution keeps moving, content keeps changing, and the only way to know your relevance held is to watch it in production. Log the queries, the retrieved candidates with their semantic and lexical contributions, and, where you can capture it, a downstream signal of whether the answer helped: a thumbs rating, a resolved ticket, an escalation. These signals feed straight back into the evaluation set as new labeled cases, so the set grows toward your real traffic instead of decaying away from it.

Watch for the specific patterns that precede a visible failure. A rising share of queries where the top candidate's score is barely above threshold means your retrieval is getting less confident, often because content changed under it. A cluster of queries that suddenly retrieve fewer golden documents points at a chunking or embedding change. Catching these in a dashboard is the difference between a quiet fix and a customer-facing incident.

Because the retrieval path in Sanity Context is a GROQ query against the Content Lake rather than an opaque external index, the same query language you tune with is the one you observe with, and content changes that would move relevance are already versioned in the store. Sanity is the intelligent backend for companies building AI content operations at scale, which in practice means the loop from a production signal, to a content or instruction fix, to a staged and reviewed release, to a fresh eval, runs inside one governed system rather than across four disconnected ones.