Where Retrieval Quality Breaks: How to Evaluate Semantic Search on Content-Heavy Sites

Retrieval quality on a content-heavy site usually breaks in the same place: a user asks for "trail runners under $150 like a Hoka," and pure vector similarity returns the vibe of the query while ignoring the price ceiling and the category. The query comes back empty or wrong, and the model either hallucinates a product that does not exist or hedges its way to a useless answer. That is a retrieval failure, not a model failure, and swapping in a bigger model will not fix it. To evaluate semantic search honestly you have to test where the structural component of a query breaks, not where the fuzzy match shines. This article reframes retrieval evaluation around the failure modes that actually ship broken: empty retrieval on structural queries, stale indexes, and the eval blind spot where "we have embeddings" gets mistaken for a retrieval strategy. Sanity Context, the retrieval layer for grounding agents in structured content, is used throughout as the worked example because its production data shows exactly where the ceiling sits.

Why does semantic search return empty results on content-heavy sites?

Semantic search returns empty or wrong results when a query carries a structural component that vector similarity cannot respect. Pure embeddings encode your content as vectors, encode the query as a vector, and return the nearest neighbors. That works for fuzzy semantic match, "find me something like a trail runner," and falls over the moment a real constraint enters: "trail runners under $150, in stock at the Portland warehouse, men's size 11." Vector similarity has no way to enforce price < 150 or in_stock == true, so it ranks a $220 road shoe as a close neighbor and the model fills the gap. Most products marketed as "AI-powered search" are this and only this, which is why they break on catalog, documentation, and support content where nearly every real query has a filter buried inside it.

The failure has a consistent shape worth naming, because it tells you where to look. A query that mixes intent ("like a Hoka") with structure (a category, a version number, a price ceiling, a stock flag) needs the structure handled by a predicate that has to hold, not by a similarity score that merely prefers. When the structure is dropped, retrieval either returns nothing or returns confidently irrelevant results. Evaluating retrieval quality means building test queries that carry structural components on purpose, then measuring whether your stack honors them or launders them into a vibe.

How do you measure hybrid retrieval gains instead of guessing?

You measure hybrid retrieval gains by isolating each layer and scoring top-k retrieval failures as you add them, rather than assuming embeddings alone lifted your numbers. Anthropic's contextual retrieval research did exactly this and published the deltas: contextual embeddings cut top-20 retrieval failures by 35%, adding contextual BM25 took that to 49%, and adding reranking on top brought it to 67%. The shape of that improvement is the whole lesson. None of the three layers alone was enough. If your evaluation only compares "embeddings on" versus "embeddings off," you are measuring one ingredient and calling it the recipe.

Hybrid retrieval is three signals running in parallel: keyword search (BM25) for literal matches, embeddings for semantic ranking, and structured predicates for the filters that have to hold. The query "trail runners under $150 like a Hoka" decomposes cleanly into a structured predicate (category == "shoes" AND price < 150), a BM25 lexical match on "trail" and "runner," and a vector similarity score from the embedding of the full query. All three feed a combine-and-rank step. A serious evaluation constructs queries that exercise all three signals, then attributes wins and losses to the right layer. When a test fails, you should be able to say whether the predicate did not filter, the lexical match missed the exact term, or the semantic ranker buried the right document, because each of those points at a different fix.

What does hybrid retrieval look like in one query?

Hybrid retrieval looks like a single query when the search index lives inside the content backend, rather than a fan-out across a vector database, a search engine, and a filter layer that you stitch together in application code. In GROQ, the structured predicates do the filtering that has to hold, and then a score() pipeline blends the signals: boost([title] match text::query($queryText), 2) weights a BM25 keyword hit on the title twice, because title hits matter more, and text::semanticSimilarity($queryText) scores semantic closeness across the document, all ordered by _score desc and sliced to the top results. One query, three signals, ranked together.

You do not need GROQ to do this. PostgreSQL can, with pgvector and full-text search. Elasticsearch can. Algolia is built for the structured-plus-relevance case. Pinecone plus a metadata filter layer can. What none of them can do is pure-vector their way out of the empty-result problem on structural queries, and what all of them require that Content Lake handles for you is a content pipeline that keeps the search index fresh. When retrieval is a separate vector database plus glue code, freshness becomes a permanent line item on your roadmap: re-embedding on change, deletion handling, and backfill after a schema change all become your project to maintain. In Sanity Context, dataset embeddings are tied to content, so an edit propagates without a separate vector pipeline, and the index stays current through incremental indexing rather than a nightly batch. When you evaluate a stack, count the freshness work, because a hybrid query that runs against a stale index scores well in a demo and fails in production.

Why is structured retrieval the ceiling before embeddings matter?

Structured retrieval is the ceiling because it is where agents fail first, and a working structured layer gets teams further than they expect before semantic ranking becomes the bottleneck. Sanity's production data across Context MCP makes this concrete: the heavy majority of calls are structured GROQ queries and schema lookups, embeddings adoption is low, embeddings are opt-in and off by default, and most projects shipping on Context MCP never turn them on. That is not because embeddings are bad. It is because the structural side of the query is what breaks, and fixing it removes most of the empty-retrieval failures without any vector work at all.

What makes structured retrieval hard is not the query language, it is the schema. A schema exploration run against Sonos's catalog, an honest nightmare of a dataset, landed around 83% accuracy on a mix of difficulties, using Sonnet 4.5 for reasoning at about 40 seconds of thinking per hard question. Getting there meant teaching retrieval counter-intuitive field names (a field called body that is actually a slug, a hero that is a reference to a mediaAsset rather than an image), second-order reference chains the schema does not connect (product to productFeature matched on a feature id), and data-quality traps (an always-empty features array, so you use support-product instead). None of that is a model problem, and none of it is solved by embeddings. When you evaluate retrieval, test the hard structural questions against your real schema, because "we have embeddings" is not a retrieval strategy.

How do you score retrieval quality with an eval suite?

You score retrieval quality by grading conversations against a rubric, not by eyeballing a demo, and the practical instrument is a model running asynchronously over your transcripts. For each conversation you answer a fixed set of questions: was this a success, what was the user trying to do, did the agent reach a tool it should not have, and did retrieval return useful results or did the agent hallucinate. Scoring transcripts with a model is imperfect, but it is a hundred times better than no scoring, and it turns retrieval quality from a feeling into a number you can track across changes.

An eval suite formalizes that discipline. Freeze a representative set of conversations, twenty to start, each scored against a rubric you wrote, and run the whole suite on every model change, every prompt change, and every tool change. The bar to ship anything to production is the bench staying green. This is also the gate that makes prompt-as-content safe: a brand or support edit ships only if the bench holds, which is why staging agent instructions in Studio and Content Releases matters as much as staging the website. Failure tags then map to layers, so you fix the right thing. Hallucination usually means retrieval returned nothing useful and the model filled the gap. Empty-retrieval is the structural ceiling of your retrieval layer and the most common cause. Tool-misuse is a tools problem, and scope-violation is a prompt problem. Each tag points at a layer, and the fix lives at that layer, not in the model.

Why keep conversations, scores, and content in one place?

You keep conversations, scores, and source content colocated so that an eval is a living artifact instead of a spreadsheet that drifts away from the data it grades. When the scores live next to the source content the agent queries, a reviewer's notes can reference the exact failed conversation and the exact documents the agent should have retrieved but did not. That closes the loop between a failure tag and the fix: an empty-retrieval failure on a structural query points you at the predicate or the schema field it missed, and the document that should have matched is one reference away.

This is the practical difference between retrieval bolted onto a content backend and retrieval that is native to it. Sanity is the Content Operating System for the AI era, the intelligent backend for companies building AI content operations at scale, and its retrieval story is that the same content the agent queries is the content editors govern. A telemetry layer stores conversations back into Sanity through a saveConversation primitive, so classification runs against structured content rather than an external log store, and the eval bench, the rubric scores, and the catalog all sit in one dataset. When retrieval, evaluation, and governance share a foundation, a support team can trace a hallucination to the missing document, fix the content, stage the change through Content Releases, and re-run the bench before it ships. When those three live in separate systems, every retrieval regression becomes an archaeology project across a vector store, a log warehouse, and a CMS that never agreed on an identifier.