How to Build a Hybrid Retrieval Pipeline in a Single GROQ Query

Your agent gets asked "trail runners under $150 like a Hoka" and returns nothing, or worse, invents a product that does not exist. The query carried a real structural component, price under $150, category shoes, in stock, that pure vector retrieval cannot resolve, because vector similarity does not respect those constraints. So the result comes back empty or wrong, and the model either hallucinates or hedges depending on the prompt. The fix is not a better model. It is better retrieval.

Sanity Context (previously Agent Context) is the AI Content Operating System surface for grounding agents in real content, an intelligent backend that runs hybrid retrieval natively rather than asking you to assemble it from a vector database and glue code. Most "AI-powered search" products are pure embeddings and only that. Real retrieval is a discipline, not an ingredient: keyword search for literal matches, embeddings for semantic ranking, and structured predicates for the filters that have to hold, all combined and ranked together.

This article shows how that combination collapses into a single GROQ query inside the Content Lake, why production data says the structural side is where agents fail first, and how that compares to building the same pipeline yourself on Pinecone, Postgres, or a headless CMS with search bolted on.

Why pure vector search returns empty or wrong

Start with the failure mode, because it is more common than teams expect. The pure-embeddings recipe is simple: encode your content as vectors, encode the query as a vector, return the nearest neighbors. It works beautifully for fuzzy semantic match, "find me something like a trail runner." It falls over for anything structural: "trail runners under $150, in stock at the Portland warehouse, men's size 11." Vector similarity has no opinion about price, stock, or size. It will happily return a $240 road shoe that is out of stock because the embedding sits close in vector space. Most AI-powered search products are this and only this, which is why so many of them feel magical in a demo and frustrating in production.

The inverse failure is just as real. Pure structured query, GROQ, SQL, or GraphQL, gives you exactly what you ask for. You write the predicate, you get the matching rows. But it falls over the moment the user says "something like X" or "the cozy one" or anything else that lives in vibes, not fields. And it requires you to know exactly what you are looking for, which is rarely the case when a person is exploring and chatting with an agent. So you have two tools, each excellent at the thing the other one cannot do.

The consequence is the hallucination most enterprise teams are actually fighting. When a query with a structural component comes back empty from a vector-only system, the model fills the gap. It does not know the warehouse was out of stock; it knows it was asked a confident question and is expected to answer. The reframe for the rest of this guide is that you do not solve this with a smarter model. You solve it by giving the model retrieval that respects both the filters and the vibe at once.

Hybrid retrieval is three signals running in parallel

Hybrid retrieval is the discipline of running all three signals together and combining their results. Keyword search using BM25 catches literal matches, the exact product name, the version number, the SKU. Embeddings provide semantic ranking for intent the user could not phrase precisely. Structured predicates enforce the filters that simply have to hold, the price ceiling, the category, the in-stock flag. None of the three is sufficient alone, and that is not an opinion, it is measured.

Anthropic's contextual retrieval research quantified the layering directly. Contextual embeddings cut top-20 retrieval failures by 35 percent. Adding contextual BM25 on top took that reduction to 49 percent. Adding a reranking step brought it to 67 percent. The shape of the improvement holds whether you read the paper closely or just notice that each layer kept removing failures the previous layers left behind. "We have embeddings" is not a retrieval strategy; it is one ingredient in a recipe that needs all three.

Walk the canonical query through it. "Trail runners under $150 like a Hoka" decomposes into three retrieval signals running in parallel. A structured predicate handles category equals shoes and price under 150. A BM25 lexical match scores the keywords "trail" and "runner." A vector similarity score comes from the embedding of the full query, which is what carries the "like a Hoka" intent that no predicate can express. All three feed a combine-and-rank step that produces a short, ordered list honoring the hard constraints and the soft preference together. The architectural question is not whether to do this. It is where the combine-and-rank step lives, and who maintains the index that feeds it.

The whole pipeline in one GROQ query

Here is where the Content Lake earns its place. In GROQ, using the text search operators documented in the Sanity docs, the entire hybrid pipeline is a single query. The predicates filter first, the score pipeline blends keyword and semantic relevance, and the order step returns a ranked slice:

*[ _type == "product" && category == $category && price < $maxPrice && stockLocation == $warehouse ] | score( boost([title] match text::query($queryText), 2), text::semanticSimilarity($queryText) ) | order(_score desc) [0...10]

Read it left to right. The predicates do the filtering that has to hold, category, price ceiling, and warehouse, so nothing out of stock or over budget can survive into the ranked set. The score pipeline then blends a BM25 keyword match on the title, weighted 2x with boost() because a title hit matters more than a body hit, with a semantic similarity score across the whole document via text::semanticSimilarity(). Finally order(_score desc)[0...10] returns a small ranked list that matches both the structural constraints and the vibe.

Notice what is not in that query: no second system, no metadata-filter syntax bolted onto a vector index, no application code stitching three result sets together. The combine-and-rank step that other architectures assemble out of multiple services is a single GROQ scoring pipeline running inside the store that already holds your content. This is the "Power anything" pillar in practice. Structured content with semantic clarity, queried in one pass, and delivered to an agent through the Sanity Context MCP endpoint. The query is the integration.

Freshness is the line item every other stack keeps

Be honest about the alternatives, because they are capable. PostgreSQL can do hybrid retrieval with pgvector and full-text search. Elasticsearch can. Algolia is purpose-built for the structured-plus-relevance case. Pinecone plus a metadata filter layer can. The query syntax is not the moat, and pretending it is would be dishonest. What none of them can do is pure-vector their way out of the empty-result problem, and what all of them require is the part the demos skip: a content pipeline that keeps the search index fresh.

That pipeline is a real project and a class of bug all its own. Incremental indexing so new content appears. Re-embedding on change so an edited document does not return a stale vector. Deletion handling so removed products stop showing up. Eventual-consistency reasoning so your agent is not answering from yesterday's index. Backfill for schema changes when a field is added or renamed. Build it yourself on a separate vector database plus glue code, and freshness becomes a permanent line item on your roadmap, owned by whoever is on call.

When retrieval is wired into the content backend, that problem stops being something you maintain. Because the embeddings in the Content Lake are tied to the content rather than living in a parallel store, the freshness machinery is the backend's job, not your roadmap's. This is one of the five differentiators in concrete form: a separate vector DB creates a silo you keep in sync, while the Content Operating System provides a shared foundation where the content and its retrieval index are the same source of truth. The agent queries live content, not a copy that drifts.

Production data: the structural side is where agents fail first

There is a counter-intuitive lesson hiding in how agents actually use retrieval, and it cuts against the industry's embedding-first reflex. When Sanity looks at how agents call the Context MCP endpoint in production, structured retrieval dominates. The heavy majority of calls are GROQ queries and schema lookups, with compressed initial context behind that. Semantic search is a small slice of the traffic, not the main event.

Embeddings adoption follows the same shape. On Context MCP, embeddings are opt-in and off by default, and most projects shipping in production never turn them on. That is not because embeddings are bad. It is because the structural side of the query is where agents fail first. An agent that cannot find the right field, cannot follow a reference chain, or does not know that "in stock" lives in a joined document will fail long before semantic ranking ever becomes the bottleneck. Vector search and RAG are one ingredient, not the whole discipline.

The evidence is in the hard cases. Running schema exploration against Sonos's catalog, an honest nightmare of a dataset, landed around 83 percent accuracy on a mix of difficulties using Sonnet 4.5, at roughly 40 seconds of thinking per hard question. Getting there meant teaching retrieval counter-intuitive field names, second-order reference chains, and data-quality issues the schema cannot reveal on its own. As the team put it, none of that is a model problem, it is a context problem. The model needed to know the shape of the data, not just its types. That is the work hybrid retrieval has to support, and it is why the structured predicate, not the vector, is usually the load-bearing part of the query.

Governing the agent's retrieval in the Studio, not in YAML

A retrieval pipeline in production is not just a query, it is a set of decisions about scope, voice, and what the agent is allowed to see. On most stacks those decisions live in a YAML file or a system prompt buried in application code, edited by an engineer, deployed on a release train, and invisible to the people who own the content. That is a governance gap, and it is where retrieval quietly drifts out of editorial control.

Sanity Context closes that gap by putting agent behavior in the Studio. The system prompt lives in a Sanity document, so an editor can tune the agent's voice and scope without a code change. Behavior is staged with Content Releases the same way you stage the website, with drafts, scheduling, version history, permission gating, and audit trails. You can preview how a new instruction changes the agent before it reaches users, then schedule it, then roll it back if it misbehaves, all through the same surface that already governs publishing. The agent stops being a separate system with separate rules.

This is what it means to call Sanity Context the intelligent backend for companies building AI content operations at scale rather than a headless content layer. Retrieval, the instructions that shape it, and the content it runs against share one governed foundation, backed by SOC 2 Type II, GDPR, regional hosting, and a published sub-processor list. And the surface area is broader than a single endpoint: alongside the Context MCP endpoint, Knowledge Bases turn datasets, websites, PDFs, and support databases into agent-readable documents that share the same retrieval path, so the hybrid query you wrote for products extends to the rest of your content without a second pipeline to govern.