Qdrant

Your Qdrant-backed retrieval works great in the notebook. Then it ships, and a user asks "what did we publish about pricing last quarter?" The top vector hit is a draft from two years ago, the second is an internal memo that should never have been indexed, and the third is right but ranked fourth behind two near-duplicates. The embeddings are doing their job. The problem is that "last quarter," "published," and "external" are structural facts, not semantic ones, and cosine similarity has no opinion about them.

Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset, with Knowledge Bases as the second surface for unstructured sources like PDFs, websites, and support data. The reason it matters to a Qdrant user is narrow and specific: when your retrieval failures trace back to structure rather than meaning, you need predicates and freshness, not a better embedding model.

This article stays on the Qdrant side first. We will walk through payload filtering, the precision-versus-recall trap with filtered HNSW search, and how to keep your collection in sync with editorial truth. Then we will draw the line: which content belongs in Qdrant, and which belongs behind a single GROQ query in Sanity Context.

The metadata-filtering gap, concretely

Qdrant has payload filtering, and it is good. You attach a JSON payload to every point, then constrain search with a `Filter` of `must`, `should`, and `must_not` clauses. So why do teams still hit a wall?

The gap is not that filtering is missing. It is that filtering and similarity ranking pull against each other, and the failure mode is silent. When you add a `must` clause for `status == "published"` and `published_at` within a date range, Qdrant has to find the nearest vectors that ALSO satisfy the filter. If your filter is selective (say, 0.5% of the collection matches), the HNSW graph that makes approximate search fast suddenly works against you: the nearest neighbors in vector space mostly fail the filter, so the search either degrades to a slow exhaustive scan or returns fewer, worse results than you expected.

The second half of the gap is that your payload is a copy. The `status` field, the `author_ref`, the `published_at` timestamp, these all live authoritatively somewhere else (a CMS, a database, an editorial tool). The moment an editor unpublishes an article or fixes a date, your Qdrant payload is stale until the next sync job runs. Vector search returns a confidently-ranked result that is simply wrong about the world. Neither the embedding nor the filter is broken. The data is just behind.

These two problems, filter selectivity and payload staleness, are the ones that actually page you at 2am, not the cosine math.

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue, DatetimeRange

client = QdrantClient(url="http://localhost:6333")

query_vector = embed("what did we publish about pricing last quarter")

results = client.query_points(
    collection_name="articles",
    query=query_vector,
    query_filter=Filter(
        must=[
            FieldCondition(key="status", match=MatchValue(value="published")),
            FieldCondition(
                key="published_at",
                range=DatetimeRange(gte="2024-10-01T00:00:00Z", lt="2025-01-01T00:00:00Z"),
            ),
        ],
        must_not=[FieldCondition(key="visibility", match=MatchValue(value="internal"))],
    ),
    limit=5,
).points

# Looks correct. But if status/published_at were synced 6 hours ago,
# an article unpublished this morning is still in here.

Tuning filtered HNSW search before you blame the model

Before you reach for anything new, there are real Qdrant knobs for the selectivity problem, and most teams never touch them.

First, build payload indexes on the fields you filter by. Without an index, Qdrant evaluates the filter by checking points one at a time during graph traversal. With a payload index, it can use the cardinality of a filter to decide its strategy. Create them explicitly with `create_payload_index` for `status`, `visibility`, and a `datetime` index on `published_at`. This is the single highest-leverage change for filtered queries.

Second, understand Qdrant's filterable HNSW. Qdrant estimates how many points a filter will match. If the filter is broad, it searches the HNSW graph and applies the filter inline. If the filter is very selective, below a configurable `full_scan_threshold`, it switches to a plain payload scan because that is genuinely faster than fighting the graph. You can tune `full_scan_threshold` per collection. Get it wrong in either direction and you pay in latency or recall.

Third, when recall matters more than latency on a hard query, raise `hnsw_ef` at search time. A higher `ef` widens the candidate set the graph explores, which recovers neighbors that a selective filter would otherwise prune away. The cost is real CPU per query, so raise it for the queries that need it, not globally.

These three changes fix a large share of "filtering returns junk" reports. Reach for them first. They are cheaper than re-architecting your stack, and they keep the workload where Qdrant is genuinely strong: high-volume approximate nearest neighbor search over a large vector space.

from qdrant_client.models import PayloadSchemaType, SearchParams

# One-time: index the fields you filter on
client.create_payload_index(
    collection_name="articles",
    field_name="status",
    field_schema=PayloadSchemaType.KEYWORD,
)
client.create_payload_index(
    collection_name="articles",
    field_name="published_at",
    field_schema=PayloadSchemaType.DATETIME,
)

# Per hard query: widen the search frontier so a selective
# filter doesn't prune the right answer out of the candidate set
results = client.query_points(
    collection_name="articles",
    query=query_vector,
    query_filter=my_filter,
    search_params=SearchParams(hnsw_ef=256),
    limit=5,
).points

The sync job is where production actually breaks

Indexes and `ef` fix ranking. They do not fix staleness, and staleness is what generates the support ticket that says "the agent recommended a product we discontinued."

Every Qdrant deployment that filters on editorial state needs a pipeline that watches the source of truth and re-upserts points when state changes. That pipeline is its own small distributed system. You have to handle the article that gets unpublished (delete the point or flip the payload), the article that gets re-embedded because its body changed (re-upsert the vector AND the payload atomically), and the reference that moves (an author merges into another author, and now `author_ref` on a thousand points is wrong). Miss any of these and the gap reopens.

The honest framing is a routing decision, not a tool war. Ask what kind of content you are retrieving. High-volume, machine-generated, or append-mostly corpora, log lines, scraped pages at web scale, embeddings you generate yourself and never edit by hand, belong in Qdrant. That is what it is built for, and a dedicated vector database earns its keep there.

But editorial content, the catalog, the articles, the help center, anything a human reviews, approves, and changes, has a different shape. Its structural facts (status, locale, references, publish dates) change independently of its text, and they change by human action, not by re-embedding. Mirroring that into Qdrant means you own a sync job whose entire purpose is to chase a database that already answers structural questions correctly. That is the moment to ask whether the structural slice of your retrieval should live where the structure already lives.

Where Sanity Context fits: structure stays where structure lives

For the editorial slice, Sanity Context lets you skip the mirror entirely. Instead of copying `status`, `published_at`, and `author_ref` into Qdrant payloads and chasing them with a sync job, you query the dataset where those fields are authoritative, and they are never stale because you are reading the source.

The production reality here is worth stating plainly, because it is the opposite of the usual RAG narrative. The heavy majority of agent retrieval calls against a Sanity dataset are structured: GROQ queries and schema lookups. Semantic search is a small slice, and in Sanity Context embeddings are opt-in and off by default. Most projects shipping on Context MCP never turn them on. So the first thing Sanity Context replaces is not your vector search. It is the filtered query you were forcing through a vector store because that is where you had put the data.

A GROQ query expresses your structural predicates directly: published state, date range, locale, dereferenced author, all in the query language, all against live data. No payload index to maintain, no `full_scan_threshold` to tune, no nightly job. The structural questions that were fighting your HNSW graph are just predicates inside `*[ ... ]`.

This is the Content Operating System framing, the intelligent backend for companies building AI content operations at scale: the structural truth about your content stays governed and current in one place, and the agent reads it through a query rather than through a copy. You keep Qdrant for the high-volume embedding work it is good at, and you stop paying the staleness tax on the part of your corpus that humans actually edit.

*[
  _type == "article" &&
  status == "published" &&
  visibility != "internal" &&
  publishedAt >= "2024-10-01T00:00:00Z" &&
  publishedAt < "2025-01-01T00:00:00Z"
]{
  _id,
  title,
  publishedAt,
  "author": author->name,
  "excerpt": pt::text(body)[0...280]
} | order(publishedAt desc)

When you do need semantics: hybrid lives inside one GROQ query

Sometimes the query genuinely has a semantic component. "Articles about pricing" is fuzzy, "published last quarter" is not, and the right answer needs both. The wrong fix is to run a structural query in Sanity, run a vector query in Qdrant, and try to reconcile two ranked lists in application code. That reconciliation is where relevance goes to die.

If you opt embeddings on in Sanity Context, the structural and semantic parts collapse into a single GROQ query. The structural facts stay as predicates inside `*[ ... ]`, and the semantic ranking happens in a `score(...)` pipeline. `text::semanticSimilarity($queryText)` contributes a vector-similarity term, `body match text::query($queryText)` contributes a BM25 keyword term, `boost()` weights them, and `order(_score desc)` ranks the combined result. One query, one ranked list, live data, no reconciliation step.

Notice what is NOT in the `score()` block: the structural filters. Status, date range, and visibility stay as hard predicates in the `*[ ... ]` selector, exactly where they belong. The semantic operators only decide ranking among the rows that already passed the structural gate. This is the hybrid discipline that the metadata-filtering gap was really asking for, and it runs against content that is current by construction.

The routing rule stays simple. Structured editorial content goes through GROQ retrieval. Unstructured sources, PDFs, scraped websites, support transcripts, go through Knowledge Bases, which turns a messy corpus into ordered documents with a table of contents. Genuinely high-volume machine-generated vectors stay in Qdrant. You are not replacing your vector database; you are no longer asking it to be a stale copy of your editorial system.

*[
  _type == "article" &&
  status == "published" &&
  publishedAt >= "2024-10-01T00:00:00Z"
]
| score(
    boost(body match text::query($queryText), 1.0),
    text::semanticSimilarity($queryText)
  )
| order(_score desc) [0...5] {
  _id,
  title,
  _score,
  "author": author->name
}

Wiring it into your agent: Context MCP first

The fastest way to give an agent access to this is Context MCP, the hosted read-only MCP endpoint. Your agent loop attaches it as an MCP server and gets schema-aware tools, including GROQ execution, reference traversal, and (if enabled) semantic search, without you hand-writing a single tool wrapper. Because the endpoint is read-only, an agent can read and query but cannot mutate the dataset through MCP; writes go through Agent Actions, not the MCP surface. That read-only constraint is a feature when you are letting a model loose on production content.

If you want full control over the exact query, the second path is a thin custom tool that runs a typed GROQ query through the client and returns the rows to your agent. This is useful when you want to pin the query shape, validate inputs, or cap result size yourself rather than letting the model compose queries freely.

Whichever path you choose, the observability lesson from production carries over directly: when an agent gives a wrong answer, the trace almost always points at a bad retrieval, not a bad model. Log what the agent actually saw, the GROQ query and its result, or the MCP tool call and response, next to what it said. With Qdrant you were logging vectors and filters and then guessing whether the payload was current. With a query against live data, the trace is the query, and the query is the truth at that moment.

Keep Qdrant for the embedding-heavy, high-cardinality work. Move the structural, governed slice of retrieval to where the structure already lives. The metadata-filtering gap closes not because you found a better filter, but because you stopped copying the data you were trying to filter.

// Path 1: attach Context MCP as a read-only MCP server
// (schema-aware GROQ + reference traversal out of the box)
// mcp endpoint: https://<projectId>.api.sanity.io/<version>/context/mcp

// Path 2: a thin typed GROQ tool for full query control
import { createClient } from 'next-sanity'

const sanity = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: 'production',
  apiVersion: '2024-10-01',
  useCdn: false, // read live, not a cached copy
})

export async function searchArticles(queryText: string) {
  return sanity.fetch(
    `*[_type == "article" && status == "published"]
     | score(
         boost(body match text::query($queryText), 1.0),
         text::semanticSimilarity($queryText)
       )
     | order(_score desc) [0...5]{ _id, title, _score }`,
    { queryText }
  )
}