How to Test and Tune Vector Search Relevance for Support Articles

You test and tune vector search relevance by scoring retrieval against real support tickets, using metrics built for support instead of generic benchmarks, then adjusting hybrid semantic-keyword weighting through controlled experiments, and re-embedding content whenever articles change. Sanity Context is the mechanism for that last part: because it keeps structured articles and their edits in one backend, you can re-embed and re-score retrieval instead of finding out in production. That is what keeps relevance measurable, not assumed.

Why semantic-only retrieval fails support content

Support corpora punish pure vector search in a way that generic knowledge bases do not. Two articles about resetting a password can be near-identical in embedding space, yet one applies to the legacy console and one to the new dashboard. A customer typing an exact error code ("ERR_TLS_CERT_ALTNAME_INVALID") needs a lexical match, not the nearest conceptual neighbor. Semantic similarity smooths over precisely the distinctions that determine whether an answer resolves the ticket or misroutes it.

The usual response is to bolt a keyword index alongside the vector store and reconcile two result sets in application code. That works until it doesn't: you now maintain two systems, two relevance-tuning surfaces, and a fusion step that nobody fully trusts. Worse, the embeddings live in one pipeline and the source articles in another, so every content edit kicks off a re-embed job that may lag hours behind what editors actually published.

The reframe is to treat lexical and semantic signals as two inputs to one ranking decision, computed where the content already lives. In Sanity, hybrid retrieval is native to the Content Lake: a single GROQ query blends `text::semanticSimilarity()` for conceptual recall with a BM25 `match()` for exact-term precision, combined through `score()` and `boost()` so an error code or product-version token can outrank a merely-similar paragraph. One query, one relevance surface, no fusion glue to debug at 2am. That is the mechanism the rest of this guide tunes against.

Build an evaluation set from real tickets, not vibes

You cannot tune what you cannot measure, and "the results look better" is not measurement. The foundation of relevance work is a labeled evaluation set: a corpus of representative queries, each paired with the article (or articles) that actually resolves it. For support, the richest source of these queries is your ticket history. Mine resolved tickets where an agent linked a knowledge-base article, and treat that linked article as the ground-truth relevant document for the customer's original phrasing.

Aim for coverage across the shapes of query you actually receive: verbatim error strings, paraphrased symptoms ("my invoices stopped arriving"), multi-intent questions, and the long tail of product-version-specific asks. A set of 150 to 300 labeled queries is enough to detect real regressions; a set of a dozen hand-picked "easy" queries will lie to you. Include known-hard cases on purpose, such as pairs of articles that differ only by product tier or platform, because those are exactly where semantic-only ranking collapses.

Store the evaluation set as structured content, not a spreadsheet that drifts out of sync. Modeling it in the Content Lake means each labeled query, its expected article references, and its query category live as first-class documents you can query with GROQ. When an editor archives or splits an article, the reference in your eval set is a real content relationship, so you can catch a now-dangling ground-truth label before it silently corrupts your metrics.

Choose metrics that reflect what support relevance means

Retrieval metrics are not interchangeable, and picking the wrong one hides the failures that matter most for support. If an agent only ever reads the single top-ranked article before answering, then whether the right doc sits at rank 1 versus rank 4 is the whole game, and you should weight Precision@1 heavily. If your agent synthesizes across the top three or five retrieved chunks, then Recall@k and Mean Reciprocal Rank tell you whether the resolving article made it into the window at all.

MRR is a strong default for support because it rewards putting the correct article high and penalizes burying it, which maps directly to deflection: a customer who has to scroll past three near-misses often gives up and files a ticket. Normalized Discounted Cumulative Gain (nDCG) is worth adding when relevance is graded rather than binary, for example when a superseded-but-related article is "partially" useful. Pair these ranking metrics with a business-facing one you already track, such as self-service deflection rate or escalation rate, so relevance gains are legible to people who do not read precision-recall curves.

Run every metric against the same labeled evaluation set on every change, and record the numbers as content in the Content Lake alongside the query set. That turns tuning from a one-off spreadsheet exercise into a versioned, queryable history: you can ask, in GROQ, which query categories regressed after a retrieval change, not just whether the aggregate ticked up or down.

Run controlled experiments on hybrid weighting

With an evaluation set and metrics in place, tuning becomes a series of controlled experiments rather than guesswork. Change one variable at a time and re-score against the same queries. The highest-leverage knob in hybrid retrieval is the balance between lexical and semantic contribution: how much weight `match()` carries against `text::semanticSimilarity()`, and how aggressively `boost()` promotes exact-term hits like error codes, API names, or version numbers.

Start with a semantic-heavy blend and watch what it does to your exact-string queries; you will typically see error-code tickets regress because a conceptually similar troubleshooting doc outranks the one that names the code verbatim. Increase the lexical weight and re-measure, and you will usually recover Precision@1 on those queries without wrecking recall on paraphrased ones. Because the whole ranking is expressed in a single GROQ query in the Content Lake, an experiment is an edit to a query and a re-run against the eval set, not a rebuild of a separate search cluster.

Beyond weighting, test chunking strategy (article-level versus section-level granularity), the effect of including structured fields such as product, platform, and version as boostable signals, and whether filtering by a `product == $tier` predicate before ranking removes an entire class of wrong-tier answers. Log the metric delta for each experiment as a content record so the reasoning behind a tuning choice is retrievable months later, when someone asks why the lexical weight is set where it is.

Keep embeddings fresh so relevance survives content edits

The relevance you tuned last quarter quietly rots if your embeddings lag your content. In a bolt-on architecture, embeddings live in a separate vector database fed by a batch re-embed job. An editor fixes a wrong instruction in an article at 9am; the vector store still serves the old version until the next re-index. For support, that gap is not academic: it is the window in which your agent confidently repeats a step your team already corrected.

This is where architecture, not tuning, determines whether relevance holds. In Sanity, dataset embeddings are tied to the content itself, so when an editor publishes a change the embeddings update within minutes and there is no separate vector pipeline to babysit. The article your agent retrieves and the article your editor governs are the same document on the same retrieval path, which collapses the freshness problem instead of scheduling around it.

Editors also need a safe way to change agent-facing content without shipping a regression blind. Studio and Content Releases let a team stage a batch of article edits, and the agent instructions that go with them, the same way they stage a website launch, then promote them together. Pair that with your evaluation set and you can re-score a Content Release before it goes live, catching a relevance regression in staging rather than in a customer's failed self-service session. Governance and measurement stop being separate activities.

Close the loop: monitor relevance in production

An evaluation set proves relevance on the queries you anticipated. Production tells you about the ones you did not. Real customers phrase problems in ways your labeled set never captured, new products ship, and the query distribution drifts, so a relevance program that stops at pre-launch scoring will slowly go stale even if no code changes at all.

Instrument the live retrieval path to capture the query, the ranked articles returned, which one the agent used, and the eventual outcome, whether the session deflected or escalated. Escalations that follow a confident retrieval are your richest signal: they are candidate additions to the evaluation set, and clusters of them by topic point straight at a chunking or weighting gap. Feed those failing queries back into the labeled set so the corpus of hard cases grows from real misses rather than imagination.

Because production agents connect to Sanity Context through its MCP endpoint, and the retrieval, the source articles, and the evaluation set all live in the Content Lake, the monitoring loop closes inside one system rather than being stitched across a vector database, a content backend, and an observability tool that each know a fraction of the story. Knowledge Bases extend the same retrieval path to PDFs, support databases, and websites, so as you fold new content sources into the agent, your relevance tuning and monitoring discipline extends to them by default instead of restarting per source.