How to Tune Semantic Search Ranking With Click Data and Feedback

You tune semantic search ranking by capturing click, dwell, and resolution signals, then feeding them back into how results are scored alongside vector similarity. Sanity Context stores that behavioral data next to semantic ranking and structured metadata so it can be queried and blended in one pass, rather than stitched together across a vector database, a logging pipeline, and a separate re-ranking service. The result is a ranking model that adjusts based on what users actually click and resolve, not just what looks similar in embedding space.

Why pure vector similarity plateaus in production

Embedding similarity answers one question well: which documents are semantically close to this query? It answers a different, more important question badly: which document will actually satisfy this user right now? Those diverge constantly. A query for "billing error" is semantically close to your entire billing subsystem documentation, but the result that resolves 80 percent of those queries is a short troubleshooting checklist that shares almost no vocabulary with the fancy conceptual overview sitting one cosine-distance tick higher.

The plateau shows up as a familiar pattern. Offline, your retrieval benchmarks look strong. In production, click-through on the top result stays flat, users reformulate queries, and support agents grounded on your retrieval start citing plausible-but-wrong pages. Nothing is broken in the model; the model was never optimizing for satisfaction, only for proximity.

This matters more, not less, once an AI agent sits downstream. A human scanning ten results self-corrects by clicking the third one. An agent takes the top one or two chunks and generates an answer. If the ranking is off by two positions, the human shrugs and the agent hallucinates. That asymmetry is why ranking quality became a retrieval problem rather than a UX nicety.

Sanity Context treats the Content Lake as the queryable store where content, structured metadata, and the signals about that content live together, so the ranking question is answered against the same source the agent reads from, not a drifting copy in a separate vector index.

Instrumenting the signals that actually predict relevance

You cannot tune ranking without honest signals, and not all clicks are equal. The three that carry the most information are click position (was the clicked result buried?), dwell time (did the user stay long enough to read, or bounce back in two seconds?), and resolution (did the session end, the ticket close, or the agent stop retrieving?). A click followed by an immediate return to results is a negative signal disguised as engagement.

Start by logging, per query, the ranked list you served, which result was clicked, the position of that click, dwell before return, and any downstream outcome you can observe. Attach a query fingerprint so you can aggregate signals across users who asked the same thing in different words. The goal is a table of query intent to preferred document, built from behavior rather than guesswork.

Beware the biases baked into raw clicks. Position bias means the top result gets clicked because it is on top, not because it is best; correct for it by comparing click rates against expected rates for each position. Presentation bias means a result with a good title steals clicks from a better body. These are solvable with standard debiasing, but only if you captured position and the full served list, which is why instrumentation comes before math.

In Sanity, these behavioral signals can be modeled as structured documents in the same dataset as the content they describe, so a click record references the actual document it rewarded rather than a foreign key into a separate analytics warehouse.

Turning click data into ranking features

Raw logs are not a ranking signal until you aggregate them into features the scorer can use. The workhorse feature is a per-document, per-intent click-through rate, debiased for position: how often does this document get chosen when it appears for this kind of query, relative to how often a document in that position normally gets chosen? Documents that consistently beat their position expectation are the ones your semantic score is underrating.

Layer on decay so freshness matters. A support article that resolved queries beautifully last quarter may describe a deprecated flow today; weight recent signals more heavily than old ones with an exponential decay on click age. Add a confidence floor so a document with three clicks does not outrank one with three hundred on a lucky streak; low-sample documents should fall back toward their semantic score until the behavioral evidence accumulates.

The output of this stage is a set of numeric boosts keyed to documents and intents. Critically, these should be stored as fields on or alongside the content, not locked inside a training job that runs monthly. Ranking that updates weekly beats ranking that updates quarterly, because content and user intent both drift faster than most retraining cadences.

With Sanity Context, dataset embeddings are tied to the content itself, so when a document changes the embedding refreshes within minutes and no separate vector pipeline drifts out of sync. The behavioral boost fields live in the same Content Lake, ready to be read by the same query that does the retrieval.

Blending semantic, lexical, and behavioral scores in one query

The temptation is to bolt a re-ranker onto the end of your pipeline: retrieve semantically, then call a separate service that reorders using click features. It works, but it adds a network hop, a second source of truth, and a place for the two systems to disagree about what a document even is. The stronger pattern is to compute the blended score where the content lives, in the retrieval query itself.

A robust production score combines three terms. Semantic similarity handles conceptual matches where vocabulary differs. Lexical match handles exact terms, product names, error codes, and acronyms that embeddings famously blur together. The behavioral boost then nudges results that real users prefer. You blend them with tunable weights so you can dial lexical up for a documentation corpus full of exact identifiers, or semantic up for conversational content.

In Sanity Context this is native rather than assembled. A single GROQ query can combine `text::semanticSimilarity()` for the vector term and a BM25-style `match()` for the lexical term, then use `score()` and `boost()` to fold in the debiased click-through fields stored on each document. Hybrid retrieval happens inside the Content Lake, so there is no separate search stack to keep consistent with your content and no re-ranking microservice to operate.

The operational payoff is that the ranking logic is one query you can read, version, and reason about, not a chain of services where a regression could hide in any hop.

Governing the feedback loop so it does not learn the wrong thing

A feedback loop is a system that changes its own behavior, which means it can degrade its own behavior. The classic failure is the rich-get-richer spiral: a document ranked first gets more clicks because it is first, its boost climbs, and it entrenches itself while genuinely better documents starve for exposure. Without governance, your ranking optimizes for whatever it happened to show yesterday.

Guard against this with deliberate exploration. Reserve a slice of traffic to surface lower-ranked candidates and measure how they perform when given a fair shot; this is how you discover the document that would win if users ever saw it. Pair that with staged rollout: change ranking weights for a fraction of queries, watch resolution rate and reformulation rate, and promote the change only if the real outcome metrics improve, not just the offline scores.

There is also a content-governance dimension. When behavioral signals reveal that users keep clicking away from a document, that is editorial intelligence, not just a ranking penalty. The document may be stale, mistitled, or wrong, and the right fix is to edit the content, not merely to down-weight it.

Sanity Studio and Content Releases give editors a place to stage ranking-weight changes and agent instructions the way they already stage the website, review them, and roll them back if resolution drops. The people who own the content own the tuning, and every change is reviewable rather than buried in a model artifact no one on the content team can see.

Closing the loop for AI agents, not just human search

Everything above assumed a human clicking results, but the highest-stakes consumer of your retrieval is increasingly an agent. Agents generate different signals than people do. Instead of a click, you get a citation: which retrieved chunk did the model actually quote in its answer? Instead of dwell, you get downstream outcomes, whether the user thumbs-upped the answer, escalated to a human, or asked the same question again in a new session.

These agent signals are gold because they are cleaner than human clicks. A cited chunk that produced a thumbs-up answer is a strong endorsement of both the retrieval and the ranking. A chunk that got retrieved, quoted, and then corrected by a human is a precise negative signal you can trace back to a specific document. Feeding these back into the same behavioral boost fields tunes ranking for the way agents actually consume content, which is the workload that will dominate.

This is where Sanity Context earns its framing as the intelligent backend for AI content operations. Agents connect through the Sanity Context MCP endpoint to query the same Content Lake, run the same hybrid GROQ retrieval, and write back citation and outcome signals that live beside the content. Knowledge Bases fold datasets, websites, PDFs, and support databases into that shared retrieval path, so the tuning you do for human search compounds for every agent grounded on the same foundation.

The result is one feedback loop serving both audiences, rather than two disconnected pipelines learning contradictory lessons about what your content is for.