How to Evaluate RAG Answer Quality (Groundedness) Before You Ship

A support agent tells a customer that your product supports SAML SSO on the Starter plan. It does not. The answer was fluent, confident, and cited nothing checkable, and it shipped because your evaluation harness only measured whether the model responded, not whether the response was true. That is the failure mode that quietly erodes trust in every RAG system: not the query that returns nothing, but the answer that sounds right and is grounded in nothing.

Sanity Context (previously Agent Context) exists to close exactly this gap. Sanity is the Content Operating System for the AI era, an intelligent backend that keeps retrieval anchored to real, versioned product, support, and documentation content instead of a stale vector snapshot. Groundedness evaluation is the discipline of proving that anchor holds before a change reaches production.

This guide reframes answer-quality evaluation away from "does the model sound good" toward three measurable properties: is the answer supported by retrieved context, did retrieval surface the right context in the first place, and can you catch regressions before they ship. We cover the metrics that matter, how to build an evaluation set that reflects real questions, and why the shape of your content backend determines how trustworthy your scores actually are.

Groundedness is not the same as fluency, and your eval should know the difference

The most dangerous RAG answers are the confident ones. A model that hallucinates a pricing tier, a compliance certification, or an API parameter does not stammer; it produces a clean, well-formed paragraph that a reviewer skims and approves. Fluency metrics, and the human intuition that tracks them, are actively misleading here because they reward the exact quality that makes a wrong answer dangerous.

Groundedness measures something narrower and more useful: is every claim in the answer supported by the context that was actually retrieved. An answer can be fluent and ungrounded (invented from the model's parametric memory), or grounded and awkward (accurate but poorly phrased). You want to optimize the second and detect the first. In practice this means scoring each atomic claim in a response against the retrieved passages and flagging any claim that has no supporting evidence, then treating unsupported claims as failures regardless of how plausible they read.

This is where the shape of your content backend starts to matter. If retrieval pulls from a pile of scraped, undated text, "supported by context" is a weak guarantee, because the context itself may be wrong. When retrieval runs against the Content Lake, the supporting passage carries structure, a document type, a status, and a last-updated timestamp, so groundedness checks can also confirm the source was current and published rather than a draft. Sanity Context ties embeddings to that content, so the evidence you score against is the same evidence editors govern, not a separate copy drifting out of sync.

Separate the two failures: bad retrieval versus bad generation

When a RAG answer is wrong, there are two independent culprits, and a single quality score cannot tell them apart. Either retrieval failed to surface the passage that contained the answer, or retrieval succeeded and the model ignored or distorted it. Debugging these requires different fixes, so your evaluation has to measure them separately.

Retrieval quality is answered by context recall and context precision. Recall asks: of the passages that actually contain the answer, how many did we retrieve? Precision asks: of the passages we retrieved, how many were relevant? Low recall means the right content never made it into the prompt, and no amount of prompt engineering will save the answer. Generation quality is answered by groundedness (also called faithfulness) and answer relevance: given the context we did retrieve, did the model use it faithfully and address the actual question?

A worked example: a user asks whether webhooks retry on failure. If your eval shows high groundedness but the answer is still wrong, retrieval handed the model the wrong passage and it faithfully summarized the wrong thing, so fix retrieval. If groundedness is low, the model had the right passage and drifted anyway, so fix the prompt, the model, or the chunking. Collapsing both into one number destroys this signal. This decomposition is also why native hybrid retrieval matters: in the Content Lake you can blend `text::semanticSimilarity()` with a BM25 `match()` in a single GROQ query, using `score()` and `boost()` to tune recall and precision together, then measure each independently instead of guessing which stage failed.

Build an evaluation set that looks like your real traffic

Most teams evaluate against a dozen questions someone wrote by hand in an afternoon, then act surprised when production behaves nothing like the test. A useful evaluation set mirrors the distribution of real questions: the common ones, the ambiguous ones, the multi-hop ones that require two documents, and the adversarial ones that tempt the model to invent an answer it does not have.

Start from real logs if you have them. Cluster actual user questions, sample across the clusters, and for each question write down the ground-truth answer and the specific source passage that supports it. That source pairing is the part teams skip and the part that makes automated groundedness scoring possible, because you can now check both whether retrieval found that passage and whether the answer matched it. Include a deliberate slice of out-of-scope questions whose correct answer is "I don't have that information," since a system that never abstains is a system that hallucinates on the long tail.

The evaluation set is content, and it decays like content. When your product changes, the ground-truth answers change with it, and a stale eval set will bless regressions. Treating your golden questions as versioned documents in the same system that holds your knowledge keeps them aligned. With Knowledge Bases turning datasets, websites, PDFs, and support databases into agent-readable documents on the same retrieval path, the eval set and the content it tests against move together, and Content Releases let you stage both a content change and its evaluation before either reaches an agent in production.

Pick metrics you can compute automatically and defend to a stakeholder

Manual review does not scale past the first sprint, so the working set of metrics has to be computable on every change. The four that carry most of the weight are context recall, context precision, faithfulness (groundedness), and answer relevance. Recall and precision judge the retriever, faithfulness and relevance judge the generator, and together they cover the two-failure decomposition from earlier.

For scoring, an LLM-as-judge approach works well enough when it is disciplined: feed the judge the question, the retrieved context, and the answer, and ask it to enumerate each claim and mark whether the context supports it, returning a faithfulness ratio rather than a vibe score. Claim-level scoring is far more reliable than asking "is this answer good, one to five," because it forces the judge to point at evidence. Calibrate the judge against a human-labeled subset so you know its error rate before you trust its verdicts.

The number that gets you buy-in is the one tied to a stake. Track the rate of ungrounded claims per hundred answers and the abstention rate on out-of-scope questions, then set thresholds that block a release when either regresses. Because the retrieval half of these metrics depends entirely on the evidence available, the freshness of that evidence is a first-class variable. Sanity's dataset embeddings update within minutes of a content change instead of requiring a separate re-embedding pipeline, so a documented fix appears in retrieval quickly and your recall metric reflects reality rather than a snapshot from last week's batch job.

Wire evaluation into the release, not the retrospective

An evaluation you run once, admire, and file away is theater. The point of groundedness scoring is to gate changes, so it has to sit in the path a change takes to production. Every modification that touches answer quality, a new model, a re-chunking, a prompt edit, or a content update, should trigger the eval suite and compare against the last known-good baseline before it is allowed to ship.

Concretely, that means treating agent behavior as something you stage rather than hotfix. In Studio, agent instructions and system prompts live as governed, reviewable content, and Content Releases let you bundle a prompt or content change, run the evaluation set against the staged version, and require the groundedness and recall thresholds to pass before promotion. This is the same review discipline you already apply to the website, applied to the agent. The alternative, editing a system prompt directly in production and hoping, is how the confident-wrong answer from the introduction reaches a customer.

Production agents connect to this through the Sanity Context MCP endpoint, which means the thing you evaluate in staging and the thing that serves users are querying the same governed retrieval path, not two parallel implementations that drift apart. Agent Actions give you schema-aware generate, transform, and translate operations that run against structured content, so the transformations you test are the transformations that run. Closing that loop, from eval set to staged release to the endpoint agents actually call, is what turns groundedness from a metric you report into a guarantee you ship.