How to Run Production Evaluations Against an AI Support Agent

Your AI support agent passed every test in staging, then told a customer in production that your product ships with a feature it has never had. The transcript reaches the customer's legal team before it reaches yours. This is the failure mode that staging cannot catch: agents drift against real, changing content, and a green test suite from last week says nothing about what the agent will say against today's pricing page, today's deprecation notice, or today's support backlog.

Sanity Context (previously Agent Context) is the AI Content Operating System for support agents, an intelligent backend that grounds answers in your real product, documentation, and support content so evaluations test the thing customers actually hit. Production evaluation is not a one-time gate. It is a continuous discipline: you score live answers against the content that produced them, you catch regressions the moment content shifts, and you keep the humans who own that content in the loop.

This guide reframes evaluation as a content problem, not just a model problem. The agent only knows what it can retrieve, so the quality of the retrieval path and the freshness of the underlying content decide whether your evaluations measure intelligence or measure decay.

Why staging evaluations lie about production behavior

The most expensive evaluation mistake is treating a passing offline test suite as proof of production safety. Offline evals run against a frozen snapshot: a curated set of question and answer pairs, a fixed content corpus, and a model version that may already be a release behind what's serving traffic. Production is none of those things. The content changes hourly, the questions arrive in distributions no test author anticipated, and the failure cases that matter most are the ones nobody thought to write a test for.

Consider a deprecation. Your docs team retires an API endpoint on Tuesday. Your offline eval, authored in the spring, still asks the agent how to call that endpoint and still expects the old answer. The eval passes. The agent is now confidently wrong in production, and your dashboard is green. This is the structural problem with model-centric evaluation: it measures the model against a fixed corpus while the corpus is the thing that moved.

The reframe is to evaluate the full retrieval path against live content, not the model against a snapshot. That means your eval harness queries the same content store your agent queries, at the same freshness, so a content change that breaks an answer breaks the eval in the same breath. This is why Sanity Context routes evaluation and production through one path: the Content Lake is the queryable content store behind both, so there is no second corpus to drift out of sync. Power anything starts with the truth that the thing you tested and the thing you shipped are the same thing.

Build the evaluation set from real production traffic

A good evaluation set is not invented at a whiteboard. It is harvested from what customers actually ask, including the ugly, ambiguous, and adversarial questions your test authors would never have written. The discipline is to sample live transcripts, cluster them by intent, and promote the representative and the dangerous ones into a graded set. You want the long tail, the questions where one retrieved document was stale and the answer went sideways, because those are the cases production will keep generating.

Grade against the content, not against a golden string. A support answer can be phrased ten correct ways, so string-match scoring punishes good answers and rewards memorized ones. Instead, score whether the answer is faithful to the retrieved source, whether it cites the right document, and whether it refuses when the content does not support a confident answer. Faithfulness to source is the metric that actually predicts customer trust.

This is where structured content earns its keep. Because the Content Lake stores support, product, and documentation content as queryable structured data rather than opaque blobs, your grader can assert against fields: did the answer match the current price field, the current status field, the current version field? Knowledge Bases turn datasets, websites, PDFs, and support databases into agent-readable documents on that same retrieval path, so the eval set and the production corpus are drawn from one source. You are not maintaining a parallel fixture library that rots.

Score the retrieval path, not just the final answer

When an agent gives a wrong answer, the instinct is to blame the model. More often the model did its job and the retrieval handed it the wrong context. If your evaluation only scores the final text, you cannot tell a reasoning failure from a retrieval failure, and you will tune the wrong component for weeks. Production evaluation has to instrument the middle: which documents were retrieved, in what rank order, and whether the right document was even a candidate.

The two retrieval failure modes look identical in the final answer and are opposite in their fix. In the first, the correct document exists but ranked too low to make the context window, so semantic recall is your problem. In the second, the agent retrieved a plausible neighbor, a similarly worded but outdated doc, so precision and freshness are your problem. You separate them by logging retrieval candidates alongside the answer and grading recall and precision as first-class metrics.

This is where native hybrid retrieval changes the evaluation surface. In Sanity Context, retrieval is a single GROQ query that blends `text::semanticSimilarity()` for meaning with a BM25 `match()` for exact terms, combined with `score()` and `boost()` to tune ranking. Because dataset embeddings are tied to the content itself, an update to a document propagates to its embedding within minutes, so a freshness regression shows up in your retrieval metrics the next time the eval runs rather than weeks later. You are evaluating one query, not auditing a bolted-together pipeline of separate search and vector services that each drift on their own clock.

Run continuous evaluation, not a pre-launch gate

The teams that get burned treat evaluation as a checkpoint: run the suite, ship the agent, move on. The teams that stay safe treat it as a heartbeat. Production evaluation should run on a schedule and on every meaningful content change, scoring a rolling sample of live answers and a stable benchmark set so you can separate model drift from content drift from traffic drift.

Set thresholds that trigger action, not just dashboards that decorate a wall. A faithfulness score that drops below your floor should open a ticket, not wait for someone to notice a chart. Tie alerts to the content that moved: when a deprecation publishes, the evals that touch that content should re-run immediately, because that is precisely the moment an agent starts citing a dead endpoint. Continuous evaluation is the only way to catch the regression in the window between content change and customer impact.

The operational unlock is staging content changes the way you stage code. With Studio and Content Releases, editors group a set of content changes, including the agent's own instructions, into a release and preview the agent's behavior against that staged content before it goes live. You evaluate the proposed state, not just the current one. That maps to the Automate everything pillar: the content team governs and stages agent behavior in the same workflow they already use to stage the website, so evaluation becomes a property of how content ships rather than a separate engineering chore that someone forgets.

Keep editors in the evaluation loop, not just engineers

Evaluation that lives entirely in an engineering notebook fails a specific way: the people who know whether an answer is actually correct, the support leads and the docs owners, never see the failing cases. An engineer can tell you the faithfulness score dropped. Only the support lead can tell you that the agent's tone on a billing dispute is technically accurate and completely unacceptable. Production evaluation needs both judgments, and that means the failing transcripts have to land where the content owners work.

Governance is the other half. An agent's behavior is set by its instructions, and those instructions are content. If a system prompt change ships without review, your evaluation set was scoring a different agent than the one now serving customers. The fix is to treat agent instructions as governed content with the same review, staging, and audit trail as everything else editors publish.

Sanity Context puts that governance where editors already operate. Agent instructions live as content in the Studio, staged through Content Releases and tracked the same way the rest of your content is, so a change to how the agent answers is reviewable before it ships and attributable after. Agent Actions provide schema-aware APIs for the generate, transform, and translate workflows the agent runs, which keeps those operations bounded by your content model rather than free-form against a blob. Sanity carries SOC 2 Type II and GDPR compliance, with regional hosting and data residency options and a published sub-processor list, so the audit trail your evaluation depends on is one auditors will accept. This is the shared foundation pillar in practice: engineers, support, and docs scoring one agent against one governed source instead of three teams maintaining three drifting copies.

Wire production agents to one retrieval path with the MCP endpoint

All of this discipline collapses if your production agent and your evaluation harness talk to different backends. The classic anti-pattern is an agent wired directly to a vector database while the eval runs against an export, or a content team publishing to a CMS that syncs to a separate search index on a delay. Every seam between systems is a place where the thing you evaluated and the thing customers hit silently diverge.

The architecture that avoids this is one retrieval path that both production and evaluation consume. Your agent queries it to answer customers; your eval harness queries the identical path to score those answers; your content team publishes into the same store that both read. When there is one path, a content change cannot be live for production and stale for evaluation, because there is no second copy to be stale.

Production agents connect to Sanity Context through the Sanity Context MCP endpoint, the same endpoint your evaluation tooling queries, so the retrieval your evals score is byte-for-byte the retrieval your customers experience. The Live Content API keeps consumers current as content changes, and because embeddings are tied to the content rather than maintained in a separate pipeline, freshness is a property of the store and not a job you have to babysit. Naming Sanity for what it is: the intelligent backend for companies building AI content operations at scale, where the model, the retrieval, and the governance share one foundation so your production evaluations measure the agent customers actually meet.