How to A/B Test Two System Prompts in Production

A prompt change ships on Friday. By Monday, support escalations are up, but nobody can say whether the new system prompt caused it or whether traffic just got weirder. The old prompt is gone, overwritten in a config file, and the only "evidence" is a Slack thread of vibes. This is how most teams ship agent behavior: one prompt at a time, no control group, no way to attribute a regression to the change that caused it.

A/B testing two system prompts in production fixes that, but only if the test is built on infrastructure that can route deterministically, version the instructions under review, and tie outcomes back to the exact prompt variant that produced them. Sanity Context (the Sanity product for grounding agents in your structured content) is the AI Content Operating System for this work, an intelligent backend where agent instructions live as governed, versioned content rather than as strings buried in application code.

This guide treats prompt A/B testing as an experimentation problem with a content-governance core: how to split traffic, what to measure, how to keep the grounding constant so you are testing the prompt and not the retrieval, and how to roll the winner out without a redeploy.

Why prompt changes regress silently

The failure mode is rarely a crash. A revised system prompt that tells the agent to "be concise" quietly drops the citation it used to include, and now answers look confident but ground out against nothing. Another revision adds a safety preamble that makes the agent refuse a class of legitimate questions. None of this throws an error. It shows up days later as a churned customer or a support ticket that says "the bot got worse," with no line in any log pointing at the prompt edit that caused it.

Three structural problems make this hard to catch. First, prompts usually live as inline strings in application code, so a change to agent behavior is indistinguishable from any other deploy and carries no review trail of its own. Second, there is no control group: the new prompt replaces the old one for everyone at once, so you compare this week's metrics to last week's and pretend traffic was identical. Third, when the prompt and the retrieval pipeline change in the same release, a regression cannot be attributed to either one.

The reframe is to treat each system prompt as a versioned piece of content with its own identity, history, and review state, and to treat the rollout as a controlled experiment rather than a swap. When instructions are content, a prompt variant gets the same governance as a marketing page: a draft, a diff, an approver, and a release it ships in. In Sanity Context, agent instructions live in the Studio as structured documents, so the question "what changed and who approved it" has an answer before the experiment even starts. That governed foundation is the prerequisite for everything that follows; without it, an A/B test is just two undocumented strings racing each other.

Designing the split: traffic, randomization, and a stable grounding

A clean experiment needs three things held straight. The first is a deterministic assignment: every request must land in variant A or variant B by a stable hash of something durable, usually a user or session identifier, so the same user sees the same prompt across a conversation. Random per-message assignment contaminates multi-turn sessions and makes the data unreadable. The second is a holdout discipline: decide the split (often 50/50, or a cautious 90/10 when the new prompt is risky) and the minimum sample before you peek, so you are not stopping the test the moment the first chart wiggles.

The third, and the one teams skip, is keeping everything except the prompt constant. If variant B also gets fresher documents, a better embedding model, or a different retrieval query, you are no longer testing the prompt. This is where the grounding layer matters more than the experiment framework. In Sanity Context, retrieval runs through the Content Lake using GROQ, where hybrid search blends `text::semanticSimilarity()` with a BM25 `match()` and combines them with `score()` and `boost()` in a single query. Because both variants query the same Content Lake through the same GROQ retrieval path, the documents reaching variant A and variant B are identical. You are isolating the one variable you meant to test.

It also matters that the grounding does not drift mid-experiment. Sanity's dataset embeddings are tied to content, so when a document changes the embeddings update within minutes, with no separate vector pipeline to reconcile. During an A/B test that property cuts both ways: it keeps the corpus current, and because both arms read the same live index, an update lands for both arms at once rather than skewing one. The experiment measures the prompt, not an accident of which arm happened to see the newer chunk.

Choosing metrics that survive a postmortem

A prompt A/B test fails most often not in the routing but in the scorecard. "Users seemed happier" is not a metric. You need a small set of measures that are defined before the test starts and can be defended in a review. Group them into three tiers.

Tier one is task outcome: did the agent resolve the user's request? For a support agent that might be deflection rate (resolved without a human) paired with its safety counterweight, escalation-to-human rate, so you catch a prompt that resolves more by stonewalling more. For a sales or docs agent it might be answer-acceptance or click-through to the cited source. Tier two is quality and safety: groundedness (does the answer trace to retrieved content), citation rate, refusal rate, and a hallucination flag from an LLM judge or human spot-check. A prompt that lifts deflection while dropping groundedness is a regression wearing a costume. Tier three is cost and latency: tokens per resolution and time to first token, because a "better" prompt that doubles cost or response time may still lose.

The governance point is that every one of these has to be attributable to a specific prompt variant by its version identifier, not to "the deploy that week." When prompts are versioned content in the Studio, each logged outcome carries the variant it came from, so the postmortem reads as a clean diff between two known documents rather than a forensic reconstruction. Define a primary metric and a guardrail metric up front: ship the winner only if the primary improves and no guardrail regresses past a threshold you agreed to in advance. That discipline is what stops an A/B test from becoming a search for the chart that confirms the decision you already made.

Versioning and staging prompts as governed content

Once a prompt is content rather than a string, you can manage its lifecycle the way you manage anything else editorial. Each system prompt variant is a document with a draft state, an edit history, and an owner. A reviewer can see the diff between variant A and the proposed variant B before either touches a user, and an approver can sign off, so a behavioral change to a production agent has the same review trail as a published article.

This is where Content Releases earns its place in the workflow. The same mechanism teams use to stage a batch of website changes and ship them atomically applies to agent instructions: you stage variant B in a release, preview it, and promote it on a schedule or by approval, rather than hand-editing a live config and hoping. If the experiment goes wrong, rolling back is reverting to the previous version, not reconstructing a deleted string from memory. Editors govern agent behavior in the Studio without a code deploy, which is the difference between a prompt change that needs an engineer and a sprint and one that a content owner can stage, review, and revert the same afternoon.

This maps directly to Sanity's automate-everything pillar. Legacy CMSes stop at publishing; the instructions that drive an agent were never their concern, so teams bolted prompt management onto application code and lost the audit trail. Treating agent instructions as first-class governed content closes that gap. The Audit logs record who changed which prompt and when, and Roles & Permissions decide who is even allowed to promote a variant to production traffic. For a regulated buyer, that is the difference between "we A/B test prompts" and "we can prove which prompt was live when that answer was generated, and who authorized it."

Wiring the experiment to production agents

The runtime path has to do three jobs: assign the variant, fetch the right prompt, and log the outcome against it. Production agents connect to Sanity Context through the Sanity Context MCP endpoint, which is the query surface your agent runtime talks to for both grounding content and the governed instructions that steer it. The assignment logic (the stable hash bucketing a user into A or B) lives in your agent orchestration layer; the prompt content and the retrieval both come from Sanity Context, so the experiment framework does not need its own copy of either.

Concretely: a request arrives, your router computes the bucket, and the agent pulls the corresponding prompt variant plus the grounding documents for that query in the same retrieval path. Knowledge Bases let you turn datasets, websites, PDFs, and support databases into agent-readable documents that share that path, so the corpus the agent reasons over is the same governed content the rest of the business uses, not a stale export sitting in a separate vector store. Both arms of the test read from it identically.

For agents that write back, generate a summary, draft a reply, transform a document, Agent Actions provide schema-aware APIs so the output conforms to your content model instead of arriving as free text you have to re-parse. That schema awareness matters in an experiment because it keeps the output of both variants in the same shape, which is what makes them comparable. The logging step closes the loop: stamp each interaction with the variant identifier from the Studio, the retrieved document references, and the tier-one through tier-three metrics, so the analysis later is a query over governed data rather than a scrape of application logs. The whole loop, route, ground, act, log, runs against one backend instead of a federation of services you have to keep in sync.

Reading results and shipping the winner without a redeploy

When the sample target is hit, the analysis is a comparison between two known documents, not a guess. Check the primary metric for a real, significant difference, then check every guardrail. The most common surprising result is a split decision: variant B lifts deflection but drops citation rate, which means it is resolving more tickets by sounding authoritative on thinner grounding. That is not a winner; that is a future incident. A disciplined readout treats a guardrail breach as disqualifying even when the headline metric looks great.

Watch for the traps that make A/B tests lie. Peeking and stopping early inflates false positives, so honor the pre-registered sample size. Simpson's paradox can hide a regression inside an aggregate that looks flat, so segment by the cohorts that matter, new versus returning users, or by query type, before you trust the top-line number. And confirm the two arms actually saw the same grounding; if a document changed mid-test, both arms should have seen it at the same time, which the content-tied embeddings make true by construction.

Shipping the winner is the part that should be boring. Because the prompt is content, promotion is a Content Release that swaps the production variant, with the losing variant preserved in version history as an instant rollback target. No redeploy, no config edit on a live box, no window where the agent is running an unreviewed string. This is the closing argument for treating prompts as governed content on a single backend: the experiment, the grounding, the instructions, and the rollout all live in the same place, so "we changed how the agent behaves" is a reviewable, attributable, reversible event rather than a Friday deploy you spend Monday explaining.