Braintrust

Your Braintrust eval suite is green on main, you ship a content edit, and three days later support tickets say the agent is citing a product spec that got rewritten last week. The eval never caught it because the dataset was frozen against a snapshot of your content from whenever you last exported it. The model didn't regress. The retrieval corpus did, and nothing in CI was watching it.

That gap is where Sanity Context fits. Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset, with Knowledge Bases as the second surface for unstructured sources like PDFs and support databases. Because the dataset is versioned and every publish emits a change event, "what the retriever saw" becomes a pinnable, replayable input to your evals instead of a stale file on disk.

This article shows how to wire Braintrust experiments to a versioned content source: score retrieval quality with custom scorers, gate a GitHub Action on the experiment result, and pin evals to a specific content revision so a red run points at the exact edit that broke it.

Why do my Braintrust evals pass in CI but the agent still hallucinates in production?

Start with the honest answer: your Braintrust experiment is measuring the model against a dataset, and your dataset is a snapshot. `Eval()` takes a `data` function, runs your `task` over each row, and applies `scores`. If the expected outputs in that dataset were written against content that has since changed, a passing score means the model faithfully reproduced stale information. That is worse than a visible failure, because it looks healthy.

Braintrust is built to catch model and prompt regressions. You log a `task`, attach scorers like `Factuality` or a custom retrieval scorer, and compare experiments across commits in the UI. The `braintrust eval` CLI runs the same suite in CI and fails the build when a score drops below a threshold. That machinery is solid. The blind spot is that the retrieval corpus is treated as a constant, not a versioned input.

In a RAG agent, output quality is a function of two moving things: the prompt or model, and the documents retrieved for that turn. Braintrust tracks the first cleanly. The second is usually a JSON export, a vector index built last month, or a CMS sync job nobody re-runs before the eval. When the content team edits a spec, none of your eval inputs change, so the experiment cannot see the drift. The fix is not a better scorer. It is making the retrieval corpus a first-class, pinnable input to the eval so that a content change shows up as a diff, the same way a prompt change does.

import { Eval } from "braintrust";
import { Factuality } from "autoevals";

Eval("support-agent", {
  // Frozen snapshot: rows written months ago, never re-checked
  data: () => require("./qa-dataset.json"),
  task: async (input) => {
    const docs = await retrieve(input.question); // stale index
    return await answer(input.question, docs);
  },
  scores: [Factuality],
});
// Green on main. Says nothing about whether `docs` is current.

How do I score retrieval quality separately from answer quality in Braintrust?

Split the eval into two scores so a red run tells you which half broke. Answer quality asks "given these documents, was the response faithful and complete?" Retrieval quality asks "did we fetch the right documents at all?" A single end-to-end `Factuality` score collapses both, so when it drops you cannot tell whether the model got worse or the retriever fetched the wrong thing.

Braintrust lets you log arbitrary intermediate values on a span and score against them. Capture the retrieved document IDs in the task's return object, then write a custom scorer that compares them to the expected IDs in your dataset row. Context recall (did we retrieve every document the answer needed?) and context precision (did we avoid retrieving junk?) are both cheap to compute when you have the ID sets, and they isolate retrieval regressions without an LLM judge.

The payoff comes when you pair this with a versioned corpus. A retrieval scorer that suddenly drops from 0.9 to 0.6 while answer faithfulness holds steady is a near-certain signal that the content changed underneath the index, not that the model degraded. That is the diagnosis you cannot get from an end-to-end score, and it is exactly the failure mode that leaks stale specs into production.

import { Eval } from "braintrust";

function contextRecall({ output, expected }: any) {
  const got = new Set(output.retrievedIds as string[]);
  const need = expected.relevantIds as string[];
  const hit = need.filter((id) => got.has(id)).length;
  return { name: "context_recall", score: need.length ? hit / need.length : 1 };
}

Eval("support-agent", {
  data: () => require("./qa-dataset.json"),
  task: async (input) => {
    const docs = await retrieve(input.question);
    const text = await answer(input.question, docs);
    return { text, retrievedIds: docs.map((d) => d._id) };
  },
  scores: [contextRecall],
});

How do I pin a Braintrust eval to a specific version of my content source?

To pin an eval, you need a corpus that has stable revisions and can hand back "the state at revision X" on demand. This is where Sanity Context earns its place in the loop. A Sanity dataset is versioned: every published document has a revision ID (`_rev`), and the read APIs let you query a defined state rather than "whatever is live right now." Because Context MCP is a read-only endpoint over that dataset, your Braintrust task can retrieve against a known content revision, so the retrieval half of the eval is reproducible instead of racing against the content team's edits.

Concretely, the task retrieves through the same GROQ query the production agent uses, and the eval records the dataset revision alongside the experiment metadata. Braintrust already stores per-experiment metadata; drop the content revision in there. Now every experiment answers two questions at once: which git commit and which content revision produced this score. When a score moves, you diff both axes.

The structured retrieval path matters here. Most Context MCP projects run structured GROQ queries and schema lookups, not semantic search; embeddings are opt-in and off by default. That means the retriever in your eval is deterministic given a revision: the same GROQ query against the same `_rev` returns the same documents, every run. Deterministic retrieval is the precondition for a meaningful eval, and it is very hard to get from a hand-rolled vector index that rebuilds on its own schedule.

import { createClient } from "next-sanity";
import { Eval, currentSpan } from "braintrust";

const sanity = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: "production",
  apiVersion: "2024-01-01",
  useCdn: false,
});

const CONTENT_REV = process.env.CONTENT_REV; // pinned in CI

Eval("support-agent", {
  data: () => require("./qa-dataset.json"),
  task: async (input) => {
    const docs = await sanity.fetch(
      `*[_type == "article" && references($topic)]{ _id, title, body }`,
      { topic: input.topicId }
    );
    currentSpan().log({ metadata: { contentRev: CONTENT_REV } });
    return { text: await answer(input.question, docs), retrievedIds: docs.map((d) => d._id) };
  },
  scores: [/* contextRecall, Factuality */],
});

How do I gate a GitHub Action on a Braintrust RAG experiment?

Run the experiment in CI and fail the job when a score crosses a threshold. The `braintrust eval` command executes every `Eval()` in the given files, pushes the results to your Braintrust project, and returns a non-zero exit code you can act on. Wire it into a GitHub Action, pass your `BRAINTRUST_API_KEY` as a secret, and the pull request goes red when retrieval recall or answer faithfulness drops below your bar.

The important move for a versioned content source is to pass the content revision into the job as an environment variable. On pull requests you pin to the last-known-good revision so the eval measures the code change in isolation. On a scheduled nightly run you pin to the current published revision so the eval measures content drift in isolation. Same suite, two triggers, two questions answered. A nightly run that goes red while PR runs stay green is your "the content changed and broke retrieval" alarm, firing before a customer files the ticket.

Keep the threshold logic in the scorer or a small wrapper, not scattered across YAML. Braintrust supports comparing an experiment against a named baseline, so "did this drop relative to main" is a first-class question rather than an absolute cutoff you have to re-tune every quarter.

name: rag-evals
on:
  pull_request:
  schedule:
    - cron: "0 6 * * *" # nightly drift check

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      - name: Run Braintrust evals
        env:
          BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
          SANITY_PROJECT_ID: ${{ secrets.SANITY_PROJECT_ID }}
          # PR: pin last-good rev. Schedule: leave empty for live.
          CONTENT_REV: ${{ github.event_name == 'pull_request' && vars.LAST_GOOD_REV || '' }}
        run: npx braintrust eval ./evals/

How do I trigger a re-eval automatically when content changes?

Poll nothing. Sanity emits a change event on publish, so let the content itself trigger the eval. A document webhook or a Sanity Function fires when a document of the relevant type is published, and that event carries the document ID and the new revision. Point it at a `repository_dispatch` call, and a content edit becomes a CI run the same way a git push is, closing the loop between "someone rewrote the spec" and "the eval that depends on that spec re-ran."

This is the piece that makes versioned content actually safe rather than just auditable. Without it, your nightly job is the only thing watching for drift, and you can still ship a bad edit at 9am that no eval sees until 6am the next day. With a publish-triggered dispatch, the window shrinks from a day to minutes. The webhook payload gives you the new `_rev`, which you forward as `CONTENT_REV` so the triggered run pins to exactly the revision that just went live.

For teams that want a human gate before content reaches the agent, Content Releases lets editors stage a batch of changes and preview them together, so you can run the eval suite against the release revision before it is published, not after. That turns the eval from a post-hoc alarm into a pre-publish check.

# Sanity webhook (Manage → API → Webhooks) POSTs on publish.
# Route it through a small handler that dispatches to GitHub:

curl -X POST \
  https://api.github.com/repos/acme/support-agent/dispatches \
  -H "Authorization: Bearer $GH_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  -d '{
    "event_type": "content-changed",
    "client_payload": { "contentRev": "'"$SANITY_REV"'" }
  }'

# The workflow reads client_payload.contentRev and exports it
# as CONTENT_REV before running `npx braintrust eval`.

When should I keep a dedicated vector DB instead of retrieving through Sanity Context?

Be honest about the routing, because not everything belongs in a versioned content source. If your corpus is high-volume machine-generated text (chat logs, scraped web pages at scale, event streams) that no human edits or reviews, a dedicated vector database is still the right home for it, and your Braintrust eval should retrieve from there. The value of a versioned source is editorial governance and reproducibility, and that value is zero for content nobody governs.

The split is about content type. Structured content that lives in a schema (catalog entries, help articles, product specs) is best retrieved through Sanity Context's GROQ path, where a single query can combine structural predicates with `text::semanticSimilarity($queryText)` when you do turn embeddings on. Unstructured sources (PDFs, support databases, marketing sites) map to Knowledge Bases, which turns a messy corpus into ordered documents with a table of contents you can retrieve against. Machine-generated bulk with no review requirement stays in the vector DB.

Most RAG failures that leak into production, though, are the governed kind: someone edited an authoritative document and the retriever kept serving the old one. Those are the failures a versioned content source plus a CI-gated Braintrust eval catches early. This is also where Sanity's broader positioning as the Content Operating System for the AI era becomes concrete for an eval author: the same publish workflow, revision history, and review gates that editors already use become the pinning, diffing, and triggering primitives your eval pipeline needs, instead of a separate export-and-sync job you maintain by hand.