Braintrust

Your Braintrust eval suite passes. The model's answers read well, the LLM-as-judge scorer gives them an 8 out of 10, and you ship. Then a user asks "what changed in the refund policy last quarter" and the agent confidently cites a document that was unpublished six months ago. Your eval never caught it, because you scored the answer and not the retrieval that produced it.

That gap is what this article is about. When an agent does retrieval, the failure is almost never the model. It is the context the model was handed. To score that, you need to log what the agent actually retrieved, and to do that cleanly you need a structured source on the other end. Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset, with Knowledge Bases as the surface for unstructured sources like PDFs and support data.

This piece walks through what to actually score in a retrieval agent: context relevance, context recall, faithfulness, and the structural correctness retrieval-only scorers miss. Then it shows how logging the GROQ query and its result into your Braintrust spans turns a vague "the answer was wrong" into a debuggable trace.

The eval that lies: scoring the answer, not the context

The default Braintrust setup for a RAG agent scores the final string. You define a dataset of questions and expected answers, run your agent, and pass output and expected to a scorer like Factuality or an LLM-as-judge prompt. It feels rigorous because the numbers move when you change the prompt.

The problem is that a high answer score is compatible with broken retrieval. If your retriever returns three documents and one of them happens to contain the answer, the model will often produce a correct response even when the other two are noise, stale, or from the wrong product line. Your scorer sees a good answer and reports green. In production the same retriever hits a question where the right document ranked fourth and got cut off at top-k, and now the model confabulates from the noise. The eval gave you no warning because it never looked at the retrieved set.

This is the single most common reason a retrieval agent looks fine in evals and falls over in production. The eval and the failure are measuring different things. You scored generation quality. The thing that broke was retrieval quality. They correlate loosely enough to fool you on a small test set and decouple completely at scale. The fix is not a better answer scorer. It is to make retrieval a first-class thing you log and score, with its own metrics, on its own spans.

Logging the retrieval span so you have something to score

Braintrust scores whatever you put in the trace, so the first move is to log the retrieval step as its own span with the query and the documents it returned. Wrap the retrieval call in traced() (or use the @traced decorator in Python) and attach the retrieved documents to the span output. Now every eval row carries not just the answer but the evidence the answer was built from.

The shape that matters: log the query the agent issued, the IDs of the documents that came back, their scores or rank, and the text that was actually injected into the prompt. The last one is easy to forget and the most useful. There is often a gap between what the retriever returned and what survived truncation, reranking, or a token budget before it reached the model. You want to score the text the model saw, not the text the retriever wished it had sent.

With the retrieval span in place, your scorers can take retrieved_context as an input alongside input and output. That unlocks the metrics that actually predict production behavior: context relevance (are these documents on-topic for the query), context recall (did we retrieve everything needed to answer), and faithfulness (is the answer grounded in what we retrieved, or did the model fill gaps from its weights).

import { traced, wrapTraced } from "braintrust";

const retrieve = wrapTraced(
  async function retrieve(query: string) {
    const docs = await runRetriever(query);
    return docs; // [{ id, title, score, text, publishedAt }]
  },
  { name: "retrieve" }
);

export async function answer(query: string) {
  return traced(
    async (span) => {
      const docs = await retrieve(query);

      // log the text the model ACTUALLY sees after truncation
      const context = docs.map((d) => d.text).join("\n\n").slice(0, 8000);
      const output = await callModel(query, context);

      span.log({
        input: query,
        output,
        metadata: {
          retrieved_ids: docs.map((d) => d.id),
          retrieved_context: context,
          top_score: docs[0]?.score,
        },
      });
      return output;
    },
    { name: "answer" }
  );
}

The four scorers that catch retrieval failures

Once the retrieved context is in the trace, define scorers that target it directly. Braintrust ships an autoevals package with RAG-shaped scorers, and you can write your own as plain functions that return a name and a score between 0 and 1.

Context relevance asks whether the retrieved documents are actually about the query. A low score here means your retriever is pulling noise, often because a pure-embedding search matched on vibe rather than on the structural part of the question. Context recall asks whether the retrieved set contains the information needed to answer; you score it against a reference answer or a set of must-include facts. Low recall means the right document exists but did not rank high enough, which is a ranking or top-k problem, not a model problem. Faithfulness checks whether every claim in the answer is supported by the retrieved context; a faithful but wrong answer means your retrieval was wrong, while an unfaithful answer means the model is ignoring its context and leaning on its weights.

The fourth scorer is the one autoevals does not give you: structural correctness. Did the agent retrieve a document that was actually published, in the right locale, for the right product variant, within the date range the user asked about. Embedding-based recall scorers are blind to this because a stale draft and the live version have near-identical embeddings. You score it by checking the metadata you logged on the retrieval span against the constraints implied by the question.

Writing a custom Braintrust scorer for retrieved context

A custom scorer in Braintrust is a function that receives the row and returns a score object. For context relevance you can call an LLM judge; for structural correctness you assert against logged metadata, which is fast, deterministic, and free. Mix both in your Eval definition.

The deterministic scorers are worth leaning on. An LLM-as-judge context-relevance scorer is useful but noisy and slow, and it costs a model call per row. A structural-correctness assertion (every retrieved document has publishedAt set and is not in the future, the locale matches the query locale) is a few lines of plain code that never flakes. Put the cheap deterministic checks first and reserve the judge for the genuinely fuzzy questions.

import { Eval } from "braintrust";
import { ContextRelevancy } from "autoevals";

// deterministic: did we retrieve only live, in-locale docs?
function structuralCorrectness({ metadata, expected }: any) {
  const docs = metadata.retrieved_docs ?? [];
  const now = Date.now();
  const allLive = docs.every(
    (d: any) =>
      d.publishedAt && new Date(d.publishedAt).getTime() <= now
  );
  const localeOk = docs.every(
    (d: any) => !expected.locale || d.locale === expected.locale
  );
  return { name: "StructuralCorrectness", score: allLive && localeOk ? 1 : 0 };
}

Eval("refund-agent", {
  data: () => loadDataset(),
  task: (input) => answer(input.query),
  scores: [
    structuralCorrectness,
    (args) =>
      ContextRelevancy({
        input: args.input,
        output: args.output,
        context: args.metadata.retrieved_context,
      }),
  ],
});

Where the retrieval went wrong: structured questions, embedding retrieval

When your context-relevance and structural-correctness scorers drop, the trace usually tells the same story. The user's question had a structural component the retriever could not honor. "The current refund policy" carries a publication-state constraint. "Pricing for the enterprise plan in the EU" carries a variant and a locale constraint. "What did we announce in Q3" carries a date range. A pure-embedding retriever encodes none of these as hard predicates; it matches on semantic similarity and hopes the right document floats to the top. Often it does not, and your faithfulness scorer stays high while your structural scorer cracks.

This is the routing decision that actually moves your eval numbers. Structured content, an article catalog, products with a schema, anything with fields like publishedAt, locale, and variant, should be retrieved with those fields as filters, not approximated with embeddings. This is where Sanity Context fits into the Braintrust picture. Its primary surface, Context MCP, is a hosted read-only endpoint that exposes GROQ queries and schema reads, so your agent issues a query that puts the structural constraints in the filter and the semantic part in the scoring. The constraint is enforced, not hoped for.

The canonical pattern is hybrid retrieval inside one GROQ query: the structural predicates live in the filter, and semantic ranking is layered on with score() and text::semanticSimilarity(). One thing worth internalizing from Sanity's production data: most projects on Context MCP never turn embeddings on. The heavy majority of retrieval is plain GROQ filters and schema lookups, because that is what structured questions actually need. Semantic search is a deeper layer you reach for when the structural query alone leaves real ambiguity, not the default.

Hybrid GROQ retrieval, logged and scored end to end

With Sanity Context as the retrieval source, the document you log on the Braintrust span carries the fields your structural scorer needs for free, because they are schema fields, not metadata you bolted on. publishedAt, locale, and variant come straight off the document. Your structuralCorrectness scorer stops being a heuristic and becomes a real assertion against the schema the content was modeled with.

For the small slice of questions that genuinely need semantic ranking, hybrid retrieval keeps the structural guarantees while adding similarity scoring in a single query. The filter still enforces published and in-locale; score() and text::semanticSimilarity() rank what survives the filter. You log the query, the _score, and the returned documents to the span, and your context-relevance and faithfulness scorers run over a result set that was structurally correct before it was ever ranked.

The integration path is the default MCP one: attach the Context MCP endpoint to your agent loop as an MCP server and the agent gets schema-aware retrieval tools, or wrap a typed GROQ query in a thin tool() if you want full control over the query shape. Either way, the retrieval step you log into Braintrust now corresponds to a query you can read, rerun, and reason about, which is the difference between an eval that tells you the answer was wrong and one that tells you why.

*[_type == "policy"
  && defined(publishedAt)
  && locale == $locale
] | score(
  boost(category == $category, 2),
  text::semanticSimilarity($queryText)
) | order(_score desc) [0...5] {
  _id, title, publishedAt, locale, _score, body
}