LangSmith

You open a LangSmith trace to debug a bad answer, and the LLM call looks fine. The prompt is well-formed, the model responded coherently, and the tool calls succeeded. The problem is upstream: the agent retrieved the wrong document, or a stale one, and LangSmith logged the confident wrong answer without logging why the input was wrong. For governance, that gap is the whole story. You can prove what the model said, but not what it was allowed to see.

Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset, with Knowledge Bases as the second surface for unstructured sources like PDFs and support databases. Because retrieval runs through explicit, typed queries rather than an opaque vector lookup, the exact query and its result are things you can capture and attach to a LangSmith run.

This article covers how to instrument LangSmith properly for governance: logging retrieval inputs, not just model outputs, tagging runs so an auditor can reconstruct a decision, and closing the loop by tracing failures back to the content the agent actually read.

Why does a LangSmith trace show a good LLM call but a bad answer?

A LangSmith trace shows a good LLM call but a bad answer when the failure happened during retrieval, before the model ran. LangSmith instruments the LLM call and the surrounding chain, so it faithfully records the prompt, the token counts, the latency, and the completion. What it does not automatically record is whether the context stuffed into that prompt was correct, current, or authorized.

This is the most common governance blind spot in production agents. The model answered a customer that a discontinued plan is still available, and the trace looks clean: coherent prompt, coherent response, no exception thrown. The root cause is that the retrieval step handed the agent a cached product document from three releases ago. Nothing in the default trace tells you that.

LangSmith gives you the hooks to fix this, but you have to use them deliberately. Every run has an `inputs` and `outputs` payload, plus `metadata` and `tags`. If your retrieval function is wrapped as a traceable step, its inputs (the query) and outputs (the documents returned) become first-class trace data. The discipline is to treat retrieval as something worth logging with the same rigor as the LLM call. An auditor reconstructing a decision six months later needs the query and the returned document IDs far more than they need the temperature setting. Governance is about reconstructing what the agent was allowed to see, and that lives in the retrieval span, not the completion.

from langsmith import traceable

@traceable(run_type="retriever", name="fetch_product_context")
def fetch_product_context(query: str) -> list[dict]:
    # your retrieval call goes here
    docs = run_retrieval(query)
    # returned value becomes the run's `outputs`, visible in the UI
    return docs

# `query` is now logged as `inputs`, `docs` as `outputs`
result = fetch_product_context("current price of the pro plan")

How do I attach the retrieval query and result to a LangSmith run?

You attach the retrieval query and result to a LangSmith run by logging them as structured metadata on the run, not as free text buried in a prompt. The cleanest path is `@traceable` on the retrieval function, as above, which captures inputs and outputs automatically. For finer control, use the tracing context to write explicit metadata onto the current run.

The reason to use `metadata` rather than folding everything into the prompt string is queryability. LangSmith lets you filter and group runs by metadata keys. If every run carries `retrieval.query`, `retrieval.doc_ids`, and `retrieval.source`, you can later pull every run that touched a specific document, which is exactly the query an auditor or an incident responder needs. A blob of text in the prompt is not filterable.

Tag runs with a stable identifier for the content version too. When your content source exposes a revision or timestamp, log it. Then a run is not just "the agent said X"; it is "the agent said X after reading document `product-123` at revision `abc` retrieved by query Q." That triple is what makes a trace legally and operationally useful. LangSmith's `metadata` is arbitrary JSON, so structure it consistently across every agent in your fleet and enforce the shape in code review. The consistency is what turns a pile of traces into an auditable record.

from langsmith import get_current_run_tree, traceable

@traceable(name="agent_turn")
def agent_turn(user_query: str):
    docs = fetch_product_context(user_query)

    run = get_current_run_tree()
    run.metadata["retrieval.query"] = user_query
    run.metadata["retrieval.doc_ids"] = [d["_id"] for d in docs]
    run.metadata["retrieval.revs"] = [d["_rev"] for d in docs]
    run.metadata["retrieval.source"] = "sanity-context-mcp"

    return call_model(user_query, docs)

How do I make retrieval itself auditable, not just the trace of it?

You make retrieval auditable by giving it an explicit, inspectable query instead of an opaque similarity lookup. This is where the shape of your content source matters. If retrieval is "embed the question, return the top 5 nearest vectors," the best your LangSmith run can log is a query string and five document IDs with no explanation of why those five. The retrieval logic is a black box, so the audit trail is thin.

Sanity Context changes what you can log because retrieval runs through GROQ, an explicit query language. When your agent calls the Context MCP endpoint, or a custom tool wrapping a GROQ query, the query itself is a readable predicate: filter by document type, publication state, date range, then rank. You log that query verbatim into the LangSmith run, and an auditor reading it later can see the exact rules that selected the content. "Return published pricing documents for the pro plan, most recent first" is self-documenting in a way that a vector distance is not.

A further point that matters for governance: Context MCP is read-only. An agent connected to it can read schema, run GROQ, and traverse references, but it cannot mutate content. Writes go through Agent Actions, a separate path. When you are reasoning about blast radius in an incident review, "this MCP endpoint physically cannot have changed our content" is a strong invariant to be able to state. The typical case here is structured retrieval, GROQ queries and schema lookups, not semantic search; embeddings are opt-in and off by default, so most of what you log is a clean, deterministic query you can reason about.

*[_type == "pricingPlan"
  && slug.current == "pro"
  && !(_id in path("drafts.**"))]
  | order(_updatedAt desc)[0]{
    _id,
    _rev,
    _updatedAt,
    name,
    monthlyPrice,
    status
  }

How do I capture the exact GROQ query and result in the run?

You capture the exact GROQ query and result by logging both the query string and the returned document in your traceable retrieval function, before you hand the document to the model. Whether you reach Sanity Context through the Context MCP endpoint or a thin custom tool, the pattern is the same: run the query, log the query text and the result revision, then proceed.

The practical benefit shows up during incident response. Say a customer reports a wrong price. You filter LangSmith for runs where `retrieval.doc_ids` contains `pricingPlan-pro`, find the offending turn, and read the logged GROQ. If the query filtered on `status == "published"` and still returned a wrong price, the problem is in the content, and you have the exact revision to inspect in the Studio. If the query was missing the publication-state filter, the problem is in the agent's tool, and you fix the query. Either way, the trace points at a specific cause rather than a vibe.

This is the closed loop that governance actually needs. LangSmith owns the evidence of what the agent did. Sanity Context owns the evidence of what the agent was allowed to read, versioned by `_rev` and governed by publication state. Logging the join between them, the doc IDs and revisions on every run, is the one piece of glue you have to write yourself. It is a few lines of metadata, and it is the difference between a trace you can defend and a trace you can only apologize for.

import { createClient } from "next-sanity";
import { traceable } from "langsmith/traceable";

const sanity = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: "production",
  apiVersion: "2024-01-01",
  useCdn: false,
});

const PRICING_QUERY = `*[_type == "pricingPlan"
  && slug.current == $slug
  && !(_id in path("drafts.**"))]
  | order(_updatedAt desc)[0]{ _id, _rev, name, monthlyPrice }`;

export const fetchPricing = traceable(
  async (slug: string) => {
    const doc = await sanity.fetch(PRICING_QUERY, { slug });
    return { query: PRICING_QUERY, params: { slug }, doc };
  },
  { name: "fetch_pricing", run_type: "retriever" }
);

How do I run LangSmith evals against a governed content source?

You run LangSmith evals against a governed content source by building your eval dataset from the same content the agent reads at runtime, so the expected answers move when the content moves. The failure mode with hand-written eval datasets is drift: someone hardcodes "the pro plan costs $49" as the expected answer, the price changes, and now your eval fails on a correct answer while nobody notices the dataset is stale.

When the source of truth is Sanity Context, you can generate eval examples from GROQ queries so the expected output is derived from live, published content rather than frozen at authoring time. Your eval for "what does the pro plan cost" pulls the current `monthlyPrice` at eval time. If content editors publish a price change through Content Releases, the next eval run reflects it automatically, and you are testing the agent against reality instead of against a snapshot.

This matters for governance because evals are your continuous proof that the agent behaves. An eval suite grounded in stale fixtures gives false confidence: green checks that mean nothing. Grounding the dataset in the same governed, versioned content the agent retrieves means a passing eval is a real statement about production behavior. LangSmith runs the evaluation, scores the outputs, and tracks the results over time. Sanity Context supplies the ground truth, kept current by the same editorial workflow that governs the live agent. You get one source of truth feeding both the runtime path and the test path, which is the only way the two stay honest with each other.

from langsmith import Client

client = Client()

# expected answer is derived from live content, not hardcoded
plan = fetch_product_context("pro plan price")[0]

client.create_examples(
    dataset_name="pricing-qa",
    inputs=[{"question": "What does the pro plan cost?"}],
    outputs=[{
        "expected": f"${plan['monthlyPrice']}/month",
        "source_rev": plan["_rev"],
    }],
)

Where does Sanity Context fit under a LangSmith-instrumented agent?

Sanity Context fits directly under your LangSmith-instrumented agent, in the retrieval and governance layer, feeding it the content the agent would otherwise have to assemble from a vector database, a JSON export, and a custom CMS sync job. It does not replace LangChain, LangGraph, or your agent loop, and it does not compete with LangSmith. LangSmith observes; Sanity Context supplies and governs the content being observed.

In Sanity's framing, this is the Content Operating System for the AI era: rather than a legacy CMS that stops at publishing, content is operated end to end, with the same schema, versioning, and review workflow serving human editors and agents alike. For your LangSmith setup, the concrete payoff is a clean join. Every run carries the query, the document IDs, and the revisions, and every one of those revisions is inspectable, diffable, and attributable in the Studio. The observability platform and the content source share one identifier space.

The fastest way in is the Context MCP endpoint: attach it to your agent as an MCP server and it exposes schema-aware, read-only tools with no glue code. For unstructured sources, PDFs, marketing sites, or a support database, Knowledge Bases turns that mess into ordered documents your agent can retrieve against, while structured content stays on GROQ. Start by wrapping retrieval in `@traceable`, log the query and revisions as metadata, and point your evals at the same content. That is the whole integration: LangSmith records what happened, Sanity Context records what the agent was allowed to see, and the two line up on document revision.