LangSmith

Your LangSmith trace shows the agent confidently answered a question about a product that was discontinued six months ago. The model looks fine. The prompt looks fine. The trace is green. So you scroll up, past the LLM span, into the tool call that fed it, and there it is: the retrieval step returned a stale document and nothing in the trace tells you why. That is the failure mode LangSmith was built to expose, and the one most teams stop one span short of reading.

Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset. Knowledge Bases is the second surface, for unstructured sources like PDFs, websites, and support data. When your agent's content comes from there, the tool span LangSmith records carries the exact query and the exact documents returned, so the trace finally explains the failure instead of just timestamping it.

This article stays on the LangSmith side first: how to read a trace down to the retrieval span, how to attach metadata that survives into your eval datasets, and how to instrument tool I/O so a green run that produced a wrong answer stops hiding. Then it shows what changes when the retrieval span is a GROQ query you can replay by hand.

Read the trace one span deeper than the LLM call

Most LangSmith debugging stops at the LLM span. You open the failing run, see the final completion, compare it to the system prompt, and start tuning wording. That is the wrong span. In an agent loop the model is usually doing exactly what it was told with the context it was handed. The defect is upstream, in whatever tool span assembled that context.

LangSmith records the full tree. A LangGraph run gives you the graph node spans, the tool spans nested under them, and the LLM span as a leaf. The retrieval tool span carries two things you need: the input (the query string or arguments the agent passed) and the output (the documents or rows that came back). When an answer is wrong, open that span and read the output verbatim before you touch the prompt.

The common discovery is mundane. The agent asked a reasonable question, the retriever returned three documents, and one of them was outdated or off-topic, and the model dutifully summarized all three. No amount of prompt engineering fixes a retriever that returns the wrong row. You confirm this by reading the tool output span, not by re-reading the completion.

from langsmith import Client

client = Client()

# Pull a failing run by id and walk its children
run = client.read_run(run_id)

for child in client.list_runs(
    project_name="agent-prod",
    parent_run_id=run.id,
):
    if child.run_type in ("retriever", "tool"):
        print("TOOL:", child.name)
        print("INPUT:", child.inputs)
        # This is the line that usually explains the failure
        print("OUTPUT:", child.outputs)

Make tool spans carry enough metadata to be replayable

A retrieval span that only logs document text is half a span. To debug it later you need to know which query ran, against which dataset, with which filters, and at what point in time. By default a generic vector retriever logs the embedding query and the chunk text and nothing else, so when you find a bad result in the trace you cannot reproduce it without rebuilding the call by hand.

The fix is to wrap your retrieval tool so the span captures structured metadata, not just prose. LangSmith merges anything you pass through the run config into the trace, and you can attach tags and a metadata dict per call. Log the literal query, any structural filters (date range, status, author), and the dataset or index name. Now a failing span tells you not just that the wrong document came back but exactly which call produced it.

This pays off twice. First during live debugging, when you can copy the logged query and rerun it directly against your source. Second in evals: LangSmith lets you turn production runs into a dataset, and runs that carry structured retrieval metadata produce eval examples you can actually filter and slice by failure type, instead of an undifferentiated pile of text.

from langsmith import traceable

@traceable(run_type="retriever", name="content_search")
def content_search(query: str, status: str, since: str):
    rows = run_retrieval(query=query, status=status, since=since)
    return {
        # Logged as span output, visible in the trace and eval dataset
        "documents": rows,
        "_meta": {
            "query": query,
            "filters": {"status": status, "since": since},
            "count": len(rows),
        },
    }

# Tag the call so you can slice failures later in the LangSmith UI
content_search(
    "return policy for refurbished units",
    status="published",
    since="2024-01-01",
    langsmith_extra={"tags": ["retrieval", "policy"]},
)

The failure is usually structural, and embeddings can't see it

Once you start reading retrieval spans carefully, a pattern shows up. The query had a structural component that pure vector similarity ignored. The user asked for the return policy for refurbished units and the retriever returned the policy for new units because both documents are semantically almost identical. The word refurbished is one token in a 200-token chunk, so cosine similarity treats the two policies as near-duplicates and the wrong one wins.

This is the class of bug that no embedding model fixes, because the discriminator is not semantic, it is a field. Product variant, publication status, effective date, author, locale: these are predicates, and a vector index does not enforce predicates. It approximates them, and the approximation fails exactly on the edge cases that matter most in production.

The trace makes this diagnosable. You see the query, you see the returned document, and you see that the returned document violates a constraint the query implied. The next question is architectural: where does retrieval need to honor a hard filter and a soft ranking in the same call? That is the question that points you off the pure-vector path and toward retrieval that combines structural predicates with similarity ranking.

Point the retrieval span at a query you can read and replay

When the structural filter and the ranking live in the same retrieval call, the LangSmith span becomes genuinely diagnostic. This is where Sanity Context fits into a LangChain or LangGraph stack. Instead of a vector lookup that returns opaque chunk ids, the agent runs a GROQ query against your dataset through the Context MCP endpoint, and the query itself is the span input. You read it, you understand it, and you can paste it into the Studio Vision tool to reproduce the exact result the agent saw.

GROQ lets the structural filter live in the query predicate, where it belongs, while similarity ranking happens in a scoring stage. The structural part, status equals published and the publish date inside a range, is a hard predicate inside the brackets. The ranking is a separate scoring expression. A vector index can't express that split. A GROQ query can, and the trace records the whole thing as plain text.

For unstructured sources, PDFs, marketing pages, support transcripts, you point the agent at a Knowledge Base instead, which turns that messy corpus into ordered documents with a table of contents. Either way the LangSmith tool span now carries a query you can rerun, not an embedding you can't.

*[_type == "policy" && status == "published"]
  | score(
      boost(category match text::query($queryText), 1.5),
      text::semanticSimilarity($queryText)
    )
  | order(_score desc)[0...5]

Wire the Context MCP endpoint into the agent and let the span fall out

The fastest way to get this into a LangGraph agent is the Context MCP endpoint. It is a hosted, read-only MCP server, so the agent attaches it as a tool source and gets schema-aware retrieval tools without you hand-writing a client. Read-only is the important constraint here: through MCP the agent can query, traverse references, and run optional semantic search, but it cannot write. Writes go through a separate path, which means your observability story stays clean, every MCP span is a read you can replay with no side effects.

LangChain's MCP adapter turns the endpoint's tools into LangChain tools, and from there every call the agent makes is traced by LangSmith automatically. The span input is the GROQ query or tool arguments, the span output is the documents returned, and because the endpoint is read-only you can replay any failing span as many times as you want while you debug.

If you want full control over the query, the second path is a thin custom tool that runs a typed GROQ query through the client directly. Same trace shape, more control, slightly more code. Most teams start with MCP and drop to a custom tool only when they need a query the default tools don't expose.

from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent

client = MultiServerMCPClient(
    {
        "sanity": {
            "transport": "streamable_http",
            # Hosted, read-only Context MCP endpoint for your dataset
            "url": "https://mcp.sanity.io/agent/your-project/production",
        }
    }
)

tools = await client.get_tools()

# Every tool call is now traced by LangSmith; the span input
# is the GROQ query, the output is the documents returned.
agent = create_react_agent("openai:gpt-4o", tools)

Close the loop: turn replayable spans into evals

The reason to make retrieval spans replayable is not just live debugging, it is regression prevention. Once a failing run carries the literal query, the filters, and the documents returned, you can add it to a LangSmith dataset and write an evaluator that checks the retrieval step in isolation, separate from the generation step. Did the right document come back for this query? That question has a deterministic answer when the span input is a GROQ query, and a fuzzy one when it is an embedding.

The practical workflow: collect the bad runs you found by reading tool spans, promote them to an eval dataset, and run a retrieval-only evaluator that reruns the logged query and asserts the expected document id is in the result set. Because the Context MCP endpoint is read-only, rerunning these queries in CI is safe and cheap. You catch the next schema change or content edit that would have silently broken retrieval before it ships, instead of finding it in a green production trace a week later.

This is also where the editorial side earns its place. The content these queries return is versioned and reviewed, so when a document changes you can preview the change against your eval set before it goes live, using Content Releases rather than discovering the drift in a LangSmith trace after a customer hit it.