LiteLLM

You swap a LiteLLM agent from gpt-4o to claude-3-5-sonnet to keep costs down, and overnight your accuracy craters. Same prompts, same tools, worse answers. The model didn't get dumber. The retrieval feeding it was always thin, and the stronger model was papering over it. LiteLLM made the provider swap trivial, which is exactly why it exposed the part of your stack that was never provider-agnostic: what the agent actually sees per turn.

Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset. Knowledge Bases is the second surface, for unstructured sources like PDFs, websites, and support data. It sits underneath LiteLLM, not beside it: LiteLLM routes the model call, Sanity Context supplies the structured content that call grounds against.

This article stays on LiteLLM's side of the problem first. We will set up provider-agnostic routing, fallbacks, and tool calling, then show why retrieval quality, not model choice, decides whether your agent holds up when you switch providers, and where structured retrieval through Context MCP fits.

One interface, a hundred providers, and the failure that hides behind it

LiteLLM's pitch is simple and it delivers: every provider gets normalized to the OpenAI chat-completions shape. You write completion(model="...") once and point it at OpenAI, Anthropic, Gemini, Bedrock, Groq, Mistral, or a local Ollama box by changing a string. No SDK rewrites, no per-provider response parsing.

The trap is that this makes the model the easiest thing to change and therefore the first thing you blame. When an agent regresses after a swap, the instinct is to tune the prompt or jump back to the bigger model. Both treat the symptom. The actual variable that moved is how much slack each model has for sloppy context. A frontier model will infer the missing product variant or reconcile two contradictory snippets you handed it. A cheaper or faster model takes your context literally. If the retrieval was already wrong or incomplete, the weaker model surfaces it as a hallucination.

So the discipline that LiteLLM enforces, treat the provider as interchangeable, has to extend one layer down. If your agent only works on one specific model, you don't have a provider-agnostic agent. You have a model-specific agent with a flexible billing endpoint. Getting genuinely provider-agnostic means making the context good enough that any competent model lands the same answer.

from litellm import completion

# Same call shape across providers; swap the model string only.
messages = [{"role": "user", "content": "Which winter jackets are in stock under $200?"}]

resp = completion(model="gpt-4o", messages=messages)
resp = completion(model="anthropic/claude-3-5-sonnet-20241022", messages=messages)
resp = completion(model="groq/llama-3.3-70b-versatile", messages=messages)

print(resp.choices[0].message.content)

Routing, fallbacks, and budgets that survive a 429

In production you don't pick one model, you build a fallback chain. LiteLLM's Router handles this: declare a model list with priorities, and on a rate limit, timeout, or context-length error it retries down the chain automatically. This is the part that keeps an agent up at 3am when your primary provider throttles you.

The Router also enforces budgets and rate limits per key, which matters once an agent loop can fan out into dozens of tool-driven calls. A retrieval-augmented agent that re-queries on every turn can quietly 10x your spend, and a runaway loop against a frontier model is a real bill. LiteLLM's max_budget and rpm caps give you a circuit breaker.

Here is the operational reality, though. Fallbacks make your model selection nondeterministic. On any given request you might be served by your primary or by the third entry in the chain, and those models have different tolerances for thin context. If your agent only behaves correctly on the primary, your fallback is not a fallback, it's a slow-motion incident. Designing for fallbacks is the same problem as designing for provider swaps: the context has to be solid enough that whichever model answers, answers the same.

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "chat", "litellm_params": {"model": "gpt-4o"},
         "model_info": {"id": "primary"}},
        {"model_name": "chat", "litellm_params": {"model": "anthropic/claude-3-5-sonnet-20241022"},
         "model_info": {"id": "fallback-1"}},
        {"model_name": "chat", "litellm_params": {"model": "groq/llama-3.3-70b-versatile"},
         "model_info": {"id": "fallback-2"}},
    ],
    fallbacks=[{"chat": ["chat"]}],
    num_retries=2,
)

resp = router.completion(
    model="chat",
    messages=[{"role": "user", "content": "..."}],
)

Tool calling is where context quality becomes visible

LiteLLM normalizes function calling too, so you can pass an OpenAI-style tools array and get tool_calls back regardless of provider. This is where most agent retrieval actually lives: the model decides it needs data, calls a tool, you run a query, you feed the result back. The model never touches your database. It only ever sees the string your tool returns.

That single fact reframes the whole accuracy problem. The model's answer is bounded by the quality of that returned string. If your tool runs a fuzzy vector search and returns five loosely related chunks, the model has to guess which one the user meant. If it returns the three exactly-matching rows with the fields the question implied, any model from Llama to GPT-4o gets it right.

So the engineering question is not "which model calls my tool best", it's "what does my tool return". A good retrieval tool resolves the structural part of the query (date ranges, in-stock state, author, product variant) deterministically, and only uses similarity for the genuinely fuzzy part. When developers say their agent hallucinates after a provider swap, the tool is almost always returning ambiguous context and the old model was covering for it.

from litellm import completion

tools = [{
    "type": "function",
    "function": {
        "name": "search_products",
        "description": "Find products by query, with stock and price filters.",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string"},
                "max_price": {"type": "number"},
                "in_stock": {"type": "boolean"},
            },
            "required": ["query"],
        },
    },
}]

resp = completion(model="gpt-4o", messages=messages, tools=tools)
tool_call = resp.choices[0].message.tool_calls[0]
args = tool_call.function.arguments  # run your retrieval here

Why vector search alone returns the wrong document

Most agent retrieval defaults to dumping the corpus into a vector store and doing pure semantic search. It works in the demo and then breaks on the first query with a structural component. A user asks for "winter jackets in stock under $200". Embeddings capture "winter jackets" beautifully and are completely blind to "in stock" and "under $200", because those are not semantic facts, they're predicates over fields. You get back the most jacket-like documents, several of which are sold out or $400, and your weaker fallback model dutifully recommends them.

This is the core misframing worth correcting: vector search and RAG are not the same thing as good retrieval. They are one ingredient. The retrieval that holds up is hybrid, structured predicates plus keyword matching, with semantic similarity reserved for the part that is actually fuzzy. In Sanity Context's production data the heavy majority of agent calls are structured (GROQ queries and schema lookups). Semantic search is a small slice, opt-in, and off by default. Most projects shipping on Context MCP never turn embeddings on, because the structural query was the whole answer.

The practical routing rule: structured content (a catalog, articles with a schema) goes through GROQ retrieval. Unstructured content (PDFs, support transcripts, marketing pages) goes through Knowledge Bases, which turns the messy corpus into ordered documents with a table of contents. High-volume machine-generated text with no editorial governance can still live in a dedicated vector DB. Not everything belongs in one place.

Wiring Sanity Context into a LiteLLM agent through MCP

The fastest way to give a LiteLLM agent structured retrieval is the Context MCP endpoint. It is a hosted, read-only MCP server over your Sanity dataset, and it ships schema-aware tools out of the box: the agent can read your content types, run GROQ queries, traverse references, and (if you have enabled it) run semantic search, without you hand-writing a tool for each. Read-only is the deliberate constraint here. The agent can retrieve and reason over content but cannot mutate it through MCP; writes go through Agent Actions, not the retrieval path. For a retrieval-augmented agent that is exactly the boundary you want.

LiteLLM speaks tool calling, not MCP natively, so the common pattern is a thin bridge: an MCP client connects to the Context MCP endpoint, lists its tools, and you expose them to LiteLLM as the standard tools array. When the model emits a tool_call, you forward it to the MCP server and return the result. The schema-awareness means the model gets accurate field names and types, so it stops inventing fields that don't exist, a frequent cause of empty or malformed queries.

The payoff is provider independence. Because the agent now receives precise, schema-correct, filtered content instead of a bag of similar-ish chunks, the answer no longer depends on the model being smart enough to disambiguate. Swap gpt-4o for a Groq-hosted Llama in your fallback chain and the answers stay consistent, because the hard part was solved before the model saw the context.

import json
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client
from litellm import completion

# Connect to the hosted, read-only Context MCP endpoint.
async def run_agent(messages):
    async with streamablehttp_client(
        "https://<projectId>.api.sanity.io/agent/context/mcp"
    ) as (read, write, _):
        async with ClientSession(read, write) as session:
            await session.initialize()
            mcp_tools = await session.list_tools()

            # Map MCP tool defs to LiteLLM's OpenAI-style tools array.
            tools = [{
                "type": "function",
                "function": {
                    "name": t.name,
                    "description": t.description,
                    "parameters": t.inputSchema,
                },
            } for t in mcp_tools.tools]

            resp = completion(model="gpt-4o", messages=messages, tools=tools)
            call = resp.choices[0].message.tool_calls[0]
            result = await session.call_tool(
                call.function.name,
                json.loads(call.function.arguments),
            )
            return result

The one GROQ query that does hybrid retrieval

When you do want full control over the query, the second integration path is a custom GROQ tool: a thin tool() wrapper that runs a typed query against your dataset. This is where the structural-plus-semantic discipline becomes a single expression instead of a multi-step pipeline you orchestrate by hand.

The pattern is that structural filters live as predicates inside the array projection, and scoring combines a keyword match with optional semantic similarity. The structural part (in stock, price ceiling, published state) is a hard filter that never gets fuzzed. The fuzzy part runs through score() with text::query() for BM25 keyword matching and text::semanticSimilarity() for embeddings, then you order by the computed _score. That is the whole "hybrid" idea: predicates decide what is eligible, scoring decides the order of what's left.

For a LiteLLM tool, you call this query, return the rows as JSON to the model, and let it phrase the answer. Because the eligibility was decided by predicates, every model in your fallback chain receives the same correct candidate set. The model is doing presentation, not retrieval. That is the separation that makes provider-agnostic actually true rather than aspirational.

Governing the content your agent reads

There is a part of the agent stack that does not belong in LiteLLM, your vector DB, or your Redis cache: the content that humans need to author, review, and version. Brand voice guidelines, approved responses, agent instructions, the knowledge base your support agent reads from. These change, they need review before they go live, and a wrong edit ships a wrong answer to every user instantly.

This is the editorial side of state, and it is distinct from ephemeral per-turn memory. Per-user chat history is Upstash or Redis territory, fast, disposable, scoped to a session. The content your agent grounds against is the opposite: durable, governed, and edited by people. Sanity is the AI Content Operating System, an intelligent backend designed to keep these AI workflows governed, reviewable, and safe inside the editorial loop, rather than scattered across JSON files and hardcoded prompts.

The concrete primitive is Content Releases: a non-technical editor can stage a change to the knowledge base, preview exactly what the agent will retrieve, and publish it as an atomic release, with version history if it needs rolling back. Your LiteLLM agent reads the published state through the same Context MCP endpoint, so a content fix does not require a code deploy. The model layer stays interchangeable, the retrieval layer stays structured, and the content layer stays under human control. That division is what lets a provider-agnostic agent stay accurate as you swap models underneath it.