Google Gemini

You moved your RAG pipeline onto Gemini 1.5 Pro because the million-token window meant you could finally stop chunking. Paste the whole product catalog, the docs, the support history, let the model figure it out. It worked in the demo. Then the bills arrived, the answers got vaguer as you added documents, and the model started confidently citing a product variant that was discontinued two releases ago. The context was in the window. Gemini just didn't weight it.

This is the long-context trap, and the fix is not a bigger window. It's giving Gemini the right few thousand tokens per turn instead of every token you own. Sanity Context is Sanity's agent-facing product, and it does exactly that. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset. Knowledge Bases is the second surface, for unstructured sources like PDFs, websites, and support data.

This article covers why long context degrades, how Gemini's "needle in a haystack" benchmarks mislead, and how to wire a retrieval step in front of Gemini so the model sees a small, current, structurally-filtered slice of your content per request.

The long-context trap: why a full window hurts accuracy

Gemini 1.5 Pro and 2.0 Flash advertise context windows of 1M to 2M tokens, and Google's needle-in-a-haystack evals show near-perfect recall when you ask the model to find one planted fact. That benchmark is misleading for production retrieval. Finding a single distinctive needle is not the same task as reasoning over a haystack where dozens of passages are plausibly relevant and several of them conflict.

Two failure modes show up once you load real data. The first is the lost-in-the-middle effect: models attend more strongly to the beginning and end of a long prompt, so a fact buried at token 400,000 gets less weight than the same fact at token 2,000. The second is distractor interference. When your window holds the current product spec and three older revisions, Gemini has no signal about which one is authoritative. It sees four passages that all look like product specs. Sometimes it picks the discontinued one.

Neither failure is a bug you can prompt your way out of. They are structural consequences of cramming everything in. The model's job is generation, not retrieval and ranking. When you ask it to do both at once over a million tokens, retrieval quality silently drops, and you only notice when a user reports a wrong answer that was technically present in the context.

What a full-window prompt actually costs

Even when accuracy holds, the economics of stuffing the window do not. Gemini bills per input token, and a long-context request bills for every token you send whether or not it influenced the answer. Send 800,000 tokens of catalog to answer a question that depended on 1,500 tokens, and you paid for 798,500 tokens of attention the model mostly ignored.

The latency cost is just as real. Time-to-first-token scales with input length because the model has to run prefill over the entire prompt before it emits anything. A 600,000-token prompt can add seconds of prefill latency per request. Gemini's context caching helps when the same large prefix is reused across calls, but caching a stale snapshot of your content introduces its own problem: the cache holds yesterday's prices. The moment your data changes, every cached prefix is wrong until it expires.

The pattern that actually scales is the opposite of full-window. Retrieve a small, relevant, current slice per request, then send Gemini a few thousand tokens it can actually reason over. You pay for what matters, prefill stays fast, and there is no stale cache to invalidate because you fetch fresh content each turn.

import { GoogleGenerativeAI } from "@google/generative-ai";

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);
const model = genAI.getGenerativeModel({ model: "gemini-1.5-pro" });

// Anti-pattern: paste the whole catalog, pay for every token,
// let Gemini sort out which variant is current.
// const context = await loadEntireCatalog(); // ~800k tokens

// Better: retrieve the few documents that matter for THIS question,
// then prompt over a small, current slice.
async function answer(question: string, retrieved: string) {
  const result = await model.generateContent(
    `Use only the context below to answer.\n\n` +
    `Context:\n${retrieved}\n\nQuestion: ${question}`
  );
  return result.response.text();
}

Retrieval is a ranking problem, and Gemini is not a ranker

The honest framing is that retrieval and generation are different jobs. Retrieval decides which 2,000 tokens out of your 2,000,000 should reach the model. Generation turns those tokens into an answer. Gemini is excellent at the second job and structurally ill-suited to the first, because attention over a flat token stream has no notion of recency, authority, publication state, or reference structure.

Those are exactly the dimensions that decide which document is correct. "Which return policy applies" depends on region and effective date. "What is the price" depends on which variant and whether it is published or in draft. "Who wrote this" depends on a reference to an author document. Pure semantic similarity, the thing a long-context model approximates internally, cannot resolve any of these. Two passages can be equally similar to the query while one is authoritative and one is three years out of date.

This is why a retrieval step in front of Gemini outperforms a bigger window. You let a system that understands structure pick the documents, filter by publication state and date, traverse references to pull the linked author or category, and hand Gemini a clean, current, minimal context. The model does what it is good at. You stop asking it to be a database.

Putting Sanity Context in front of Gemini

When your content already lives in Sanity, the retrieval step in front of Gemini is Sanity Context. The fastest way in is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search over your dataset. You attach it as an MCP server to a Gemini agent loop and the model gets schema-aware tools without you writing a retrieval service.

The read-only constraint is deliberate and worth knowing. Through Context MCP, Gemini can query and read your content but cannot mutate it. Writes go through Agent Actions, not MCP. For a retrieval-augmented Gemini app that is the shape you want anyway: the model reads current content, generates an answer, and never touches the source of truth.

The routing rule matters too. Structured content, your catalog, your articles with a schema, is served by GROQ retrieval. Unstructured content, the PDFs, marketing pages, and support transcripts that do not have a clean schema, goes through Knowledge Bases, which turns a messy corpus into well-ordered documents with a clear table of contents. Most production setups lean heavily on the structured side, because that is where authority, publication state, and references live, the exact signals a long window cannot weight on its own.

Hybrid retrieval in one query when keywords aren't enough

Sometimes structural filters and keyword match leave a gap. The user asks in their own words, the exact term never appears in the document, and you need semantic similarity to bridge it. The mistake is to bolt a separate vector database onto the side and reconcile two result sets in application code. With GROQ you express the structural filter, the keyword match, and the semantic component in a single query, and order by a combined score.

The structural predicates live inside the outer filter and run first, so a date range, a publication state, or a product variant constraint narrows the candidate set before any scoring happens. Then `score()` combines a keyword match via `text::query()` with semantic similarity via `text::semanticSimilarity()`, and you order by the resulting `_score`. One round trip, one ranked list, and the structural constraints are non-negotiable rather than something the model has to honor on faith.

This is the hybrid discipline that a long context window cannot replicate. Gemini reasoning over a flat dump has no way to say "only published articles from this quarter, ranked by a blend of keyword and meaning." The query layer enforces it before the model ever sees a token.

*[_type == "article" && published == true && publishedAt > $since]
  | score(
      boost(title match text::query($queryText), 2),
      text::semanticSimilarity($queryText)
    )
  | order(_score desc)[0...5]{
    title,
    publishedAt,
    "author": author->name,
    body
  }

Wiring the retrieved slice into a Gemini call

Once Context MCP hands back a ranked, current slice, the Gemini call is small and boring, which is the point. You take the few documents the query returned, format them as context, and let Gemini generate over a window measured in thousands of tokens rather than millions. Prefill is fast, the bill reflects the question, and there is no stale cache because you fetched fresh content this turn.

If you prefer full control over the query rather than the MCP tool surface, a thin tool wrapper around the GROQ client gives Gemini a typed retrieval function it can call. Either way the architecture is the same: retrieval picks the documents, Gemini writes the answer, and the two responsibilities stay separate. The reference traversal is doing quiet work here. When the query pulls an article, it resolves the author reference and the category in the same round trip, so Gemini never sees a dangling ID it has to guess about.

The broader frame is that Sanity is the AI Content Operating System, an intelligent backend for teams building AI content operations at scale, where the same content humans edit and govern in the Studio is the content your Gemini agent retrieves. The model gets a current, reviewed, structurally-filtered view instead of a frozen dump, and your editors keep control of what the agent is allowed to say.

import { GoogleGenerativeAI } from "@google/generative-ai";
import { createClient } from "next-sanity";

const sanity = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: "production",
  apiVersion: "2024-01-01",
  useCdn: false,
});

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);

// A thin tool() wrapper: Gemini decides when to retrieve,
// the GROQ query enforces the structural constraints.
async function retrieveArticles(queryText: string, since: string) {
  return sanity.fetch(
    `*[_type == "article" && published == true && publishedAt > $since]
      | order(publishedAt desc)[0...5]{
        title, publishedAt, "author": author->name, body
      }`,
    { queryText, since }
  );
}

const model = genAI.getGenerativeModel({
  model: "gemini-1.5-pro",
  tools: [{ functionDeclarations: [{
    name: "retrieveArticles",
    description: "Fetch current published articles relevant to the question.",
    parameters: {
      type: "object",
      properties: {
        queryText: { type: "string" },
        since: { type: "string", description: "ISO date lower bound" },
      },
      required: ["queryText", "since"],
    },
  }] }],
});