Pinecone

Your Pinecone index returns the right neighborhood of documents and the wrong document. A user asks for "the return policy for orders placed after the March update," and top-k cosine similarity hands back last year's policy because it reads almost identically. The query has a structural component, a date, a publication state, a product variant, that embeddings flatten into vibes. You reach for metadata filters, and now you are hand-syncing every field you might ever filter on into Pinecone metadata, and the sync job drifts the moment an editor changes something.

Sanity Context is Sanity's agent-facing product. Its primary surface today is Context MCP, a hosted, read-only MCP endpoint that exposes schema reads, GROQ queries, reference traversal, and optional semantic search across a Sanity dataset. Knowledge Bases is the second surface, for unstructured sources like PDFs, websites, and support data. The point of pairing it with Pinecone is not to replace your vector index. It is to handle the part of retrieval that is structural, not semantic.

This article stays Pinecone-first. We cover where metadata filters actually break, how to think about the structured-versus-semantic split, and then how to route the structured half of a query through a single GROQ call instead of a brittle metadata mirror, so the date range, the author, and the published state resolve as predicates rather than as embeddings you hope are close enough.

Where Pinecone metadata filters actually break

Pinecone metadata filtering is real and it works. You attach a metadata dict to each vector, then pass a `filter` to `query()`, and Pinecone applies the predicate before ranking by similarity. For low-cardinality categorical fields, this is exactly right. The trouble starts on three fronts.

First, cardinality. Pinecone's filtering is tuned for selective, low-cardinality fields like `category` or `language`. High-cardinality fields (a unique author ID per document, a timestamp at second resolution) blow up the index metadata and degrade query latency. The docs warn against indexing metadata you do not filter on, but teams index everything just in case, then wonder why p99 latency climbs.

Second, range queries on stale data. You can filter `{"published_at": {"$gte": 1710000000}}`, but only if `published_at` is correct in Pinecone. The vector was written at ingest time. If an editor unpublishes the document, retracts it, or changes its effective date, Pinecone keeps serving the old metadata until your sync job catches up. Your agent confidently cites a document that is no longer live.

Third, relationships. Embeddings have no notion of a reference. "Articles by authors on the marketing team" requires a join from article to author to team. Metadata can flatten one level of that if you denormalize at ingest, but denormalized joins go stale exactly like everything else, and two-hop relationships are not expressible in a flat metadata filter at all.

The structured-versus-semantic split most RAG pipelines get wrong

The instinct after the first stale-filter incident is to make the metadata mirror more complete: sync more fields, add a webhook, tighten the cron. That treats the symptom. The underlying issue is that you are storing two copies of the same facts (the source of truth, and the Pinecone metadata mirror) and asking them to stay in lockstep forever.

Split the query instead. Almost every real retrieval request has two halves. The semantic half is "what is this about," which is what Pinecone is genuinely good at: fuzzy, meaning-based nearest neighbors over an embedding space. The structural half is "which of these am I allowed to return," which is a set of hard predicates: date ranges, authorship, publication state, product variant, locale, access level. These are not fuzzy. There is a correct answer, and it lives in your content's source of truth, not in a vector's frozen metadata snapshot.

The right architecture keeps each half where it belongs. Let Pinecone own semantic recall. Let the structured predicates resolve against live, governed data at query time. The mistake is forcing the structural half into the vector store because that is the tool already in your hand. A timestamp does not become more reliable by being stored next to an embedding.

When the corpus is genuinely unstructured and high-volume, machine-generated logs, scraped pages with no editorial owner, a dedicated vector DB like Pinecone is still the right home for all of it. The split matters most when the structural predicates point at content a human curates and edits.

Keep Pinecone doing what it is good at

Before adding anything, tighten the Pinecone side. Index only the metadata you actually filter on. Use namespaces to partition by tenant or locale so a filter you would otherwise express as metadata becomes a cheaper namespace scope. And keep the embedded text and the structured facts as separate concerns from the start.

A clean Pinecone query for the semantic half looks like this. Note there is no attempt to encode the date logic or the publication state in the filter; those are deliberately left out, to be resolved against live data in the next step.

from pinecone import Pinecone
from openai import OpenAI

pc = Pinecone(api_key="...")
index = pc.Index("docs")
client = OpenAI()

query_text = "return policy after the March update"

emb = client.embeddings.create(
    model="text-embedding-3-small",
    input=query_text,
).data[0].embedding

# Semantic half only. Low-cardinality, slow-changing filter is fine here.
# Date range and published state are NOT encoded as metadata.
results = index.query(
    vector=emb,
    top_k=25,          # over-fetch; we filter structurally downstream
    namespace="en-us",
    filter={"doc_type": "policy"},
    include_metadata=True,
)

# Collect the source IDs to resolve against live content.
candidate_ids = [m["id"] for m in results["matches"]]

Resolve the structured half against live content with Sanity Context

Now take the candidate IDs Pinecone returned and resolve the structural predicates against the source of truth, at query time, with no metadata mirror to keep in sync. This is where Sanity Context fits. The fastest way in is Context MCP, the hosted, read-only MCP endpoint. Your agent attaches it as an MCP server and gets schema-aware tools, so it can run a GROQ query against the live dataset directly.

GROQ resolves the structural half as predicates inside the document filter. Date ranges, publication state, and reference traversal are first-class, and they read the current value of every field, not a snapshot frozen at embed time. If an editor unpublished a policy this morning, the predicate `defined(publishedAt) && !retracted` excludes it on this turn, with no re-ingest.

A GROQ query that takes Pinecone's candidate IDs and applies the live structural predicates looks like the snippet below. Pinecone decided what is relevant. Sanity Context decides what is currently allowed and live, then traverses the author reference your flat metadata could never join.

*[
  _type == "policy" &&
  _id in $candidateIds &&
  defined(publishedAt) &&
  publishedAt >= $since &&
  !retracted
]{
  _id,
  title,
  body,
  publishedAt,
  "author": author->{ name, "team": team->title }
} | order(publishedAt desc)

When the structured content is also where semantic search should live

There is a case where you do not need two systems at all. If the corpus the structural predicates point at already lives in a Sanity dataset, articles, policies, product entries with real schema, Sanity Context can do the semantic half too, inside the same GROQ query that applies the predicates. That collapses the two-step round trip into one call.

This is hybrid retrieval the way the production data actually shows it works. The structural predicates stay in the document filter where they belong. The semantic ranking is expressed with `text::semanticSimilarity()`, combined with a keyword match via `text::query()` inside a `score()` pipeline, then ordered by `_score`. Worth being precise here: semantic search in Sanity Context is opt-in and off by default. The heavy majority of production calls are plain structured GROQ and schema lookups, the compressed structured context behind most agents. Embeddings are the deeper layer you turn on when keyword plus structure is not catching enough, not the default everywhere.

For unstructured sources, the PDFs, scraped pages, and support exports that have no clean schema, the right surface is Knowledge Bases, which turns a messy corpus into well-ordered documents with a table of contents. That is the case where keeping Pinecone for raw high-volume vectors still makes sense, and Sanity Context owns the governed, editable slice.

Wiring Context MCP into an existing agent loop

The integration is deliberately small. Context MCP is a hosted, read-only endpoint, so there is nothing to deploy. Your agent attaches it as an MCP server and gets the schema-aware tools, including the ability to run GROQ, with no custom tool code. Read-only is the important constraint: an agent can query, traverse references, and read schema through MCP, but it cannot write. Writes go through Agent Actions, not MCP, which is what you want when an autonomous loop is touching governed content.

If you want full control over the exact query rather than letting the model compose it, a thin custom tool that runs a typed GROQ query is the second path. With `next-sanity` you create a client once and expose a single `tool()` whose handler runs the structured resolution from the section above.

import { createClient } from "next-sanity";
import { tool } from "ai";
import { z } from "zod";

const sanity = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: "production",
  apiVersion: "2024-01-01",
  useCdn: false, // read live data, not a cached snapshot
});

export const resolvePolicies = tool({
  description: "Filter Pinecone candidate IDs by live date range and publication state",
  parameters: z.object({
    candidateIds: z.array(z.string()),
    since: z.string(), // ISO date
  }),
  execute: async ({ candidateIds, since }) => {
    return sanity.fetch(
      `*[_type == "policy" && _id in $candidateIds &&
         defined(publishedAt) && publishedAt >= $since && !retracted]{
         _id, title, body, publishedAt,
         "author": author->{ name, "team": team->title }
       } | order(publishedAt desc)`,
      { candidateIds, since }
    );
  },
});

Choosing the split for your own stack

The decision is not Pinecone or Sanity Context. It is which half of each query goes where. Route on the nature of the content and the predicate.

For structured content with real schema, catalogs, articles, policies, anything an editor owns, push the structural predicates to Sanity Context GROQ retrieval, and consider letting it handle the semantic half too once you confirm keyword plus structure is not enough. For unstructured content, PDFs, websites, support transcripts, route ingest through Knowledge Bases so the messy source becomes ordered documents you can query. For high-volume, machine-generated vectors that need no editorial governance, keep them in Pinecone; that is what it is built for, and not everything belongs in a governed content system.

The governance angle is the part Pinecone was never meant to solve. The reason to resolve structural predicates against live content is the same reason you want a human in the loop on what an agent retrieves: publication state, effective dates, and approvals change, and they should change in one place that is versioned, reviewable, and previewable before it goes live. Sanity is the Content Operating System for the AI era, the intelligent backend for teams keeping AI retrieval governed inside the editorial loop. Pinecone finds what is relevant. The split lets the source of truth decide what is currently true. Your agent stops citing the policy that was retracted this morning, because the predicate read the live value, not last week's embedding.