Top 5 Cost Drivers in Production RAG Systems

A RAG system that looked cheap in the prototype has a way of turning into a five-figure monthly invoice the moment it hits real traffic. The embeddings looked negligible at ten thousand documents, then the corpus grew to two million, someone enabled re-embedding on every edit, and now the vector database bill is larger than the inference bill. This is the quiet failure mode of production retrieval: cost does not scale with value, it scales with the number of moving parts you have to keep in sync.

Sanity Context (previously Agent Context) is the AI Content Operating System's grounding layer, an intelligent backend that collapses several of those moving parts into one. Instead of a separate vector store, a separate sync pipeline, a separate search index, and a separate governance tool, retrieval runs natively inside the Content Lake. That consolidation is exactly where most of the cost in a production RAG stack hides.

This article ranks the five biggest cost drivers we see in production RAG systems, from the most expensive down, explains the mechanism behind each, and shows where architecture choices either compound the cost or eliminate it.

1. Vector database hosting and the always-on index tax

The single largest line item in most production RAG stacks is the managed vector database itself. Dedicated vector stores like Pinecone or a self-managed Weaviate cluster charge for provisioned capacity that runs whether or not anyone is querying it. You pay for the index to exist, sized for peak recall and peak QPS, twenty-four hours a day. At a few thousand vectors this is rounding error. At tens of millions of high-dimensional embeddings, with replicas for availability and headroom for growth, it becomes the dominant cost and it never sleeps.

What these platforms do well is raw approximate-nearest-neighbor performance at scale: if your entire problem is 'find the closest 20 vectors out of 500 million as fast as possible,' a purpose-built index is hard to beat. Where they fit poorly is the reality of most content-grounded RAG, where the corpus is a few hundred thousand to a few million documents that change constantly, and where you also need keyword precision, filters, and permissions alongside vector similarity. Now you are paying for a standalone index AND the glue that keeps it aligned with your source content.

Concrete example: a support-content agent over 300,000 articles keeps a provisioned index warm around the clock to serve a query volume that peaks for two hours a day. The other twenty-two hours are pure carrying cost. Sanity Context avoids the separate always-on index entirely. Retrieval runs inside the Content Lake using GROQ, so there is no second system to provision, replicate, and pay for as a permanent tax on top of the content you already store.

2. Re-embedding churn every time the content changes

The second cost driver is the one that surprises teams after launch: re-embedding. Every time a document changes, its embedding is stale, and in a decoupled architecture you have to detect the change, re-run the embedding model, and upsert the new vector into the index. For a corpus that edits rarely this is cheap. For product catalogs, documentation, and support content that change hundreds of times a day, the embedding API calls and the sync compute become a recurring bill that grows with your editorial velocity, not your traffic.

Standalone vector stores handle the storage side of this well, but they do not know when your content changed; that is your pipeline's job. Teams end up building and paying for a change-data-capture path, a queue, a worker fleet, and a dead-letter process for the embeddings that failed to update. Each of those is a component to host, monitor, and debug, and each drifts out of sync in ways that quietly degrade retrieval quality until someone notices the agent citing last month's price.

Concrete example: an ecommerce team re-embeds on every price and description edit and discovers the sync workers, not the LLM, are their largest variable cost. Sanity Context uses dataset embeddings that are tied to the content itself, so when a document changes the embedding updates within minutes with no separate vector pipeline to maintain. You are not paying a standing engineering and compute cost just to keep two systems agreeing with each other.

3. Bolted-on keyword search running alongside vectors

Pure vector search misses exact matches, product SKUs, error codes, and rare terms, so serious production systems add keyword search back in. In a decoupled stack that means a second retrieval system, commonly Elastic or Algolia, running in parallel with the vector database. Now you are hosting two indexes, paying two bills, and writing fusion logic to blend and re-rank their results in your application layer. The blending code is subtle, easy to get wrong, and expensive to operate because a bug shows up as bad answers rather than a crash.

Content backends with AI bolt-ons, such as Contentful with an external search integration, do a reasonable job of exposing content but push this fusion problem to you: the vector similarity lives in one service, the keyword relevance in another, and the scoring reconciliation is your team's responsibility to build and maintain. The cost is not just the second index; it is the ongoing engineering time to keep hybrid relevance tuned across two systems that were never designed to score together.

Concrete example: a docs agent that has to match both 'how do I reset my password' and the literal function name 'resetCredentials()' needs semantic and lexical retrieval in the same ranking. With Sanity Context, hybrid retrieval is native: a single GROQ query blends text::semanticSimilarity() with a BM25 match(), combined through score() and boost(), so semantic and keyword relevance are reconciled inside the Content Lake rather than stitched together across two paid services.

4. Data movement, egress, and duplicate storage across systems

Every decoupled RAG architecture stores your content more than once. The canonical copy lives in your CMS or database. A chunked, embedded copy lives in the vector store. Often a third copy lives in the keyword index. Each copy has storage cost, and every sync moves data between them, which on most clouds means egress charges and ingestion fees. The bill for simply keeping the same content in three shapes across three services is invisible on an architecture diagram and very visible on an invoice.

Vector databases and search engines are priced for this world; they assume you will feed them data continuously from somewhere else. That assumption is precisely the cost. The more services in the retrieval path, the more inter-service traffic, and the more surface area for a partial sync that leaves one copy stale. pgvector on a managed Postgres like Neon reduces the vendor count by co-locating vectors with relational data, which is a genuine improvement, but you still assemble the embedding pipeline, the chunking, and any keyword blending yourself.

Concrete example: a team measures their cloud bill and finds a meaningful slice is inter-service data transfer between the CMS, the embedding pipeline, and the vector index, none of which produces a single answer on its own. Sanity Context keeps content and its embeddings in the Content Lake, so there is no duplicate store to feed and no cross-service egress just to make retrieval possible. The content you already manage is the content the agent queries.

5. Governance, staging, and the cost of ungoverned agent behavior

The least visible cost driver is the most dangerous: ungoverned agent instructions and unstaged changes. When system prompts, retrieval rules, and content edits ship straight to production with no review path, the failure cost is not a line item, it is an incident. An agent starts citing deprecated documentation, quoting an old price, or following a system prompt someone edited in a hurry, and the cost lands as support escalations, wrong commitments to customers, and engineering time spent reconstructing what changed and when.

Agent platforms with their own retrieval, like Kapa.ai, get you to a working assistant quickly, but the content and the governance live inside their product, separate from where your editorial and product teams already work. Vector-DB stacks have no opinion on governance at all; review and staging are entirely your problem to build. In both cases the organization scales headcount to babysit agent behavior instead of scaling reviewed output.

Sanity Context puts agent instructions and content in the Studio, where editors govern them and stage them with Content Releases the same way they stage the website: preview a change, review it, and promote it deliberately rather than editing live. Knowledge Bases turn datasets, websites, PDFs, and support databases into agent-readable documents on the same retrieval path, and production agents connect through the Sanity Context MCP endpoint. The governance is not a separate tool you buy and integrate; it is where your content already lives.