Top 5 Ways AI Agents Break in Production

A support agent confidently quotes a return policy that was retired two quarters ago. A shopping agent insists a speaker is in stock when it was discontinued last week. The model did not lie on purpose. Retrieval returned nothing useful for a query that carried a real structural constraint, a version number or an "in stock" flag, and the model filled the silence. That is the most common way agents break in production, and no bigger model fixes it.

Sanity Context (the AI Content Operating System for grounding agents in your structured content) treats each failure mode as a property of a specific layer in the harness, not a mystery of the model. Sanity is the intelligent backend for companies building AI content operations at scale, where retrieval, the system prompt, and the eval bench live next to the content itself. Hallucination is a retrieval problem. Scope violations are a prompt problem. Auth confusion is a tools-and-auth problem.

This is a ranked field guide to the five ways agents actually fail under real traffic, what each one costs you, and where the fix belongs. Read each entry for the mechanism first, then the layer that resolves it.

1. Retrieval returns nothing useful, so the model fills the gap

This is where most agents fail, and the failure has a consistent shape. A user query carries a real structural component, a product feature, a version number, a category, or "in stock", and pure vector similarity cannot resolve any of those constraints because semantic distance does not respect them. The query comes back empty or wrong, and depending on the prompt the model either hallucinates a confident answer or hedges its way out. The fix is not a better model. The model needed to know the shape of the data, not just its types.

The discipline that resolves it is hybrid retrieval, and it maps to the Model your business pillar. In Content Lake, predicates do the filtering that has to hold, then a single GROQ query ranks results with score(boost([title] match text::query($queryText), 2), text::semanticSimilarity($queryText)) followed by order(_score desc). A BM25 keyword match on the title, weighted twice because title hits matter more, blends with a semantic score across the document. Running schema exploration against Sonos's catalog, an honest nightmare of a dataset, landed around 83% accuracy on a mix of difficulties using Sonnet 4.5 with about 40 seconds of thinking per hard question. None of that was a model problem. It was a context problem.

Where a pure vector store fits poorly: it owns the fuzzy semantic slice well, but you assemble the structured predicate layer alongside it and then own a content pipeline that keeps the index fresh. Content Lake handles incremental indexing, re-embedding on change, and deletion handling for you.

2. Empty retrieval hits the structural ceiling of your retrieval layer

Empty retrieval is the close cousin of hallucination, and it is the most common cause the field guide reports. The difference is honesty: instead of inventing an answer, the agent comes back with nothing and tells the user it cannot help, on a question your content clearly could answer. The user assumes the product is broken. From the outside, a confident wrong answer and a blank "I don't know" look like two different bugs, but they share one root: the retrieval layer hit its structural ceiling and could not represent the query against the shape of your data.

A concrete example: a shopper asks for "trail runners under $150 like a Hoka." That is a price predicate, a category, and a similarity request in one sentence. A vector-only index treats the whole thing as a fuzzy blob and either returns shoes over budget or nothing in the right category. The structural constraint, under $150, is exactly what semantic similarity cannot enforce. This is the Model your business pillar again: the ceiling lifts when the predicate filters first and the semantic score only ranks what survives the filter.

Where bolt-on stacks fit poorly: stitching a vector database to a content backend through glue code means the structured filter and the semantic rank live in two systems that must be kept in sync. The query author has to remember to apply both. Inside Content Lake the filter and the rank are one GROQ expression, so the structural component of the query is never silently dropped. The eval bench then proves it: empty-retrieval cases belong in the frozen suite so a regression in the retrieval layer fails CI before it reaches a user.

3. Scope violation, the agent answers something the never-say list should have caught

Scope violation is when the agent answers a question it was never supposed to touch: a competitor comparison, medical advice, a pricing commitment outside its authority. The mechanism is almost always the same. The forbidden-topics list either does not exist or lives somewhere Compliance cannot see, so nobody who owns the rule can confirm it is in force. Engineering buries the system prompt as a string literal in src/agents/prompts.ts, and the marketing team cannot read it, the compliance team cannot review it, and the people accountable for the boundary have no way to touch it.

The fix maps to the Automate everything pillar by treating the application prompt as customer-facing behavior, governed like content rather than buried as code. In Studio it becomes an agentPrompt document split into fields with role-based ownership: voice owned by Brand, userContext owned by Product, escalation owned by Support, and a mustNotSay forbidden-topics array owned by Compliance. None of them files a pull request. Because it is content, you get real-time collaboration, version history, scheduled publishing, and rollback. Vipps came to Sanity wanting the whole organization to contribute to prompt writing, with product managers owning it, not just engineers.

Where a prompt-as-string approach fits poorly: the people who define the boundary are the people who cannot edit it, so the never-say list drifts out of date and scope violations slip through. Splitting the prompt into fields is not cosmetic. It is access control. Author it like content, and gate it like code: a prompt change runs the eval bench in CI before it can ship.

4. Auth confusion, the agent acts under the wrong identity

Auth confusion is the failure that does not show up in a demo, because demos do not have users. A demo agent runs as a service account, so it can see everything and act on anything. Ship that same agent to production and it now answers logged-in users, but if it still runs as the service account it can read another customer's order, expose a record outside someone's permission scope, or take an action the user was never entitled to take. The agent's reach is the user's reach, and that is the single most underbuilt thing in production agents.

The fix lives in the tools-and-auth layer, which maps to the Power anything pillar. The production agent must forward the user's session token through the agent runtime and the tool layer to the backend API, so retrieval and actions run under the user's permissions and are audited against the user, not the model. The token never leaves the user's identity context. The side benefit is that the agent inherits your existing security model: the same row-level permissions, the same rate limits, the same regulatory boundaries you already enforce. You do not rebuild authorization for the agent; you let the agent operate inside it.

Where bundled agent platforms fit poorly is subtler. MCP tool descriptions are trusted text injected into the prompt every turn, so a sloppy or malicious server can prompt-inject your agent before the user has typed anything. Read the source of every MCP you install. The Sanity Context MCP endpoint is model-agnostic, so the retrieval, schema, and tool surface stay constant no matter which LLM you point at it, and identity enforcement does not change when the model does.

5. Prompt drift, a change ships without the eval bench catching the regression

Prompt drift is the quietest failure on this list and the one that erodes trust over weeks. Someone tweaks a line in the system prompt to fix one annoying behavior, and the change quietly degrades three others. The escalation path stops firing. The voice goes off-brand on edge cases. Nobody notices until support tickets climb, because there was no gate between the edit and production. Prompt drift means somebody shipped a prompt change without the eval bench catching the regression, and once prompt-as-content lets more people edit the prompt, the blast radius of an ungated change only grows.

The fix is the discipline that makes every other fix on this list safe, and it maps to the Automate everything pillar. The eval suite is a frozen set of about 20 representative conversations, each scored against a rubric you wrote, run on every model change, every prompt change, and every tool change. The bar to ship anything to production is the bench staying green. This is also the gate that makes prompt-as-content safe: a Brand or Support edit ships only if the bench holds. Author it like content, gate it like code.

Where standalone eval and observability tools fit poorly is not capability, it is location. Tools in that tier do real scoring and tracing, but they sit beside the content backend, so the eval gate, the prompt content, and the source documents the agent retrieves live in separate systems. When the bench, the agentPrompt document, and the Content Lake the agent queries are the same foundation, a prompt edit and its eval run are one workflow, not a handoff across three vendors. That shared foundation is the whole point: legacy stacks create silos, while Sanity provides one place where retrieval, governance, and evaluation meet.