The Anatomy of a Modern AI Customer-Service Agent

A customer asks your support agent whether the trail runners they're eyeing ship before the weekend, and the agent confidently quotes a return policy that was retired two quarters ago. Nobody wrote that answer. The model paraphrased a stale document, filled a gap with a plausible guess, and a real customer acted on it. Multiply that across thousands of conversations and the failure mode isn't a quirk, it's a liability with your brand's name on it.

Most teams treat an AI customer-service agent as a single thing: a model with a prompt. In production it is several systems with different owners and different failure modes, and the gap between a demo and a dependable agent is almost entirely in those systems. Sanity is the Content Operating System for the AI era, the intelligent backend designed to keep retrieval grounded, agent instructions governed, and content fresh inside the editorial loop rather than copied into a brittle pipeline downstream.

This article takes the agent apart. We walk the harness and the agent loop, the four kinds of context with different lifetimes, hybrid retrieval that actually holds up, the three tool categories every agent needs, and where the system prompt should live. The thesis: an agent is mostly architecture, and the architecture is mostly content.

The harness is most of what you build

The model is the part everyone fixates on, but it is the smallest part of a production agent. Wrapped around it is the harness: the code that decides what goes into the context window on every turn, how much of it, and when the conversation is done. The harness runs an agent loop. Every model turn sees the same three things: the system prompt, the full conversation history so far, and the tool menu. The model either calls a tool or returns text. A tool call produces a result, that result is pushed onto a history array, and the loop runs again. No tool calls means the model is finished and its text is the answer.

That history array matters more than it looks. It is the model's only memory between calls. The LLM is stateless; the harness is what makes it feel continuous. If you do not push tool inputs and outputs back onto history, the model forgets what it just did. To stop a runaway loop, you cap iterations with a maxIterations limit, fifty being a reasonable ceiling. Some of those turns the user never sees: the model reasoning about which tool to call next, the internal monologue your interface renders as a 'Thinking…' indicator.

Understanding the loop reframes every other decision in this article. Retrieval is not a magic property of the model; it is a tool the harness exposes. Governance is not a setting; it is the system prompt the harness injects on every turn. Once you see the agent as a loop over a context window you control, the question stops being 'which model' and becomes 'what does the harness put in front of the model, and who decides.' That is an architecture question, and it is the one that separates a convincing demo from an agent you can put in front of paying customers.

Context is four things, not one

The single most expensive mistake in agent design is treating context as one bucket. It is four distinct things, each with a different lifetime, a different owner, and a different access pattern. Collapse them and you put the wrong people in charge of the wrong decisions.

First, static instructions: what the agent is, what it can do, and what it must never say. These are release-scoped and owned by product and brand, not by the runtime. Second, per-turn runtime state: who the user is, what page they are on, what is in their cart. This is single-turn, owned by your application, and it is why an agent is not personalized out of the box; somebody has to inject it. Third, retrieved content: product catalogs, documentation, policy documents, and knowledge bases. This is comparatively stable and owned by the content team. Fourth, agent-authored notes: summaries and distilled memory the agent writes for its own future self, spanning sessions and owned, in a sense, by the agent.

When these blur together, ownership breaks. If the never-say list lives in the same place as runtime state, Compliance cannot touch it without an engineer. If retrieved content is pasted into the prompt, the content team cannot update it without a deploy. Sanity Context (previously Agent Context) is built around this separation. Static instructions live as governed documents in the Studio. Retrieved content lives in Content Lake and is queried at runtime rather than frozen into a prompt. The application supplies per-turn state through its own tools. Each kind of context ends up where its real owner can reach it, which is the only version of this that survives contact with a large organization.

Hybrid retrieval, or how the agent finds the right thing

Retrieval is where confident hallucination usually begins. Ask a vector search 'do you have anything like a Hoka for trail running under $150' and pure semantic similarity will happily return a $300 road shoe because it reads as close in meaning. Ask a keyword index the same thing and it misses the intent entirely. Neither layer alone is enough, and the evidence is unusually clean here. Anthropic's contextual retrieval research measured contextual embeddings cutting top-20 retrieval failures by 35 percent, adding contextual BM25 taking that to 49 percent, and adding reranking on top bringing it to 67 percent. Three layers, each compounding.

Good retrieval is therefore hybrid by design: keyword (BM25) for literal matches, embeddings for semantic ranking, and structured predicates for the filters that simply have to hold, like 'under $150' or 'in stock.' In most stacks you assemble that from separate systems. In Sanity it is one GROQ query. Structured predicates filter first, then a score pipeline blends the signals: `| score(boost([title] match text::query($queryText), 2), text::semanticSimilarity($queryText)) | order(_score desc)`. The BM25 title match is weighted 2x because title hits matter more, and the semantic similarity score runs across the document. One query, three layers, no glue.

The freshness problem is the quiet one. When retrieval lives in a separate vector database plus glue code, keeping the index current (incremental indexing, re-embedding on change, deletion handling, backfill) becomes a permanent line item on your roadmap. Because Content Lake keeps the search index fresh on content change, that line item disappears. There is a counter-intuitive twist worth naming, covered in the callout, that should change how you budget retrieval engineering.

Tools: how the agent does things, not just says things

An agent that can only talk is a chatbot. The moment it needs to move a flight seat, cancel a subscription, or check whether a logged-in customer is over quota, it needs tools, and production agents converge on three categories. Read tools query content and fetch user state; their auth boundary is usually the user's session token, so the agent reads as the user and sees exactly what that user is allowed to see. Write tools mutate state: move a seat, cancel a plan, open a ticket. These almost always run on the user's token too, because the agent acts as the user and the strength of the identity caps the strength of the action. Composite tools wrap a multi-step backend workflow so that one tool call maps to three API calls behind the scenes, keeping the model's decision space small.

The design rule that prevents a whole class of hallucination: tools should return structured, schema-shaped data, not prose. When a tool answers in prose, the model paraphrases it, and paraphrasing is where facts go to die. A shipping date returned as a typed field stays a date. The same date returned in a sentence becomes something the model might round, reword, or quietly invent around.

This is where the agent's portability is decided. What stays constant across channels is the content backend, the retrieval layer, the tool definitions, the system prompt, and the eval bench. The Sanity Context MCP endpoint exposes the same tools whether the user touches your website, a Slack bot, or a proactive surface, so the agent loop connects to one read-only endpoint instead of a bespoke integration per channel. You still build the tool catalog per channel where identity and risk differ, but the retrieval and content surface underneath is shared, which is what keeps four channels from becoming four divergent agents.

The system prompt is customer-facing behavior, so govern it

The application system prompt is not configuration. It is the agent's voice, its escalation rules, and the list of things it must never say to a customer. That is customer-facing behavior, and behavior that customers see should be governed like any other published content, not buried in a repository where only engineers can change it and only after a deploy.

The move that makes this work is splitting the prompt into fields, and the split is not cosmetic. It is access control. Store the prompt as a Sanity document with separate fields: role, voice, userContext, escalation, and mustNotSay. Now Brand owns voice, Product owns how the agent uses user context, Support owns escalation, and Compliance owns the never-say list. None of them files a pull request. Because it is content in the Studio, you get real-time collaboration, version history, scheduled publishing, and rollback for free, and the same Content Release that ships a homepage change can ship the matching prompt change, with an eval gate in CI guarding the door.

This is not theoretical. Nearform reported that editors tuned the agent's voice with no code changes. Vipps came to Sanity wanting the whole organization to contribute to prompt writing, with product managers owning it rather than engineers. That is the shape of a healthy agent program: the people responsible for what the agent says are the people who can change what it says, staged and reviewed exactly the way they already stage the website. Sanity is the intelligent backend for companies building AI content operations at scale, and governing the prompt as content is that positioning made concrete.

What production data says about what to build first

There is a strong industry instinct to lead with embeddings. Stand up a vector database, embed the corpus, and call it a retrieval strategy. Sanity's own production data on the Context MCP endpoint complicates that instinct in a useful way. When you look at how agents actually call the endpoint, structured retrieval dominates: GROQ queries and schema lookups are the heavy majority of calls, with compressed initial context behind that. Semantic search is a small slice. Embeddings are opt-in, off by default, and most projects shipping on the endpoint never turn them on.

The reason is not that semantics do not matter. It is that agents fail on the structural side of the query first. Before an agent needs the nuance of 'something like a Hoka,' it needs to reliably filter to in-stock items under a price, fetch a specific order by id, or look up a policy by category. Those are structured operations, and if they are unreliable, no amount of embedding quality saves the answer. 'We have embeddings' is not a retrieval strategy; it is one optional layer on top of a structured foundation that has to be solid first.

This reorders the build plan. Get schema-aware structured retrieval right, make tool outputs typed and trustworthy, govern the prompt, and stand up an eval bench before you reach for vector search. Then add embeddings where intent-heavy queries genuinely need them, knowing the layer is there and, because dataset embeddings in Sanity are tied to the content, they update within minutes of a content change instead of requiring a separate re-embedding pipeline. The architecture rewards getting the unglamorous structural layer right first, which is exactly the layer most demos skip.