How to Make Your AI Agent Decline Gracefully (And Why It Matters)

A customer asks your support agent whether a discontinued medication is safe to combine with alcohol, and the agent, eager to help, answers with confidence. It should have refused. That single response is the kind of failure that ends up in a screenshot, a compliance review, or a lawsuit. Most teams obsess over making their agent say the right thing and never design what happens when it should say nothing at all.

Sanity Context (previously Agent Context) treats graceful declining as a governance problem, not a personality quirk. Sanity is the Content Operating System for the AI era, the intelligent backend for companies building AI content operations at scale, and that framing matters here: whether an agent refuses cleanly, escalates to a human, or hallucinates an answer is decided by named layers you can author, version, and test, not by hoping the model behaves.

This guide reframes "decline gracefully" as the intersection of two engineering decisions: what the agent must never say, and what it does when retrieval comes back empty. Get those right and refusal stops being an embarrassing edge case and becomes a measurable, governed behavior.

Why a graceful decline is a feature, not an apology

An agent that cannot say no is a liability dressed as helpfulness. The dangerous failures in production are rarely the agent that gives a slightly wrong shipping estimate. They are the agent that answers a medical, legal, or financial question it had no business touching, or the one that invents a product feature because retrieval came back empty and the model filled the gap to avoid looking stupid. A clean decline, "I can't advise on that, let me connect you to a specialist," is the behavior that keeps those incidents off your front page.

The trouble is that most teams treat refusal as a tone problem. They add a line to the prompt that says "be careful with sensitive topics" and consider it handled. It is not handled. Tone is not a control. A control is a specific, owned, testable rule that fires the same way every time, and a fallback path the agent takes when it genuinely does not know.

Sanity Context's field guide separates the failure modes precisely here. Scope-violation means the agent answered something the never-say list should have caught, which is a prompt problem. Hallucination usually means retrieval returned nothing useful and the model filled the gap, which is a retrieval problem. They look identical to the customer, a confidently wrong answer, but they fail at different layers and they get fixed at different layers. The team's blunt summary: the fix is at that layer, not in the model. Treating both as "the agent needs better instructions" is how teams burn a quarter tuning prompts against what is actually an empty-retrieval ceiling. Designing graceful declines means designing for both layers deliberately, and knowing which one fired when something goes wrong.

The two roots of a bad answer: scope and the empty result

Every undesirable answer your agent gives traces back to one of two roots, and conflating them is the most expensive mistake teams make. The first root is scope. The user asked about something the agent should have refused outright, and it answered anyway. The agent had the information, or could generate something plausible, and nothing stopped it. This is a prompt problem: the never-say list either did not exist or did not cover the case. No amount of better retrieval helps, because the agent was never supposed to engage.

The second root is the empty result. The user asked a legitimate, in-scope question, and retrieval failed to find the answer. As the field guide puts it, a query carries a real structural component, a product feature, a version number, a category, "in stock," that pure vector retrieval can't resolve, because vector similarity doesn't respect any of those constraints. The query comes back empty or wrong, and then, in their words, the model either hallucinates or hedges depending on the prompt.

That last clause is the whole game. Whether an empty retrieval becomes a hallucination or a graceful "I don't have that information" is a prompt decision layered on a retrieval decision. You cannot fix it purely at the prompt, because a better refusal instruction does not conjure the missing data. You cannot fix it purely at retrieval, because some questions genuinely have no answer in your content. You need both: retrieval that actually finds what exists, and a prompt that tells the agent to decline cleanly when retrieval truly comes up empty. Empty-retrieval, the field guide notes, is the structural ceiling of your retrieval layer and the most common cause they see.

Where the never-say list should live (and who should own it)

If your agent's refusal rules live in a string in src/agents/prompts.ts, the wrong people own the most customer-facing behavior you ship. Compliance cannot read the never-say list. Support cannot edit the escalation language. When the agent gives an answer that lands you in legal review, the fix is a pull request, scheduled behind whatever else engineering is shipping that sprint. That is backwards.

Sanity Context's model is to author the application prompt as structured content in the Studio, split into fields with explicit owners. The escalation field, titled Escalation Policy, carries "When to hand off to a human" and is owned by Support. The mustNotSay field, titled Forbidden Topics, is an array of "Topics the agent must refuse" owned by Compliance. As the field guide says, splitting the prompt into fields isn't cosmetic, it's access control. Brand owns voice, Product owns how the agent uses user context, Support owns escalation, Compliance owns the never-say list, and none of them files a pull request.

This is the difference between governance and a string. The real choice, the team argues, isn't "content, loose" versus "code, rigorous." It's "governed, with the right people able to edit and a test gate on the way out" versus "a string only engineering can touch." Author it like content, gate it like code. A clean decline depends on static instructions, what the agent must never do, written once and edited rarely, plus per-turn runtime state so the agent knows who it is talking to and can escalate to the right place. When Compliance can open the Studio, add a forbidden topic, and see it staged before it ships, refusal stops being a backlog ticket and becomes editable policy.

Retrieval that knows when to come up empty

A graceful decline is only honest if the agent actually looked. An agent that refuses because retrieval failed on a question your content could answer is not graceful, it is broken, and it erodes trust just as fast as a hallucination. So the precondition for a trustworthy "I don't have that" is retrieval good enough that an empty result genuinely means empty.

This is where pure vector search fails the test. Vector similarity does not respect hard constraints. Ask for "a soundbar under $400 that supports Dolby Atmos" and a semantic-only search will happily return the closest-sounding products regardless of price or feature, then the model presents them and you have manufactured a wrong answer out of a retrieval miss. The fix, per the field guide, is not a better model. It is teaching retrieval the shape of the data. Running schema-aware exploration against Sonos's catalog, an honest nightmare of a dataset, the team landed around 83% accuracy on a mix of difficulties using Sonnet 4.5 with about 40 seconds of thinking per hard question, only after retrieval understood the structure.

Hybrid retrieval inside Content Lake is native, not assembled. A single GROQ query uses predicates for the hard filters that must hold, then a score pipeline: score(boost([title] match text::query($queryText), 2), text::semanticSimilarity($queryText)) ordered by _score. The predicates do the filtering that has to hold, in stock, under $400, supports Atmos, and the score pipeline blends a BM25 keyword match on the title, weighted 2x because title hits matter more, with a semantic similarity score across the document. When the predicates legitimately match nothing, the agent gets a real empty result it can decline on, not a fuzzy near-miss it will dress up as an answer.

Proving the decline works before customers see it

A refusal rule you have not tested is a hope, not a control. The whole reason teams fear letting Compliance edit a prompt is that they imagine a well-meaning edit silently breaking a behavior nobody notices until a customer does. The answer is not to lock the prompt in code. The answer is a gate that makes editing safe.

Sanity Context's recommended gate is an eval suite: a frozen set of representative conversations, twenty to start, each scored against a rubric you wrote. You run the suite on every model change, every prompt change, and every tool change, and the bar to ship anything to production is the eval bench staying green. Crucially, this is also the gate that makes prompt-as-content safe. A brand or support edit ships only if the bench holds. Refusals belong in that frozen set: include the conversations where the agent must decline, the medical question, the off-topic legal ask, the request for data the user is not entitled to, and score whether it refused cleanly and escalated correctly.

Governing the prompt as content makes this workflow tangible. The application prompt gets versioning, attribution, scheduled release, rollback, and role-based edit, and you stage agent behavior with Content Releases the same way you stage your website, previewing before you ship. After deployment, conversation classification scores transcripts asynchronously with a model, asking whether the conversation was a success, what the user was trying to do, whether the agent reached a tool it shouldn't have, and whether retrieval returned useful results or the agent hallucinated. Graceful declines are scoreable this way. As the field guide admits, it's not perfect, it's a hundred times better than no scoring.

Who governs the decline: managed agents versus your own backend

The deepest decision in agent governance is not how the agent refuses but who owns the rules that make it refuse. Two paths dominate the market, and they put ownership in very different places.

The managed support-agent tier, platforms like Kapa.ai, Decagon, and Sierra AI, will own retrieval and refusal logic for you. That is genuinely attractive when you want to ship fast. But the never-say list and the escalation policy live inside their platform, not in a content backend your Compliance and Support teams can author and version. They can refuse and escalate. The governance of those rules, though, is theirs, not yours, and the day you need to prove to an auditor exactly when a forbidden topic was added and by whom, that ownership gap becomes the problem. Some teams also hit data ceilings: one company couldn't get logged-in-user data into a managed agent and ended up building their own.

The assembled path, a vector database like Pinecone plus glue code, gives you control but hands you the maintenance. You hard-filter and rank, but you assemble re-embedding, deletion handling, and index freshness yourself, and you still can't pure-vector your way out of the empty-result problem that triggers hallucination instead of a clean decline. Content backends with AI bolt-ons, like Contentful, support automated AI steps, but the never-say list and escalation policy aren't first-class governed content fields with role-based edit and an eval gate; refusal rules tend to drift back into code or external config.

Sanity Context's position is that the rules governing a decline are content, your content, owned by the teams accountable for them, retrieved over a backend where embeddings are tied to content and freshness stops being a roadmap line item.