How to Build an Internal Q&A Agent for Engineering Teams

Your engineering team ships a new authentication service, deprecates three internal APIs, and rewrites the deployment runbook in the same sprint. Two weeks later a new hire asks in Slack how to rotate a service token, gets three contradictory answers from teammates, and follows a fourth one from a wiki page that has been stale since the last reorg. The token rotation fails in staging. This is the everyday failure mode an internal Q&A agent is supposed to fix, and it is exactly where most of them break: they retrieve confidently against documentation that no longer reflects reality.

The hard part of building an internal Q&A agent for engineering teams is not wiring up a chat interface. It is grounding the agent in content that stays current, structured, and governed as your systems change underneath it. Sanity Context (previously Agent Context) is the AI Content Operating System for this job, an intelligent backend that keeps the retrieval path tied to live structured content so answers move when your docs move.

This guide walks through the architecture decisions that actually determine whether your agent earns trust: how you model engineering knowledge, how retrieval blends keyword and semantic search, how you keep embeddings fresh, and how editors govern what the agent is allowed to say.

Why internal Q&A agents fail in week three

The demo always works. You point an LLM at last quarter's runbooks, ask it a question you already know the answer to, and it responds fluently. The failure shows up later, when the agent confidently cites a procedure that was rewritten after the embeddings were generated, or when it answers a question about the payments service using documentation that was actually about the legacy billing service because the two share half their vocabulary.

There are three structural causes, and none of them are fixed by a better prompt. The first is stale retrieval: most teams build a vector pipeline as a one-time export, so the moment a runbook changes, the index drifts out of sync and nobody notices until an answer is wrong in production. The second is shape: engineering knowledge is not a pile of undifferentiated text. A service has owners, dependencies, on-call rotations, and deprecation status, and a flat chunk of prose throws all of that structure away, so the agent cannot distinguish a current API from a deprecated one. The third is governance: when an answer is wrong, who fixes it, and how do they verify the fix before the agent starts repeating it?

These map cleanly to the first pillar of building on Sanity, model your business. Before you write a single retrieval query, you decide what an engineering answer is made of. Treat the agent as a consumer of structured content rather than a consumer of scraped text, and most of the week-three failures stop being possible. The rest of this guide is about making that structure real.

Model engineering knowledge as structured content, not chunks

Start by refusing the default. The default RAG tutorial tells you to take every document, split it into 500-token chunks, embed the chunks, and retrieve the nearest neighbors. That works for a homogeneous corpus and falls apart on engineering knowledge, which is heterogeneous by nature. A service definition, an incident postmortem, an architecture decision record, and an onboarding runbook are different document types with different fields, different freshness expectations, and different authority.

Model them as such. In the Content Lake, Sanity's queryable content store, each of those becomes a document type with explicit fields: a service document carries an owning team, a status (active, deprecated, sunset), upstream and downstream dependencies, and a link to its current runbook. A postmortem carries the affected services as references, a date, and resolved action items. Because these are references and not free text, the agent can answer a question like "who owns the service that caused last month's checkout outage" by traversing structure rather than hoping the right two sentences ended up in the same chunk.

This is the difference between content that is merely searchable and content that is queryable. When a service is marked deprecated, every answer that touches it inherits that status, because the status lives on the document, not buried in prose an embedding might miss. Structured content also means you can scope retrieval: limit a question about deployment to documents of type runbook owned by the platform team, and the agent stops bleeding billing docs into payments answers. The model you build here is the single highest-leverage decision in the whole system, and it is the one most teams skip on the way to the chat box.

Blend keyword and semantic retrieval in one query

Engineering questions punish pure semantic search. Ask "what does error E1492 mean" and a vector index will happily return passages that are semantically about errors in general while missing the one document that contains the literal string E1492. Ask "how do I roll back a canary deploy" and keyword search alone will miss the runbook that calls it a "progressive rollout reversal." You need both: exact-match precision for identifiers, error codes, and flag names, plus semantic recall for paraphrased intent.

Most stacks solve this by standing up two systems, a vector database and a keyword search engine, then writing glue code to merge and re-rank the results. That glue is where bugs and latency live, and it is a second pipeline to keep in sync with your content. Inside the Content Lake the blend is native. A single GROQ query can combine text::semanticSimilarity() for semantic recall with a BM25-style match() for exact terms, then weight the two with score() and boost() so identifiers win when they appear and meaning wins when they do not. One query, one source of truth, no merge layer to debug.

The practical payoff is that you tune relevance in one place using the same query language your application already uses to read content. You can boost documents whose status field is active over ones marked deprecated, or boost runbooks owned by the team the asker belongs to, all in the retrieval query itself. Hybrid retrieval stops being an architecture you assemble and becomes a query you write, which is exactly what you want when the corpus is changing every week and you cannot afford a separate ranking service to rot alongside it.

Keep embeddings fresh so answers move when docs move

The most damaging class of Q&A agent error is the confident, fluent, out-of-date answer, because nothing about the response signals that it is wrong. A human who is unsure hedges. An ungrounded agent does not. The root cause is almost always an embedding pipeline that ran once and was never reconciled with the content it indexed.

The usual architecture makes this hard on purpose without meaning to. Content lives in one system, embeddings live in a separate vector store, and a batch job ferries between them on a schedule. Every change to a runbook opens a window where the prose is correct but the embedding the agent retrieves is stale, and the window is as long as your reindex cadence. Teams paper over this with nightly rebuilds, which means an answer can be wrong for most of a day.

With dataset embeddings, the embeddings are tied to the content itself, so when an editor updates a runbook the corresponding embedding updates within minutes, with no separate vector pipeline to maintain. There is no second system to fall out of sync, because there is no second system. This is what it means in practice for Sanity to operate content end to end rather than stopping at publishing: the same update that changes what an editor sees changes what the agent retrieves. For an engineering Q&A agent, where the corpus changes constantly and a stale answer can route someone into a deprecated procedure, freshness measured in minutes rather than hours is not a nicety. It is the difference between an agent your team trusts and one they learn to route around.

Pull in PDFs, support tickets, and external docs without a second stack

Engineering knowledge is never confined to one tidy repository. The authoritative answer to a question might live in a vendor PDF, a SOC 2 control narrative, a closed support ticket where someone already debugged this exact failure, or an external API reference your team depends on but does not own. An agent that only sees your wiki gives confidently incomplete answers, which is its own kind of failure.

The instinct is to bolt on another ingestion pipeline per source, a PDF parser here, a support-database connector there, each with its own embedding model and its own staleness problem. That is how you end up maintaining four retrieval systems and reconciling four sets of results. Knowledge Bases collapse this. They turn datasets, websites, PDFs, and support databases into agent-readable documents that share the same Sanity Context retrieval path, so a vendor PDF and an internal runbook are queried through the same hybrid GROQ query and ranked against each other consistently.

This maps to the pillar of providing a shared foundation rather than silos. The error code in a support ticket, the procedure in a PDF, and the service definition in your structured content all become first-class documents the agent can retrieve and cross-reference. Production agents reach this through the Sanity Context MCP endpoint, a single connection point rather than one integration per source. The win is not just less code. It is that an answer can synthesize across sources, citing the ticket that already solved the problem and the runbook that prevents it, because both live on the same foundation instead of in four disconnected stores.

Govern what the agent is allowed to say

An internal Q&A agent is a publishing surface, and the moment you treat it as one, the right controls become obvious. The questions that matter are editorial: who decides the agent's behavior, how is a wrong answer corrected, and how do you verify a change before the whole engineering org starts acting on it? Most agent stacks answer this with configuration buried in code, which means the people who actually own the content, your staff engineers and tech writers, cannot fix a bad answer without a deploy.

Move that governance into the same place editors already work. In Studio, the people who own engineering knowledge curate the documents the agent retrieves and govern its instructions directly, without filing a ticket against the platform team. When a procedure changes, they stage the new version with Content Releases and review how the agent will answer against the updated content before any of it goes live, the same way they would stage a website change. Agent Actions, the schema-aware APIs for LLM-driven workflows like generate, transform, and translate, run inside that same governed environment rather than as an ungoverned script hitting a model endpoint.

This is governance as workflow, not as an afterthought. On compliance, Sanity is SOC 2 Type II compliant and GDPR-aligned, with regional hosting for data residency and a published sub-processor list, which matters when your Q&A corpus includes internal architecture and security procedures you cannot leak. Roles and Permissions scope who can change agent instructions, and Audit logs record who changed what. The result is an agent whose behavior is reviewable and reversible, owned by the people closest to the content rather than by whoever last touched the retrieval code.