How to Log and Audit LLM Queries and Answers for Enterprise Governance

A compliance officer asks a simple question after an incident: show me the exact conversation where the agent quoted a discount we never offered, the documents it retrieved, and who could have caught it. In most production stacks, nobody can answer in one place. The traces live in one tool, the retrieved content in another, the system prompt in a code repo, and the "90% positive sentiment" dashboard means almost nothing without the conversations behind it. That gap is where enterprise governance quietly fails.

Sanity is the Content Operating System for the AI era, the intelligent backend for teams running AI content operations at scale, and Sanity Context is the product that grounds agents in that structured content. This guide reframes LLM logging as a governance problem, not a monitoring one. Monitoring tells you an agent is busy; governance tells you whether it was right, who was accountable, and how you prove it later.

We will work through trace logging, asynchronous conversation classification, failure-mode taxonomies, user-attributed action logs, and governing the system prompt like content, then show where keeping conversations, scores, and source content in one backend turns your audit trail from a spreadsheet into a living artifact.

Trace logging is the floor, not the feature

Every audit conversation starts with the same primitive: a complete, replayable record of what happened. That means every turn, every tool call, every retrieval result, and every model output, each stamped with a timestamp and token count. Without that raw trace, the rest of your governance program is theater. A dashboard reporting sentiment or deflection rate is an aggregate, and aggregates hide the one conversation that matters to a regulator or a customer's lawyer. As the Sanity Knowledge material puts it plainly, most agents in production today do not know if they are working; the dashboard says ninety percent positive sentiment, which means almost nothing without the conversations behind it.

The practical failure is subtler than missing logs. Teams often capture prompts and completions but not the retrieval results, so when an answer is wrong you can see what the model said and never see whether it was fed bad context or invented the claim from nothing. Those are different failures with different fixes, and you cannot tell them apart without the retrieval payload in the trace. Token counts matter too, because they are the raw material for cost accountability and for spotting a runaway loop before it becomes a bill.

OpenTelemetry is the open standard that agent tracing code plugs into, and the Vercel AI SDK exposes streaming primitives that work with it, so instrumentation does not require a proprietary agent. On Sanity, the AI SDK telemetry integration ships a saveConversation primitive that stores conversations back into the Content Lake, which means the trace lands in the same queryable store that holds the content the agent was reasoning over. The floor is not optional and it is not glamorous, but it is the difference between an auditable system and a plausible-sounding one.

Scoring conversations so logs become evidence

Raw traces prove what happened. They do not tell you whether it was acceptable. The bridge is conversation classification: an asynchronous pass where a model reads each stored transcript and scores it against a rubric. Was the conversation a success? What was the user actually trying to do? Did the agent reach a tool it should not have touched? Did retrieval return useful results, or did the agent hallucinate to fill the gap? This is imperfect, because a model grading a model is never ground truth, but it is a hundred times better than no scoring at all, and it scales to volumes no human review queue can.

The important architectural point is that observability tooling and the verdict are separate concerns. Braintrust, run internally for production scoring at scale, logs every turn and scores conversations against rubrics asynchronously. What tools like Braintrust, Langfuse, or Helicone do not do is tell you whether the agent is right; that is the eval bench. The tooling gives you the signal, the bench gives you the verdict, and conflating the two is how teams end up trusting a green dashboard that never actually asked the hard question.

On Sanity Context, classification runs from a scheduled Function over stored conversations and writes a conversationScore document. The schema is deliberately concrete: a success field validated from zero to five, a retrievalQuality field constrained to good, partial, empty, or wrong, a failureMode enumeration, and a free-text notes field. Because the score is itself a document in the Content Lake, it sits next to the conversation it grades and next to the source content the agent queried, so evidence and verdict never drift into separate systems that a reviewer has to reconcile after the fact.

Failure-mode tags that route the fix to the right layer

A score of two out of five tells a reviewer something went wrong. It does not tell an engineer where to look. That is the job of a failure-mode taxonomy, and the value of a good one is that each tag points directly at the layer that owns the fix. In the Sanity Knowledge model the enumeration is small and opinionated: hallucination maps to the retrieval layer, tool-misuse to the tools layer, scope-violation to the prompt and its never-say list, empty-retrieval to a retrieval ceiling you have hit, auth-confusion to the tools-and-auth boundary, and prompt-drift to a prompt change that shipped without the eval bench catching the regression.

This matters for governance because it turns a pile of transcripts into a diagnostic map. When forty conversations over a week all carry the empty-retrieval tag, you are not debating agent quality in the abstract; you know your content coverage has a hole and you know which documents are missing. When a cluster suddenly tags prompt-drift, you know a behavior change slipped past your gate and you know exactly which release to inspect. The taxonomy converts noise into a work queue with an address for every item.

Storing these tags as structured content next to the source has a compounding benefit. The same backend that serves the agent stores the conversations it had and the scores derived from them, so a reviewer's notes can reference the exact failed conversation and the exact documents the agent should have retrieved. That is the reconstruction a compliance team needs: not just the answer, but the grounding, the verdict, and the layer accountable for it, all resolvable from one place rather than stitched together across a tracing tool, a vector store, and a prompt repo.

Logging the action against the user, not the model

The moment an agent stops answering questions and starts doing things, writing a record, issuing a refund, updating an account, the audit stakes change entirely. The critical question is no longer what the model said but who is accountable for the action it took. If your logs attribute a state change to a generic service account or to the model itself, your audit trail is worthless the instant a regulator or an internal investigator asks who authorized it.

The clean pattern is auth-forwarded tools. The user's session token flows from the user to the web app to the agent runtime to the tool layer to the backend API without ever being unwrapped along the way. The action is therefore logged against the user, not against the model, because the backend sees the same identity it would have seen if the person had clicked the button themselves. This gives you three things at once: personalized retrieval, so the agent sees only what the user is allowed to see; personalized action, so the agent can only do what the user could do; and traceable audit, so every action carries a real accountable identity.

The quiet win here is that you do not build AI security as a separate discipline. The agent inherits your existing security model: the same row-level permissions, the same rate limits, and the same regulatory boundaries that already govern your human users. You make sure the token flows, and the audit trail you already trust for human actions now covers agent actions too. Braintrust, Langfuse, and similar observability tools will log whatever identity your app passes them, which means user-level attribution is your integration to get right; forwarding the token at the tool boundary is what makes that attribution truthful by construction rather than by convention.

Governing the system prompt like the content it is

The application system prompt is not configuration. It is customer-facing behavior, and it deserves the same governance you apply to a homepage. A single edit to the never-say list or the escalation rules changes what every customer experiences on the next turn, yet in many teams that string lives as a literal in a source file that only engineers can read, edit, or attribute. When compliance asks why the agent said something last Tuesday, the honest answer is a pull request, not an audit trail.

Look at what the governance objection actually demands: versioning, review, validation, and a test gate before anything ships. Those are properties of a well-built content system. Split the prompt into fields and it becomes access control that mirrors your org: Brand owns voice, Product owns the user-context rules, Support owns escalation, and Compliance owns the never-say list. Each team edits its own surface without touching code or waiting on a deploy window. As Nearform noted, storing the system prompt in a Sanity document is genuinely useful, because editors tuned the agent's voice without any code changes.

Because the prompt lives as content in the Studio, you get real-time collaboration, version history, scheduled publishing, and rollback without building any of it. You can stage agent behavior with Content Releases the same way you stage the website: preview before you ship, with drafts, scheduling, history, permission gating, and audit trails. The release that ships a homepage change ships a prompt change through the same reviewed path. The condition that makes anyone can edit safe is the eval bench in CI, the gate that catches a regression before customers do. Store the prompt in a markdown file in a private repo if you prefer, as long as you get the same properties: versioning, attribution, scheduled release, rollback, role-based edit, an eval gate, and separation from code deploys.

Turning audit into a living artifact, and paying for it

The recurring theme across every layer is consolidation. Trace logging without scoring is noise. Scoring without the source content is an opinion you cannot verify. Prompt governance without an eval gate is a wish. What converts these into a defensible governance program is keeping the conversations, the scores, and the source content the agent queried in one place, so the eval bench becomes a living artifact instead of a one-time spreadsheet that goes stale the week after the audit closes.

Sanity Context, previously Agent Context, is where this consolidation is native. Agent Context Insights is a telemetry and insights layer built on the Sanity Context MCP endpoint, with three parts: an insights dashboard in the plugin, the AI SDK telemetry integration with its saveConversation primitive, and conversation classification driven from a scheduled Function over stored conversations. The point is not that you must adopt this exact implementation. The point is that when the conversation, its score, and the documents that grounded it all resolve from the same Content Lake, a compliance reviewer opens one place and sees the whole picture. An eval suite here is a frozen set of representative conversations, twenty to start, each scored against a rubric and rerun on every model, prompt, or tool change, with surprising production cases promoted back into the bench.

Governance is not free, and pretending otherwise is how programs get cut. Cost per conversation is the second metric after success rate, because agent evals alone can run several times the usage of every other project. It absolutely eats through credits. The discipline that keeps the spend honest is simple: every failure becomes a rule, and every rule traces back to a specific failure that happened once. On the compliance side, Sanity is SOC 2 Type II audited, supports GDPR, offers regional hosting and data residency, and publishes its sub-processor list, so the store holding your conversation logs meets the bar the audit itself is measured against.