LLM Observability Beyond Traces: Instrumenting Quality Signals in Production

Three weeks into production, every dashboard was green. Latency median at 380ms, error rate at 0.3%, token spend exactly on budget. The engineering team had signed off on the SLO review and moved to the next sprint. User satisfaction scores had been dropping every single day for two weeks.

It took a VP of Product escalating before anyone traced the degradation back to a prompt regression. A developer had cleaned up the system prompt eleven days earlier, removing nine words that told the model to format responses as structured JSON. Latency didn't change. Error rate didn't budge. Token cost was unaffected. The only signal was user feedback that nobody had connected to the engineering timeline.

This is the quality gap in LLM observability: the signals most teams instrument first — latency, error rate, token cost — can all sit healthy while output quality collapses. Infrastructure metrics measure whether your system functioned. They say nothing about whether it worked.

Traditional distributed tracing assumes determinism. If a function returns 200, it succeeded. You can write an SLO against it, alert on deviations, build dashboards. LLMs break this assumption completely.

A response can be syntactically valid, arrive in under 500ms, return a 200 OK, consume exactly the expected tokens, and still be factually wrong, poorly formatted, or entirely off-topic for the user's intent. The span looks clean. The service looks healthy. The user experience is quietly deteriorating.

The gap between 'request completed' and 'request was useful' is where most LLM quality failures hide. Teams commonly discover this the hard way: their monitoring alerts on model API outages but goes silent during slow, multi-week quality degradation events that users notice long before engineers do.^[3]

For RAG systems, this compounds. A retrieval step that returns documents with slightly lower relevance scores won't trigger any infrastructure alert. The model does its best with worse context. Outputs degrade subtly. Tracking retrieval and generation quality separately is the only way to diagnose which layer is failing — aggregating them into a single health metric hides the root cause.

The core technique: after a request completes, route a sample of it through an evaluator that scores the response against criteria you define — format compliance, relevance, factual coherence. Aggregate those scores over time and alert when trends break.

The sampling rate is a cost decision. Adaline's 2026 observability guide recommends 5–10% of production traffic for LLM-as-judge evaluation — enough for statistical confidence without doubling inference costs.^[1] Start lower, at 1–2%, until you've validated that your judge model produces scores consistent with human judgment on the same samples. Scaling to 5–10% before you trust the scores means collecting misleading data at scale.

Two rules regardless of sampling rate: evaluate 100% of requests that triggered an error or a user complaint, and stratify by intent type so each request category gets proportional coverage. Sampling only from the most recent requests will give you recency bias — you'll miss gradual degradation that accumulates across intent categories unevenly.

The metric that matters is trend, not absolute score. A quality dimension sitting at 3.8/5.0 today is meaningless without history. That same dimension dropping from 4.2 to 3.5 over two weeks is a meaningful signal. Teams running continuous quality monitoring report that a 5% drop in any single dimension over a 7-day rolling window is the threshold that surfaces real degradation before users escalate.^[2]

One honest caveat: non-determinism means you can't expect consistent scores on individual requests. Don't alert on single-request variance. Alert on distribution shift — when the population of scores changes shape, not when one score fluctuates.

Prompt drift is invisible by design. Nobody announces it. A developer refactors a helper function, a template variable changes scope, a conditional branch that used to include an instruction now skips it. The effective prompt the model receives changes — and unless you're hashing your templates, you have no record it happened.

The fix requires one function and one span attribute. Hash your prompt template before rendering it with user data, and attach that hash to every LLM call span. When the hash changes, you know the template changed — regardless of whether any deployment log mentions it.

Correlate hash changes with quality score trends in the following 24–48 hours. A hash change accompanied by a quality score drop is almost always the root cause. A hash change with no quality impact is usually safe to let stand. The combination tells you what a deployment log alone cannot: whether the change actually mattered to output quality.

One thing to note: this only catches changes to the template itself, not changes to the data interpolated into it. For cases where changing context (like user history, retrieved documents, or business rules) is the source of quality drift, you need embedding-based drift detection on the rendered input distribution.^[6] Prompt hashing is a fast, zero-cost signal for the most common cause of silent regressions — it's not a complete solution.

The most common mistake is starting with quality evaluators before having cost tracking in place. This seems backwards until you realize: your sampling rate for LLM-as-judge is directly constrained by how much that evaluation costs. Without knowing your baseline inference cost per request type, you have no principled basis for choosing a sampling rate. You'll either undersample (missing real degradation) or oversample (running quality evals that cost more than the system they're monitoring).

Cost tracking also delivers fast value on its own. Adding trace IDs and cost-per-span takes a few hours. It immediately shows which request types are expensive and whether any single workflow is responsible for disproportionate spend. That data shapes every downstream sampling decision.

There's also the uncomfortable truth about observability comprehensiveness: teams that try to build the full quality monitoring stack in the first sprint typically ship nothing usable in the first month. Phased delivery — something working at each stage — consistently outperforms a complete design that's half-implemented.

The teams that catch quality degradation before it escalates to executive complaints share one characteristic: they treated output quality as an engineering concern from the start, not a customer support problem to triage after the fact.

The infrastructure for this isn't complicated. It's a hash function, a sampling decision, a judge model, and the discipline to build baselines before you need them. What's hard is the organizational habit of treating semantic quality as a first-class signal alongside latency and error rate — not as a soft metric that belongs on a different team's dashboard.

Your traces are already recording that requests completed. Start recording whether they worked.