LLM Observability: Catch Output Quality Drift Your Green Traces Can't See

Three weeks into production. Median latency 380ms. Error rate 0.3%. Token spend on budget. SLO review signed off. The team moved on.

User satisfaction had been dropping every day for two weeks. It took a VP escalating before anyone correlated the curve with a prompt cleanup eleven days earlier — nine words removed from the system prompt that had told the model to format responses as structured JSON. Latency did not move. Errors did not move. Cost did not move. The only signal was user complaints, and nobody had wired user complaints into the engineering timeline.

This is the gap. The signals teams instrument first — latency, error rate, token cost — all sit healthy while semantic quality collapses underneath them. Infrastructure metrics tell you the request completed. They have no opinion on whether it worked. Logging that records 'request returned 200' is not observability for an LLM. It is an alibi.

Distributed tracing was built on a contract: a function returns 200, the request succeeded. SLOs hold. Alerts fire. Dashboards mean something. LLMs break that contract.

A response can be syntactically valid, arrive under 500ms, return 200, consume the budgeted tokens, and still be wrong, malformed, or off-intent. The span is clean. The service is healthy. The user experience is degrading. Nothing in your tracing stack was built to notice the difference.

The gap between 'request completed' and 'request was useful' is where every LLM quality failure hides. Teams find out the same way every time — monitoring fires on model API outages and stays silent through multi-week quality regressions that users notice long before engineers do.^[3] By the time someone escalates, the curve has been bending for two weeks.

The failure taxonomy is wider than most teams expect. Research on production LLM failure modes identifies at least five distinct classes that infrastructure metrics are blind to: output format deviation, semantic drift, factual incoherence, context handling failures (long context truncation, retrieval miss), and safety degradation (refusal rate changes from provider-side model updates).^[10] Each class requires a different signal to detect.

RAG compounds it. Retrieval scores that drift a few percent will not trigger any infrastructure alert. The model does its best with worse context. Outputs degrade subtly. Aggregating retrieval and generation into one health number is how the root cause stays hidden — track them as separate signals or don't bother tracking them at all.

The OpenTelemetry GenAI semantic conventions standardize how LLM calls are recorded across providers.^[7] They're the right foundation. But understanding exactly what they cover — and what they don't — determines where you have to build.

What the conventions give you out of the box:

• gen_ai.system — the provider (e.g., anthropic, openai, aws.bedrock)
• gen_ai.request.model — the exact model name requested
• gen_ai.usage.input_tokens / gen_ai.usage.output_tokens — token counts
• gen_ai.response.finish_reasons — why generation stopped (stop, length, tool_calls)
• gen_ai.client.operation.duration — a histogram of LLM call latencies^[8]

As of early 2026, most GenAI semantic conventions are in experimental status — the API isn't fully stabilized, and major vendors like Datadog and Grafana are beginning native support but with version-specific caveats. Pin your instrumentation library version and test collector compatibility before assuming pass-through.

What the conventions do not cover:

• Output evaluation or quality scoring
• Safety and hallucination detection
• Prompt template identity or versioning
• Intent-level quality tracking

Those are yours to instrument. The convention doesn't stop you — OpenTelemetry collectors pass custom span attributes through without modification. You add gen_ai.prompt.hash, gen_ai.prompt.version, quality.score.format, quality.score.relevance as custom attributes on the same span. They flow through the same pipeline your infra metrics use. No separate pipeline, no separate backend.

The mechanism is simple. After a request completes, route a sample of it through an evaluator that scores the response against the criteria you actually care about — format compliance, relevance to intent, factual coherence. Aggregate the scores. Alert when the trend breaks.

Sampling rate is a cost decision, not a statistics decision pretending to be one. Adaline's 2026 guide puts the range at 5–10% of production traffic — enough for confidence intervals without doubling inference spend.^[1] Don't start there. Start at 1–2% and validate that the judge agrees with human raters on the same samples. Scaling to 5–10% before you trust the judge means collecting misleading data faster.

Two rules hold regardless of sampling rate. Evaluate 100% of requests that errored or got escalated. Stratify by intent so each category gets proportional coverage. Sampling only from the most recent traffic gives you recency bias and hides slow degradations that compound across intent categories unevenly.

The metric that matters is trend, not absolute score. 3.8/5.0 today is meaningless. 4.2 → 3.5 over two weeks is a structural signal. Continuous quality monitoring reports a 5% drop in any single dimension over a 7-day rolling window as the threshold that surfaces real degradation before users escalate.^[2]

One honest constraint: non-determinism means individual scores will move even on identical inputs. Don't alert on single-request variance. Alert on distribution shift — when the population of scores changes shape, not when one number fluctuates.

Most teams skip calibration. They pick a judge model, write a rubric, and start collecting scores. The scores look plausible. The trend lines look smooth. What they've actually built is an expensive random number generator with a good aesthetic.

Calibration is the step that separates measurement from theater. Rate 50–200 production samples by hand, have your judge score the same samples, then compute Cohen's kappa between the two. For balanced criteria on a binary pass/fail rubric, 50 stratified traces will pin kappa to within ±0.10–0.15 at a 95% bootstrap confidence interval. That's enough to know whether your judge is tracking human judgment.^[9]

When your criteria involve rare-but-expensive failure classes — safety violations appearing in 6% of traces, for example — 50 samples isn't enough. The variance of kappa is dominated by the count of minority-class examples, not the total sample size. Plan for 200+ when minority classes matter.^[9]

The threshold that matters: kappa ≥ 0.6 before scaling to 5–10% of production traffic. Below that, the judge isn't tracking human quality judgment consistently enough to detect real regressions. Between 0.6 and 0.8 is workable for most quality dimensions. Above 0.8 is the bar for high-stakes criteria like safety detection.

Calibration also has a shelf life. Judge drift is real: without a monthly recalibration cadence, agreement degrades over 60–90 days as production distribution shifts. Run your gold set monthly. Alert if kappa drops below threshold. Treat the judge as a system that also needs monitoring.

Prompt drift is invisible by design. A developer refactors a helper. A template variable shifts scope. A conditional branch that used to include an instruction now skips it. The effective prompt the model receives changes, and unless you're hashing the template, no deployment log will mention it.

The fix is one function and one span attribute. Hash the template before rendering it with user data. Attach the hash to every LLM call span. When the hash moves, the template moved — whether or not anyone wrote a changelog entry.

The leverage is in correlation. A hash change followed by a quality score drop within 24–48 hours is almost always the root cause. A hash change with no quality impact is the safe case — note it and move on. The combination tells you what a deployment log alone cannot: did the change actually matter.

Model provider updates are a second, subtler form of drift. Providers update model weights, safety policies, and serving infrastructure without changing API endpoints. An August 2025 postmortem documented bugs affecting Claude Sonnet 4's performance without any API modifications, including routing errors affecting up to 16% of requests. Prompt hashing doesn't catch this — the template didn't change. What catches provider-side drift is a continuous baseline: quality scores should be stable between template changes, and any shift that isn't correlated with a hash change needs a different root cause hypothesis.

One constraint: hashing catches changes to the template itself, not to the data interpolated into it. When user history, retrieved documents, or business rules drift, you need embedding-based drift detection on the rendered input distribution.^[6] Hashing is a zero-cost signal for the most common silent regression. It's not the whole story, and a team that claims otherwise is selling something.

There's a failure class that prompt hashing misses entirely: the input distribution shifts while your prompt stays constant. Users start asking questions your system wasn't built for. A seasonal event changes query intent. A product update attracts a different user segment. The model handles these new queries worse than your training distribution, but every span shows the same template hash.

Embedding-based drift detection addresses this. Embed a rolling sample of incoming queries using the same encoder your retrieval system uses. Cluster them over time. When the centroid of the production distribution moves outside a threshold of your golden dataset coverage, you're receiving queries meaningfully different from what you built for — before failure rates climb.^[2]

The mechanics: compute cosine distance between the production query embedding centroid (rolling 7-day window) and the golden dataset centroid. A 15–20% shift in centroid distance is a reasonable threshold for investigation. It's not a reason to page; it's a reason to sample manually and decide whether the new distribution requires prompt updates or retrieval index refreshes.

For RAG pipelines, a variant of this applies at the retrieval layer. Track the distribution of retrieval scores across a rolling window. A 10% drop in the median retrieval score over 3 days means the knowledge base is drifting relative to the queries — not a model problem, a data freshness problem. Track it as a separate signal from generation quality, or you'll spend two days debugging the wrong layer.

Before you spend inference dollars on an LLM judge, run deterministic checks on everything. These execute in microseconds, cover 100% of requests, and catch the class of failures that are easiest to produce and hardest to notice: structural regressions.

Format validation is the highest-ROI heuristic. If your application expects JSON, validate the schema on every response. Track the JSON-parse failure rate as a metric. A spike from 0.1% to 3% failure rate is an earlier signal than any quality score — and it doesn't cost you an LLM call to detect. Similar logic applies to response length: if your summarization workflow should produce 100–300 words, a response histogram that collapses to 20-word outputs is detectable without a judge.

Other high-value heuristics: scan for refusal patterns (I'm sorry, I can't, As an AI) to track safety calibration drift from model updates; check for unexpected language or character sets in international deployments; validate that expected structure markers appear (section headers, numbered lists, code fences) when the task requires them.

The pass/fail rate of each heuristic check is itself a monitoring signal. Logging pass/fail rates by intent type over time gives you the earliest warning of model drift or provider-side API changes — visible within hours, not days.

The most common mistake is starting with quality evaluators before having cost tracking in place. This seems backwards until you realize: your sampling rate for LLM-as-judge is directly constrained by how much that evaluation costs. Without knowing your baseline inference cost per request type, you have no principled basis for choosing a sampling rate. You'll either undersample (missing real degradation) or oversample (running quality evals that cost more than the system they're monitoring).

Cost tracking also delivers fast value on its own. Adding trace IDs and cost-per-span takes a few hours. It immediately shows which request types are expensive and whether any single workflow is responsible for disproportionate spend. That data shapes every downstream sampling decision.

There's also the uncomfortable truth about observability comprehensiveness: teams that try to build the full quality monitoring stack in the first sprint typically ship nothing usable in the first month. Phased delivery — something working at each stage — consistently outperforms a complete design that's half-implemented.

The teams that catch quality degradation before it escalates to executive complaints share one characteristic: they treated output quality as an engineering concern from the start, not a customer support problem to triage after the fact.

The infrastructure for this isn't complicated. It's a hash function, a sampling decision, a judge model, and the discipline to build baselines before you need them. What's hard is the organizational habit of treating semantic quality as a first-class signal alongside latency and error rate — not as a soft metric that belongs on a different team's dashboard.

Your traces are already recording that requests completed. The question is whether you're recording whether they worked. That gap — between completion and usefulness — is where production LLM quality lives and dies.