When the Stack Trace Is Clean and Production Is Already Wrong

At 2:14am the loan-eligibility agent returned HTTP 200. No exceptions. No alerts. The income-verification tool logged status: ok. Over the next six hours the agent rejected 847 valid applications because a third-party API had quietly renamed a JSON field overnight. The model received a key it had never seen, hallucinated null for income, and declined every pending case. ^[8]

This is the failure class agentic systems were built to expose. Infrastructure healthy. Dashboard green. The fault is semantic — distributed across a reasoning chain where one wrong interpretation at step 3 shapes every decision 40 calls deep. There is no exception to read. There is no line number to fix.

This runbook gives you four things: a failure-mode taxonomy you classify against before reading the trace, the minimum observability stack that catches silent failures before users do, a five-step triage sequence that works when the error log is empty, and a postmortem template that does not pretend non-determinism is a typo.

Traditional SRE asks what crashed. Agentic SRE asks something else: which step produced the wrong premise, and how far did it travel before anyone noticed?

The most dangerous agent failures are the ones your monitoring stack calls successes.

Latency histograms, error rates, CPU graphs — every one of those signals was designed for deterministic systems where failure means the process exited nonzero. Agents break that assumption. A clean HTTP 200 from an LLM API is not evidence of correct behavior. Valid JSON from a tool is not evidence the agent interpreted it correctly. A formatted final response is not evidence the response is true. ^[2]

The canonical pattern: a third-party tool changes its response schema. No exception fires. The JSON is still valid. The model receives a field name absent from training and, instead of escalating, generates a plausible value from prior context. That value enters the session state, gets cited in the next three calls, and by the time the user sees the output the entire reasoning chain has been built on a hallucinated foundation. None of it shows up in your Datadog dashboard. ^[8]

In early 2026, 89% of teams running agents in production had implemented some form of agent observability. Only 52% had eval pipelines that actually validated whether the agent's behavior was correct, not just syntactically complete. ^[2] The 37-point gap between "we have traces" and "we catch behavioral failures" is where production incidents live.

Here is the part most teams miss: the model is rarely the root cause. The dominant failure is a boundary failure — a tool returned partial JSON, retrieval pulled the wrong chunk, a planner looped, an API contract changed without notice. ^[4] If the LLM's reasoning is the actual root cause, your team got unlucky. If the tool interface failed, that is just production.

The intuitive assumption is that a 40-call session distributes failures proportionally across all steps. The evidence contradicts it.

Across 1,200 logged agent runs with verified hallucinations, analysis reported from multiple benchmark evaluations found 71% of errors were introduced in the first two steps — typically during query reformulation or initial retrieval. ^[1] The final output looks like a complex multi-step failure. Trace it backward and the root cause is almost always an error in how the task was understood or how the first retrieval was executed.

This changes the triage path. When you are staring at a 40-step session, start at steps 1–3. Read what the agent understood the user to be asking. Read what came back from the first retrieval or tool call. Read whether the initial plan was sound. If those steps are clean, the problem is unlikely to be a fundamental reasoning failure — it is more likely a specific tool boundary issue later in the chain.

Hallucination probability scales nonlinearly with tool call count. Across agentic benchmark evaluations, the probability of at least one hallucination in a run climbs from roughly 12% at 2 tool calls, to 67% at 10, to above 85% at 15 or more. ^[1] This is not an argument against complex agents. It is the structural reason any agentic workflow that requires 10+ tool calls is production-critical infrastructure and needs dedicated semantic observability — not standard LLM monitoring with extra fields.

Benchmark numbers come from controlled environments. Real production data is dirtier — noisier inputs, inconsistent schemas, retrieval systems that drift over time. Real-world hallucination rates in enterprise deployments are likely higher than the benchmarks suggest. Treat the directional insight as the load-bearing claim: errors are front-loaded, complexity compounds risk nonlinearly, and the precise numbers depend on how clean your inputs are.

Latency, token count, API errors — necessary for any production LLM workload, insufficient for agentic incident response. The minimum stack captures four things generic LLM monitoring does not.

Session-level correlation. Every LLM call, every tool invocation, every state mutation must carry the same session_id. Without it, a 40-step session decomposes into 40 disconnected events in your log store. The session ID is the single most important field you can add — and the one most commonly missing from first-generation agent deployments.

Tool call payloads at every step. Not "tool was called with status 200" — the actual input arguments and the actual response payload, including the field structure. Schema drift detection — spotting when a tool response has different field names than expected — is one of the highest-signal automated checks available. It caught the loan agent incident above in under two minutes on replay. The TOOL_SCHEMA_DRIFT flag on step 3 pointed directly to the renamed field. ^[5]

Step-level anomaly flags. Structured fields that mark specific failure patterns the moment they occur: ENTITY_HALLUCINATION when the model references an entity ID not in prior context, TOOL_SCHEMA_DRIFT when a tool response does not match its documented schema, LOOP_DETECTED when the same tool fires with near-identical arguments three or more times in a session. ^[6] These flags do not catch everything. They reduce the backward trace from "read 40 events manually" to "check the flagged steps first."

Session outcome classification. Every session ends in one labeled outcome from a small set: Completed, Tool Error, Bad Output, Timeout, Budget Exceeded. The label enables cohort queries — "show me every Bad Output session from the last week" — that single-trace inspection cannot answer. ^[4]

Before reading any trace in detail, classify the failure. Classification determines which 20% of the trace you actually need to examine. Getting this wrong costs you the first 30 minutes of every agent incident.

The decision tree below encodes the four operationally distinct failure modes. Not because the taxonomy is academically satisfying. Because each one has a different blast radius and a different first action. A cost runaway needs a session kill order before forensics begins. A tool misfire needs a downstream system audit before forensics begins. Starting forensics without classification means an hour spent debugging the wrong thing while the agent continues to affect production.

The first time we ran an agent postmortem on a standard SRE template, the conclusion read: "root cause: LLM hallucination." That is about as useful as writing "root cause: gravity" for a structural failure.

Standard templates ask for a single root cause — one line, one changeset, one decision that went wrong. Agent failures do not cooperate with that framing. Every significant agent incident we investigated had at least three contributing factors, each insufficient on its own: a tool response schema changed without notice, a prompt that did not constrain entity references, an eval suite with no coverage for this failure class. Fixing any one of them in isolation would not have prevented the incident. The gravitational pull toward a singular root cause mislabels the problem and produces narrow fixes that block the exact failure without addressing the class.

The agent postmortem template replaces "root cause" with "contributing factors" and refuses to close the incident until all of them are named.

1. Session context — Session ID. Time range. Total LLM calls. Total tokens. A plain-language description of intended session behavior. This grounds every other finding in concrete evidence rather than generalized claims about model behavior.

2. Failure classification — Which of the four failure modes applies. This is not bureaucracy. It determines which prevention layers were absent and which architectural change is actually needed.

3. Fault origin and cascade path — The step number that introduced the wrong premise. The full context window at that step. The cascade path from fault origin to final output, with specific step numbers — not "around step 10" but "step 7."

4. Impact assessment — Every external action taken after the fault origin, classified reversible or irreversible, with a concrete remediation plan for each irreversible one.

5. Contributing factors — Name all three or four. Model limitation? Prompt design gap? Tool interface contract change? Retrieval contamination? List each one explicitly. Refuse to collapse them into a single cause.

6. Detection gap — Why didn't observability catch this before users were affected? Missing anomaly flag? No eval coverage for this failure pattern? No guard rail on the tool call? The detection gap is the most important finding in the entire postmortem. It is the only one that drives an infrastructure change rather than a one-off prompt edit.