Agent Incident Playbook: Debug a Failure Spread Across 40 LLM Calls

2:37am. The order agent returns a confirmation for an order that does not exist. The on-call pulls logs and finds nothing. No stack trace. No 500. No exception. HTTP 200, well-formatted prose, full confidence — confirming the wrong order against the wrong customer account.

The failure is not in your code. It's distributed across 40 LLM calls, where the actual fault happened around step 7 — a small hallucination about a customer identifier — and every subsequent call built confidently on top of the poisoned premise. By the time the user saw the output, that error had been laundered through 33 more reasoning steps.

Tool calling fails between 3–15% of the time in production, even in well-engineered systems ^[8]. When it does, there's no stack trace. Error rates look normal. Latency is fine. The SLO dashboard is green. The user is filing a ticket.

Traditional SRE asks one question: what crashed? Agentic systems demand a different one: which of these 40 reasoning steps produced the wrong context that cascaded into the wrong action? That's not debugging. That's distributed session forensics.

This playbook maps the four operational failure modes to concrete triage steps, gives you a backward-trace method for locating fault origins inside long sessions, shows you the exact OpenTelemetry span attributes that matter for forensics, and provides a postmortem template built for systems where non-determinism is the default state.

Traditional runbooks assume systems that fail loudly. A service crashes. A timeout fires. A null pointer propagates. You find the exception, walk the call stack, identify the line. The bug has a location.

Agent incidents violate every assumption in that model. The agent didn't crash. It ran to completion and returned a result. The result was wrong, and nothing in your observability stack knows that. The failure is closer to a reasoning defect than a code defect. And reasoning defects compound.

A wrong assumption at step 7 shapes the framing at step 8, which selects the wrong tool at step 9, which returns data that reinforces the wrong assumption at step 10. By step 20, the agent has constructed a coherent internal narrative that is entirely wrong — and it has the tool call logs to prove it. The model is not confused. It is confidently wrong, and its own earlier output is now the evidence it cites.

This isn't a hypothetical mode. In multi-agent systems, hallucination cascades are a documented failure pattern: one subagent produces a plausible-sounding but false intermediate result, the orchestrator accepts it as ground truth, and every downstream agent operates on poisoned inputs ^[5]. Without span-level tracing that captures what was in the context window at each step, that cascade is invisible.

The agentic AI fault taxonomy literature ^[1] catalogs dozens of failure patterns. For incident response, they collapse into four operationally distinct modes — each with a different signal, a different blast radius, a different triage path.

1. Hallucination propagation

The model generates a false assertion early in the session. Because agents accumulate context across calls, that assertion gets referenced and re-affirmed in later steps. By call 20, the hallucination is an established fact inside the session. The model is not wrong because it's confused. It's wrong because its own earlier output is now evidence.

Signature: confident, coherent, structured output, internally consistent. Built on a false premise. No tool call failure. No error code.

2. Tool misfire

The model selects the wrong tool, passes malformed arguments, or misinterprets tool output. Unlike hallucination, tool misfire produces real side effects immediately — deleted records, sent emails, processed payments, triggered workflows. The session looks healthy on latency and token counts while causing irreversible damage downstream. Schema drift is a common trigger: a dependency update changes how tool schemas are generated, making them incompatible with the model provider's format ^[6].

Signature: a tool was called with an unexpected argument pattern, or a tool returned data the model processed without validating against the session's stated goal.

3. Context poisoning

A hallucination or injected content makes it into persistent context — goal state, working memory, retrieved documents. The agent's framing of the entire task warps. Long-running agents are especially exposed because they carry context across many turns and the poisoning compounds ^[5]. Context poisoning differs from hallucination propagation in what it corrupts. Hallucination poisons factual claims about the world. Poisoning corrupts the agent's self-model — what it thinks it's trying to do.

Signature: the agent's stated goal drifts across turns without user instruction. The world model becomes internally inconsistent.

4. Cost runaway

The agent loops: repeated tool calls with similar arguments, infinite retry logic, self-spawning subagents, circular reasoning chains. In November 2025, a market research pipeline running four LangChain agents entered an unintended infinite loop that ran for 11 days and cost $47,000 ^[11]. Alerts fired. Nobody acted on them in time. Token consumption compounds silently until a budget alarm fires or the session times out.

Signature: per-session token counts spike well above baseline. Tool call frequency is abnormally high. The same tool gets called multiple times with near-identical arguments.

The industry is converging on OpenTelemetry (OTel) as the standard for agent telemetry, and the GenAI semantic conventions — currently at v1.37 — define the span attribute schema ^[10]. Datadog, Langfuse, and Arize Phoenix all map to these conventions natively, so you instrument once and export to any backend ^[9].

For forensics, not every attribute is equally useful. These are the ones that determine whether you find the fault origin in ten minutes or three hours:

Per-LLM-call spans (gen_ai.client.inference or equivalent)

• gen_ai.request.model — which model version processed this step. Model version drift is a silent cause of behavioral regression, especially after provider rollouts.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — token counts per call. An unexpectedly large input token count often means the context window accumulated garbage. A sudden spike is your loop detection signal.
• gen_ai.response.finish_reasons — why the model stopped. tool_calls means the model handed off. stop means it gave a final answer. length means the output was truncated — a common cause of malformed tool arguments.
• gen_ai.system_instructions / full message capture — opt-in, sensitive, but essential. Without capturing what was actually in the context window at the fault step, backward tracing is guesswork.

Per-tool-call spans

• gen_ai.operation.name: tool_call — labels this as a tool execution span
• Tool name, input arguments, status code, and response payload — these are your audit trail for blast radius assessment

Custom fields to add on every span

• session_id — links all spans from one agent run into a single forensic unit
• step_number — without this, backward tracing requires time-sorting across potentially out-of-order events
• anomaly_flags — structured array, not free text. Free text is not searchable evidence.

A complex agent run produces a trace hundreds of events long. Reading sequentially from step one is the wrong move — you'll spend most of your time on steps that were fine. The correct method is forensic: start from the known bad output and work backwards.

The backward trace method

Take the final wrong output — a hallucinated order confirmation, a malformed API call, an incorrectly processed payment — and ask: what would have to be true for this output to make sense? Then find the step where that information first appeared. That is the fault origin.

In the example trace above, the wrong output is a confirmation linking cust_789 to ord_555. Working backward: the agent called get_order with ord_555 at step 4. Where did ord_555 come from? It appeared in the LLM output at step 3 — an order ID that was never in the conversation context before that point. Entity hallucination. Step 3 is the fault origin.

Anomaly flags as accelerants

If your tracing infrastructure captures structured anomaly flags, the backward trace collapses from forty events to two or three. Useful flag types to instrument ^[3]:

• ENTITY_HALLUCINATION — the model referenced an entity ID not found in prior context
• TOOL_ARG_MISMATCH — a tool was called with arguments that violate its schema
• CONTEXT_DRIFT — the agent's stated goal changed between turns without user instruction
• LOOP_DETECTED — the same tool was called with near-identical arguments inside one session
• ENTITY_MISMATCH — a tool returned data belonging to a different entity than the one being processed
• OUTPUT_TRUNCATED — finish_reason: length on a tool-call step; the model's arguments were cut mid-JSON

Reconstruct the context, not just the output

LLM call summaries tell you what the model said. They don't tell you what it believed. At the fault origin step, rebuild the full prompt — system message, conversation history, tool results, working memory — and ask: given exactly this context, is this model output surprising? Sometimes the answer is yes and you've found a model limitation. More often the answer is no: given what was in that context window, the output was entirely predictable. Which means the fix is upstream — in the context, not the model.

This is the most common finding in agent postmortems that get honest: the model did exactly what the context implied it should do. The fault is in what made it into the context.

Standard postmortem templates ask deterministic questions. What changed in the deployment? What was the root cause line of code? How did the change ship? Agent postmortems need different questions because the failure is rarely in the deployment. It's in the combination of model behavior, prompt design, tool interfaces, and the specific data that appeared in the session.

We ran our first three agent postmortems on a standard SRE template. All three came out inconclusive. The template demanded "the root cause" — a single line, a single change — and every agent failure we investigated had three or four contributing factors that were each insufficient alone. Renaming the field "contributing factors" instead of "root cause" changed the remediation conversations immediately. Postmortems that end in "add a unit test" produce different follow-up than ones that end in "redesign the entity validation layer."

A useful agent postmortem answers six things:

1. Session context — What was the session trying to do? Provide session ID, time range, total LLM calls, total tokens consumed, plain-language description of intended behavior. Every other finding grounds in this.

2. Failure classification — Which of the four modes applies? Not a bureaucratic label — it determines which prevention layer was missing and what has to change.

3. Fault origin and cascade path — Which step was the fault origin? What was in the context window at that step? Trace the cascade from origin to final wrong output. Specific step numbers, no hand-waving.

4. Impact assessment — What external actions did the agent take after the fault origin? Reversible or not? User-facing impact? Cost impact?

5. Contributing factors — Model limitation (the base model hallucinates reliably on this input type)? Prompt design flaw (the system prompt didn't constrain entity references)? Tool interface issue (the tool returned ambiguous data)? Data contamination (a retrieved document carried misleading content)? Name the forces clearly.

6. Detection gap — Why did this reach the user? No eval test for the failure pattern? Insufficient monitoring? Blast radius larger than expected because no guardrails? The detection gap is the most important finding. It drives all prevention work. Everything else is description.

Prevention architecture falls into two tiers. The first tier is visibility — making failures findable. The second tier is containment — limiting blast radius when a failure does occur. Teams that skip straight to containment without visibility end up with circuit breakers that trip for the wrong reasons and no evidence to diagnose why.

Start with visibility. Then contain.

The agent postmortem that ends with "the model hallucinated" is not a postmortem — it's a shrug. Every agent model hallucinates. The question is which structural condition let that hallucination propagate through 33 more reasoning steps and reach the user. That condition is in your observability stack, your entity validation layer, your eval harness, or your containment design. Name it specifically, fix it structurally, and add it to the failure library. The next incident will test whether you actually did.