The Agent Incident Playbook: Debugging a Failure Across 40 LLM Calls

2:37am. The order agent returns a confirmation for an order that does not exist. The on-call pulls logs and finds nothing. No stack trace. No 500. No exception. HTTP 200, well-formatted prose, full confidence — confirming the wrong order against the wrong customer account.

The failure is not in your code. It is distributed across 40 LLM calls, where the actual fault happened around step 7 — a small hallucination about a customer identifier — and every subsequent call built confidently on top of the poisoned premise. By the time the user saw the output, that error had been laundered through 33 more reasoning steps.

Traditional SRE asks one question: what crashed? Agentic systems demand a different one: which of these 40 reasoning steps produced the wrong context that cascaded into the wrong action? That is not debugging. That is distributed session forensics.

This playbook maps the four operational failure modes to concrete triage steps, gives you a backward-trace method for locating fault origins inside long sessions, and provides a postmortem template built for systems where non-determinism is the default state, not the anomaly.

Traditional runbooks assume systems that fail loudly. A service crashes. A timeout fires. A null pointer propagates. You find the exception, walk the call stack, identify the line. The bug has a location.

Agent incidents violate every assumption in that model. The agent did not crash. It ran to completion and returned a result. The result was wrong, and nothing in your observability stack knows that. Error rates look normal. Latency is fine. The SLO dashboard is green. The user is filing a ticket.

The failure is closer to a reasoning defect than a code defect. Reasoning defects compound. A wrong assumption at step 7 shapes the framing at step 8, which selects the wrong tool at step 9, which returns data that reinforces the wrong assumption at step 10. By step 20, the agent has constructed a coherent internal narrative that is entirely wrong — and it has the tool call logs to prove it.

The agentic AI fault taxonomy literature ^[1] catalogs many failure patterns. For incident response, they collapse into four operationally distinct modes — each with a different signal, a different blast radius, a different triage path.

1. Hallucination propagation

The model generates a false assertion early in the session. Because agents accumulate context across calls, that assertion gets referenced and re-affirmed in later steps. By call 20, the hallucination is an established fact inside the session. The model is not wrong because it is confused. It is wrong because its own earlier output is now evidence.

Signature: confident, coherent, structured output, internally consistent. Built on a false premise.

2. Tool misfire

The model selects the wrong tool, passes malformed arguments, or misinterprets tool output. Unlike hallucination, tool misfire produces real side effects immediately — deleted records, sent emails, processed payments, triggered workflows. The session looks healthy on latency and token counts while causing irreversible damage downstream.

Signature: a tool was called with an unexpected argument pattern, or a tool returned data the model processed without validating against the session's stated goal.

3. Context poisoning

A hallucination or injected content makes it into persistent context — goal state, working memory, retrieved documents. The agent's framing of the entire task warps. Long-running agents are especially exposed because they carry context across many turns and the poisoning compounds ^[5].

Context poisoning differs from hallucination propagation in what it corrupts. Hallucination poisons factual claims about the world. Poisoning corrupts the agent's self-model — what it thinks it is trying to do.

4. Cost runaway

The agent loops: repeated tool calls with similar arguments, infinite retry logic, self-spawning subagents, circular reasoning chains. There is no wrong output in the traditional sense — the agent may never surface a result. The failure is financial and operational. Token consumption compounds silently until a budget alarm fires or the session times out ^[6].

Signature: per-session token counts spike well above baseline. Tool call frequency is abnormally high. The same tool gets called multiple times with near-identical arguments.

A complex agent run produces a trace hundreds of events long. Reading sequentially from step one is the wrong move. You will spend most of your time on steps that were fine. The correct method is forensic: start from the known bad output and work backwards.

The backward trace method

Take the final wrong output — a hallucinated order confirmation, a malformed API call, an incorrectly processed payment — and ask: what would have to be true for this output to make sense? Then find the step where that information first appeared. That is the fault origin.

In the example trace above, the wrong output is a confirmation linking cust_789 to ord_555. Working backward: the agent called get_order with ord_555 at step 4. Where did ord_555 come from? It appeared in the LLM output at step 3 — an order ID that was never in the conversation context before that point. Entity hallucination. Step 3 is the fault origin.

Anomaly flags as accelerants

If your tracing infrastructure supports it, attach automated anomaly flags to trace events ^[3]. Useful flag types:

• ENTITY_HALLUCINATION — the model referenced an entity ID not found in prior context
• TOOL_ARG_MISMATCH — a tool was called with arguments that violate its schema
• CONTEXT_DRIFT — the agent's stated goal changed between turns without user instruction
• LOOP_DETECTED — the same tool was called with near-identical arguments inside one session
• ENTITY_MISMATCH — a tool returned data belonging to a different entity than the one being processed

These flags do not catch every failure. They dramatically accelerate the backward trace. In the example above, the ENTITY_HALLUCINATION flag at step 3 points directly to the fault origin without manually comparing 40 events.

Reconstruct the context, not just the output

LLM call summaries tell you what the model said. They do not tell you what it believed. Examine what was actually in the context window at the fault origin step. For agents using structured working memory or tool-call history as context, the question is: what did the model believe to be true when it made the wrong inference? Rebuild the full prompt — system message, conversation history, tool results, working memory — at that exact step. The fault origin is visible in the context, not just the output.

Standard postmortem templates ask deterministic questions. What changed in the deployment? What was the root cause line of code? How did the change ship? Agent postmortems need different questions because the failure is rarely in the deployment. It is in the combination of model behavior, prompt design, tool interfaces, and the specific data that appeared in the session.

We ran our first three agent postmortems on a standard SRE template. All three came out inconclusive. The template demanded "the root cause" — a single line, a single change — and every agent failure we investigated had three or four contributing factors that were each insufficient alone. We renamed the field "contributing factors" instead of "root cause" and the remediation conversations changed shape immediately. Postmortems that end in "add a unit test" produce different follow-up than ones that end in "redesign the entity validation layer."

A useful agent postmortem answers six things:

1. Session context — What was the session trying to do? Provide session ID, time range, total LLM calls, total tokens consumed, plain-language description of intended behavior. Every other finding grounds in this.

2. Failure classification — Which of the four modes applies? Not a bureaucratic label. It determines which prevention layer was missing and what has to change.

3. Fault origin and cascade path — Which step was the fault origin? What was in the context window at that step? Trace the cascade from origin to final wrong output. Specific step numbers, no hand-waving.

4. Impact assessment — What external actions did the agent take after the fault origin? Reversible or not? User-facing impact? Cost impact?

5. Root cause — Model limitation (the base model hallucinates reliably on this input type)? Prompt design flaw (the system prompt did not constrain entity references)? Tool interface issue (the tool returned ambiguous data)? Data contamination (a retrieved document carried misleading content)? One root cause, clearly named.

6. Detection gap — Why did this reach the user? No eval test for the failure pattern? Insufficient monitoring? Blast radius larger than expected because no guard rails? The detection gap is the most important finding. It drives the prevention work. Everything else is description.