Agentic Incident Triage: Debug Silent Agent Failures That Return HTTP 200

At 2:14am the loan-eligibility agent returned HTTP 200. No exceptions. No alerts. The income-verification tool logged status: ok. Over the next six hours the agent rejected 847 valid applications because a third-party API had quietly renamed a JSON field overnight. The model received a key it had never seen, hallucinated null for income, and declined every pending case. ^[8]

This is the failure class agentic systems were built to expose. Infrastructure healthy. Dashboard green. The fault is semantic — distributed across a reasoning chain where one wrong interpretation at step 3 shapes every decision 40 calls deep. No exception to read. No line number to fix.

This runbook gives you five things: a failure-mode taxonomy to classify against before reading any trace, the minimum observability stack that catches silent failures before users do, the gen_ai semantic conventions that are now the industry standard for agent instrumentation, a five-step triage sequence that works when the error log is empty, and a postmortem template that doesn't pretend non-determinism is a typo.

Traditional SRE asks what crashed. Agentic SRE asks something harder: which step produced the wrong premise, and how far did it travel before anyone noticed?

The most dangerous agent failures are the ones your monitoring stack calls successes.

Latency histograms, error rates, CPU graphs — every one of those signals was designed for deterministic systems where failure means the process exited nonzero. Agents break that assumption. A clean HTTP 200 from an LLM API is not evidence of correct behavior. Valid JSON from a tool is not evidence the agent interpreted it correctly. A formatted final response is not evidence the response is true. ^[2]

The canonical pattern: a third-party tool changes its response schema. No exception fires. The JSON is still valid. The model receives a field name absent from training and, instead of escalating, generates a plausible value from prior context. That value enters the session state, gets cited in the next three calls, and by the time the user sees the output the entire reasoning chain has been built on a hallucinated foundation. None of it shows up in your Datadog dashboard. ^[8]

In early 2026, 89% of teams running agents in production had implemented some form of agent observability. Only 52% had eval pipelines that actually validated whether the agent's behavior was correct, not just syntactically complete. ^[2] The 37-point gap between "we have traces" and "we catch behavioral failures" is where production incidents live.

The model is rarely the root cause. The dominant failure is a boundary failure — a tool returned partial JSON, retrieval pulled the wrong chunk, a planner looped, an API contract changed without notice. ^[4] Categorically, tool-use and agentic prompt injection incidents were negligible in H2 2025 and are now meaningful shares of the H1 2026 incident dataset. ^[12] If the LLM's reasoning is the actual root cause, your team got unlucky. If the tool interface failed, that's just production.

The intuitive assumption is that a 40-call session distributes failures proportionally across all steps. The evidence contradicts it.

Across 1,200 logged agent runs with verified hallucinations, analysis reported from multiple benchmark evaluations found 71% of errors were introduced in the first two steps — typically during query reformulation or initial retrieval. ^[1] The final output looks like a complex multi-step failure. Trace it backward and the root cause is almost always an error in how the task was understood or how the first retrieval was executed.

This changes the triage path. When staring at a 40-step session, start at steps 1–3. Read what the agent understood the user to be asking. Read what came back from the first retrieval or tool call. Read whether the initial plan was sound. If those steps are clean, the problem is unlikely to be a fundamental reasoning failure — it's more likely a specific tool boundary issue later in the chain.

The TRAIL benchmark from Patronus AI puts a sharper number on the debugging difficulty: 148 human-annotated agent execution traces, 841 total errors, and the best available model (Gemini-2.5-Pro) scored 11% joint accuracy at identifying fault origins. ^[9] Modern long-context LLMs are not yet reliable debuggers of their own trace output. They'll miss most of what you need them to find. That's not an argument against automated triage — it's a specific reason automated triage needs structured anomaly flags rather than asking the model to self-diagnose from raw text.

Hallucination probability scales nonlinearly with tool call count. Across agentic benchmark evaluations, the probability of at least one hallucination in a run climbs from roughly 12% at 2 tool calls, to 67% at 10, to above 85% at 15 or more. ^[1] This is not an argument against complex agents. It's the structural reason any agentic workflow requiring 10+ tool calls is production-critical infrastructure and needs dedicated semantic observability — not standard LLM monitoring with extra fields.

Latency, token count, API errors — necessary for any production LLM workload, insufficient for agentic incident response. The minimum stack captures four things generic LLM monitoring doesn't.

Session-level correlation. Every LLM call, every tool invocation, every state mutation must carry the same session_id. Without it, a 40-step session decomposes into 40 disconnected events in your log store. The session ID is the single most important field you can add — and the one most commonly missing from first-generation agent deployments.

Tool call payloads at every step. Not "tool was called with status 200" — the actual input arguments and the actual response payload, including the field structure. Schema drift detection — spotting when a tool response has different field names than expected — is one of the highest-signal automated checks available. The AgentTelemetry benchmark found that vanilla OpenTelemetry misses 57% of agent faults (FDR 0.429) while a comprehensive span taxonomy covering agent-specific spans achieves a fault detection rate of 1.000 across 14 fault types. ^[10] The gap comes from exactly the missing tool payload and planning-layer spans.

Step-level anomaly flags. Structured fields that mark specific failure patterns the moment they occur: ENTITY_HALLUCINATION when the model references an entity ID not in prior context, TOOL_SCHEMA_DRIFT when a tool response doesn't match its documented schema, LOOP_DETECTED when the same tool fires with near-identical arguments three or more times. ^[6] These flags don't catch everything. They reduce the backward trace from "read 40 events manually" to "check the flagged steps first."

Session outcome classification. Every session ends in one labeled outcome from a small set: Completed, Tool Error, Bad Output, Timeout, Budget Exceeded. The label enables cohort queries — "show me every Bad Output session from the last week" — that single-trace inspection can't answer. ^[4]

For platform selection: the six platforms anchoring the 2026 space are LangSmith (deepest LangChain integration), Langfuse (open-source leader, self-hostable, acquired by Clickhouse Jan 2026), Arize Phoenix (ML-grade drift detection and embeddings analysis, OpenTelemetry-native via OpenInference), Helicone (drop-in proxy, simplest install), Datadog LLM Observability (enterprise default for Datadog shops), and Honeycomb LLM Observability (event-based deep tracing). ^[3] All support the OTel gen_ai semantic conventions. Pick the one that fits your data residency and cost model, not the one with the best marketing.

OpenTelemetry's GenAI semantic conventions reached broad adoption in 2025-2026, with LangChain, CrewAI, AutoGen, AG2, and LlamaIndex all emitting OTel-compliant spans either natively or via thin instrumentation packages. ^[11] The conventions define four span categories: LLM client spans, agent spans, tool/function execution spans, and events (for capturing prompt and completion content).

The practical benefit for incident response: if every agent, tool, and LLM call in your system emits standardized attributes, any OTel-compatible backend — Jaeger, Datadog, Honeycomb, Arize Phoenix — can reconstruct the full session trace without custom parsers. Agent spans carry gen_ai.agent.id, gen_ai.agent.name, and gen_ai.conversation.id. LLM call spans carry gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. Tool execution spans wrap each tool call in a child span with its input arguments and response payload. ^[11]

The gap the conventions don't yet cover: five critical agent orchestration phases — planning, reasoning, safety monitoring, inter-agent delegation, and memory management — still lack standardized span-level representation. ^[10] Until that gap closes, you need custom attributes for those phases. The AgentTelemetry benchmark suggests those custom spans are the difference between catching 43% and 100% of fault types. They're not optional for production workflows.

Before reading any trace in detail, classify the failure. Classification determines which 20% of the trace you actually need to examine. Getting this wrong costs you the first 30 minutes of every agent incident.

The decision tree below encodes the four operationally distinct failure modes. Not because the taxonomy is academically satisfying — because each one has a different blast radius and a different first action. A cost runaway needs a session kill order before forensics begins. A tool misfire needs a downstream system audit before forensics begins. Starting forensics without classification means an hour spent debugging the wrong thing while the agent continues to affect production.

The first time we ran an agent postmortem on a standard SRE template, the conclusion read: "root cause: LLM hallucination." That's about as useful as writing "root cause: gravity" for a structural failure.

Standard templates ask for a single root cause — one line, one changeset, one decision that went wrong. Agent failures don't cooperate with that framing. Every significant agent incident had at least three contributing factors, each insufficient on its own: a tool response schema changed without notice, a prompt that didn't constrain entity references, an eval suite with no coverage for this failure class. Fixing any one of them in isolation wouldn't have prevented the incident. The gravitational pull toward a singular root cause mislabels the problem and produces narrow fixes that block the exact failure without addressing the class.

The agent postmortem template replaces "root cause" with "contributing factors" and refuses to close the incident until all of them are named.

1. Session context — Session ID. Time range. Total LLM calls. Total tokens. A plain-language description of intended session behavior. This grounds every finding in concrete evidence rather than generalized claims about model behavior.

2. Failure classification — Which of the four failure modes applies. This determines which prevention layers were absent and which architectural change is actually needed.

3. Fault origin and cascade path — The specific step number that introduced the wrong premise. The full context window at that step. The cascade path from fault origin to final output, with specific step numbers — not "around step 10" but "step 7."

4. Impact assessment — Every external action taken after the fault origin, classified reversible or irreversible, with a concrete remediation plan for each irreversible one.

5. Contributing factors — Name all three or four. Model limitation? Prompt design gap? Tool interface contract change? Retrieval contamination? List each explicitly. Refuse to collapse them into a single cause.

6. Detection gap — Why didn't observability catch this before users were affected? Missing anomaly flag? No eval coverage for this failure pattern? No guard rail on the tool call? The detection gap is the most important finding in the entire postmortem. It's the only one that drives an infrastructure change rather than a one-off prompt edit.

Most teams running agents in production are somewhere in the middle — they have some logging, they don't have session correlation, they have no anomaly flags, and they've debugged one or two incidents by staring at logs for three hours. This is the priority sequence.

Week 1: Session correlation and tool payloads. Add gen_ai.conversation.id (or your own session_id) to every LLM call and every tool call. Log the actual tool input arguments and response field names at each step. These two changes alone cut mean debug time by more than half on the next incident. Nothing else matters until these are in place.

Week 2: Outcome classification and TOOLSCHEMADRIFT. Add the five outcome labels to your session lifecycle: Completed, Tool Error, Bad Output, Timeout, Budget Exceeded. Wire TOOL_SCHEMA_DRIFT detection to your tool wrappers — hash the expected field set per tool, compare on every call. This flag catches the most common production failure class (API contract change) at the boundary rather than hours later in user reports.

Week 3: ENTITYHALLUCINATION, BUDGETWARNING, and LOOP_DETECTED. Add entity-ID format detection to your LLM call wrappers. Set a soft budget warning at 60% of session budget. Add loop detection at the two near-identical-call threshold. These three flags cover hallucination propagation, cost runaway, and the most obvious form of context poisoning.

Month 2: Eval harness and CONTEXTDRIFT. Build a regression eval that replays historical session traces and asserts against expected outcomes. Add CONTEXTDRIFT detection at session mid-points using embedding comparison. File the first postmortem using the six-section template rather than the standard RCA format.

You don't need all of this before shipping your first agent to production. You need session correlation and tool payloads. Everything else is a force multiplier on that foundation.