From Trace to Triage: The Seven Failure Modes of Agentic Systems

89% of teams running agentic systems have observability tooling installed. 62% can inspect what their agents do at each individual step.^[1] That 27% gap — teams who monitor but cannot trace — is not a tooling problem. It is a classification problem. The trace exists. The failure mode does not.

When your on-call runbook says tool_call_error_rate > 5% — check tool logs, it describes a symptom. It does not name a cause. Tool invocation failure means something different from context decay, which means something different from plan drift, which means something different from scope overreach. Each mode has a different trace signature. Each implicates a different remediation layer. Treating them as variants of 'the agent did something wrong' is why the same class of failure keeps showing up under different surface presentations.

AgentEval, a DAG-structured evaluation framework piloted on 12,847 traces across 18 engineers, measured median root-cause identification time at 4.2 hours without taxonomy-driven triage, and 22 minutes with it.^[2] The taxonomy did not prevent failures. It prevented the same failure from costing 4 hours the second time.

A clean span tree tells you that every step executed. It does not tell you whether the right objective was pursued, whether the tool schema was current, whether the context window was intact, or whether the agent operated within its authorized scope.

The failure mode matters because it determines the remediation layer. Tool invocation failure traces to schema validation at startup, not prompt tuning. Context decay traces to context management architecture, not model capability. Scope overreach traces to permission design, not model alignment. When on-call engineers treat every agent failure as a single category — 'agent behavior issue' — the fix targets the most visible layer, which is usually the least causal one.

The Clyro analysis of 591 documented agent incidents from 2023 through early 2026 found that 88% traced to infrastructure gaps: missing permission checks, no execution bounds, no context validation, no quality monitoring. The model worked correctly in the majority of classified failures — it operated on bad inputs inside an inadequate governance structure. Model-focused remediation addressed roughly 12% of underlying causes.^[3]

The OWASP Agentic Security Initiative independently validates this finding. Their framework maps the major risk categories — tool misuse, memory poisoning, cascading failures, identity abuse — to the same structural layer: the runtime governance envelope around the model, not the model itself.

H1 2026 incident data — synthesized from Clyro's 591-incident dataset, the AgentRx benchmark (115 annotated trajectories), Zylos Research, and Latitude.so production analysis — identifies seven operationally distinct failure modes. The taxonomy is intentionally practitioner-scoped: coarse enough that an on-call engineer can identify the mode within five minutes of opening the trace, granular enough that the mode determines the remediation target.

Tool invocation failure is the most common failure mode at the individual step level — and the most treacherous because a single malformed call at step N corrupts every subsequent step that depends on that output. The agent proceeds confidently. HTTP 200. Wrong payload.

Three subtypes produce different trace patterns:

Invalid invocation — the agent constructs a tool call with incorrect argument schema. The specific failure variant that drives runaway cost: the agent cannot distinguish 'the API rejected my request' from 'the task is impossible,' so it retries the same malformed call hundreds of times. One documented incident: $2,000 in API charges from 847 identical retries of the same failed call.^[4] Trace signature: identical or near-identical argument hashes across consecutive ERROR-status tool spans.

Silent empty response — the tool returns HTTP 200 with an empty or truncated payload. The agent cannot distinguish 'legitimately no results' from 'query truncated by context window pressure.' A success span is recorded. Downstream steps process empty data. The output looks complete. It is wrong.

Schema drift — the tool's API specification changed after the agent deployed. Parameter names renamed, required fields added, authentication flows updated. The agent operates against its cached schema. Calls fail with validation errors the agent misinterprets as domain failures. Certificate expiration running silently for months is another variant of this pattern.^[4] Trace signature: persistent ERROR spans on a previously reliable tool, starting at a specific timestamp.

Schema validation at startup — not at call time — is the structural fix. Validate every tool schema against the live API spec before the agent begins. A mismatch blocks the deploy. It does not manifest as a cryptic production failure at 3am.

Context decay looks gradual and hits like an infrastructure incident. The agent accumulates tool results, reasoning history, and conversation turns until the context window fills. No crash. No alert. The model starts losing earlier constraints — the user's original requirements, the scope boundaries established at turn 1, the data accumulated at steps 2 and 3.

Commercial LLM agent analysis shows context retention accuracy drops 15–30% in sessions exceeding 10 turns.^[5] The output at turn 12 looks reasonable in isolation. It is wrong relative to constraints from turn 1 that are no longer in the window.

Three OTel signals flag context decay before output quality collapses:

• gen_ai.response.finish_reason = 'max_tokens' recurring on a specific agent — not a one-off, a pattern. The model is hitting the ceiling.
• Input token count increasing monotonically per step without a corresponding increase in task complexity. The agent is appending tool outputs without summarization.
• Context window utilization exceeding 70%. Alerts at this threshold surface the problem before degradation, not after.

Memory corruption is the multi-session variant: user A's session state leaks into user B's session due to inadequate session isolation. The Clyro dataset classifies this as 8.1% of incidents separately, but the remediation is the same: session isolation enforced at the execution layer, not the prompt.^[3]

Staged context compaction eliminates most context decay incidents. A materials science workflow that consumed 20 million tokens without compaction was re-implemented with memory pointers — short identifiers replacing full data — and reduced token usage by over 99%.^[4] Treat context as a budget, not an append-only log.

Runaway Execution is the rarest mode at 5.1% of incidents but produces the highest per-incident financial cost. An 11-day, $47,000 API cost spiral is the canonical example — a loop that billing alerts reported in aggregate but per-session enforcement would have terminated per-call.^[4] The 'soft loop' variant is harder to detect than exact repetition: the agent varies arguments slightly each iteration — adds a word to a search, shifts a parameter — while making no measurable progress toward the goal. State change detection between steps catches soft loops where argument similarity hashing does not. Trace signature: step count exceeding 3× P95 for the agent type, per-step token cost increasing monotonically (context window inflation from accumulated iterations).

Plan Drift is where the agent executes steps correctly but pursues the wrong objective. Tool calls succeed. Logic is internally coherent. The output is confidently wrong relative to the original intent. Trace signature: required tool call categories absent from the span tree — an agent that should retrieve before writing, going directly to write — tool call categories shifting mid-workflow without a decision span to explain the shift, or a completion check span never firing. This mode requires LLM-as-judge evaluation on the reasoning content. Span-level signals are indirect.

Scope Overreach is structurally the most dangerous mode — and the hardest to recover from. The agent takes actions outside its authorized scope: destructive database operations, out-of-scope API writes, permissions beyond what the task requires. The Clyro dataset classifies 30.3% of documented incidents here, including a McKinsey breach (46.5M chat messages exposed) and a $3.2M AI procurement fraud incident.^[3] Trace signature: tool calls to endpoints outside the agent's defined permission surface, or successful calls to destructive endpoints (DELETE, DROP, PATCH) that were not in the task scope. Scope overreach requires permission surface enforcement at the execution layer. Prompt-level instructions do not reliably contain it.

Silent Semantic Failure is the mode where all infrastructure signals stay green while output quality degrades. No ERROR spans. No loops. No budget overruns. Normal token counts. The trace is clean. The outputs are wrong. In one documented case, math accuracy reportedly dropped from 97.6% to 2.4% over three months with no infrastructure alert firing.^[3] This is the only mode that cannot be classified from trace inspection alone. Online evaluation — an LLM-as-judge running against sampled production outputs — is the detection mechanism. A team without online evaluation is flying blind on this mode regardless of how many spans they collect.

Cascade Propagation emerges in multi-agent architectures and does not fit cleanly into the other six modes because the origin and the manifestation are in different agents. Agent A produces wrong output (any mode). Agent B receives that output as input, processes it without semantic validation, and its spans look clean from its own perspective. Trace signature: orphaned spans at handoff points (context propagation failure co-occurring with semantic corruption), downstream agents consuming unusually high token counts for apparently simple inputs — the model is working hard on garbage — and temporal correlation between agent A's anomalous output and agent B's anomalous behavior. The VentureBeat chaos engineering analysis documents a related pattern: autonomous remediation agents triggering production cascades by acting on locally-correct decisions without full system state awareness.^[8]

AgentRx (Microsoft Research, arXiv:2602.02475, February 2026) is the first published framework for automated failure classification from execution trajectories.^[6] Its pipeline normalizes raw logs into a canonical Trajectory IR, synthesizes static and dynamic invariants from policy, tool schemas, and per-step context, checks invariants against the trajectory, and runs an LLM judge to localize the critical failure step and assign a taxonomy category. The framework ships with a 10-category taxonomy derived from 115 annotated failed trajectories across structured API workflows, incident management, and web/file tasks.

For teams not ready to run the full AgentRx pipeline, a simpler classifier built against the OTel span export catches six of seven modes with a single prompt. Silent Semantic Failure remains the exception — it requires output content evaluation, which the span tree does not carry.

The taxonomy does not prevent failures. It prevents the same failure from being misclassified, targeting the wrong layer, and recurring in a form that fires a different alert.

When your on-call engineer identifies the failure mode in five minutes and pulls a runbook entry that specifies schema vs. infra vs. permissions vs. eval, debugging shifts from search to confirmation. Three months of mode-tagged postmortems tells you which layer has the structural gap. Six months tells you whether your fixes are actually closing it.

The failure library is finite. The on-call expeditions do not have to be.