Agentic System Failure Modes: 7 Trace Signatures On-Call Teams Miss

89% of teams running agentic systems have observability tooling installed. 62% can inspect what their agents do at each individual step.^[1] That 27-point gap — teams who monitor but can't map traces to causes — isn't a tooling problem. It's a classification problem. The trace exists. The failure mode does not.

When your on-call runbook says tool_call_error_rate > 5% — check tool logs, it describes a symptom. It doesn't name a cause. Tool invocation failure means something different from context decay, which means something different from plan drift, which means something different from scope overreach. Each mode has a different trace signature. Each implicates a different remediation layer. Treating them all as variants of 'the agent did something wrong' is why the same class of failure keeps showing up under different surface presentations.

AgentEval, a DAG-structured evaluation framework piloted on 12,847 traces across 18 engineers, measured median root-cause identification time at 4.2 hours without taxonomy-driven triage, and 22 minutes with it.^[2] The taxonomy didn't prevent failures. It prevented the same failure from costing 4 hours the second time.

A clean span tree tells you that every step executed. It doesn't tell you whether the right objective was pursued, whether the tool schema was current, whether the context window was intact, or whether the agent operated within its authorized scope.

The failure mode matters because it determines the remediation layer. Tool invocation failure traces to schema validation at startup, not prompt tuning. Context decay traces to context management architecture, not model capability. Scope overreach traces to permission design, not model alignment. When on-call engineers treat every agent failure as a single category — 'agent behavior issue' — the fix targets the most visible layer, which is usually the least causal one.

The Clyro analysis of 591 documented agent incidents from 2023 through early 2026 found that 88% traced to infrastructure gaps: missing permission checks, no execution bounds, no context validation, no quality monitoring. The model worked correctly in the majority of classified failures — it operated on bad inputs inside an inadequate governance structure. Model-focused remediation addressed roughly 12% of underlying causes.^[3]

AgentFixer (IBM Research, arXiv:2603.29848, 2026) applied fifteen failure-detection tools and two root-cause analysis modules across input handling, prompt design, and output generation. Applied to IBM's CUGA agent on the AppWorld and WebArena benchmarks, it surfaced systemic weaknesses — controller indexing errors, planner misalignments, schema non-compliance — that neither standard logging nor manual review had caught.^[11] The lesson: failure visibility requires structured detection, not more log volume.

H1 2026 incident data — synthesized from Clyro's 591-incident dataset, the AgentRx benchmark (115 annotated trajectories), Zylos Research, and Latitude.so production analysis — identifies seven operationally distinct failure modes. The taxonomy is intentionally practitioner-scoped: coarse enough that an on-call engineer can identify the mode within five minutes of opening the trace, granular enough that the mode determines the remediation target.

AgentRx's research taxonomy runs to nine categories.^[6] For production triage, seven is the right level of granularity — mode 6 (Plan Adherence Failure) and mode 5 (Intent–Plan Misalignment) collapse into Plan Drift because the remediation layer is the same: behavioral spec and eval gate at deploy. The distinction matters for research; it doesn't change the on-call action.

Tool invocation failure is the most common failure mode at the individual step level — and the most treacherous because a single malformed call at step N corrupts every subsequent step that depends on that output. The agent proceeds confidently. HTTP 200. Wrong payload.

AgentRx maps this to two of its nine categories: Invalid Invocation (tool call malformed, missing args, schema-invalid) and Misinterpretation of Tool Output (agent acted on incorrect assumptions from the response).^[9] Three subtypes produce different trace patterns in practice:

Invalid invocation — the agent constructs a tool call with incorrect argument schema. The dangerous variant: the agent can't distinguish 'the API rejected my request' from 'the task is impossible,' so it retries the same malformed call hundreds of times. One documented incident: $2,000 in API charges from 847 identical retries of the same failed call.^[4] Trace signature: identical or near-identical argument hashes across consecutive ERROR-status tool spans — argument similarity above 0.8 on the gen_ai.tool.call.arguments attribute across three or more consecutive calls is a reliable signal.

Silent empty response — the tool returns HTTP 200 with an empty or truncated payload. The agent can't distinguish 'legitimately no results' from 'query truncated by context window pressure.' A success span is recorded. Downstream steps process empty data. The output looks complete. It's wrong.

Schema drift — the tool's API spec changed after the agent deployed. Parameter names renamed, required fields added, authentication flows updated. The agent operates against its cached schema. Certificate expiration running silently for months is another variant.^[4] Trace signature: persistent ERROR spans on a previously reliable tool starting at a specific timestamp — the diff in error rate before and after that timestamp is the schema change event.

The OpenTelemetry GenAI semantic conventions (experimental as of mid-2026) define execute_tool as a child span under invoke_agent, with gen_ai.tool.name and gen_ai.tool.call.id as the key attributes.^[12] Any alerting rule on tool ERROR rate should group by gen_ai.tool.name first — a spike on a single tool name is schema drift; a spike across all tools is an execution environment problem.

Context decay looks gradual and hits like an infrastructure incident. The agent accumulates tool results, reasoning history, and conversation turns until the context window fills. No crash. No alert. The model starts losing earlier constraints — the user's original requirements, scope boundaries established at turn 1, data accumulated at steps 2 and 3.

Chroma's 2025 research, testing 18 frontier LLMs on multi-hop reasoning tasks across 10,000–500,000 token contexts, found that all 18 models showed monotonically decreasing F1 scores as input length grew, with the steepest degradation in the 100,000–500,000 token range.^[10] The 'lost-in-the-middle' effect causes 30%+ accuracy drops for content positioned away from context boundaries. In a 12-turn agent session, turn 1 constraints are structurally disadvantaged by turn 12.

A complementary finding from a 2026 study on 4,416 trials: omission constraints (things the agent should not do) decay faster than commission constraints (things it should do) as context grows. Hard limits buried in conversation history don't hold.^[10]

Three OTel signals flag context decay before output quality collapses:

• gen_ai.response.finish_reasons containing max_tokens recurring on a specific agent — not a one-off, a pattern across multiple sessions. The model is hitting the ceiling.
• Input token count (gen_ai.usage.input_tokens) increasing monotonically per step without a corresponding increase in task complexity. The agent is appending tool outputs without summarization.
• Context window utilization exceeding 70%. Alert at this threshold — it surfaces the problem before degradation, not after.

Memory corruption is the multi-session variant: user A's session state leaks into user B's session due to inadequate session isolation. The Clyro dataset classifies this as 8.1% of incidents separately, but the remediation is the same: session isolation enforced at the execution layer, not the prompt.^[3]

Staged context compaction eliminates most context decay incidents. One materials science workflow that consumed 20 million tokens without compaction was re-implemented with memory pointers — short identifiers replacing full data — and reduced token usage by over 99%.^[4] Treat context as a budget with a hard ceiling, not an append-only log.

Runaway Execution is the rarest mode at 5.1% of incidents but produces the highest per-incident financial cost. An 11-day, $47,000 API cost spiral is the canonical example — a loop that billing alerts reported in aggregate but per-session enforcement would have terminated per-call.^[4] The 'soft loop' variant is harder to detect than exact repetition: the agent varies arguments slightly each iteration — adds a word to a search, shifts a parameter — while making no measurable progress toward the goal. State change detection between steps catches soft loops where argument similarity hashing doesn't.

Circuit breakers are the structural fix. Production implementations trip on four conditions: cost velocity (spend rate exceeding the per-session ceiling), repeated prompts (argument hash similarity above threshold across N consecutive calls), error rate (ERROR spans exceeding 30% of calls in a window), and growing context (token count increasing faster than a configurable rate per step). A lightweight progress-check node running every 3 steps — asking the model whether measurable progress toward the goal occurred — catches soft loops before cost accumulates.

Plan Drift is where the agent executes steps correctly but pursues the wrong objective. Tool calls succeed. Logic is internally coherent. The output is confidently wrong relative to the original intent. AgentRx categorizes this under Plan Adherence Failure and Intent–Plan Misalignment — the distinction is whether the agent departed from a correct plan or started with a wrong one.^[9] Trace signature: required tool call categories absent from the span tree, tool categories shifting mid-workflow without a decision span to explain the shift, or a completion check span never firing. This mode requires LLM-as-judge evaluation on the reasoning content. Span-level signals are indirect. The eval gate at deploy time — running behavioral spec checks before the agent reaches production — is the right intervention point.

Scope Overreach is structurally the most dangerous mode — and the hardest to recover from. The agent takes actions outside its authorized scope: destructive database operations, out-of-scope API writes, permissions beyond what the task requires. The Clyro dataset classifies 30.3% of documented incidents here.^[3] Trace signature: tool calls to endpoints outside the agent's defined permission surface, or successful calls to destructive endpoints (DELETE, DROP, PATCH) not in the task scope. Scope overreach requires permission surface enforcement at the execution layer. Prompt-level instructions don't reliably contain it across 10+ turns. Define the authorized tool endpoints and resource scopes for each agent role at initialization and check every tool span's target against it before execution, not after.

Silent Semantic Failure is the mode where all infrastructure signals stay green while output quality degrades. No ERROR spans. No loops. No budget overruns. Normal token counts. The trace is clean because the infrastructure is healthy. The outputs are wrong because the reasoning is wrong. In one documented case, accuracy dropped sharply over a three-month period with no infrastructure alert firing.^[3] This is the only mode not classifiable from trace inspection alone. Online evaluation — an LLM-as-judge running against sampled production outputs — is the detection mechanism. Without it, you're blind to 24.9% of documented incidents.

Cascade Propagation emerges in multi-agent architectures and doesn't fit cleanly into the other six modes because the origin and the manifestation are in different agents. Agent A produces wrong output (any mode). Agent B receives that output as input, processes it without semantic validation, and its spans look clean from its own perspective. Trace signature: orphaned spans at handoff points — context propagation failure co-occurring with semantic corruption — and downstream agents consuming unusually high token counts for apparently simple inputs (the model is working hard on garbage). Temporal correlation between agent A's anomalous output and agent B's anomalous behavior is the key diagnostic signal. The VentureBeat chaos engineering analysis documents a related pattern: autonomous remediation agents triggering production cascades by acting on locally-correct decisions without full system state awareness.^[8]

The OpenTelemetry GenAI semantic conventions (Development status, mid-2026)^[12] define a span hierarchy for agent execution: a top-level invoke_agent span with child chat spans for each LLM call and execute_tool spans for each tool invocation. The attributes that distinguish failure modes are specific and measurable — not all of them are in the spec yet, but the core ones are stable enough to build alerting rules on.

The attributes below are those confirmed in the OTel GenAI spec or derived directly from span data, not invented for this taxonomy:

AgentRx (Microsoft Research, arXiv:2602.02475, February 2026) is the first published framework for automated failure classification from execution trajectories.^[6] Its pipeline normalizes raw logs into a canonical Trajectory IR, synthesizes static and dynamic invariants from policy, tool schemas, and per-step context, checks invariants against the trajectory, and runs an LLM judge to localize the critical failure step and assign a taxonomy category. Evaluated on 115 annotated trajectories, it achieves +23.6% improvement in failure localization and +22.9% improvement in root-cause attribution over prompting baselines.^[9]

AgentFixer (IBM Research, 2026) takes a complementary approach: fifteen rule-based and LLM-as-judge detection tools that surface weaknesses before incidents, rather than classifying failures after the fact.^[11]

For teams not ready to run the full AgentRx pipeline, a simpler classifier built against the OTel span export catches six of seven modes with a single prompt. Silent Semantic Failure remains the exception — it requires output content evaluation, which the span tree doesn't carry.

This taxonomy works for the common case: single-tenant or multi-tenant agentic systems where tools have defined schemas, execution is bounded, and outputs have a quality standard that can be evaluated. It works well enough to replace symptom-level on-call runbooks in the first week.

Three situations where it needs extension:

Heavily domain-specific tool failure modes. In financial services or healthcare workflows, what looks like 'Plan Drift' at the span level may actually be regulatory non-compliance — a distinct failure category with different remediation and a very different incident severity. Add a subtype under Plan Drift for your domain, don't invent a new top-level mode. Taxonomy proliferation kills frequency analysis.

Untooled or black-box agents. If your agent doesn't emit execute_tool spans — because it's a legacy system or a vendor black-box — the classifier is operating on incomplete evidence. Instrument tool boundaries first. Until you can distinguish tool failure from reasoning failure at the span level, all classification results should carry reduced confidence.

True multi-agent orchestrator patterns. When orchestrator and subagents run in different processes or services, context propagation requires explicit trace header passing. Cascade Propagation is both a failure mode and a diagnostic tool for propagation gaps — if you're seeing frequent 'inconclusive' results from the classifier on multi-agent sessions, fix context propagation first. Most inconclusive results in practice trace to orphaned spans that make the failure mechanism invisible.^[7]

The taxonomy doesn't prevent failures. It prevents the same failure from being misclassified, targeting the wrong layer, and recurring in a form that fires a different alert.

When your on-call engineer identifies the failure mode in five minutes and pulls a runbook entry that specifies schema vs. infra vs. permissions vs. eval, debugging shifts from search to confirmation. Three months of mode-tagged postmortems shows which layer has the structural gap. Six months shows whether your fixes are actually closing it.

The failure library is finite. The on-call expeditions don't have to be.