Three Questions Before You Touch the Prompt

IBM Research applied MAST to 310 real IT automation traces in early 2026 and produced something more operationally useful than failure rates: each model class had a different dominant failure pattern, and each pointed to a different structural intervention.^[4] GPT-oss-120b showed systematic FM-2.6 failures — reasoning mismatches that poisoned the task context over multiple turns and caused total session derailment. The fix was context hygiene and early error detection. Not a prompt rewrite. Kimi-K2 couldn't recognize task completion, generating runaway loops. The fix was a deterministic state machine. Gemini-3-Flash's most common fatal failure was FM-3.3: the agent self-terminated based on its own assessment rather than tool-mediated evidence that the alert was actually cleared. The fix was requiring a hard verification step — an AlertManager clearance or a Kubernetes state change — before marking any run successful.

Three architectures. Three dominant structural failures. None solved by touching the system prompt.

This is the wrong-layer problem. Every agent failure produces visible symptoms that look like model behavior — repeated steps, agent disagreement, degraded output. The team reaches for the nearest control surface. But MAST's 14 failure modes cluster into three structural categories that each require interventions at different stages:^[1] specification failures (FC1, 41.77%) are fixable pre-deploy; coordination failures (FC2, 36.94%) require runtime architecture changes; verification failures (FC3, 21.30%) need an eval layer that infrastructure observability cannot replace. Applying the wrong fix delays the right one by exactly one incident cycle. Enough incident cycles and the pattern becomes invisible — every postmortem says 'model behavior was inconsistent' because nobody classified the structural root cause.

This protocol is for engineering leaders who need to classify a live agent failure in under 20 minutes and route it to the right team. MAST's categories become a speed-ordered triage sequence: fastest check first, ownership built in.

The critical structural insight from MAST isn't the count of 14 failure modes. It's that the three categories map to fundamentally different ownership domains. The wrong-layer problem happens when incident response ignores that mapping.

FC1 — Specification Issues originate before the first agent turn. The agent misunderstood the task, stepped outside its role definition, repeated work already done, lost conversational context, or ran without a termination signal it could evaluate. Every FC1 failure is readable from the spec. The owner is whoever writes and reviews behavioral specifications before agents deploy — the system architect, the platform engineer who owns the agent's system prompt, or the team that governs the deployment review gate. The fix window is pre-deploy, making FC1 the cheapest failure category to close.

FC2 — Inter-Agent Misalignment emerges during execution from agent-to-agent interaction. Agents proceed on wrong assumptions (FM-2.2), reasoning doesn't match action taken (FM-2.6), task objectives drift mid-run without a decision point (FM-2.3). The owner is the platform or orchestration team — responsible for coordination protocol design, inter-agent message schemas, and shared task state. This isn't fixable from the system prompt. It requires architecture changes: clarification gates, explicit handoff validation, reasoning-action logging at each step. The MAST authors are direct: communication protocol improvements are insufficient for FC2 failures — coordination design is required.^[1]

FC3 — Task Verification produces the most dangerous diagnostic signal: everything completes, all infrastructure checks pass, output is wrong. Premature termination (FM-3.1), missing verification (FM-3.2), incorrect verification (FM-3.3) — all three look like successful runs from the infrastructure layer. The owner is the eval or quality team running LLM-as-judge or domain validators against production outputs. Infrastructure observability cannot catch FC3 failures. The eval layer either exists or this entire failure category is invisible.^[1]

Under incident conditions, you need to route to the right fix team fast. The three checks are ordered by time to answer, not by complexity of the failure.

The 30-second check (FC1): Pull up the system prompt and role definition alongside the failed trace. Can you point to a missing constraint, an ambiguous role boundary, or a termination condition the agent couldn't evaluate? If yes — that's FC1. You don't need logs, tool traces, or inter-agent messages. You need the spec and the first point of divergence. FC1 is the largest category at 41.77%^[1] and the cheapest to diagnose.

The 5-minute check (FC2): Go to the inter-agent message log at the step where behavior diverged. Compare stated reasoning to the actual tool call or message at that step. Check whether the agent had a decision point to ask for clarification before it proceeded on an assumption. If individual agent outputs look valid but the coordination produced the wrong result, you're in FC2.

The 20-minute check (FC3): If the first two checks come up clean, verify what the verifier actually checked. Did verification run at all? Did it run at intermediate steps or only at terminal output? Did it test task-level correctness rather than format or schema? FC3 takes longer because confirming absent verification requires understanding what eval coverage was supposed to exist.

Teams that skip the spec check and go directly to logs are doing the 20-minute check on a 30-second problem. Start with the fastest check that can falsify the hypothesis.

The most common wrong fix across all three failure categories is the same: reaching for the system prompt.

For FC1 failures, the prompt already contains an imprecise or absent constraint. More natural language doesn't add missing structure. An agent without an evaluatable termination condition cannot infer one from 'make sure to stop when the task is done.' The missing constraint has to be stated in terms the agent can check against its current state.

For FC2 failures, the wrong fix is more aggressive natural language coordination: 'be sure to ask for clarification when uncertain,' 'always verify with other agents before proceeding.' FM-2.2 requires a conditional in the execution flow — a structural gate that evaluates confidence before proceeding — not a sentence in the system prompt. You can add that instruction and FM-2.2 will recur on the same ambiguous input at the same rate.

For FC3 failures, the wrong fix is infrastructure alerting: circuit breakers that trip on span duration, budget ceiling alerts, tool error monitoring. FC3 failures produce clean infrastructure signals while output quality degrades. A Kubernetes automation agent completing a task and returning incorrect configuration is undetectable by span duration or error rate. Automated attribution frameworks like ErrorProbe are beginning to tackle this at scale by operationalizing MAST classification against live traces^[8] — but the core fix is simpler: an LLM-as-judge or domain validator sampling real production outputs against task-level success criteria.

Fix ordering matters when a failure looks hybrid. FC1 is cheapest. If a trace shows both a spec gap (FC1) and a coordination failure (FC2), fix the spec first — the coordination failure may be a downstream effect of the ambiguous spec. Fixing FC3 before FC1 or FC2 means your eval layer is running over outputs from a broken spec or broken coordination protocol. You'll catch more failures but continue producing them at the same rate.

MAST classification per incident is triage. MAST distribution across 30 incidents is diagnosis.

If FC1 failures cluster — different agents, different tasks, same pattern of spec ambiguity or missing termination conditions — the problem is upstream of the agents. The behavioral spec review process doesn't exist or doesn't check the right properties. The fix is a deployment gate: a pre-deploy checklist that blocks agent shipments without explicit, evaluatable termination conditions and role constraint statements.

FC2 clusters indicate the coordination layer was designed as an afterthought. Inter-agent message schemas, clarification gates, and shared task state are infrastructure decisions, not prompt additions. More than three FM-2.6 or FM-2.2 failures in 30 postmortems almost always means there's no coordination protocol documentation. The fix is redesigning that layer from the protocol up.

FC3 clusters — multiple incidents that passed all infrastructure checks but produced degraded output — mean the eval coverage is measuring format instead of correctness. These are the most dangerous failures because downstream systems act on wrong outputs before engineering knows there's a problem.

IBM Research's ITBench application converted 'open models struggle on SRE tasks' into three concrete per-model architectural prescriptions.^[4] Your production incident history can do the same. Tag 30 postmortems with FC category and failure mode code. Query the distribution. That distribution is the architectural backlog — not a list of individual model failures.

Every incident that exits with 'we adjusted the prompt' — without first running the 30-second FC1 check — is a misdiagnosis on record. The structural gap that caused the incident is still open. The next incident in the same FC category is already scheduled.

MAST doesn't require an ML research background to apply. It requires three questions, in speed order, before anyone changes the prompt. The first question takes 30 seconds. The model is usually not the answer.