The Spec Is Where Production Failures Start

Every FC1 failure in the MAST dataset was writable before the agent ran. Specification Issues — 41.77% of all failures across 1,642 annotated traces from seven open-source multi-agent frameworks — don't emerge from runtime surprises.^[1] They're embedded in the spec the author wrote before the first token generated. The agent violated its task constraints because the constraints were absent. It repeated completed steps because the execution flow had no state-tracking guard. It ran past task completion because "return when done" is not a terminal condition an agent can evaluate against anything.

Researchers at UC Berkeley published MAST at NeurIPS 2025 after analyzing traces across MetaGPT, ChatDev, HyperAgent, OpenManus, AppWorld, Magentic, and AG2. Failure rates ranged from 41% to 86.7% depending on the system.^[1] That range isn't primarily explained by model choice. It's explained by how completely each system's specs, coordination protocols, and verification layers were designed before execution. The MAST paper's targeted interventions on ChatDev improved performance by +15.6% without touching the model — only the spec.^[1]

The MAST triage playbook answers: "My agent failed — which of the 14 failure modes is this, and which layer do I fix?" This playbook answers a different question: what must be true before the agent ships so that those failure modes cannot occur. The answer is three sequential pre-production review gates — one per MAST category — with 12 questions, explicit pass criteria, and assigned ownership. No postmortem required.

FC1's five failure modes — Disobey Task Specifications (FM-1.1), Disobey Role Specifications (FM-1.2), Step Repetition (FM-1.3), Conversation Loss (FM-1.4), and Agents Unaware of Termination Conditions (FM-1.5) — all share an origin: they're consistent with the specification the agent was given.^[1] The model isn't malfunctioning. The design is underspecified.

The test for FC1 is cheap: pull the system prompt and role definition next to the failed trace. If you can point to a line that's missing — a termination condition, a role constraint, a state check — that's FC1. If you find yourself writing "I assumed the model would know not to do that," you found the gap.

FM-1.5 (Termination Conditions) is the most consequential because it compounds. Without a verifiable terminal state, the agent continues until a hard limit fires — token budget, API timeout, or circuit breaker. "Return when the task is complete" is not a termination condition. "Return when the diff is clean, all tests pass, and the output matches the expected schema" is. Every agent needs a terminal condition it can evaluate against current state.

FM-1.3 (Step Repetition) shows up most in long-horizon tasks. AppWorld had the highest concentration in MAST-Data.^[1] The prevention is a state-tracking guard at step start: before the agent executes a step, it checks whether the target state already exists. Without this, loops are the default state when the task is partially complete.

The Agent Behavioral Contracts (ABC) framework formalizes what MAST implies here: contracted agents operating with explicit preconditions, invariants, and recovery rules detect 5.2–6.8 soft violations per session that uncontracted baselines miss entirely.^[5] The gap between an underspecified system prompt and a behavioral contract is exactly where FC1 failures live — and it's entirely preventable before deploy.

FC2's most common mode is Reasoning-Action Mismatch (FM-2.6, 13.98% of all MAST failures^[1]). The agent states an intention and executes something different. This isn't a prompt problem — it's a monitoring gap. You can't catch FM-2.6 without logging reasoning-action pairs at each step, and you can't build that logging after the agent ships without changing the execution flow. The logging decision belongs in the pre-production design review.

Failure to Ask for Clarification (FM-2.2, 11.65%^[1]) is structural, not behavioral. An agent that receives an ambiguous instruction will proceed on an assumption — that's what language models do. The prevention requires a designed clarification gate: a conditional in the execution flow that evaluates confidence before proceeding. Adding this after production is an architecture change. Doing it before deploy is a design decision.

The MAST paper's FC2 finding is direct: communication protocol improvements — cleaner message formats, structured handoffs, turn-taking confirmations — are insufficient for coordination failures.^[1] You can enforce message schemas across every agent boundary and still ship a system where no gate stops an agent from proceeding on a wrong assumption. The gate is an architectural decision, not a protocol tweak.

Task Derailment (FM-2.3, 7.15%^[1]) is the FC2 mode most likely to produce plausible-looking output. The agent is productive — completing steps, returning results, interacting with other agents — but on a drifted objective. Prevention requires shared task state visible to all agents: a single authoritative task definition that any agent can check against its instructions before continuing. This is a design artifact. If it doesn't exist at review time, it won't exist at runtime either.

A practical signal for FC2 readiness: when you review the architecture diagram, can you point to where the clarification gate sits in the execution flow? Can you show where reasoning-action pairs are logged? If both answers are "we'll add that later," the FC2 gate fails.

FC3's three modes — Premature Termination (FM-3.1, 7.82%), No or Incomplete Verification (FM-3.2, 6.82%), and Incorrect Verification (FM-3.3, 6.66%^[1]) — share one signature: the workflow completes, no alert fires, infrastructure is clean. The problem is in the output.

This makes FC3 the most expensive category to catch in production because it looks like no incident. Output quality degrades. Downstream systems process the output. The problem surfaces in a sprint retro or a customer complaint three weeks later.

The ReliabilityBench study found that pass@1 metrics overestimate production reliability by 20–40%.^[6] That gap is the FC3 failure surface — agents that pass staging tests and fail at correctness in production, because staging verified format and production revealed correctness. The root cause is the same across modes: verification checked the wrong thing, or ran only at the terminal output instead of at intermediate steps.

FM-3.3 (Incorrect Verification) is the trap in evaluation design: you built an eval, but it tests the wrong property. A code agent verified by a linter that checks syntax but not execution. A research agent verified by citation count rather than claim accuracy. An FC3 eval that checks format is FM-3.3 applied to your own quality process.

The design question for FC3 pre-production review: what does "correct output" mean for this specific agent, and can you evaluate that before shipping? An LLM-as-judge sampling 10–20% of production outputs provides FC3 coverage without reviewing everything — but only if the judge tests task-level correctness. Defining the correctness criterion is the work that must happen before the eval layer ships alongside the agent. Teams that defer it ship an eval layer that catches the wrong failures.^[8]

FC1 review belongs to the agent author and tech lead. It's the cheapest review to run — pulling the spec and working through the four FC1 gate questions takes under an hour. The failure mode for FC1 ownership is diffuse accountability: everyone knows the spec exists, nobody is responsible for verifying it's complete. Make the FC1 review an explicit deployment ticket item with a named approver, not an implicit assumption about what passed staging.

FC2 review belongs to the architect who designed the coordination protocol. If no architect owns the coordination design, FC2 review ownership is undefined — which means FC2 failures are shipping by default. This is a structural assignment the engineering lead makes before the agent is built, not after it's built. FC2 review is not asking whether the agent "communicates correctly with other agents." It's asking: does a clarification gate exist in the design spec? Are reasoning-action logs part of the implementation plan? Is shared task state a named artifact? These are architecture decisions, reviewed against the design doc, before implementation is considered complete.

FC3 review belongs to the platform or eval team with LLM evaluation tooling. FC3 review is the hardest to complete before shipping because it requires the eval layer to already exist. Teams that don't have an eval layer can't do FC3 review — and that itself is a deployment blocker. If the eval layer doesn't exist when the agent is ready to ship, the sequencing problem is upstream of the agent: the agent is blocked on platform infrastructure, not development. That's the right resolution. Shipping without FC3 coverage is a named risk decision, not an unnoticed gap.

The most common failure of this ownership model is treating FC3 as optional for "low-risk" agents. The MAST paper doesn't distinguish risk tiers — all three verification modes occur across agent types regardless of task complexity.^[1] The team decides the blast radius, not the taxonomy.

The MAST paper showed that targeted FC1 spec fixes improved performance by +15.6%, then immediately noted that simple fixes are insufficient — more fundamental system design changes are required.^[1] That's not a pessimistic conclusion. It's a precise one: the failure modes are structural, the fixes are structural, and the right time to make them is before the system runs.

Thirty deployed agents without FC1 gates, FC2 coordination design, or FC3 eval coverage are thirty scheduled postmortems. The failures are already written in the specs. The production incidents just haven't run yet.

The three gates force three decisions that every agent deployment requires anyway — made before shipping, with named owners, against explicit pass criteria, instead of discovered under pressure after a failure with an ambiguous root cause and no clear fix layer.