Inference Budget Governance: The Hidden Finance Problem in Scaling Agents

Three months after shipping their research pipeline, a team found an $89,000 API bill. The agents were running successfully — every quality metric was green, latency was acceptable, users were happy. But each research request was feeding a fresh 150,000-token context window to Claude Opus on every reasoning step. Nobody modeled that. The bill arrived on Tuesday. The CFO asked for an explanation on Wednesday. By Thursday, the team had manually throttled the pipeline to a trickle.

This is inference bill shock, and it follows a pattern almost every team scaling agents eventually hits. They build cost estimates from single-call benchmarks — accurate for a clean system prompt plus response. Then they deploy agents that iterate: accumulating context across steps, spinning up subagents that inherit that context, retrying on failures without any budget awareness. A 10-turn research agent with 20,000 tokens of context per turn doesn't cost 200,000 tokens — it costs 1.1 million, because each step retransmits everything that came before it.

Agents make 3–10x more LLM calls than simple chatbots per user request^[1], output tokens cost 3–5x more than input tokens across all providers^[2], and multi-agent pipelines multiply these effects by the number of agents operating in parallel. The combination turns a $0.05 per-request estimate into a $5 per-request reality. Inference budget governance — treating token capacity as a managed resource with per-agent quotas, model routing policies, and a Finance-Engineering operating cadence — is how teams stop discovering this math on the billing page.

The core problem isn't that agents are expensive — it's that single-agent cost models don't predict multi-agent costs at all.

When engineering estimates cost per request, they measure a clean single call: system prompt + user message + response. That measurement is accurate. It just doesn't describe what happens in production, where agents iterate.

A research agent analyzing a competitor landscape doesn't ask one question. It asks eight, accumulates the answers into a growing context window, then runs a synthesis pass against all of it. Each reasoning step retransmits the full accumulated context. Add multi-agent orchestration, and the multiplier compounds: each subagent receives a context including the orchestrator's full reasoning trace, its own history, and every tool call output accumulated so far. A three-agent pipeline doesn't cost 3x the single-agent price — it costs 6x, because every fan-out step carries accumulated context from everything upstream.

Then there's the loop problem. In November 2025, two LangChain-based agents entered an infinite conversation cycle that ran for 11 days, generating a $47,000 API bill before anyone caught it^[3]. According to a 2025 State of AI Cost Management survey, 80% of enterprises underestimate their AI infrastructure costs by more than 25%^[4]. The teams that avoid surprises share one trait: they treat inference budget as a first-class design constraint before launch, not a monitoring dashboard to check after something goes wrong.

96% of enterprises report that AI costs exceeded their initial projections^[1]. That's not a forecasting failure — it's an architectural one. The cost model wasn't wrong; it wasn't modeling the right system.

Most inference budget problems trace to three structural failures that compound each other.

Context accumulation without pruning. Multi-turn agents retransmit full conversation history on every step. A 20-step research agent with 10,000 tokens of context per turn spends 1.1 million input tokens total, not 200,000 — because step N pays for all N-1 previous steps plus the new input. The fix isn't reducing context size; it's passing structured summaries between steps rather than raw transcripts. Agent output contracts — structured JSON rather than conversational history — cut this multiplier significantly.

Reflexive frontier model use. Teams default to their best model for everything, which feels safe but doesn't make economic sense. Sonnet 4.6 runs at $3/$15 per million tokens; Opus 4.6 runs at $5/$25^[5] — a 1.67x per-token gap. On tasks where Sonnet delivers 98% of Opus quality (most everyday coding, classification, and content generation), using Opus is a 67% cost premium for near-zero quality gain. Routing 90% of production requests to Sonnet instead of Opus cuts that portion of the bill by 40%.

Hard stops without graceful degradation. Engineering teams often implement budget caps as exceptions: when the limit is hit, the agent throws an error. Hard stops cause silent failures, retry storms, and compounding costs. Users see errors and retry. The retry itself may breach the next budget tier. The correct pattern is three-tier response: warn at 80% of budget, downgrade model and compress context at 90%, return a partial result at 100%. An honest partial result preserves more user trust than an unhandled exception.

The organizational piece of inference budget governance is systematically underbuilt. Platform engineers implement cost controls. Finance tracks monthly invoices. Neither talks to the other until the bill is already a surprise. By that point, the conversation is adversarial rather than collaborative.

The contract that prevents this has three components.

A shared cost model built before launch. Engineering provides Finance with: estimated token counts per agent task class at P50 and P95 confidence, expected request volume by workflow, and model tier assignments per task type. Finance provides: a monthly budget envelope per product area and escalation thresholds that trigger review. The shared output is a per-workflow cost forecast. P50 is the operating target; P95 defines the alert boundary. Without this document, nobody knows whether a 40% month-over-month cost increase signals success (volume grew 60%) or failure (efficiency collapsed).

Per-agent cost attribution. Every inference request needs to emit a structured cost event: agent ID, workflow ID, product area, model tier, input tokens, output tokens, and cache hit status. Without attribution, you can see total spend but not the cause. With it, anomalies become diagnosable: "the document analysis agent in the compliance workflow consumed 3x its P95 budget because a new document format triggered verbose extraction chains" is actionable. "Our total API spend was 40% over forecast" is not.

A weekly governance cadence. Finance and platform engineering review the cost report together — weekly, not monthly. The latency between anomaly and detection should be measured in days. A team reviewing weekly catches a runaway agent in the current sprint. A team reviewing monthly catches it after four weeks of damage. This cadence also builds the usage history needed to negotiate provider discounts: committed spend tiers typically yield 15–30% reductions at moderate volumes^[2], but you need reliable projections to negotiate confidently.

Budget enforcement has to happen at the SDK layer. Monitoring tells you what you spent. SDK-level enforcement controls what you can spend — and shapes how the agent behaves when approaching a limit.

The minimal pattern: wrap every agent execution with a budget tracker that holds a per-trace token ceiling. As the ceiling approaches, the agent downgrades to a cheaper model. At 90% of budget, it compresses working context. At 100%, it returns a partial result rather than an exception. The user gets a degraded-but-useful response. No retry storm. No silent failure.

Model routing is the highest-impact inference cost control available. Teams either route everything to the frontier model (expensive by default) or try to route everything to the cheapest model (breaks tasks, causes retries that compound costs). Both miss the point.

The routing decision lives at the task level, not the session level. A single agent session might use Opus for the planning step, Sonnet for document extraction and analysis, and Haiku for formatting the final output. This isn't a quality compromise — it's matching cognitive load to model capability.

Production data from routing implementations supports this: routing 90% of requests to Sonnet 4.6 while reserving Opus 4.6 for high-complexity tasks typically yields 40–60% cost reduction with less than 5% measurable quality degradation^[6]. The word measurable matters — there's a common assumption that cheaper models perform dramatically worse on real workloads, but benchmark data shows the gap is narrow for the task types that dominate most production traffic.

The routing decision needs its own benchmark. Public benchmark scores (SWE-bench, GPQA) are proxies. Your task distribution is the ground truth. Before committing to routing rules, run your specific task mix through both models and measure quality on your own evaluation criteria. Teams that get routing wrong almost always set rules based on intuition rather than their own task benchmarks.

Model routing handles the model selection decision. Context caching handles the repeated-context problem — and it's often the easier win to implement first.

Most agent systems send the same system prompt and shared knowledge base on every request. Without caching, those tokens are billed at full input rate every call. Anthropic's prompt caching prices cache reads at 10% of standard input cost^[5]. For a 50,000-token system prompt called 1,000 times per day, that's the difference between paying $150 per day and $15 per day — just for the system prompt portion.

The implementation is lightweight: mark stable context with cache_control in the API request. The cache is valid for up to one hour per write. For high-frequency agents, the cache write cost (1.25x standard input price for 5-minute TTL) is recovered after a single subsequent cache read^[5].

Caching and routing compose. A cached Haiku call costs approximately 50x less on the system prompt portion than an uncached Opus call — because the cache read costs 10% of Haiku's already-low input rate, versus full Opus input pricing. At 30,000 requests per month, that difference on system prompt alone can reach $7,350^[7].