Cost-Aware LLM Architecture: Heterogeneous Model Routing from Day One

The bill arrives from Anthropic or OpenAI, and the number is wrong. Not wrong as in the math does not add up — the math is perfectly accurate. Wrong as in nobody budgeted for this during architecture review. A team running 50,000 requests per day on Claude Sonnet is spending roughly $22,500 per month on input tokens alone, assuming an average 300 tokens per request at $3/1M. Route 65% of that traffic to Claude Haiku at roughly $1/1M and the same volume costs closer to $7,700/month. Nobody designed that routing. Nobody decided to pay the premium. It just happened, because the default is always the capable model.

The pattern is structural, not accidental. Teams pick a frontier model for development — sensibly, because frontier models surface edge cases faster and fail more expressively. Then the feature ships. Traffic grows. Six months later, the model choice is load-bearing infrastructure, the billing alarm fires, and the engineers who built it are in a different org. Retrofitting routing now means a migration that often costs more in engineering time than the accumulated overspend.

A heterogeneous LLM stack routes each request to the cheapest model that meets its quality requirement, rather than sending all traffic to a single default model. The mechanism is a task profile audit: an explicit map of which task types live in the system, what quality bar each requires, and which model tier meets that bar at minimum cost. Teams that build this before launch consistently report 40–85% lower inference spend from day one ^[1][2]. That is not a post-launch optimization — it is a design decision made before writing the first prompt.

Development economics favor capability over cost. During prototyping, frontier models produce better outputs faster, surface problems earlier, and fail expressively — the error is legible, not silent. Those properties have high value when you are figuring out the right behavior. They have zero value in production when the task is a JSON extraction you have run ten thousand times.

The cost signal arrives too late. API costs aggregate to monthly bills. A model choice made on day one generates no visible feedback until month two or three, by which point the architecture has calcified. Every new feature built on top of the expensive default extends the blast radius of the original decision. The compounding is slow enough to be invisible until it is not.

Nobody owns the routing decision. Whoever writes the first prompt picks the model. That choice does not go through architecture review. It has no owner. It defaults to whatever the documentation examples show — typically the highest-capability tier — and becomes permanent by inertia. Cost-first architecture requires treating routing ownership the way you treat database schema ownership: explicitly assigned, reviewed before merge, not implicit.

The task profile audit is a structured list of every discrete operation your LLM system performs. For each task, you answer three questions: What is the input distribution? What constitutes an acceptable output? What is the downstream consequence of a wrong answer?

Those three answers determine the model tier — and the third question is the one teams consistently skip. Task complexity and downstream consequence are not the same variable. A simple classification that feeds an automated refund decision needs more careful model selection than a complex synthesis that a human reviews before acting. Model tier selection is not only a cost decision — it is a risk calibration. The blast radius column in your audit is as important as the complexity column.

A short prompt does not equal a low-stakes task. A long, nuanced prompt does not always require a frontier model. Map them separately. Teams that classify only by input complexity and skip the consequence column build routing configs that look cheap on paper and expensive in production incident reports.

The three-tier model is a practical approximation covering most production workloads. Small models (Gemini Flash, Claude Haiku, DeepSeek V3) run at $0.14–$1 per million input tokens. Mid-tier (Claude Sonnet, GPT-4o) at $3–$10. Frontier (Claude Opus, o-series reasoning models) at $15–$75 ^[5]. The price gap between frontier and small is roughly 100x. The capability gap on most real-world tasks is not 100x — it is closer to 5–15%, and often zero.

Rule-based routing uses application-layer metadata — task type tag, input length, structured-vs-open flag — to assign tiers without an additional LLM call. Zero added latency, zero added cost, roughly 60–75% classification accuracy ^[4]. The right default for systems where task type is deterministic (the same application path always produces the same task type). Most systems have more deterministic task distribution than engineers assume — the classification already lives in your route handlers, it just has not been wired to the model selector.

Content-based classification analyzes the actual prompt text using a small classifier — an embedding model, a BERT-scale model, or a cheap LLM call — to assign complexity. The RouteLLM framework, published at ICLR 2025 by researchers from UC Berkeley, Anyscale, and Canva, demonstrated that a trained matrix factorization router achieves 95% of GPT-4's quality using only 26% of frontier model calls — roughly 48% cheaper than random routing ^[9]. With augmented training data, the same router drops frontier call share to 14%, cutting cost by 75% ^[11]. The catch: RouteLLM's routers require training data (Chatbot Arena preference data or similar); teams without historical annotations start with rule-based routing and evolve. Off-the-shelf content-based classifiers cost $0.001–$0.003 per query overhead ^[4]. Use content-based routing when your application layer cannot reliably tag task type, or when a single endpoint receives genuinely mixed complexity.

Cascade routing sends requests to the cheapest tier first, then escalates on quality failure. Highest accuracy (because failure is empirically measured, not predicted), highest latency, most complex failure modes. Reserve it for workloads with reliable programmatic quality signals and users who can tolerate 2–3 second escalation delays. Customer-facing interactive features are usually the wrong fit for cascade routing. Background processing pipelines are the right one.

The standard routing advice assumes you have production traffic logs to analyze. You do not, on day one. Synthetic profiling fills that gap — and most competing articles on LLM routing skip it entirely.

The method: collect 200–500 representative prompts from analogous systems, internal dogfooding, or hand-constructed edge-case scenarios. Run them through each model tier in a staging environment. Apply your quality criteria to each output. The result is an empirical routing threshold — what percentage of your task corpus does Tier 1 handle to acceptable quality? — measured on your actual prompt distribution before a single user touches the system.

This is harder than it sounds for one reason: you must define "acceptable" before you start. Teams that skip this definition end up calibrating routing thresholds to a gestalt of "looks good," which does not survive engineer turnover or model upgrades. The quality oracle — the acceptance criteria applied to each output in the profiling run — becomes the canonical definition of correctness for the task. Write it down. Version it. Treat it like a spec.

LiteLLM is the most common open-source routing proxy in production today, with over 33,000 GitHub stars and a unified OpenAI-compatible interface to 100+ providers ^[10]. The YAML config below wires up a three-tier stack with automatic fallback. The Python snippet adds the task-type classifier and context-length guard that the YAML alone does not give you.

Context window as a feasibility ceiling: Small models have smaller effective context windows. Older small models cap out at 32K–128K tokens. For most tasks this does not matter. For tasks involving long documents, multi-turn conversation history, or large retrieval contexts, routing to a small model is not just a quality question — it is a feasibility question. The request that works on Sonnet fails on Haiku because the context exceeds what the configuration will serve reliably. Teams that discover this mid-production end up with two bad options: limit context to the smallest model in the fleet (wasting capability on every high-tier task) or add a second routing dimension — complexity plus context length — that was not in the original design. Build a context-length guard into the classifier from the start. Any request exceeding 80% of the small model's effective context window should bypass Tier 1 automatically.

Output tokens compound faster than input tokens at scale: Development prompts tend to produce short outputs. Production traffic does not. A frontier model generating 2,000 output tokens per request costs roughly $0.15 per call at Opus pricing ($75/1M output). The same request on a budget model costs under $0.01. The task profile audit should capture expected output length explicitly — not just input complexity. Generative tasks with open-ended outputs need output-length-aware routing, not just input-length-aware routing.

One more failure mode: model upgrade cycles invalidate thresholds. Routing configurations calibrated for one pricing generation become economically wrong when the same capability tier drops 10x in the next model release — a pattern that has repeated consistently since 2023. Build thresholds as configuration values, not hardcoded constants. You will update them quarterly.

The bill you are looking at is a record of design decisions that had no cost dimension when they were made. The model was picked for capability during development, traffic grew, nobody updated the routing, and the accounting arrived six months late.

Cost-first architecture does not require a sophisticated ML router. It requires a task profile audit, a synthetic profiling run, and a configuration layer that routes by task type. Three weeks of work done before launch compounds into substantial savings as traffic scales. The measurement layer — logging response.model on every request, attributing cost per task type, alerting on escalation rate changes — is what keeps the system honest as models and pricing evolve.

Teams surprised by their quarterly bill skipped the audit. Teams that ran it do not get surprised.