LLM Eval Pipelines: Catch the Production Failures Your APM Can't See

200 OK in 1.2 seconds. The dashboard is green. The model invented a refund policy that does not exist, quoted the wrong deadline, and shipped code that compiled cleanly while corrupting a column in production.

The HTTP layer never saw any of it. To latency monitoring, a fabricated refund and a correct one are identical bytes. Status codes are not a quality signal for generative output. The output is the system. Evaluating that output — continuously, before users become your QA team — is the only layer that closes the gap.

This article walks the full stack: why the binary test contract breaks, how eval maturity ladders work, what judge models actually score reliably, how to wire CI gates and async production sampling, and what the dataset needs to look like. It ends with the two-week bootstrap that gets a team from zero to meaningful CI coverage.

A unit test asserts equality. Input X returns Y, every run, or the test fails. The contract is binary. The contract is stable.

Model output is neither. It drifts between runs at fixed temperature. It reacts to phrasing the test author never anticipated. It is frequently correct in three different ways at once. A response can be factually right and uselessly long. Concise and dangerously incomplete. Authoritative in tone and entirely fabricated in substance.

The binary contract does not survive that surface area. What replaces it is graded scoring across quality dimensions — measured continuously, calibrated against human labels, owned as infrastructure rather than checked once before launch.

Roughly 57% of organizations report agents in production, and quality is the cited blocker for further rollout — 32% of respondents in LangChain's 2026 State of AI Agents report name it as the top barrier^[2]. The same survey shows most teams stuck at Level 0 or Level 1: manual review, no automation, no systematic coverage of the failure surface.

The ladder, and what each rung actually buys you:

Most teams reading this sit at Level 1 or 2. Reliability starts to meaningfully improve at Level 3 — LLM-as-judge in CI. The jump from 3 to 4 is where production failures stop surprising you.

A caution before the climb. One team spent three months on a Level 5 flywheel before discovering 80% of their quality problems traced to a single ambiguous system prompt. The pipeline faithfully measured a broken baseline. Fifty golden examples and two weeks of manual output review will teach you more than instrumentation pointed at the wrong target. Build the dataset before the harness.

The structural point: the pipeline is a loop, not a linear gate. Production failures route back into the eval dataset. The dataset grows sharper every time the system fails in a new way. This is the flywheel. Without it, eval coverage stagnates the day you stop adding examples by hand.

LLM-as-judge is the practice of using a capable model — usually stronger or separately tuned — to score the outputs of your application model. It is the only semantic eval mechanism that scales to production traffic without crippling cost.

The economics are decisive. Roughly 500x to 5000x cheaper than full human review, with about 80% agreement against human preferences in calibrated settings^[6] — comparable to how often two humans agree with each other on the same task. After calibration against human labels, precision and recall can approach 0.9 in some domains, though numbers vary by task complexity and domain specificity^[1].

The failure modes are systematic, not random. Plan for them or the score becomes theater. Frontier models in 2026 exceed 50% error rates on bias tests^[8] — meaning the judge is wrong more often than a coin flip on certain evaluation axes before mitigation. The biases are not random noise; they are structural artifacts of how these models were trained.

The judge prompt structure matters as much as the judge model. A rubric that says 'Is this response helpful?' is not a rubric. A rubric that says 'Does the response directly answer the user's question without introducing unsupported claims, in 150 words or fewer?' is. Specificity drives agreement. Vague criteria propagate vague scores, and vague scores cannot gate deployments.

Here is a minimal but concrete judge prompt pattern:

Which metrics depend on what the application does. Every production LLM system needs at least three layers covered: output quality, behavioral safety, operational efficiency. The dimensions below are the minimum surface.

Agent systems need one more dimension: trajectory quality. Not just whether the agent landed on the right answer, but whether it took a defensible path. An agent that completes a five-step task in fifty steps is not a win. It is a reliability risk wearing a green check mark.

For RAG pipelines, faithfulness splits into two distinct failure modes that need separate scores. Retrieval faithfulness: does the response reflect what was retrieved? Context faithfulness: does the retrieved context actually match the user's intent? A system can pass retrieval faithfulness while completely failing context faithfulness — it accurately reported information that was irrelevant to the question. RAGAS measures both separately, which is why it became the default for RAG eval^[10].

The golden dataset is the set of (input, expected behavior) pairs CI runs before every deploy. It is the highest-leverage asset in the eval stack and the most commonly neglected.

Golden datasets fail in two ways. They are too small to catch regressions reliably. Or they go stale as the application evolves and nobody owns the curation. Both are expensive. Both are silent until production breaks.

On size: industry data puts the minimum viable dataset at 50–100 examples — enough to catch obvious regressions — and production-ready at 200–500 covering major use cases and edge cases^[12]. Microsoft's Copilot teams recommend 150 question-answer pairs for complex domains. What matters more than count is coverage per failure-mode category. A dataset of 500 happy-path examples cannot detect a regression on edge inputs; 75 adversarial examples structured around known failure modes can.

The goal is to make shipping a regression harder than fixing it. CI fails the build when scores drop below threshold. Not a comment on the PR. A red X. The engineer cannot merge through it without an explicit override.

A practical CI eval job:

If you are starting with DeepEval rather than a custom script, the integration is tighter. DeepEval uses pytest-style assertions so failures block the CI job without any custom exit code handling:

Pre-deploy evals catch regressions against known behavior. They cannot catch what breaks when real users hit the system in ways the dataset never modeled — which is, predictably, every day.

Production monitoring for LLM quality is not latency monitoring with extra steps. There is no health endpoint. The pattern is asynchronous sampling, off the critical path, scored by a judge, alerted on sustained score degradation.

The load-bearing word is sustained. Individual outputs vary. A single low score is noise. A 3% drop in faithfulness across 500 sampled requests over 48 hours is signal.

The temptation is to automate everything and never read the outputs again. Resist it.

Human review carries two roles automated judges cannot. The first is calibration: judge scores are only meaningful relative to human judgment. Stop sampling real outputs and the calibration drifts, the scores become self-referential noise, and the dashboard turns into theater. The second is novel failure discovery: automated judges only catch failure modes you have already defined. Humans catch the weird ones — outputs technically within spec, clearly wrong in context.

A lightweight weekly review workflow:

The ecosystem matured. Almost no team needs to build eval infrastructure from scratch. The harder problem is selecting the right tool per layer and wiring them so the dataset, judge, and production sampler share a single source of truth.

A natural pairing that many production teams land on: RAGAS defines the metric suite (particularly for retrieval evaluation), and DeepEval enforces the thresholds in CI^[11]. They share a compatible test case structure, so the dataset travels between them without transformation.

The gap between no evals and useful evals is smaller than it looks from outside. Two engineers in two weeks can land Level 2 and meaningful CI coverage. The plan below is the minimum viable bootstrap.

Eval theater is real. A system that scores 0.90 on the golden dataset and 0.91 next week looks healthy. If the dataset has not grown in three months and production traffic has drifted, those scores measure yesterday's failure modes, not today's.

Three conditions that make eval scores actively misleading:

Dataset staleness. The application evolved — new features, different user populations, updated prompts — but the golden set did not. The eval is measuring a product that no longer exists.

Distribution shift. Production traffic changed in ways the seeding process did not model. A customer support bot that was trained and evaluated on English queries then deployed to a multilingual market will score fine in CI and fail in production every night.

Judge capture. The application model learned to write in a style that triggers high scores from the judge model, without improving quality for users. This happens when the same judge model is used for both development feedback and production gating, with no human calibration loop.

The mitigation is structural, not metric-level. Review the dataset age alongside the score. Track score trend against the date of the last dataset update. If scores are flat and the dataset is old, that is a warning sign, not a success signal.

LangChain's coding agent moved from 52.8% to 66.5% on Terminal Bench 2.0 — outside the top 30 to the top 5 — without changing the model^[3]. They changed the harness. Same model. Different eval and correction infrastructure. Dramatically different results.

The insight most teams internalize too slowly: the underlying model matters less than the system around it. The model is a replaceable component. The eval infrastructure, the golden dataset, the feedback loops — those compound. They get sharper every time production fails and the team handles it correctly.

Teams that built eval infrastructure in 2024 now hold datasets of hundreds or thousands of real failure cases. Teams starting today start from zero but inherit better tooling and a clearer playbook.

The uncomfortable counterpoint: eval infrastructure can become its own theater. High scores are not evidence of quality — they are evidence that the system handles the cases you remembered to test. Distinguishing between the two is a discipline, not an output of the tooling. The eval stack only compounds if the dataset keeps growing. A frozen dataset is a false ceiling.