Explore all articles

Karpathy’s four coding-agent principles are useful, but production agents need scoped edits, test-gaming controls, trust-boundary calibration, and calibrated reporting.

Detection tells you something is wrong. The four-step diagnostic pipeline — behavioral telemetry, failure clustering, root cause attribution, eval generation — tells you what failed, why, and how to stop it from shipping again. Most teams build partial detection and stop there.

Most teams architect for capability and optimize for cost after the invoice lands. Here is the playbook for building cost constraints in from day one: task profile audits, three-tier routing, and synthetic benchmarking before your first deploy.

A deep teardown of the production CMS pipeline that turns GitHub Issues into merged PRs while you sleep. 10 workflows, 6 issue types, AI media generation, and the exact DAGs that make it work.

Most production agents run on intentions nobody wrote down. Here is how to write the behavioral spec — scope, invariants, testable success criteria, and failure modes — that translates business intent into something your infrastructure can enforce.

Amazon's Kiro deleted production in December 2025. The model didn't malfunction — it executed inside the permissions it had been given. The fix is not a better model. It's an enforcement stack the prompt cannot override. Four layers, executable constraints, no theater.

Most production agent failures are not model failures. They are missing constraints — business rules carried in four engineers' heads with no formal representation agents can query. The fix is a versioned, governed context store the data team owns instead of answers.

Four agents coordinate. The trace backend shows 3 to 10 orphaned root spans, no causal thread. The model is not the failure. Context propagation is. Five gaps, the minimal code to close each, and the build order that actually ships.

Why production inference bills always exceed estimates — and the Finance-Engineering governance framework for per-agent budgets, model routing, and cost forecasting without capability degradation.

Agentic tools push engineering past 2–3x velocity and product definition becomes the binding constraint. Hiring more PMs makes it worse. The fix is a three-tier decision rights model that moves authority to where the information actually lives.

Valid JSON, clean dashboards, no alerts — and the agent's reasoning depth dropped 67% between two model updates. Three detection layers catch what HTTP error rates structurally cannot: execution fingerprinting, semantic drift, and user-signal triangulation.

How to apply semantic versioning and consumer-driven contract testing to AI agent system prompts — treating prompts as versioned API contracts with explicit breaking change classification, agent manifests, and CDC-style registration for multi-agent production systems.

Most teams promote to multi-agent before proving the single agent. Three gates — observability, override readiness, behavioral consistency — decide whether orchestration is earned or inherited. Skip them and a $3.50 task becomes a $47,000 incident.

Eight in ten agentic AI projects stall on data, not models. Score your environment on ten dimensions before the agent surfaces the gaps. Four tiers, calibrated thresholds, structural fixes ordered before operational ones.

Latency, error rate, and token cost stay green while LLM output quality degrades for weeks. The infrastructure layer cannot see semantic failure. Sampled evals, prompt hash drift, and distribution alerts are the signals that catch it before users do.

60% of agentic projects stall on data, not models. A 30-minute, three-tier gate — Foundation, Workflow, Autonomous — that decides what autonomy your data can actually support, with a retrofit pattern for legacy systems you cannot rewrite.

Train once, control the weights, call it sovereignty. Twelve months later the model is confidently wrong about pricing, policy, and headcount. The playbook for when to retrain, what to retrain on, and how to validate without breaking live agents.

Two LangChain agents burned $47K in eleven days. The model worked. The budget math didn't. Multi-agent cost is a heavy-tailed distribution, monitoring is structurally too late, and only synchronous SDK-level enforcement stops the spiral.

Most agent failures return HTTP 200. The dashboard stays green while the reasoning chain quietly compounds the wrong premise. Here is the triage runbook, the failure-mode field guide, and the postmortem template that survives non-deterministic systems.

Push automation onto an absent substrate and you get usage numbers without capability. Four layers — Literacy, Sandbox, Playbooks, Feedback Loops — a scored readiness rubric, and the sequencing rhythm that holds after the mandate memo fades.

46% of AI proofs of concept never ship. The gap is not technical. It is structural: PoC culture rewards experimentation and punishes shipping. A 90-day decision gate, an operational owner, and an incentive rewrite — or pilot purgatory wins again.

Launches get conference talks. Retirements get archived repos and live credentials. Five sequential phases — audit, extract, shadow, communicate, shut down — and the security blast radius when you skip any of them.

Third-party MCP servers run inside your agent's reasoning loop with privileged tool access. Most teams added them without a review process. A 0-100 scorecard across provenance, scope, code, network, and runtime — gated in CI before they ship.

Agents generate code overnight. Humans still review at human speed. Story points lie. The sprint board fills up while cycle time flatlines. The fix is not more agents — it is inverting the planning logic and capping agent output at what reviewers can clear.

AI tools landed as net-new line items. Nobody owns the kill decision. Run the overlap matrix, the 30-day silent run test, the contract clause review, and the procurement reclaim — and bring the CFO a real number.

You approved Copilot. Then Claude Code. The invoice is a surprise and nobody owns the line item. The window for token FinOps is open right now — proxy, attribution, routing, anomaly detection. Build it before the next quarterly review.

Developers report 40% faster code generation. Cycle time barely moves. The gain lands on a non-constraint stage and accumulates as WIP in front of review and QA. A flow-metrics framework for engineering leaders who want the actual answer.

AI ROI math is contaminated at the inputs. The 40% time savings is self-reported. The 3x PR throughput is a review-queue traffic jam. The board number is one cherry-picked team. Four measurement layers, the rework tax nobody applies, and the attribution problem.

Eighty-eight percent of organizations deploy AI. Fewer than six percent see results. The gap is not a model problem — it is a rollout problem. Incentives, champions, friction, and the change-management work nobody budgeted for.

Your employees are already running AI on personal cards because procurement moves at geological speed. Crackdowns don't kill usage — they kill visibility. Build the discovery-to-sanctioned pipeline that makes the official channel faster than workarounds.

The MCP spec describes a protocol, not a security posture. Most production deployments shipped with a static secret in a header, no identity propagation, and error messages that leak internals. Four enforcement layers, executable, before the next incident review.

Quality at five users is self-regulating. At fifty, it is a liability. Build the rubric layer, gate stack, and federated ownership model before consensus rots into theater — or your AI program gets cancelled with the next budget cycle.

Forty entries scored 1-5 in a SharePoint folder is not governance. It is theater. A risk register the board acts on has five entries, dollar ranges, named owners, and a regulatory deadline next to each one.

Compliance is not the brake. The single review queue is. Risk-tier the routing, codify the patterns, automate the checks — and 70% of AI requests stop touching a human. The bottleneck is architectural, not regulatory.

Four signal layers, scored monthly per service, produce a fragility register that names your next outage weeks before it happens. Size is not risk. Neglect is risk. The heat map measures neglect.

Most AI use case selection is workshop theater. Process mining reads the actual event logs and ranks workflows by volume, variance, and structure — so you find out whether you need an LLM, an RPA bot, or nothing before spending a dollar.

GMV is the scoreboard, not the game. Marketplace teams that wait for revenue to confirm a category is dying have already lost the merchants whose absence caused it. Four signals, one weekly brief, three to six weeks of warning before the line bends.

Distributed teams burn productivity at the timezone seam. Decisions buried in threads. Phantom blockers. Parallel divergence. The fix is not better Slack hygiene. It is a structured brief that extracts decisions, blockers, and active work from the tools the team already uses.

Visibility bias is a management failure mode, not a character flaw. Five signal channels, a recognition debt modifier, and a queue that surfaces the contributors your attention misses. Calm correction, not surveillance.

Engineers say it three times before managers hear it. The structural fix is not better listening. It is a delta-aware brief auto-generated 30 minutes before each 1:1, pulling Jira, GitHub, and 5/15s into one page that tags every signal as new, continuing, or resolved.

New hires don't lack capability. They lack context. Three onboarding agents — orientation, historical reasoning, starter-ticket matching — index the institutional knowledge that already exists in PRs, ADRs, and post-mortems. Ramp compresses.

Every signal that would have caught the bad hire was already in your stack — sitting in scorecards nobody opened, Slack threads buried under 200 others, comp data in a different tool. The synthesizer compresses it into one structured recommendation before the offer goes out.

Job posts, changelogs, pricing diffs, and key hires expose a competitor's roadmap weeks before the press release. Run a weekly agent that fuses five channels into strategic inferences, not news summaries — and act on the lead time before it closes.

Static spreadsheets stop working around contract forty. Auto-renewals fire silently, SLA credits expire unclaimed, and concentration risk hides in plain sight. The fix is a three-check radar that runs against contract source data every week.

Engineering budgets do not blow up at quarter end. They drift quietly for ten weeks while nobody is reading the right numbers. A weekly agent over headcount, contractors, cloud, and tooling catches drift in seven days, not ninety.

Manual Monday attribution is the loudest voice winning the narrative. Replace it with an agent that pulls the delta, queries five evidence systems, and ships a ranked hypothesis list with explicit confidence — and an unexplained remainder.

Twenty-six retros a year, three platforms, zero memory. The same friction points keep resurfacing because nobody re-reads 26 documents. An agent that normalizes, clusters, and ranks the patterns turns retro output into a longitudinal record the team can act on.

Single-metric attrition dashboards die in two weeks because their false-positive rate is too high to trust. The signal that holds is four independent metrics drifting together, on one person, across the same fortnight. Architecture, scoring, and the surveillance line.

App Store reviews, NPS verbatims, Zendesk tickets, interview notes, community mentions — five inputs, five biases, five cadences. Treat them equal and the loudest channel wins. The fix is a normalization and weighting layer that produces one weekly brief.

Engineering directors burn 45 minutes every morning reconstructing a picture five tools could have assembled. Replace the loop: five parallel collectors, one orchestrator, a confidence score, a 90-second RED/AMBER/GREEN brief. Triage out of working memory, into code.

Most production deploys that break did not break because of bad code. They broke because of context the deployer could not see. A pre-deploy risk score replaces gut feel with six measurable signals and a HOLD/PROCEED/WATCH verdict the pipeline enforces.

SRE runbooks assume one process, one stack trace, one bad line. Agent failures are distributed across dozens of reasoning steps — the wrong premise gets laundered through 33 more calls before the user sees it. Here is the taxonomy, the triage, the postmortem.

Billing anomaly alerts run on a 24–48 hour lag. The retry loop is already an invoice by the time anyone sees it. The control that catches it is per-session, in-process, and lives in the orchestration layer — profiled envelope, 3x P95 trip, defined degradation.

Five enforcement layers anchored to documented production incidents. Permission scoping, dry-run gates, deletion protection, blast radius scoring, and audit trails the agent cannot reach. Built before you need them, not after the first escape.

The dashboard goes green while the model invents a refund policy. Status codes are not a quality signal for generative output. The fix is an eval stack: CI gates, judge models, sampled production scoring, and a dataset that compounds with every failure.

Every dismiss, modify, and escalate is a labeled training signal. Most teams log it as a debug artifact and move on. Here is the audit schema, the weekly tuner, and the human approval gate that turn that signal into thresholds that converge in eight weeks.

One orchestrator decomposes the question. Four subagents work the threads in isolation. Synthesis weighs the evidence. The brief lands in twenty minutes — not because the model is faster, but because the topology stopped wasting wall-clock on serial wait.

Your private /deploy shortcut saves you twenty minutes a day and helps exactly one person. Plugins move the same workflow into a parameterized package every team installs in minutes. Here is the full lifecycle — skill, context files, MCP wiring, marketplace.

Roughly nine in ten skill files fail one of five basic checks. The body is rarely the problem. The description is — that 100-token blurb is the only thing the agent reads when deciding whether to load you. Engineer it, or stay invisible.

Most enterprise AI lives between pilot and replacement. Five patterns for the 12-18 months it actually takes — strangler fig, sidecar, parallel run, dual-write, eval-based rollback — with the rollback signals that catch silent quality drift.

Modernizations die in the comprehension gap, not the rewrite. The gap has no owner, so it stays open. Five extraction patterns bind every rule to a source line, build the lineage map, and force a behavioral test suite to go green before the new system ships.

Seven patterns for moving DB2, IMS, and VSAM data into RAG: nightly EBCDIC export, CDC, federation, event sourcing, dual-write, schema-on-read, and RAG over the COBOL itself. Pick by freshness budget, not preference.

Senior engineers carry the runbooks nobody wrote. Then they retire. "Document everything" is the ask that produces nothing. A structured-interview pipeline that turns one hour into searchable institutional memory before the bus-factor goes to zero.

Meeting transcripts produce decisions. The decisions vanish into a Notion graveyard within thirty days. A two-agent Cowork workflow extracts structured records and attaches review triggers that fire when conditions actually change — not on a calendar.

Four layers between a generic assistant and a colleague: always-on slow facts, on-demand skill files, live MCP data, and a persistent entity graph. One architecture. Zero fine-tuning. The teams that ship all four cut correction cycles in half.

Every Claude session starts from zero unless something carries the org forward. CLAUDE.md is that something — a persistent context layer that encodes team topology, current priorities, and the decisions you have already paid for. Treat it as a config file and you keep paying the coordination tax.

Most enterprise AI failures are not model failures. They are retrieval failures. Chunking, embeddings, vector stores, knowledge graphs, and the context budget — what actually breaks at scale and how to build the memory layer that holds.

RBAC was built for humans clicking pages. Agents fire hundreds of retrievals per session across permission domains the role-to-resource map never reconciled. The fix lives in the pipeline, not the prompt: pre-retrieval filters, delegated identity, RLS, audit trails that outlive ACL changes.

Business logic stored in employee heads, PDFs, and Slack threads is logic the model cannot enforce. The fix is not better prompts. It is structured rules — decision tables, rule engines, policy-as-code — that the agent calls instead of guesses at.

The primary consumer of your documentation is no longer a human. It is an agent making code changes, retrieving context, executing workflows. Treat docs as infrastructure — versioned, tested, owned — or ship guesses every time the model runs.

Most broken RAG deployments are not model failures. They are upstream failures the model is forced to ventriloquize. The fix is a data pipeline that does the judgment work before retrieval — staleness gates, canonical resolution, business rules as first-class content.

Roughly 88% of experiments do not produce a clean primary-metric win. The bottleneck is interpreting the ones already concluded — not running more. An agent that pulls results, retrieves related history, cross-references releases, and proposes the next three tests closes the gap.

Calendar presence, response latency, and meeting drift carry more signal about who actually decides than the reporting hierarchy. Build a monthly influence map that compares observed decision flow against the org chart — and flag the gap.

A weekly agent reads PRDs, roadmaps, ADRs, and OKRs, extracts the implicit assumptions buried between paragraphs, and ranks the conflicts by blast radius. Surface the contradictions before code gets written. The agent finds. Humans decide.

$3.48M vs $610K revenue per employee. The gap is not measuring AI cleverness — it is measuring how much of traditional engineering headcount was scaffolding for slow handoffs. A role-by-role rebuild for the math you cannot escape.

Fifty engineers running fifty private AI workflows is not adoption. It is a coordination tax with no owner. Audit what is already running, isolate the workflows with org-wide leverage, ship a versioned skills repo, and govern the blast radius before a shared skill drops a column in production.

A week-by-week operating plan for the new VP of AI, CAIO, or CTO who just inherited a transformation mandate. Stakeholder map, named failure modes, the quick-win shortlist, and the board brief that earns a second 90 days.

Most first AI picks fail because the workflow was wrong, not the model. Score risk, value, and signal quality as separate axes. Treat your first three pilots as three different questions about the organization. Pick boring. Pick measurable. Pick diverse.

A diagnostic that scores your org on five independent dimensions, names the anti-stages most maturity models hide, and ends with a 30-minute artifact review you can run without a consulting engagement.