Measuring AI ROI: An Honest Framework Finance Will Actually Accept

Your vendor reports 40% developer time savings. Your VP of Engineering shows a 3x increase in pull request volume. The CEO tells the board the AI investment is paying back ahead of schedule. Everyone is happy.

None of those numbers mean what anyone in the room thinks they mean.

The 40% figure comes from a self-reported survey where developers estimated how long the task would have taken without AI — about as reliable as asking someone how much they would have spent without a coupon. The 3x PR throughput is real, and it is a problem: AI generated more small, trivial changes that now clog the review queue. The board slide projected one team's output across the entire org.

This is the operating state of AI ROI measurement in most companies. A polite fiction nobody examines closely because everyone needs the number to be true.

Gartner forecasts worldwide AI spending hits $2.5 trillion in 2026^[1] — roughly 44% above 2025. The same firm predicts over 40% of agentic AI projects get cancelled by end of 2027 due to escalating costs and unclear value^[2].

Read those two numbers next to each other. Spend is accelerating. Measurement is not. Approximately 29% of executives say they can confidently quantify AI ROI. Deloitte's 2026 State of AI report found 74% of organizations hope to grow revenue through AI — only around 20% report actually doing so^[3]. Hope is not a metric.

The most rigorous data we have now contradicts what vendors sell. METR ran a randomized controlled trial with 16 experienced open-source developers completing 246 tasks — tasks randomly assigned to allow or disallow early-2025 AI tools. The result: AI increased task completion time by 19%^[10]. Before the study, developers forecast AI would reduce completion time by 24%. After completing it, they estimated a 20% reduction. The self-reported belief and the measured reality pointed in opposite directions.

Faros AI went wider: they analyzed telemetry from 10,000 developers across 1,255 engineering teams and found developers merge 98% more PRs and complete 21% more tasks with AI tools — yet DORA metrics at the organizational level stayed flat^[11]. Review time increased 91%. PR size surged 154%. More code, faster, with no improvement in how quickly software reached production or how stable it was when it got there.

A Workday study found roughly 37-40% of time saved through AI is offset by time spent correcting, verifying, or rewriting low-quality outputs^[5]. Only 14% of employees consistently reported net-positive outcomes.

Organizations are not bad at arithmetic. They are calculating against poisoned inputs. Five structural failure modes show up in nearly every AI ROI deck.

Honest measurement separates what AI tools do from what the business gets. Different things, different time horizons, different metrics. Conflating them is where almost every ROI claim breaks.

The stack is four layers. Each answers a specific question. You need all four. Most organizations stop at Layer 1 and file the win report.

Layer 1 tracks whether people use the tools you bought. Active users, session frequency, feature adoption, prompts per day. This is where every vendor dashboard lives, and it is the layer most organizations mistake for ROI.

Activity answers exactly one question: are people using the tool? That matters — a tool nobody uses has zero ROI by definition. But a tool everyone uses also has zero ROI if it does not change outcomes. Email has 100% adoption. Nobody claims email has positive ROI.

Treat activity as a health check. If adoption is low, investigate. If adoption is high, move to Layer 2. Either way, activity is not the answer.

Layer 2 measures whether AI makes individual tasks faster or higher quality. Time-and-motion studies, A/B experiments, before-and-after comparisons. This is also the layer most exposed to the biases above.

The discipline that separates real Layer 2 measurement from theater is the rework discount. For every time-savings claim, you need a paired measurement of time spent on error correction, review, and revision of AI-generated output. Net savings — gross saved minus rework — is the only number you can hand to finance with a straight face.

The METR experiment^[10] is the sharpest illustration of why this matters. Developers believed they were faster. The clock said they were slower. The belief came from the first-draft experience (AI autocompleted the function in seconds). The clock captured the full cycle: reading the output, spotting the edge-case the model missed, rewriting the error handling, running tests, debugging the subtle off-by-one that only appeared at scale. First-draft time and total task time are different numbers. Most Layer 2 measurement only captures the first one.

Layer 3 is where individual task improvements either compound into delivery gains or evaporate into bottleneck shifts. This is the most important layer and the one that demands the most patience — meaningful delivery outcome data takes 8-12 weeks to stabilize, longer in complex orgs.

The LSE Business Review named the failure mode: current measurement focuses on minutes saved and cost reduced, almost nothing on the quality or novelty of what gets produced^[4]. Quality and novelty are harder to observe than time savings. That difficulty is not a reason to skip them.

The Faros AI data^[11] is the clearest available evidence of what Layer 3 looks like when you skip Layer 2 discipline. Ninety-three percent of developers at surveyed organizations used AI coding tools. PR volume nearly doubled. Review time increased 91%. PR size surged 154%. Bug rates ticked up 9%. DORA metrics — change lead time, deployment frequency, change failure rate, time to restore — stayed flat at the organizational level. More code was generated faster. None of it translated into faster, more reliable software reaching users.

Measure at the team level, not the individual level. Individual metrics produce toxic incentives — people optimize for looking productive with AI rather than being productive with AI. The question is whether the team ships better work faster, not who accepted the most suggestions.

Here is the counterintuitive pattern most organizations miss. Some of the best-performing AI-augmented teams show lower story point velocity than comparable non-AI teams at week 8 — because they are shipping fewer, higher-impact deliverables. Points measure throughput. Throughput is not value. A team shipping 20 high-impact features beats one shipping 45 minor tickets, and the dashboard will tell you the opposite if you let it.

Layer 4 connects delivery improvements to business outcomes. This is where ROI actually lives. It is also the layer that requires the closest collaboration between engineering, finance, and product — the three functions that usually disagree about what counts.

The formula is straightforward:

ROI = (Revenue delta + Margin improvement + Avoided cost) - Total cost of ownership

The formula is not the problem. Honest inputs are the problem. Revenue delta from AI is nearly impossible to isolate — did the feature drive revenue because it shipped faster, or because it was the right feature regardless of build speed? Margin improvements require accounting for the full cost stack, not just license fees. Avoided cost is inherently speculative.

McKinsey's 2025 State of AI^[13] puts numbers on the gap: only 39% of respondents attribute any EBIT impact to AI at all. Of those, most report that less than 5% of their organization's EBIT is attributable to AI use. At the function level, the picture is better — software engineering, manufacturing, and IT report 10-20% cost reductions, while marketing and product development see revenue uplifts above 10% in high-performing organizations. The key variable McKinsey identifies: high performers set explicit growth targets and redesign workflows, not just tool access.

Gartner introduced two adjacent frameworks worth using: Return on Employee (ROE) measures how AI changes employee capability and satisfaction. Return on Future (ROF) quantifies strategic optionality — the future opportunities AI capabilities create^[8]. Neither is traditional ROI. That is the point. Traditional ROI was built for capital expenditures with predictable returns, not for capability investments where the upside is uncertain and potentially structural.

You do not need to redesign your measurement infrastructure this week. You need to stop producing numbers you cannot defend. Three checks expose the worst distortions without waiting for a control group to mature.

Check 1: Pull your PR review time trend. If your team adopted AI coding tools in the last 6 months, compare median review cycle time before and after. If review time increased, you have PR bloat. Faros AI found a 91% increase in review time^[11] across 10,000 developers — your team may not be an outlier. AI accelerated code generation and shifted the bottleneck downstream.

Check 2: Apply the rework ratio to your last productivity claim. Take the last time-savings number your team reported to leadership. Ask whoever reported it: what share of that saved time was spent reviewing, correcting, or rewriting AI output? If they do not have a number, the original claim is not ROI — it is activity data relabeled. The Workday research^[5] puts the rework rate at 37-40% across knowledge-work tasks. Apply that discount as a conservative floor and see if the story holds.

Check 3: Count the variables. List every material change that happened in the same quarter your AI ROI number was measured. New team members, process changes, infrastructure upgrades, project-mix shifts, new tooling beyond AI. If there are more than two concurrent changes, you cannot attribute the outcome to AI with confidence. Reframe the number explicitly: 'We saw X improvement in a quarter that included Y and Z other changes. AI contributed — we cannot isolate the share.'

Attribution — how much of an improvement was caused by AI versus everything else changing at the same time — is the hardest problem in AI ROI measurement. There is no perfect solution. Three approaches get you closer to a defensible number than the default of attributing every win to AI and hoping nobody asks.

Approach 1: Parallel team experiments. Match teams on size, seniority, project type, sprint cadence. Assign one to treatment (with AI), one to control (without AI). Run for 12 weeks minimum. Compare Layer 3 delivery outcomes, not Layer 2 task metrics. This is the gold standard. It requires organizational will to temporarily withhold AI tools from some teams. Most orgs cannot stomach that. The ones that do get the cleanest numbers.

Approach 2: Alternating sprint design. When sustaining a control group is politically impossible, alternate sprints with and without AI tools. Two on, two off, repeated three times. Compare delivery metrics across the alternating periods. Controls for team composition. Does not control for project variation.

Approach 3: Regression discontinuity. If you rolled out AI on a specific date, compare delivery trends before and after, controlling for other known changes. Weaker than experiments. Works retrospectively when nobody planned ahead. Use team-level data, not org-level — Simpson's paradox is real and it lives in aggregated AI ROI dashboards.

An honest AI ROI dashboard is an exercise in restraint. The temptation is to fill it with up-and-to-the-right activity charts. Resist. Every metric on the dashboard answers a decision: expand the tool, reduce the tool, change how the tool is used, or investigate further. If a metric does not move a decision, cut it.

The biggest barrier to honest AI ROI measurement is not technical. It is political. Nobody wants to be the person telling the CEO that the AI investment the board approved is showing ambiguous returns. So numbers get massaged, uncomfortable findings get footnoted, and the executive summary stays optimistic.

This pattern does not break with better dashboards. It breaks with structural changes to who owns the measurement.

A note on what we got wrong the first time. The first measurement frameworks we built shared Layer 3 delivery data with the same team that owned the AI rollout. The data going into board decks improved every quarter. Not because results improved — because measurement ownership was misaligned. Moving measurement to the finance and people analytics function, with engineering as a consumer rather than an owner, produced numbers 40% lower on average and dramatically more credible to the board. Uncomfortable. Necessary. Drift is the default state of any measurement system without an owner whose incentives point the other way.