The Experiment Interpreter: Turn Concluded A/B Tests Into Your Next 3 Tests

The experiment concluded. Variant B beat variant A by 4.2% on conversions, p < 0.01. Now what?

For most growth teams, the expensive work starts here. Someone drafts a debrief in a Google Doc. Someone else forgets to check whether the pricing test running in parallel may have skewed the read. Nobody connects this win to the three failed checkout experiments from last quarter that pointed in the same direction the whole time.

The gap between getting a result and knowing what to do next is the most expensive bottleneck in modern experimentation programs. According to Optimizely's analysis of more than 127,000 experiments, roughly 12% of tests produce a statistically significant win on their primary metric^[2] — which means about 88% require careful interpretation to extract any value at all. Your specific win rate depends on design rigor and product maturity. The interpretation problem applies regardless.

The structural read: most teams optimize the wrong constraint. They invest in running more tests when the binding constraint is interpreting the ones they already shipped.

Spotify's engineering team named the failure mode and built around it. Their Experiments with Learning (EwL) framework reports that roughly 64% of experiments produce actionable learning^[1] — even when the test does not "win." Caught regressions and well-powered neutral results carry real business signal. By 2025, 58 teams were running 520 experiments on Spotify's mobile home screen, averaging 10 new experiments every week — and EwL learning rates across those teams ranged from 16% to 76%^[1]. The teams at the higher end are not running more tests. They invest in adequate statistical power and bold-enough implementations to produce unambiguous answers, and they treat every concluded result as an input to the next hypothesis, not a closed chapter.

Capturing that signal requires every result to land inside a context that most teams do not maintain. That is the role of what we are calling the Experiment Interpreter — an agent wedged between your experimentation platform and your strategy process, doing the cross-referencing humans skip.

The agent's value compounds or collapses on one decision: the quality of the related experiments it retrieves. Pull irrelevant tests and the debrief turns into noise. Miss the critical predecessor and you repeat the mistake. The retrieval query is the hardest engineering problem in this system — and the one most teams hand-wave past on the way to building the LLM call.

Looking at how Statsig, Optimizely, and GrowthBook handle experiment metadata, a workable retrieval strategy combines five dimensions of relatedness. Weight them deliberately.

The weighting is load-bearing. Two experiments that share the same feature area and the same primary metric are almost certainly related, even if they ran a year apart. Two experiments that ran the same week on different parts of the product almost certainly are not.

In practice, the retrieval query works as a hybrid: a structured filter on feature area and metrics to find candidates, then semantic similarity on hypothesis descriptions to rank them. SQL WHERE clauses for the hard filters. Vector search for the fuzzy matching. Treat the structured filter as the primary mechanism. Treat embeddings as a re-ranker.

One concrete gotcha: hypothesis text is usually too short and too formulaic to carry much semantic signal on its own. Most teams write hypotheses as ritual — "We believe that X will cause Y" fills in the template. Embeddings trained on that corpus cluster on surface form, not conceptual relatedness. The fix is to embed the combination of hypothesis + feature area description + primary metric name. That compound representation has enough signal to meaningfully re-rank the candidate list the structured filter returns.

A debrief that says "the green button outperformed the blue one by 4.2%" is not analysis. It is paraphrase. The strategist needs three things: why the result happened, what it confirms or contradicts from prior tests, and where the team should put the next dollar.

The Interpreter builds causal narratives by triangulating three evidence layers.

The operative word in causal inference is triangulation. No single experiment proves a mechanism. When multiple lines of evidence converge on the same explanation, confidence rises. The agent's job is to do that triangulation automatically, every time, without requiring someone to remember.

A concrete read. The pricing page experiment showed a 7% lift in trial signups when the annual plan moved to the left column. A naive debrief stops there. The Interpreter retrieves three related tests from the past six months: a test that made the annual plan visually larger (3% lift), a test that added a "most popular" badge to the annual plan (5% lift), and a test that changed the annual price (neutral). The pattern resolves: users respond to visual prominence of the annual option, not to price. That mechanism reshapes what gets tested next.

A failure mode we hit early. The agent triangulated a confident causal story from three experiments that all ran during a holiday season. Seasonal effect was the real driver. Adding release-event cross-referencing — and flagging calendar anomalies as candidate confounds — caught this class of error. Drift toward over-confident causal claims is the default state when the agent has no view of the calendar.

Growth teams at any throughput run dozens of experiments in parallel. Statsig has written extensively on the tension between experiment velocity and interaction risk^[3]. The pragmatic consensus: overlap is fine most of the time, and the rare interactions that do occur produce wildly wrong conclusions if nobody is looking for them.

Interaction effects appear when the combined impact of two experiments diverges from the sum of their individual effects^[4]. If experiment A lifts conversion by 3% and experiment B lifts it by 2%, an additive world expects 5%. The interaction might produce 8% — or 0%. The failure mode is not frequency. It is invisibility. Interactions do not announce themselves.

The practical scale of the problem: at 400+ experiments per year — the throughput Statsig reports from customers like Whatnot^[8] — you have 160,000 possible experiment pairs. Even a 0.1% interaction rate produces 160 collisions a year, each one silently corrupting two reads.

The Interpreter checks three dimensions of interaction risk for every active pair: audience overlap (are the same users in both?), metric collision (do both target the same primary metric?), and feature adjacency (do the changes touch related parts of the experience?). When all three score high, the pair gets flagged and the debrief carries an interaction analysis. The check runs by default. The check is not optional.

One implementation note: the audience overlap check is the cheapest to compute and the most reliable early signal. Run it first. If overlap is under 5%, the other two checks rarely matter — the exposed populations are too small for the interaction to contaminate either read meaningfully.

Recommendations are where the Interpreter shifts register. Reporting becomes proposing. The mechanism: identify the highest-value knowledge gaps surfaced by the debrief, and rank candidate next tests against the team's roadmap.

The scoring weighs four factors.

Back to the pricing-page case. After the Interpreter resolves that visual prominence — not price — drives annual plan adoption, it proposes three follow-ups. First, a comparison table that visually favors the annual plan, to test whether the mechanism amplifies. Second, the same prominence treatment on the upgrade page for existing free users, to test whether the mechanism transfers. Third, removing the monthly option entirely, to test whether choice reduction helps or hurts.

Each recommendation carries a hypothesis grounded in the historical evidence, a predicted effect range, and a list of metrics to track. The team can evaluate and launch the next experiment within hours of reading the debrief — not days inside a planning meeting that exists to make planning feel like progress.

The expected-impact estimate deserves more precision than most implementations give it. "High impact" is not a score. The agent should derive an effect-size range from the distribution of related historical results — something like "prior prominence experiments have shown 3–7% lift in this feature area" — and flag when the current hypothesis has no historical analog to draw from. Unanchored predictions are speculation with a confident interface.

The Interpreter runs as an event-driven agent. Your experimentation platform emits a webhook when a test hits its stopping criteria. The agent picks up the event and orchestrates a pipeline: result ingestion, context retrieval, release event cross-reference, interaction check, debrief generation, recommendation scoring. Stateless. Restartable. No new platform.

Most experimentation platforms — Statsig, LaunchDarkly, Optimizely, GrowthBook — emit webhooks or expose API polling for status changes. The agent runs as a serverless function on the event, with a vector database for hypothesis embeddings and a structured store for experiment metadata. The whole topology is unremarkable. That is the point.

You shipped the agent. Now prove it changed something. The metrics that matter are not about the agent's speed. They are about decision quality and learning velocity on the team that depends on it.

Spotify's read is instructive. Their shift from measuring experiment velocity to measuring learning rate exposed the same pattern across programs: the quality of insight per experiment matters more than the count of experiments shipped^[1]. The EwL framework defines a "successful" experiment as one that is both statistically valid and decision-ready — meaning it definitively supports one of three actions: ship, abort, or iterate. Everything that does not clear that bar is a failed experiment regardless of statistical significance.

The Kameleoon and Speero joint research on mature programs found that businesses with mature testing programs are 69% more likely to achieve significant growth than programs still optimizing for throughput^[9]. The specific mechanisms cited: shared metric definitions across teams, post-experiment documentation that feeds future hypotheses, and active cross-team experiment review. The Interpreter automates those last two mechanisms directly.

Practitioners cited by GrowthMethod report that growth teams measuring learning velocity — validated insights per quarter relative to experiment duration — outperform teams optimizing purely for experiment throughput^[6]. The Interpreter compresses time-from-result-to-insight from days to minutes. The realized gain depends on retrieval quality and team adoption, neither of which arrive automatically.

The Interpreter changes how growth teams operate at a structural level. Each A/B test stops being an isolated event — run, read, move on — and starts feeding through a lens of accumulated organizational memory. Related experiments surface automatically. Release events get cross-referenced without anyone remembering to check. Interaction risks get flagged before they corrupt the data.

The hardest part is not the agent. It is designing the context retrieval query that defines what "related" means inside your specific product and team. Get that wrong and the system produces confident noise. Get it right — and the question of how the team interpreted experiments without it stops having a comfortable answer.