Anatomy of an Ab Test

Anatomy of an A/B Test

The one idea: Most A/B tests fail because of decisions made before launch, not because of the statistics. Decide the single metric that settles the ship/no-ship call in advance, and remember that rare events and stand-in metrics can quietly mislead you.

⚠️ The trap: I was watching conversion, revenue, average order value, and bounce rate, and one of them came out clearly up — that's our win. That feels right and it's wrong. Watching several metrics and reporting whichever moved is running several tests and quoting the luckiest one; pre-register the single metric that owns the decision before launch.

An A/B test has a deceptively short parts list: two (or more) treatments — the competing versions of the thing being tested; subjects — the users, sessions, or accounts exposed to them; random assignment of subjects to treatments; and a metric you'll use to compare the arms. One arm is usually the control — the existing experience — so the test answers "is the new thing better than what we have?" rather than "which of two novelties is less bad?"

That's the whole machine. Yet most experiment failures aren't statistical — they're failures of specification, decided before a single user was bucketed. This lesson is about the choices that happen at the design table: what to measure, when to decide that, and what can go quietly wrong even when the randomization is flawless.

Pick one decision metric — before the test starts

The single most important discipline in experimentation costs nothing and is skipped constantly: decide, in advance, the one metric that will make the ship/no-ship call.

Bruce and Bruce state the…