Experimentation · Cheat Sheet
Running Tests Well
Cheatsheet: Running Tests
Power: the four moving parts
Fix any three, solve the fourth:
| Quantity | Convention | Notes |
|---|---|---|
| Alpha | 0.05 | False-alarm rate; stricter for irreversible calls |
| Power | 0.80 | P(detect effect if real at MDE); 0.80 still misses 1 in 5 — use 0.90 for big bets |
| MDE (effect size) | From the business | Smallest effect that would CHANGE THE DECISION ("must beat incumbent by 10%") |
| n per arm | Solve for it | Or invert: given available traffic, what MDE can we see? |
Two-proportion n per arm ≈ (z_α/2 + z_pow)² × [p1(1−p1) + p2(1−p2)] / (p1 − p2)²
(1.96 + 0.84)² ≈ 7.84 at alpha .05 / power .80Symbols: z_α/2 and z_pow are z-table lookup values (a calculator or the AI supplies them; you never memorize them); ^ = to the power of; ² = squared. You never compute this by hand — a sample-size calculator does.
Scaling law: n grows with the inverse square of effect size — halve the MDE, 4x the sample; tenth the MDE, 100x. A large gap (20% vs 25% conversion) needs little data; a small one (20% vs 20.5%) needs a vastly larger sample.
Reality check: detecting a 10% relative lift on a 1.1% CTR ≈ 148,000 impressions per arm. 300 per arm ≈ 3 clicks expected — a result you can't distinguish from random noise.
Folk rules debunked: "1,000 visitors" / "one week" ignore base rate, variance, and MDE. A week matters for seasonal coverage, never for sample size. "Until it looks stable" = peeking.
Peeking and snooping
| Behavior | What it really is | Cost |
|---|---|---|
| Stop when p < 0.05 mid-test | Selecting the random walk's luckiest moment | ~14 daily peeks → real false-alarm rate ~20-25%, not 5% |
| Extend a flat test "one more week" | Peeking's mirror image | Same voided guarantee |
| Slice flat result until a cut wins | Vast search effect; many implicit tests reported as one | 12 subgroups → ~46% chance of a fake win |
"Torture the data long enough and it will confess" — and the confession is worthless.
Disciplined alternatives: fixed horizon analyzed once; pre-registered group-sequential plans (alpha budgeted across scheduled looks); always-valid inference; bandits when optimizing beats proving. Exploration is fine — as hypothesis generation, confirmed in FRESH data.
Multiple comparisons
P(at least one false positive) = 1 − 0.95^K
K = 12 → 46% | K = 20 → 64% | K = 100 → 99.4%| Error rate | Protects | Tool | Use when |
|---|---|---|---|
| Per-comparison | Each test alone | None | Independent one-off questions |
| Familywise (FWER) | Zero false claims | Bonferroni: test at alpha/K | Few, high-stakes comparisons |
| FDR | Fraction of wins that are false | Benjamini-Hochberg: p(i) ≤ (i/K)·alpha | Screening many candidates |
- Subgroup-fishing tells: oddly specific, unpredicted, no mechanism ("unmarried women under 30")
- Bayer replicated only ~14 of 67 published findings — uncorrected multiplicity at scale
- Corrections handle counted comparisons only; snooped results need fresh data
- An omnibus (F-test) pass is not a license to cherry-pick the biggest pairwise gap
Pre-launch checklist
Before a single user is bucketed, confirm:
- One decision metric named in the test plan; guardrails and secondaries listed separately
- MDE agreed with the decision-owner: the smallest lift that changes the call
- Power calc run: n per arm at alpha/power; for binary metrics, sized on events, not rows
- Expected CI width narrower than the ship/no-ship threshold gap
- Horizon fixed (or a sequential plan pre-registered); covers ≥ 1 full weekly cycle; no anomalous windows (holidays, launches)
- Randomization unit matches treatment experience; analysis planned at that unit
- Assignment mechanism named (hash/RNG — not alternation or convenience); A/A test or SRM alert in place
- Arms run simultaneously, identical infrastructure and logging; instrumentation path verified in both arms
- Planned comparisons and subgroups pre-listed, with the correction (Bonferroni/BH) chosen
- Stopping rule written down — and "the dashboard turned green" is not one
- Ethics pass: no emotion manipulation, withheld help, or price discrimination without review
- Relaunch expectation set: winner's readout will be haircut for selection before any forecast
Aligned to the Oehlert’s A First Course in Design and Analysis of Experiments and Bruce & Bruce’s Practical Statistics · last reviewed 2026-07-08.
Make this cheat sheet yours
Personalize this sheet — focus it however you study, or build one from the exact questions you keep getting wrong.