Back to Experimentation
Experimentation · Cheat Sheet
Why We Experiment
Tip: Use your browser's print function (Ctrl+P / Cmd+P) to save as PDF for offline study.
Cheatsheet: Foundations
Randomness intuitions worth memorizing
| Fact | Number | Why it matters |
|---|---|---|
| Runs in 100 real coin flips | Streaks of 5-6 heads are normal | Real noise is streakier than intuition; don't narrate streaks |
| Stadium of 20,000 flippers, 10 flips each | ~20 people flip 10 straight heads (20000/1024) | With many metrics, spectacular flukes are guaranteed |
| Five above-average days in a row | ~3% per metric-week — routine across a dashboard | Price the streak before explaining it |
| Winner of a noisy multi-variant test | Systematically overrated | Expect relaunch below the test number (regression to the mean) |
Regression to the mean
- Anything selected for being extreme (rookie of the year, winning variant) = skill + good luck; only skill repeats
- Bigger haircut when: smaller sample, noisier metric, more variants compared
- Fix: confirm winners in a fresh two-arm test; never forecast from the selected readout
Bias vs random error
| Random error (noise) | Bias | |
|---|---|---|
| Direction | Symmetric, averages out | Systematic, one direction |
| More data | Shrinks with sqrt(n) | Does nothing |
| Detectable from inside the sample | Yes — quantify with SE | No — reason about selection mechanism |
| Canonical example | Sample-to-sample wobble | Literary Digest: 2.4M biased ballots lost to Gallup's ~2,000 random |
Data quality beats quantity (Bruce & Bruce). Self-selection is the everyday bias: Yelp reviews, NPS responders, beta opt-ins.
Key formulas
SE of a proportion: SE = sqrt( p(1-p) / n )
SE of a difference: SE_diff = sqrt( SE_A^2 + SE_B^2 )
Noise shrinks like 1/sqrt(n): halving noise costs 4x the dataSymbols: sqrt = square root; ^ = to the power of; n = number of data points; p = the proportion. You never compute these by hand — a calculator or one line of code does.
CLT: averages of many observations are approximately bell-shaped even when raw data is skewed — the license behind classical inference.
Experiment vs observational study
| Experiment | Observational | |
|---|---|---|
| Who assigns treatment | You, by a random mechanism | The world (self-selection, policy) |
| Confounders | All neutralized in expectation — including unknown ones | Only measured ones adjustable |
| Claim supported | Causation | Association (+ assumptions) |
- "Randomization is somewhat like insurance" (Cochran & Cox)
- Haphazard ≠ random: alternation, convenience, "no particular rule" — if you can't name the RNG/hash, assume bias
- Randomize everything that can leak: the $75k soil study died because analysis day was confounded with site — walk the whole path from assignment to logged number
- Balance is guaranteed in expectation, not per draw — check pre-treatment covariates; extreme/recurring imbalance = broken mechanism
A/B test anatomy
| Part | Rule |
|---|---|
| Decision metric | Exactly ONE, chosen BEFORE launch; post-hoc metric choice "opens the door to researcher bias" (Bruce & Bruce) |
| Guardrails/secondary | Watch freely; they never make the ship call |
| Binary metrics | Effective n ≈ number of events: 45,000 sessions at 0.85% = ~382 conversions of information |
| Continuous metrics | Skew and whales inflate variance — decide winsorization before launch |
| Proxy metrics | Hypotheses, not truths — CAST: drug fixed the surrogate heartbeat, increased deaths; validate against the true outcome |
| Randomization unit | Match how the treatment is experienced; analyze at the unit randomized |
| Ethics | Emotion manipulation, withheld help, price discrimination → formal review (Facebook 2014) |
Red-flag phrases in readouts
- "Conversion is up six weeks straight" — priced against how many metrics you watch?
- "We surveyed our users" — who selected into responding?
- "We controlled for everything" — not the unknowns; only randomization insures those
- "Revenue was the metric we cared about" (said in week 3) — was it pre-registered?
Aligned to the Oehlert’s A First Course in Design and Analysis of Experiments and Bruce & Bruce’s Practical Statistics · last reviewed 2026-07-08.
Make this cheat sheet yours
Personalize this sheet — focus it however you study, or build one from the exact questions you keep getting wrong.
Sign up free to create a personalized cheat sheet.