Experimentation · Cheat Sheet
Getting More Signal
Cheatsheet: More Signal
Variance-reduction toolbox
| Technique | When applied | Requires | Gain |
|---|---|---|---|
| Blocking | At randomization | Known nuisance factor, measured pre-assignment | Removes block-to-block variance from the comparison |
| Covariate adjustment (ANCOVA) | At analysis | Pre-treatment covariate correlated with outcome | Variance × (1 − R²) — covariate's predictive power is free sample |
| CUPED | At analysis (platform) | Pre-experiment value of the same metric | 30-50% variance cuts routine on engagement metrics |
| More sample | Anytime | Traffic and patience | Noise shrinks only like 1/sqrt(n) |
Symbols: R² (R-squared) = how much of the outcome the covariate predicts, so 1 − R² is the fraction of noise left; sqrt = square root; SD = standard deviation, a measure of how spread out the data is.
Oehlert's maxim: "Block on what you know, randomize what you don't."
Blocking rules
- Blocks = groups similar on a known nuisance factor (platform, country, tier); randomize within each block
- Pre-treatment information ONLY
- Cannot be constructed after the fact — blocking is how randomization was performed; post-hoc grouping is weaker, model-dependent adjustment
Covariate adjustment rules
- Mechanism: compare arms at a common covariate value; covariate soaks predictable variance
- The poisoned covariate: anything the treatment could have affected (typing hours under the comfier keyboard) → post-treatment bias, can subtract real effect or manufacture fake ones
- Safe test: was it measured before the coin flip? If in doubt, leave it out
- Full theory (mediators, colliders) lives in the Causal Inference track
The unit rule
Experimental unit = the thing independently randomized. Analyze at that level.
| Design | Looks like n = | Real n = |
|---|---|---|
| 6 classrooms randomized, 150 students tested | 150 | 6 |
| 40 cities geo-randomized, 1M users measured | 1,000,000 | 40 |
Units inside a cluster are correlated (shared teacher, economy, weather); pretending independence gives standard errors several times too small. Aggregate to one number per experimental unit, or model the clustering explicitly.
Validity threats field guide
| Threat | Signature | Fix |
|---|---|---|
| Novelty effect | Early spike that decays | Plot daily effect; wait for plateau; check new-user segment |
| Primacy / change aversion | Early dip that recovers | Same — don't kill slow-burn winners early |
| Time confound | "A this week, B next week" | Arms simultaneous, always; the soil-lab error at product scale |
| Instrumentation | Latency or logging differs by arm | Walk the event path in both arms; run A/A tests |
| Seasonality | Weekday/weekend, holiday mix | Full weekly cycles; "take the visitors as they come," end on cycle boundaries |
| Regression to the mean | Relaunched winner underperforms | Haircut multi-way winners; confirm big bets in a fresh 2-arm test |
| Interference | Treatment leaks to control (marketplaces, social) | Cluster/geo/switchback designs — see advanced cheatsheet |
Relaunch haircut arithmetic
Winner of K variants ≈ true effect + (expected max of K noise draws). Five variants, true lift +4%, noise SD 5pp → winning readout averages ~+9.8%. The more arms and the noisier the metric, the bigger the haircut.
Readout hygiene
- Report the daily-effect time series, not just the cumulative average
- State the experimental unit and the n at that unit
- Name covariates/CUPED used and confirm all are pre-treatment
- Note any calendar anomalies inside the window
Aligned to the Oehlert’s A First Course in Design and Analysis of Experiments and Bruce & Bruce’s Practical Statistics · last reviewed 2026-07-08.
Make this cheat sheet yours
Personalize this sheet — focus it however you study, or build one from the exact questions you keep getting wrong.