Experimentation · Cheat Sheet
Beyond the Basic A/B
Cheatsheet: Advanced
Factorials
| Term | Plain words |
|---|---|
| Factorial design | Test all combinations of several factors at once (headline × image × price) |
| Main effect | A factor's average effect across the other factors' levels |
| Interaction | The factor's effect depends on another factor — the whole ≠ sum of parts |
| Hidden replication | Every subject informs every factor's estimate — cheaper than separate tests |
Oehlert: use factorials — a win either way. Interaction present → one-at-a-time is misleading (and can't see the best cell). Interaction absent → factorial is more efficient than separate tests.
Worked 2×2 reference
| Old image | New image | |
|---|---|---|
| Old headline | 4.0% | 4.4% |
| New headline | 4.6% | 6.0% |
Headline main effect +1.1pp; image +0.9pp; headline worth +0.6pp with old image but +1.6pp with new → interaction of +1.0pp. One-at-a-time never observes the 6.0% cell.
Interaction plots
- Parallel lines = no interaction (effects add)
- Diverging/crossing = interaction; crossing = helps segment X, hurts segment Y (chick-weight pattern) → think targeting, not ship/kill
- Judge non-parallelism against noise via the ANOVA (analysis of variance) interaction test — never by eye
- Pure interaction (two catalysts: each alone inert, together vigorous) defeats one-at-a-time testing entirely
- ANOVA = omnibus + decomposition (between- vs within-group variation, split into mains + interactions); a significant F is not a cherry-picking license
Bandits
| A/B test | Bandit | |
|---|---|---|
| Goal | Prove (error-controlled verdict) | Optimize (earn while learning) |
| Allocation | Fixed to horizon | Shifts toward winners |
| Best for | Permanent, high-stakes ship calls | 3+ arms, perishable decisions (headlines, promos) |
| Effect estimates | Unbiased, powered | Distorted by adaptive allocation — don't dual-purpose |
- Epsilon-greedy: exploit best-so-far with prob 1−ε, explore randomly with prob ε (epsilon — the fraction of traffic set aside for random exploration); ε=0 can lock onto an early-lucky loser
- Thompson sampling: route traffic by each arm's probability of being best; exploration fades automatically as certainty grows
When the standard A/B breaks
| Situation | Why it breaks | Escape design | Watch out for |
|---|---|---|---|
| Marketplaces, social, shared inventory | Treatment leaks to control (interference) | Cluster / geo randomization | n = number of clusters (40 cities = n 40) |
| Same, when clusters are few | Not enough geos | Switchbacks (randomize time blocks) | Carryover — use washout gaps; ≈ crossover designs |
| Tiny samples (8 warehouses, 12 accounts) | Large-sample approximations fail | Permutation / Fisher exact tests | Power is still tiny; exact ≠ powerful |
| Long-horizon outcomes (LTV = lifetime value, annual retention) | Forced onto fast proxies | Validate proxies; long-term holdouts | CAST redux: fixing the surrogate while harming the real outcome |
| Factor only applies at coarse grain (warehouse layout) + user-grain factors | Two different error scales | Split-plot designs (Oehlert) | Naive user-grain analysis repeats the unit error |
Escalation rule
Standard playbook (powered 2-arm, fixed horizon, pre-registered metric): run it yourself. High-stakes + non-standard design (marketplace interference, carryover-prone switchbacks, split-plot structure): bring in an experimentation specialist before data collection — design errors, unlike analysis errors, cannot be fixed afterward.
Cross-track pointers
- Potential outcomes, matching, DiD, IV, and everything for when you can't randomize → Causal Inference track
- General inference logic (distributions, estimation, the p-value in full) → Statistics track
Aligned to the Oehlert’s A First Course in Design and Analysis of Experiments and Bruce & Bruce’s Practical Statistics · last reviewed 2026-07-08.
Make this cheat sheet yours
Personalize this sheet — focus it however you study, or build one from the exact questions you keep getting wrong.