Experiments and Their Limits

Cheatsheet: Experiments

Why randomization certifies

• The coin flip makes potential outcomes (Y1, Y0 — the outcome if treated / if not treated) independent of assignment — not "outcome independent of treatment." The groups would have looked the same in either world.
• Consequence: selection bias = 0 by construction, so difference in means = ATE (average treatment effect). No controls needed, no cleverness required.
• Randomization is the only design that certifies a causal estimate. Everything else (matching, DiD, IV) grades one against untestable assumptions.
• Your A/B platform is this machinery in production. Anything that breaks the coin flip breaks the certification: opt-in betas, "engaged users first" launches, dropping non-compliers or churned users post-assignment.

Balance-check list

• Randomization guarantees balance in expectation, not in your one draw. Small-sample imbalance is normal.
• Check pre-treatment covariates only (tenure, prior spend, plan, signup date). Never anything treatment could have touched.
• With 30 covariates, ~1-2 "significant" differences at 5% is what chance predicts — look for meaningful imbalance on outcome-relevant variables.
• Imbalanced strong predictor of outcome → adjust for it in regression.
• Imbalance that is extreme, or recurs across experiments → suspect the bucketing code, not bad luck (check for sample ratio mismatch).

SUTVA / spillover red flags

SUTVA: one user's treatment doesn't affect another's outcome. Ask before trusting any per-user A/B result:

• Network features — treated users message/interact with control users → control is partially treated → lift understated
• Marketplaces — treated and control compete for shared inventory, supply, or ad slots → measured lift may be cannibalization; check the total
• Viral loops — treated users invite friends who land in control → contamination
• Equilibrium effects — a 5% test runs in a market the 100% launch will change (prices, competitors, norms)
• Fixes to ask about: cluster randomization (cities/social clusters), switchbacks, market-level holdouts

Most-dangerous-equation reminder

SE = sigma / sqrt(n) — the standard error (how much a sample average bounces around) equals sigma (how spread out the data is) divided by the square root of the sample size.

• For the SE of an A/B lift (a difference between two groups), use SE = sigma·sqrt(2/n) — each arm adds its own noise, so the plain sigma/sqrt(n) understates a lift's wobble by a factor of sqrt(2).
• Small cells produce extreme averages in both directions — small schools top both the best and worst lists; small segments show the biggest "wins" and "disasters" from the same true effect.
• A dramatic lift in a tiny segment is variance until it replicates in a pre-registered follow-up.
• Overlapping group CIs (confidence intervals) can still hide a significant difference — always test the difference directly, don't judge it by eye from whether error bars touch.

External-validity questions

An experiment certifies its own population, period, and conditions. Before extrapolating, ask:

1. Who was in the test — this quarter's signups, one campaign's traffic, one platform? Who will the launch hit?
2. When did it run — promo period, seasonal peak, news cycle?
3. How long — is this novelty-effect lift that decays after week one?
4. Who was excluded — opt-outs, unsupported platforms, churned users?
5. State the certified claim narrowly ("1.8% lift among March mobile signups"); every extension is an assumption. Verify with a post-launch holdout.

Ideal-experiment heuristic

• For any causal question, first ask: "if I could, what perfect experiment would I run?" (Facure)
• It forces precision on treatment, population, and outcome — and defines the target every quasi-experimental method is approximating.
• If you can't describe the ideal experiment, no method will save the analysis.
• When you can't run it: don't give up, don't fake it with a before/after — use quasi-experimental designs with assumptions stated out loud.