Causal Inference · Cheat Sheet
When You Can’t Randomize
Cheatsheet: Quasi-Experimental Methods
Five ways to get causal answers when you can't randomize. Every one trades the clean guarantee of an A/B test for an untestable assumption. The skill is knowing which assumption you're buying, what diagnostics can probe it, and when to walk away.
Master table
| Method | The trick | Untestable assumption | What you CAN probe | Canonical study | Business use |
|---|---|---|---|---|---|
| Matching / propensity scores | Build statistical twins: pair each treated unit with untreated units that look the same on observed covariates | Selection on observables — nothing unmeasured drives both treatment and outcome | Overlap (do comparable untreated units exist?), covariate balance after matching | Titanic class-survival; LaLonde/NSW job-training reanalysis | Look-alike account comparisons: adopters vs matched non-adopters when only observational data exists |
| Difference-in-differences (DiD) | Borrow the control group's trend: effect = (treated change) − (control change) | Parallel trends — absent treatment, both groups would have moved alike | Pre-period trends (did they move together before?), placebo periods and placebo outcomes | Card-Krueger minimum wage (New Jersey vs neighboring Pennsylvania); geo-holdout ad campaigns (launch city vs holdout) | Launch-vs-holdout markets: feature or campaign in some regions, compare change against untouched regions |
| Regression discontinuity (RDD) | Near-twins at a threshold: compare units just above vs just below a cutoff that assigns treatment | Continuity — nothing else jumps at the cutoff | Bunching/manipulation (McCrary density), covariate balance at the cutoff, placebo cutoffs, bandwidth robustness | Drinking age at 21 → mortality jump; Hoekstra flagship admission → ~10% earnings; Clark-Martorell diploma null | Free-shipping minimums, loyalty-tier cutoffs, lead-score routing, credit limits — any score-based rule |
| Instrumental variables (IV) | Find a lever that moves treatment but touches the outcome only through it; effect = (lever's effect on outcome) ÷ (lever's effect on treatment) | Exclusion restriction — the lever has no other path to the outcome | Relevance/first-stage strength (weak levers make estimates explode), plus hostile attack on the exclusion story | Oregon Medicaid lottery; Angrist-Krueger quarter-of-birth → schooling → earnings | Encouragement designs: randomize the nudge (prompt, email), not the feature; measures compliers only |
| Synthetic control | Build a weighted blend of untreated donors that tracks the treated unit's pre-period; the blend's post-path is the counterfactual | Treated unit would have kept tracking its synthetic twin absent treatment | Pre-period fit quality, in-time placebos, placebo runs on every donor, multiple specifications, leave-one-out donors | California Prop 99 tobacco tax (~25 fewer packs per capita, placebo p ≈ 0.026); Cunningham's Texas prisons | Single-market launches, one-region pricing changes, one-country rollouts — "we launched in Austin and Austin grew" claims |
Sources: Cunningham's Causal Inference: The Mixtape, Facure's Causal Inference for the Brave and True, Shalizi's lectures, and the CausalML book.
Which design fits my situation?
Work down this list — it's keyed on what randomness or structure already exists in how the treatment got assigned:
- A threshold or cutoff rule assigns treatment (score ≥ X, spend ≥ $Y, age ≥ N) → RDD. The most credible option on this list when the cutoff isn't gameable. Remember the estimate is local to the marginal unit at the line.
- A lottery, random nudge, or arbitrary lever pushed people toward treatment → IV. Strongest when you randomized the encouragement yourself. Check the first stage is strong; then attack your own exclusion story before anyone else does. The answer describes compliers only.
- Exactly one treated unit (city, market, country), many untreated ones, long pre-history → Synthetic control. Demand good pre-period fit and run placebos on every donor before quoting the gap.
- A group got treated at a known time, and a comparable group didn't → DiD. Plot the pre-trends first — if the groups weren't moving together before treatment, the borrowed trend is a fiction.
- None of the above — all you have is rich covariates on who chose treatment → Matching. The weakest claim on this list: it only removes confounding you measured. Check overlap and balance, present results as "adjusted comparison," and resist causal verbs.
Two cross-cutting rules:
- The picture is the argument. RDD's jump at the cutoff, DiD's parallel pre-trends, synthetic control's pre-period lockstep — if the key chart doesn't persuade, no coefficient table will rescue it.
- Every method's headline assumption is untestable. Diagnostics probe the perimeter; the core is always a story about the world. Write the story down explicitly, and invite someone to attack it.
The escalation rule
These designs are enough for most product and marketing decisions. But when the decision is big, unrandomizable, and irreversible — a pricing overhaul, a market exit, a regulatory commitment — don't stretch a cheatsheet across it. Bring in a causal-inference specialist. The methods above have failure modes (weak instruments, gamed cutoffs, spec-searched synthetic gaps) that are cheap to catch in review and expensive to discover in production. Knowing when you're past your depth is part of the skill.
Aligned to the Cunningham’s Mixtape, Facure’s Brave and True, Shalizi’s ADAfaEPoV, and Chernozhukov et al.’s CausalML · last reviewed 2026-07-08.
Make this cheat sheet yours
Personalize this sheet — focus it however you study, or build one from the exact questions you keep getting wrong.