Causal Inference · Cheat Sheet
Doing It With AI
Cheatsheet: Causal Inference in the AI Era
The 5 audit prompt patterns
| Pattern | One-line template |
|---|---|
| Steelman-then-kill | "Make the strongest case this claim is causal, then the strongest case it isn't — argue both at full force." |
| Confounder brainstorm | "List 10 variables that could drive both X and Y in this setting; for each, the mechanism and how I'd check it with data we have." |
| Skeptic's DAG | "Draw the DAG this claim assumes; then a rival DAG where the true effect is zero but the data looks identical." |
| Collider check | "Here's how the dataset was assembled — what did the analysis implicitly condition on?" |
| Design-the-test | "What's the cheapest experiment or quasi-experiment that would settle this? Randomized holdout, geo split, encouragement, natural experiment, pre-trend check." |
Meeting deliverable: one sharp question + one cheap test. Not twelve hypotheticals.
Anti-patterns
- Asking "is this causal?" — the model hedges or agrees with your framing. Assign tasks, never request verdicts.
- Letting the AI's verdict replace the checklist — it brainstorms, you grade.
- Prompting only for confirmation ("explain why our campaign drove signups") — models follow framing.
- Accepting AI arithmetic or "studies show" citations unverified — audit the skeptic too.
DML (Double Machine Learning) in one sentence
Compare the surprise in the outcome to the surprise in the treatment, after your best model has predicted both from everything else — and never let the model grade its own homework (cross-fitting).
Why naive ML fails: regularization bias — methods tuned for prediction shrink exactly the coefficient you care about.
Questions to ask when shown DML/uplift output: What controls went in? Was it cross-fit? Does the targeting rule beat random on a randomized holdout?
Prediction vs effect targeting
| Outcome model | Uplift model | |
|---|---|---|
| Predicts | What will happen (purchase probability) | What the treatment changes (lift) |
| Targets | Likeliest buyers — including sure things who'd buy anyway | Persuadables — where the treatment flips the outcome |
| Coupon result | Discounts wasted on committed customers | Spend concentrated where it moves behavior |
| Validation | Accuracy, R² (model-fit score) on holdout | Uplift curves, policy value on a randomized holdout — R² is meaningless for effects |
Mantra: prediction answers what will happen; heterogeneity answers for whom it works.
The 10-question audit checklist
- Who chose the groups — randomization, self-selection, or a targeting rule?
- Bias equation: what would the treated group have done without the treatment?
- What DAG does the claim assume — and what rival DAG yields identical data with zero effect?
- How was the dataset assembled — what did the filters implicitly condition on?
- Does the effect size look like treatment or selection (2–5x gaps are selection's signature)?
- Could the arrow run backwards (reverse causation)?
- Does the projection apply a volunteers' effect to marginal recruits (LATE trap)?
- Is the arithmetic verified? (Usually right — the design is what's wrong.)
- What's the cheapest settling test, and what assumption does it lean on?
- Is the honest answer "cannot be determined with this data" — and what data would determine it?
Verdict format
- What the data supports — credit what's true.
- What's unsupported, and specifically why — name the mechanism, not the person.
- Cheapest upgrading test — with owner and timeline.
The escalation rule
Big decision + randomization genuinely impossible → bring in a causal-inference specialist. Use the AI to brief them well: DAGs drawn, confounders enumerated, candidate designs listed. And sometimes the sophisticated output is: "cannot be determined with this data — here's the data we'd need."
Aligned to the Cunningham’s Mixtape, Facure’s Brave and True, Shalizi’s ADAfaEPoV, and Chernozhukov et al.’s CausalML · last reviewed 2026-07-08.
Make this cheat sheet yours
Personalize this sheet — focus it however you study, or build one from the exact questions you keep getting wrong.