Potential Outcomes

The one idea: Every customer has two possible futures — one where you act and one where you don't — but you only ever get to see one of them. So any "the treated group did better" claim is trustworthy only if the two groups were interchangeable to begin with, which they rarely are once someone has chosen who gets treated.

⚠️ The trap: "To get the effect, just compare how the treated group did against how the untreated group did." That feels right and it's wrong. You only ever see one of each person's two futures, so that comparison is causal only if the two groups were interchangeable to begin with — and the moment someone chooses who gets treated, they almost never are.

Every customer, account, and user has two futures: what happens if you treat them (send the email, run the retention program, ship the feature) and what happens if you don't. Call the first Y1 ("the outcome if treated") and the second Y0 ("the outcome if not treated"). The causal effect of your treatment on that one unit is Y1 − Y0 — the gap between the two futures.

The catch is brutal and permanent: you only ever get to see one of them. The customer either got the email or didn't. The other future — the counterfactual, the outcome that didn't happen — is gone. Scott Cunningham's Causal Inference: The Mixtape puts it bluntly: "We never know a causal effect. We only estimate it." This isn't a data-quality problem you can fix with better logging. It's the fundamental problem of causal inference, and no amount of data engineering makes it go away.

Everything…