Matching and Propensity Scores

The one idea: Matching compares each treated person to an untreated look-alike, but it only works if you measured every trait that drove both the choice to take the treatment and the outcome. Miss one important trait and a confident, tidy-looking comparison can point the wrong way.

⚠️ The trap: "If I match treated and untreated people on every trait I can measure, the comparison becomes causal." That feels right and it's wrong. Matching only works if you measured every trait that drove both the choice to take the treatment and the outcome — miss one (intent, severity, account health) and a tidy, confident comparison can point the wrong way, because matching estimates a comparison, it never makes one causal.

You didn't randomize, and now you need an answer anyway. Users chose to adopt the feature, customers chose to open the email, patients chose the treatment — and the choosers differ from the non-choosers in ways that pollute every naive comparison. Matching is the most intuitive response ever proposed: for every treated unit, find an untreated unit that looks just like it on everything you can measure, and compare those pairs.

It's the method business audiences trust most readily, because it sounds like common sense. That's exactly why you need to understand what it actually assumes — the common-sense packaging hides one enormous, untestable bet, and analysts who don't know they're making it produce confident numbers with the wrong sign.

The trick: statistical twins

If users self-selected into treatment, undo the selection by hand. Pair each treated user with an untreated user who has the same pre-treatment traits — same tenure, plan,…