Why Randomize

The one idea: Assigning the treatment by a coin flip is what turns "these things are linked" into "this caused that." Randomizing balances every hidden difference between the groups at once — even the ones nobody thought to measure.

⚠️ The trap: We adjusted for age, tenure, plan tier, and a dozen engagement metrics, so this comparison is as good as a real experiment. That feels right and it's wrong. Adjustment only balances the confounders you thought to measure; only a coin flip also balances the ones nobody in the room has ever heard of, and no regression buys that back afterward.

Here's a headline-style finding: children who are spanked go on to have lower IQs than children who aren't. Suppose the data is real and the gap is large. What did we learn — that spanking harms cognitive development? Or that households under stress (poverty, chaos, less parental time) both spank more and provide fewer of the things that build IQ? Or that children who are harder to manage for pre-existing reasons get spanked more? From the data alone, you cannot tell. Oehlert opens A First Course in Design and Analysis of Experiments with exactly this ambiguity, because it draws the line the whole subject rests on.

The line is this: in an observational study, the world decides who gets the treatment. In an experiment, you decide — and you decide with a coin flip. That single difference in who controls assignment is what separates "these things are associated, for one of a dozen possible reasons" from "this treatment caused this effect."

The Causal Inference track covers what to do when you can't flip the coin —…