External Validity and Generalizing Results

The one idea: A well-run experiment answers two separate questions, and it's easy to only notice you got one of them. Internal validity asks "did I correctly measure the effect in this sample?" External validity asks "does that effect hold anywhere else?" A test can score perfectly on the first and fail completely on the second.

⚠️ The trap: "The test was randomized and the result was significant, so we can roll this out company-wide at that lift." That feels right and it's wrong. A perfectly-run experiment on early adopters can fail to generalize to your mainstream user base at all — not just shrink, but land near zero or reverse — because the people who volunteer first are systematically different from the people who show up later.

Two lessons ago you learned what randomization buys you: internal validity, certified. The treated-minus-control gap in your sample is the true effect for your sample, no debate. That lesson also flagged, in passing, that an experiment "certifies the effect for the people, time, and conditions it ran on — nothing more." This lesson stays on that second half and takes it seriously, because it's where clean experiments quietly stop being useful without anyone noticing the switch.

Two validities, two different failure modes

Internal validity is about the comparison inside the experiment: did treatment and control actually differ only in treatment status, and did you measure the gap correctly? Randomization, a clean rollout mechanism, and correct analysis (analyze-as-randomized, no post-hoc filtering) are what buy internal validity. When people say "the experiment was rigorous," this is usually what they mean.

External validity is about the…