Multiple Comparisons

The one idea: The more things you test, the more likely at least one looks like a "win" by pure chance. Before you trust any significant result, count how many comparisons you actually ran — and correct for that number.

⚠️ The trap: We tracked 20 metrics and one came back significant — that's the result. That feels right and it's wrong. With 20 true-null tests there's about a 64% chance at least one "wins" by luck, so report how many comparisons you ran and correct for that count before trusting any single winner.

Lesson 9 showed how implicit, unrecorded searching corrupts results. But even a perfectly pre-registered analysis plan can involve many comparisons on purpose: five variants against control, twelve metrics, eight segments you genuinely committed to examining in advance. Multiplicity out in the open is healthier than snooping — but the arithmetic that punishes snooping punishes honest multiplicity just the same, and this lesson is about doing that arithmetic and choosing the right correction.

Start with the number every analyst should be able to derive on a whiteboard. Run 20 independent tests at alpha = 0.05 (alpha — the false-positive rate you accept on each test, here 5%) where every null is true — nothing works. The chance any single test stays quiet is 0.95. The chance all twenty stay quiet is 0.95 multiplied by itself 20 times, where ^ means "to the power of":

0.95^20 ≈ 0.358

So the probability of at least one false positive is 1 − 0.358 ≈ 64%. Twenty honest tests of twenty duds, and you'll probably have a "discovery" to present. At 100 tests, the…