Anova and Comparing Multiple Groups

The one idea: Comparing two groups, use a t-test. Comparing three or more groups, do not just run a t-test on every pair — that quietly stacks up your false-positive risk until "5% chance of being wrong" becomes a coin flip. ANOVA (Analysis of Variance) tests all the groups at once by comparing how much the group averages spread out from each other against how much individual scores naturally spread out within each group.

⚠️ The trap: "We're comparing conversion rates across 5 landing page variants, so I'll just t-test each pair against the control at α=.05 — that's the standard we always use." Standard for one comparison, not for ten of them. Run every pairwise t-test at α=.05 across enough groups and the chance that at least one comes back "significant" purely by chance climbs past 40% — you're no longer testing whether the pages differ, you're running a lottery for a false positive and calling the winner real.

Why you can't just run a bunch of t-tests

Each individual t-test carries its own 5% (α = .05) risk of a Type I error — a false "significant" result. That 5% sounds small in isolation. But run several of those tests on data drawn from the same experiment, and the risks don't stay isolated — they compound, because each additional test is another roll of the dice for a false alarm.

This distinction has a name: the comparisonwise Type I error rate is the α for one single pairwise comparison (.05). The experimentwise (overall) Type I error rate is the probability of making at least one Type I error across…