P Values What They Arent

P-Values: What They Aren't

The one idea: A small p-value tells you that pure chance would rarely produce a result this extreme — not that your result is probably real. Turning it into "the effect is real" needs something the p-value doesn't contain: how likely the effect was to begin with.

⚠️ The trap: p = 0.03, so there's only a 3% chance this was a fluke and a 97% chance the effect is real — ship it. That feels right and it's wrong. The p-value is how often pure chance fakes a result this big when the true effect is zero; turning it into "probably real" needs the base rate of good ideas, which the number never contains.

No number in analytics is quoted more often and understood less than the p-value. Ask a room of smart product people what "p = 0.03" means and most will say some version of: there's a 3% chance this result is due to chance — equivalently, a 97% chance the effect is real. That is what everyone wants the p-value to mean. It is not what it means, and the gap between the wish and the reality is where careers' worth of bad ship decisions live.

Bruce and Bruce, in Practical Statistics for Data Scientists, are unusually blunt about this: the p-value cannot deliver the "probability my result is real" number people crave, and much of the ritual around significance testing survives because people believe it delivers exactly that. This lesson is about what the number actually says, what alpha actually is, and how to use both like an adult.

The wish versus the reality

The wish: "p…