Permutation Tests and the NULL

The one idea: To judge whether a difference between two groups is real, pretend the labels mean nothing: shuffle them thousands of times and see how often chance alone produces a gap as big as the one you observed. If chance produces it often, your result is nothing special.

⚠️ The trap: B beat A by 21 seconds — that's a big gap, so something real happened. That feels right and it's wrong. The question is never how big the gap is but how big a gap chance produces at this sample size and variance; a large gap in a small, noisy sample is exactly what pure shuffling generates for free.

Your A/B test just finished. Variant B's average session time beats variant A's by 21 seconds. Someone asks the only question that matters: "is that real?" What they mean is: if the variants actually made no difference at all, could a gap this size show up anyway, just from which users happened to land in which bucket?

Here is the beautiful thing: you can answer that question almost without theory. No distributional formulas, no table lookups — just a deck of cards and a loop. The technique is the permutation test, and Bruce and Bruce's Practical Statistics for Data Scientists treats it as the conceptual heart of significance testing, because it makes visible what every hypothesis test is secretly doing: simulating a world where nothing works, and asking whether your result would be remarkable there.

The null hypothesis, made physical

Start with the claim to be challenged. The null hypothesis says: the treatment makes no difference — variant labels are…