Fooled by Randomness

The one idea: Random chance produces streaks and amazing-looking results all on its own, especially when you are watching many things at once. Before you explain why a number jumped, ask how often plain luck would have produced it anyway.

⚠️ The trap: Conversion has climbed five days straight — the new copy is obviously working. That feels right and it's wrong. Real randomness is streaky, so work out how often pure chance would produce that run before crediting a cause — across the metrics a team watches, the answer is usually "more than once a month."

Ask someone to write down 100 imaginary coin flips, then hand them a real sequence of 100 flips, and you can almost always tell which is which. The invented sequence alternates politely — heads, tails, heads, heads, tails — because that's what randomness feels like. The real sequence contains runs of five or six heads in a row, and the person who invented the fake one would never dare write that down. Real randomness is streakier, clumpier, and weirder than human intuition allows.

This matters for experimentation because your job as an analyst is essentially to answer one question over and over: is this pattern signal, or is it the kind of streak randomness produces for free? Every A/B readout, every "conversion jumped this week," every "variant B is crushing it after two days" is that question in disguise. And the humans around you — stakeholders, executives, and yes, you — are wired to answer it wrong, in the same direction, every time: we see signal where there is only noise.

Statisticians have known this for a century, which is why…