Peeking and Data Snooping

The one idea: If you let the results decide when to stop a test or which slice of the data to report, you will find "wins" that are pure luck. Decide your stopping point and your comparisons before you look, not after.

⚠️ The trap: It already crossed p < 0.05 on day 4 — let's just ship it now. That feels right and it's wrong. A running p-value dips below 0.05 by chance even when nothing works, so stopping at the first dip harvests luck and inflates false positives several-fold; hold to the horizon you fixed before launch.

Day 4 of your two-week test, and the dashboard shows variant B up 8%, p = 0.03. Someone posts a screenshot in Slack: "It's already significant — can we just ship it?" This is the most tempting moment in all of experimentation, and giving in to it is one of the two habits this lesson is about. The other is its quieter sibling: after a test ends flat, slicing the data — by country, device, weekday, user tenure — until some cut shows a win, then reporting that cut.

Both habits feel like diligence. Both are the same statistical crime: letting the data decide which comparison gets reported. Lesson 1 warned that randomness produces streaks; these are the two main ways analysts hand those streaks a microphone.

Stopping when it looks good is selection bias

Here's the mechanism, stripped bare. A p-value wanders as data accumulates — it's a running statistic of a noisy process, and like any random walk it wiggles. Even when the treatment does nothing, the running p-value will occasionally…