Validity Threats in the Wild

The one idea: Some experiments give the wrong answer even when the statistics are flawless, because something about the setup broke the comparison. No p-value flags these — you catch them by recognizing their patterns: effects that drift over time, arms that differ in more than the treatment, and users who affect each other.

⚠️ The trap: Engagement jumped 9% the first week after the redesign — clear winner. That feels right and it's wrong. A jump that fades week over week is the novelty effect, not the treatment; plot the daily effect over time and judge the stabilized plateau, not the launch-week bump.

By this point in the track you can design a powered test, keep your hands off the peek button, and correct for multiplicity. This lesson is about the failures that survive all of that — the ones where the statistics are impeccable and the answer is still wrong, because something about how the experiment met the real world broke the comparison. These are validity threats, and they share a nasty property: no p-value, interval, or correction will flag them. You catch them by knowing their faces.

Think of this as the field guide. Five threats, each with a tell and a fix.

Novelty and primacy effects

Ship a redesigned homepage and engagement jumps 9% in week one. Is the design better — or is it new? Users poke at anything that changed: curiosity clicks, exploration of moved menus, "what's this?" taps. That's the novelty effect, and it decays as the newness wears off. Its mirror image is the primacy effect (change aversion): entrenched users are *worse off*…