Sequential Testing and Always Valid P Values

The one idea: Fixed-horizon testing breaks under peeking not because peeking is inherently sinful, but because the math behind the classic p-value assumes exactly one look. Sequential testing methods change the math itself — spending or budgeting the false-positive rate across looks — so continuous monitoring is safe by construction, not by willpower.

⚠️ The trap: We're using a sequential testing platform now, so I can peek whenever I want and stop the moment the dashboard shows p < 0.05. That feels right and it's wrong. A sequential method only controls the false-positive rate for the specific stopping rule it was designed for — mixing a classic z-test's p-value with continuous peeking is still the same broken procedure, badge or no badge.

Peeking and Data Snooping diagnosed the problem in full: a running p-value wanders as data accumulates, dips below 0.05 by chance even under a true null, and "stop the moment it's significant" harvests those dips — turning a nominal 5% false-positive rate into something several times worse. That lesson's prescription was mostly discipline: fix the horizon, don't peek at the decision metric, or use a sequential design if you plan for it in advance. This lesson is the "plan for it in advance" part — what a sequential method actually does differently, and why it's a real answer rather than a workaround.

Why the classic p-value can't survive peeking

A classic (fixed-horizon) p-value answers one specific question: "if the null is true, what's the probability of a result this extreme at this one, pre-planned sample size?" The 5% false-positive guarantee is a promise about a single scheduled look. Nothing…