Holdback Tests and Long Term Effects

The one idea: A two-week A/B test measures a two-week effect — and some real effects only show up over months, either because novelty fades and reveals the true steady state, or because small changes compound (habit formation, network effects, trust erosion) in ways a short window can't see. A holdback — a small slice of users kept on the old experience indefinitely — is how you find out what actually happens in the long run.

⚠️ The trap: We shipped the new onboarding flow to everyone three months ago, the two-week test showed a clean win, so we're done. That feels right and it's wrong. Shipping to 100% destroys your ability to ever again measure "old experience vs. new" — without a small holdback kept on the old flow, you can never check whether that two-week win held up, decayed, or quietly reversed.

Bandits, and When A/B Fails named long-horizon metrics as one of the standard-test's structural failure modes and pointed at the fix in passing: "maintain long-term holdouts — a small percentage of users kept on the old experience for months as ground truth against surrogate-chasing drift." Validity Threats in the Wild separately covered novelty and primacy effects — early readouts that lie in both directions because users react to newness itself before settling into a steady state. This lesson connects those two threads and builds out the holdback pattern in full: why short tests systematically miss long-run truth, and how to design a holdback that actually catches it.

Two different ways a short test misses the long-run answer

Novelty and primacy decay (Validity Threats in the Wild's territory):…