Cuped Variance Reduction

CUPED: Variance Reduction in Practice

The one idea: Much of the noise in an outcome metric is just "some users were always going to be high or low" — a fact you already knew before the test started. CUPED subtracts out that predictable part using each user's own pre-experiment behavior, shrinking the noise without touching the treatment effect, so the same traffic detects the same lift faster.

⚠️ The trap: Our engagement metric is just noisy, so we need a bigger sample to detect anything. That feels right and it's wrong. Much of that "noise" is predictable from each user's own pre-experiment history — power users were always going to out-engage casual ones, treatment or not — and CUPED removes exactly that predictable part instead of paying for more traffic to average it out.

Blocking and Covariate Adjustment introduced CUPED (Controlled-experiment Using Pre-Experiment Data) as the industrial version of ANCOVA, specialized to the covariate that works almost everywhere: the same metric, measured on the same user, before the experiment started. That lesson stated the headline (30-50% variance reductions are routine) without walking the arithmetic. This lesson walks it — the adjustment formula, a worked numeric example with real computed numbers, and what the variance cut actually buys you.

The adjustment, in one line

For each user, you have two numbers: X, their value on the same metric during a pre-experiment window (say, the two weeks before the test launched), and Y, their value on the outcome metric during the test itself. CUPED computes an adjusted outcome:

Y_cuped = Y − θ(X − X̄)

where X̄ is the average pre-experiment value across the whole sample, and θ…