Blocking and Covariate Adjustment

The one idea: You can get a sharper answer from the same traffic by using what you already know about your users to cancel out predictable noise — but only with facts measured before the test starts. And always count your sample size at the level you actually randomized, not the number of people you measured.

⚠️ The trap: I randomized 6 classrooms but measured 150 students, so n = 150 and this tiny p-value is rock solid. That feels right and it's wrong. Your sample size is the number of units you actually randomized — 6 classrooms, not 150 students — because students in one classroom are correlated, and counting them as independent makes the p-value confidently fake.

Randomization (lesson 3) buys you unbiasedness — your estimate is centered on the truth. It does not buy you precision. A perfectly randomized test on a wildly variable metric still produces wide confidence intervals, and lesson 8 showed the brute-force fix: more sample, at inverse-square prices. This lesson is about the smarter fix: using what you already know about your subjects to cancel noise, so the same traffic yields a tighter answer.

The core insight is that most of the variance in your metric isn't mystery — it's structure you can name. Power users spend more than casual users. Mobile converts differently from desktop. Last month's spend predicts this month's. None of that variation is caused by your treatment, and every drop of it that leaks into the treatment-vs-control comparison is noise you paid for but didn't have to.

Blocking: handle the known problems at randomization time

Blocking is the design-time version. Group…