Confidence Intervals and the Bootstrap

The one idea: Report a plausible range for the effect, not just a yes/no verdict — a range tells people both what you learned and how much you still don't know. The bootstrap builds that range for almost any metric by resampling your own data, and you can size a test in advance so the range is tight enough to actually decide.

⚠️ The trap: The confidence interval includes zero, so the result isn't significant — kill the variant. That feels right and it's wrong. A range like [−0.2pp, +1.4pp] is compatible with a large win and nearly incompatible with a real loss, so read the interval's shape and extend the test rather than auto-killing on the verdict alone.

A significance test returns a verdict: chance-compatible or not. But your stakeholders don't actually need a verdict — they need a number they can plan with. "The new checkout lifts conversion" is a verdict. "The lift is somewhere between 0.5% and 6%, most likely around 3%" is a planning input: finance can model the revenue range, and everyone can see at a glance both what you learned and how much you still don't know.

That second style of readout is the confidence interval, and this lesson makes the case that it — not the p-value — should be the headline of every experiment report. It also introduces the tool that makes intervals available for any metric with no formulas: the bootstrap, which Bruce and Bruce in Practical Statistics for Data Scientists treat as the workhorse of modern uncertainty estimation.

Effect sizes with uncertainty, not verdicts

Recall from lesson 6 what a p-value…