Experimentation · Cheat Sheet

Reading the Results

Tip: Use your browser's print function (Ctrl+P / Cmd+P) to save as PDF for offline study.

Cheatsheet: Core Inference

The permutation test (six steps)

Combine both groups' results into one pool
Shuffle the pool
Redeal into groups of the original sizes (without replacement)
Recompute the statistic; record it
Repeat 1,000+ times → the permutation distribution
Compare: fraction of shuffled results ≥ observed = p-value estimate

Works for ANY statistic (mean, median, percentile, custom metric)
Re-enacts the randomization itself — analysis mirrors design
Reference result: +21.4s stickiness gap on n = 36 sessions fell well inside the shuffle cloud — big-looking gaps are routine at small n

Permutation vs bootstrap

	Permutation test	Bootstrap
Resampling	Without replacement, labels shuffled	With replacement, from your own sample
Simulates	The null world (treatment does nothing)	Repeated sampling from the population
Answers	"Is this distinguishable from chance?" (verdict)	"How precise is my estimate?" (interval)
Output	p-value	Confidence interval

P-values: wish vs reality

	Statement
The wish	"Probability the result is due to chance" — P(hypothesis \| data)
The reality	"Frequency the chance model produces something this extreme" — P(data \| null)

(Read | as "given": P(A | B) is the chance of A given B.) Converting reality into the wish requires the base rate of true effects, which the p-value doesn't contain.

Base-rate arithmetic: 1,000 tests, 10% real effects, alpha 0.05, power 0.80 → winners' pile = 80 real + 45 false = 125, of which 36% are noise despite every p < 0.05.

Error types and their knobs

Error	Meaning	Controlled by
Type 1	False alarm (effect declared, none exists)	Alpha — a PRE-commitment, set before the test, matched to stakes
Type 2	Missed real effect	Power / sample size — reframe: "sample too small to detect"

"Not significant" = effect not proven, NOT "no effect" — check power before concluding there is no effect
Alpha 0.05 is convention, not physics: 0.01 for irreversible calls, 0.10 for cheap reversible ones
ASA principles: p-values measure neither truth-probability nor effect size nor importance; never decide on the threshold alone

Confidence intervals and the bootstrap

Bootstrap recipe: resample n rows with replacement → recompute statistic → repeat 1,000+ → middle 95% of results = 95% CI.

Readout style	Example	Information
Verdict	"Not significant"	1 bit, often misleading
Interval	"+0.6pp, 95% CI [−0.2, +1.4]"	Estimate + plausible range + verdict (derivable)

An interval spanning zero can still mean "compatible with a big win, nearly incompatible with a loss" → extend, don't auto-kill
The point estimate is not the answer: plans built on it alone overpromise (winners are partly lucky)
Width shrinks like 1/sqrt(n) (one divided by the square root of the sample size) → halving width costs 4x sample
Design to width: interval no wider than the decision needs (else the test can't decide), no narrower (else wasted traffic)

One-line answers for stakeholders

Question	Answer
"Is it significant?"	"The lift is X ± Y — here's what that range means for the decision"
"p = 0.03 means 97% sure, right?"	"No — it means chance alone produces this 3% of the time; how often our ideas are real also matters"
"It's flat, so the feature failed?"	"It's unproven at this sample size — the test had Z% power at the effect we care about"
"Can we call 0.06 close enough?"	"Alpha was set before the test for a reason; renegotiating it after seeing the result voids the guarantee"

Aligned to the Oehlert’s A First Course in Design and Analysis of Experiments and Bruce & Bruce’s Practical Statistics · last reviewed 2026-07-08.

Make this cheat sheet yours

Personalize this sheet — focus it however you study, or build one from the exact questions you keep getting wrong.

Sections in this topic

Permutation Tests and the NULL P Values What They Arent Confidence Intervals and the Bootstrap The Bootstrap in Practice