Sample Size and Test Duration

The one idea: Sample size math answers "how many users," not "how long to run." Turning users-needed into a calendar tells you when you're actually done — and the honest answer almost never lines up with a tidy folk rule like "run it for a week."

⚠️ The trap: We hit our target sample size at 10am Wednesday — end it now and ship the readout. That feels right and it's wrong. Traffic mix shifts across the week (weekday vs. weekend users behave differently), so stopping mid-cycle even after "enough" users have arrived leaves the two arms comparing unequal slices of the week — always round a test's duration up to a whole number of full weeks.

Power analysis (previous lesson) tells you the shape of the problem: sample size, effect size, alpha, and power are locked together, and solving for sample size is the standard move. This lesson is about doing that arithmetic for real, on a real number, and about the practical mechanics of turning a "users needed" figure into an actual test-launch and test-end date — where a handful of predictable gotchas eat teams that get the math right and the operations wrong.

The formula, once, for reference — then never again by hand

For a conversion-rate (binary) metric, the standard two-proportion sample-size formula is:

You never compute this by hand — a sample-size calculator or the AI does. Skip the formula; it's here so you recognize what the calculator is doing, and the worked example below is what matters. n per arm ≈ (z_α/2 + z_power)² × [p1(1−p1) + p2(1−p2)] / (p1 − p2)²