Samples and Sampling Variability

The one idea: How a sample was chosen matters more than how big it is. A huge sample drawn the wrong way is confidently wrong, because more data shrinks random noise but never removes bias.

⚠️ The trap: We surveyed thousands of our own users and they love it — that's the voice of our users. That feels right and it's wrong. Reviews, NPS replies, and beta feedback come only from people motivated enough to answer, so no sample size makes them speak for the silent majority; report the number with its selection mechanism attached.

In 1936, the Literary Digest — a popular US magazine — ran the largest opinion poll in history. It mailed out 10 million ballots for the Landon-vs-Roosevelt US presidential election and got 2.4 million back — a sample size that dwarfs anything your analytics team will ever touch. The Digest confidently predicted Landon in a landslide. Meanwhile, a young pollster named George Gallup surveyed a few thousand people. Gallup called it for Roosevelt.

Roosevelt won 46 of 48 states. The Digest folded within two years.

What happened? The Digest drew its mailing list from telephone directories, car registrations, and its own subscribers — in Depression-era America, a roster of the relatively wealthy, who leaned Landon. Ten million ballots measured the wrong population with extraordinary precision. Gallup's small sample was drawn to represent the electorate, and representativeness beat volume by a factor of a thousand. Bruce and Bruce distill the lesson in Practical Statistics for Data Scientists: data quality beats data quantity, and for samples, quality means one thing above all — how the sample was selected.

Two ways a sample lies to…