The one idea: A clean A/B test only answers for the exact people, moment, and conditions it ran on. It can quietly mislead when users affect each other, when a segment is too small to trust, or when the group you tested isn't the group you'll ship to.
⚠️ The trap: "The A/B test was properly randomized, so its number is the truth and we can roll it out at that lift." That feels right and it's wrong. A clean test only answers for the exact people, moment, and conditions it ran on — when users affect each other (marketplaces, networks), when a winning slice is a tiny sample, or when you ship to a different population, the real-world number can shrink or flip.
The last lesson argued that randomization is the only design that certifies a causal claim. This lesson is the honest counterweight: the certification comes with fine print. Experiments mislead when users interact with each other, when cells are too small, and when the population you tested isn't the population you'll ship to. And sometimes you can't run one at all.
None of this demotes experiments. It tells you when a green A/B dashboard is answering a different question than the one you asked — and "we can't randomize this" is where the rest of this track begins.
When users leak: SUTVA and spillovers
The clean math of the previous lesson quietly assumed SUTVA — the stable unit treatment value assumption. In English: one user's treatment doesn't affect another user's outcome. In social and marketplace products, that assumption is routinely false:
- Network features. You ship a new messaging feature…
Keep reading: Limits of Experiments
Unlock the full Causal Inference track — every lesson, the AI tutor, and full mock assessments.
- Full lesson content
- AI tutor for this section
- Practice questions