The Testing Process and Hypothesis Generation

The Testing Process, and Writing a Real Hypothesis

The one idea: A test is only as good as the sentence that started it. "Let's test a green button" isn't a hypothesis — it's a guess wearing a hypothesis's clothes. A real hypothesis names the mechanism (why the change should work), the direction (which way the metric moves), and a magnitude (roughly how much) — and every step of the testing process downstream depends on having gotten that sentence right.

⚠️ The trap: We have a great idea — let's just build it and A/B test it. That feels right and it's wrong. Skipping straight from idea to build means you can't tell, when the test ends, whether you learned something about your users or just got a random result on an untheorized change — write the mechanism and the predicted effect down before a single line of code changes, so the test can actually teach you something.

Every A/B test worth running follows the same six-stage arc: identify a problem, generate a hypothesis, design the test, run it, analyze the result, decide. The earlier chapters of this track spent most of their time on stages three through five — design (randomization, unit of assignment), running (peeking, sample-size), and analysis (p-values, confidence intervals, multiple comparisons). This lesson is about the two stages that come before any of that: identifying what's actually worth testing, and writing the hypothesis that turns a hunch into something a test can actually confirm or kill.

The six stages, and why order matters

1. Identify the problem. Something in the data or in user feedback suggests friction: a drop-off point in a funnel,…