Designing a Test from a Business Question

The one idea: A vague business question ("does the new checkout flow work better?") has no metric attached — "better" could mean five different, sometimes contradictory things. Designing the test is the work of forcing that word to name one primary metric, plus the guardrails that keep the winning metric from hiding a loss somewhere else.

⚠️ The trap: Let's measure completion rate, revenue per session, time to checkout, and cart abandonment, and we'll know if it's "better." That feels right and it's wrong. "Better" measured four ways at once is Anatomy of an A/B Test's five-metrics problem wearing a business-question costume — pick the one metric that owns the ship decision, and let the rest be guardrails that can veto but can't win on their own.

The Testing Process, and Writing a Real Hypothesis laid out the six-stage arc from problem to decision and showed what a structured hypothesis looks like once you have one: a stated mechanism, a direction, and a magnitude. This lesson zooms into the step that hypothesis-writing depends on and skips lightly past — stage two's quiet prerequisite, where a business question first gets turned into something with a metric attached. Get this step wrong and a beautifully structured hypothesis just ends up structured around the wrong number.

"Better" is not a metric

"Does the new checkout flow work better?" sounds like it's already halfway to a test. It isn't — it's a question with no operational content yet, because "better" is doing all the work and naming nothing. Unpack what "better" could mean for a checkout flow, and you get a genuine list…