Factorials and Interactions

The one idea: Test several changes together in every combination instead of one at a time. It costs less, and it is the only way to catch when two changes only work together — or when a change that helps one group quietly hurts another.

⚠️ The trap: I'll test the headline, then the image, then the price one at a time so each result stays clean and isolated. That feels right and it's wrong. One-at-a-time is both more expensive and blind to interactions — it can never see the winning combination, or spot a change that helps one segment while hurting another; test all combinations together.

Your team wants to improve a landing page, and there are three ideas on the table: a new headline, a new hero image, and a new price display. The instinctive plan is to test them politely, one at a time: headline test this month, image test next month, price test after that. Three months, three answers.

Factorial design says: run them together — every combination of headline × image × price in one experiment — and get more information in less time. This sounds like it should cost more; the enduring surprise of experimental design, developed by Fisher in the 1920s and driven home in Oehlert's A First Course in Design and Analysis of Experiments, is that it costs less, answers the one-at-a-time questions better, and answers a question the one-at-a-time plan can't even ask: do the ideas interact?

Main effects and interactions, in plain words

In a factorial experiment with two factors — say headline (old/new) and image (old/new) — users are randomized across all…