The Placebo Effect and Hawthorne Effect

The Placebo Effect and the Hawthorne Effect

The one idea: Users can react to knowing something changed or knowing they're being watched — independent of whether the treatment does anything real. A placebo arm, one that looks like the treatment but has no active ingredient, is the only way to tell "the feature worked" apart from "people noticed a feature."

⚠️ The trap: Engagement rose 5% the week we shipped the new recommendation engine — the algorithm is clearly better. That feels right and it's wrong. Users who merely notice "something is different" or sense they're part of a test often behave differently for that reason alone, so without a placebo arm you can't separate the algorithm's real lift from the lift of being noticed.

Lesson 1 of this track (Fooled by Randomness) trained you to ask "how often would chance alone produce this?" before crediting a cause. This lesson trains a close cousin of that instinct: before crediting the treatment for a lift, ask whether the lift came from what the treatment actually does, or from the simple fact that users experienced some change, or sensed they were being studied. Anatomy of an A/B Test (Anatomy of an A/B Test) named the CAST trial as a warning about proxies that move for the wrong reason; this lesson is about outcomes that move for a wrong reason of a different flavor — not a bad proxy, but a real metric moved by the wrong mechanism.

Two names for "reacting to the test itself"

The Hawthorne effect takes its name from a series of 1920s-30s productivity studies at Western Electric's Hawthorne Works plant. Researchers changed factory lighting…