Bandits and When Ab Fails

Bandits, and When A/B Fails

The one idea: Use an A/B test when you need to prove whether something works; use a bandit when you just want to earn the most while a short-lived decision sorts itself out. When the setup is unusual or high-stakes, recognize it early and bring in a specialist before you collect any data.

⚠️ The trap: A bandit is just a smarter A/B test, so I'll run one and read off a clean effect size too. That feels right and it's wrong. Bandits deliberately starve losing arms of traffic to earn more during the run, which distorts the effect estimate; optimize with a bandit, but prove with a fixed A/B test.

The standard A/B test is a beautifully engineered instrument for one job: proving, with controlled error rates, whether a treatment works. But not every question is a proving question, and not every setting satisfies the test's quiet assumptions. This closing lesson covers two frontiers: multi-armed bandits, for when the goal is earning rather than learning; and the gallery of situations where the standard test breaks — with pointers to the designs that handle each.

Optimize versus prove

An A/B test spends traffic evenly to the end, even on arms that are clearly losing, because its contract is inferential: hold the false-positive rate, measure the effect precisely, deliver a defensible verdict. That's the right contract when the decision is consequential and permanent — a pricing change, a platform migration, a result you'll cite for years.

But consider a subject line for Tuesday's email blast, or which of five promo banners to show this holiday weekend. Nobody needs a p-value for posterity;…