Epsilon Greedy and Exploration Strategies

The one idea: The simplest possible answer to "explore or exploit?" is to flip a biased coin: with probability ε (epsilon) pick a completely random arm, and the rest of the time (1-ε) pick whichever arm currently looks best. It's crude, it's the first thing anyone tries, and it has a specific, predictable flaw that UCB and Thompson Sampling were both built to fix.

⚠️ The trap: A fixed ε — say ε=0.1, forever — never converges to pure exploitation, even after a million rounds, even once you're functionally certain which arm is best. You keep spending 10% of all future traffic on a random arm out of pure habit, not because you're still learning anything from it — that's regret you're choosing to keep paying, and it compounds forever unless ε is explicitly decayed toward zero over time.

The problem, restated one more time

Same setup as the UCB lesson and the Thompson Sampling lesson: K arms (five ad variants, A through E), each with an unknown true click-through rate, and a sequence of choices where every pull both earns a reward and teaches you something. Before reaching for either of those two more sophisticated algorithms, it's worth sitting with the simplest possible strategy, because it's what most people invent on their own the first time they think about the problem, and understanding exactly where it breaks is what makes UCB and Thompson Sampling's design choices make sense rather than feel arbitrary.

The algorithm: one coin flip per round

Each round, for every arm a, keep a running estimate Q(a) — the average observed reward from that arm so far…