Exploration vs. Exploitation, Playable

EP 01

Pull the arms yourself — a casino with hidden odds

Three machines, each with a fixed but hidden chance of paying a coin. Your job: earn as many coins as you can. The red curve is your regret — how many coins you gave up, in expectation, compared to a psychic who always pulls the truly best arm. Every pull of a worse arm adds to it. Feel the tension: to learn which arm is best you must waste some pulls on worse ones.

Payout rates are drawn once per casino and shuffled between machines — the regret counter knows the truth, you don't. That asymmetry is the whole game.

The good move: spend a few pulls on every arm before committing — a small, deliberate exploration budget. This tiny game is the exact skeleton of A/B testing, ad placement, and clinical trial design: each pull is a user, a patient, a dollar.

EP 02

The greedy trap — one lucky win ruins everything

Meet the pure greedy agent: try each arm once, then forever pull whichever looks best so far. Sounds reasonable — but if the best arm loses its first pull and a mediocre arm wins its first pull, greedy locks onto the mediocre arm and never checks again. It has no mechanism to discover its mistake. Run one agent to watch it happen, then run 500 to see how often it happens.

Machines pay A: 50%, B: 60% (best), C: 40% — greedy doesn't know that.

The colored strip shows which machine the agent pulled at each step. When the strip is one solid color from step 3 on, the agent has locked in — right or wrong.

Why it breaks: greedy confuses "best estimate so far" with "best". With 3 pulls of evidence, estimates are mostly noise. A recommender that only ever re-serves your first click, a startup that scales its first working channel — same trap, bigger stakes.

EP 03

ε-greedy — fix the trap with a coin flip

The classic fix is almost embarrassingly simple: with probability ε, ignore your estimates and pull a random arm; otherwise be greedy. Even a small ε guarantees every arm keeps getting sampled, so a wrong lock-in always gets corrected eventually. But ε is a tax: every random pull on a known-bad arm costs coins forever. Drag the slider and watch average regret (150 simulated casinos) respond.

ε 0.10

The shape to remember: pure greedy's regret grows in a straight line when it locks wrong. Small-ε regret bends flat — it found the truth and mostly exploits it. Big-ε regret is a straight line too: it knows the answer but keeps gambling anyway. Exploration is medicine; the dose makes the poison.

EP 04

Optimism as a strategy — assume every arm is amazing

A sneakier fix: stay greedy, but start every arm's estimate absurdly high. Reality can only disappoint, so each pull drags an arm's estimate down — and greedy naturally rotates to the next not-yet-disappointing arm. Exploration emerges for free, then fades once estimates are honest. But optimism is spent once: flip the switch to make the world change at step 400 (machine C secretly becomes the best) and see who notices.

Initial optimism Q₀ 2.0

Curves are the moving-average payout per pull, averaged over 120 simulated casinos. The ε-greedy rival uses a constant learning rate, so recent evidence never stops mattering.

The lesson: optimistic initialization is brilliant in a stationary world — front-loaded exploration, then pure profit. In a drifting world it goes blind: it stopped exploring long ago. Real systems (ads, markets, users) drift constantly, which is why production bandits never fully stop exploring.

EP 05

The grand race — who actually earns the most?

All three strategies, same casino, averaged over 200 runs: pure greedy, ε-greedy (ε=0.1), and optimistic greedy. The dashed line is the psychic upper bound. Now the twist: drag the horizon slider. With 2,000 pulls, exploration pays for itself many times over. With only 50 pulls, exploration is an expense you never live long enough to recoup — greedy often wins short games.

Horizon (total pulls) 1000

Why it matters: "how much should I explore" has no universal answer — it depends on how long you'll keep playing. This exact trade-off, scaled up, is inside every RL agent: AlphaGo's tree search balances trying new moves against deepening known-good ones, and RLHF'd LLMs balance safe answers against informative ones.