ε-Greedy Exploration, Playable

EP 01

Four slot machines and the greed trap

Each machine pays 1 credit or nothing, with fixed but hidden odds. Pull a few arms yourself and watch the estimates form. Then unleash a pure greedy agent: it always plays whichever machine currently looks best. Watch what it does with the very first machine that happens to pay out.

Pull some arms, or let the greedy agent loose.

Greedy with zero-initialized estimates: any machine that has never paid looks worth 0, so the first one that pays a single credit wins the agent's loyalty forever.

The trap: once one machine's estimate rises above zero, greedy never touches the others again — so it can never learn it was wrong. This is the core dilemma of reinforcement learning: you only learn about the actions you take, and greed decides you're done learning after one lucky pull.

EP 02

One coin flip fixes it — turn the ε knob

Same four machines. Now, with probability ε, the agent ignores its estimates and pulls a random arm. Drag the slider from 0 to 0.5 and run 2,000 pulls — each curve is the agent's average reward over time. Find the sweet spot: enough randomness to discover the best machine, not so much that you keep playing the bad ones on purpose.

ε 0.10

Curves are averaged over 40 independent runs so you see the trend, not one lucky game.

ε = 0 sometimes gets lucky and looks fine on one run — averaging 40 runs exposes how often it locks onto a loser. That's why RL papers always report means over many seeds.

✓ Good case: ε ≈ 0.1 reliably finds the 70% machine and climbs near the ceiling. ✗ Bad case: ε = 0 flatlines wherever luck dropped it, and ε = 0.5 knows the best machine but throws away half its pulls on random junk. Exploration is a cost you pay for information — the knob sets the price.

EP 03

The schedule wars — explore forever, or taper off?

A constant ε pays the exploration tax forever. The obvious upgrade: start exploratory, then decay ε over time. But decay speed is a bet on how fast you'll learn. Run the 10,000-pull marathon and compare three agents: constant ε = 0.1, a patient decay, and an impatient decay that stops exploring after ~50 pulls. The chart shows how often each is pulling the truly best machine.

Press run — takes a blink, simulates 900,000 pulls.

The x-axis is square-root scaled so you can see the early game where all the drama happens. The two best machines pay 70% vs 55% — telling them apart takes many samples, which is exactly what impatient decay never collects.

✓ Good case: patient decay gets the best of both worlds — it explores while ignorant and approaches 100% exploitation once sure. ✗ Bad case: decay ε too early and whatever the agent believed at that moment gets frozen in forever — some runs spend 10,000 pulls loyally feeding the wrong machine. An early habit, locked in permanently.

EP 04

The escape artist — a tiny ε breaks out of a rut, nearly free

Now a real (tiny) Q-learning agent in a gridworld. One step from the start sits a +1 treat that ends the episode. Across the map, behind a wall with one gap, sits a +10 jackpot. Train it three ways and watch the learned route. Pure greed grabs the treat on day one and never questions it; a whisper of randomness eventually stumbles through the gap.

Train an agent, then watch its final learned route.

Real tabular Q-learning runs in your browser each time you click — thousands of training episodes, so results can vary between clicks. ε = 0.05 occasionally fails to find the jackpot within its budget: retrain and see.

✓ Good case: the exploration was nearly free — check the average training reward: the ε = 0.05 agent sacrificed a few percent of its pulls to random moves and multiplied its final policy's value ~7×. This is why even sophisticated systems like DQN playing Atari shipped with plain ε-greedy: a tiny ε buys escape from local optima at a tiny price.

EP 05

When the coin flip must stop — random steering at 70 mph

Constant ε has an ugly meaning after learning is done: your deployed agent still does something random, forever. This car has fully mastered lane-keeping — but ε-greedy is still switched on, so every so often it "tries a random steering action" it learned nothing new from years ago. Drag ε, watch the swerve counter, then hit anneal.

deployment ε 0.12

The lane-keeper is a toy controller, not a trained policy — the point is the schedule, not the driving. Real systems separate a training policy (exploratory) from a deployment policy (greedy or near-greedy) for exactly this reason.

✗ Bad case: exploration randomness is only useful while there is something left to learn — at mastery it's pure self-sabotage, which is why nobody deploys a self-driving stack that "explores" on a public road. ✓ Good case: anneal ε toward 0 as competence grows and the same coin flip that once found the jackpot quietly retires. But remember EP 03: anneal too early and you retire the coin before it has finished its job.