Q-Learning, Playable

EP 01

The Q-table — a scorecard for every square

A robot (white dot) wanders a tiny world looking for the star. Each square is split into 4 wedges — up, right, down, left — one per action, and the wedge's brightness is its live Q-value: the discounted reward the robot expects if it takes that action there. Everything starts black (knows nothing). Run episodes and watch knowledge appear next to the goal first, then bleed outward.

Learning rate α 0.50

Real Q-learning with ε=0.25 exploration, γ=0.9 discount, reward +1 at the star only. Every brightness you see is a live number in the table, updated by the standard Q-learning rule.

The key picture: after one lucky episode, only the wedge pointing into the star lights up. Each later episode extends the lit region one hop further back. This table-plus-update-rule is the entire algorithm — everything else in RL is scaffolding around it.

EP 02

Bootstrapping — value leaks backwards, one step per pass

Why does knowledge spread backwards? Because each cell's update copies from its neighbor's current guess: Q(here) ← reward + γ · Q(next). A guess built on a guess — that's bootstrapping. This 12-cell corridor makes it exact: walk it once, only the last cell learns. Walk again, the second-to-last learns from it. Each pass, the wave moves one cell left, shrinking by the discount γ each step.

Discount γ 0.95

Changing γ wipes the corridor — the values are only meaningful for one discount.

Good case: γ near 1 carries value all the way home — the start cell "smells" a goal 11 steps away. Bad case: at γ=0.5 the start cell's value is 0.5¹⁰ ≈ 0.001 — mathematically present, practically invisible. A too-small discount makes distant goals fall over the horizon, and the agent near the start acts blind.

EP 03

The α dial — trust the new sample or trust the old belief?

Every update blends old belief with new evidence: Q ← Q + α·(sample − Q). Here one cell receives 300 noisy reward samples whose true average is 1.0 (green dashed line). Drag α and watch the same evidence produce totally different beliefs. The faint red trace is α=1: believing only the latest sample, forever.

Learning rate α 0.20

The trade: big α learns fast and never stops jittering; tiny α is smooth but glacial. In noisy worlds (which is all of them), practitioners start α high and decay it — learn boldly, then settle down. The same dial exists in EP 01: crank α to 1 there and watch the wedges flicker.

EP 04

A policy emerges — arrows out of numbers

The table isn't the point — the policy is: in each square, just take the brightest wedge. Arrows show that best action wherever the table has learned one. Train with the ε slider, then press Test to make the robot follow the arrows greedily. Now the trap: set ε to 0, wipe, and train. With no exploration and an empty table, ties always break the same way — the robot paces one dead-end corridor forever and the table stays black. Zero exploration, zero learning.

Exploration ε 0.20

Training here breaks ties deterministically (always “up” first) — exactly the pathology that makes ε=0 lethal. EP 01 used random tie-breaking, which hides it.

Why it matters: Q-learning only converges if every state–action pair keeps getting tried. That's the exploration requirement from the bandit lesson, resurfacing inside every RL algorithm — DQN played Atari with exactly this ε-greedy trick bolted on.

EP 05

The table explodes — add one key and watch it double

Here's the fatal flaw. Add a locked door and a key: "where am I" is no longer enough — the robot needs "where am I and do I have the key". The table doubles: one full copy of the world per key-state. Add a second key: it doubles again (4 world-copies). Each layer must be learned by visits — value can't jump between layers except where they connect. Train each version and compare how many episodes it takes.

K = key, D = its locked door. The robot must fetch key 2, open door 2, fetch key 1, open door 1, then reach the star. Training is real Q-learning (ε=0.3, γ=0.95) — episode counts vary run to run.

Good case: tiny worlds — the plain grid solves in a few dozen episodes; tables are perfect there. Bad case: every extra fact multiplies the table. Ten keys → 1,024 world-copies; a chessboard → ~10⁴³ rows; an Atari screen of raw pixels → more rows than atoms in the universe. No table fits, and no agent can visit every row. That's why DQN replaced the table with a neural network that generalizes across states — function approximation exists because tables explode.