Self-Play, Playable

EP 01

Two blank slates walk into a game

The game is Race to 21: players take turns adding 1, 2 or 3 to a running total; whoever says 21 wins. The agent below starts knowing nothing — it just plays itself and nudges a value table after every move (real tabular Q-learning, running in your browser). The strip shows every game position: green = the agent's move there is provably perfect, red = still wrong. Train it, watch the board turn green — then try to beat it yourself.

Train it a little (or don't!), then challenge it.

This is genuine Q-learning (α=0.12, ε=0.3), not a script: wipe the brain and it really does play randomly again. Race to 21 has a perfect strategy — always leave the total at 1, 5, 9, 13 or 17 — which is what the agent discovers on its own.

Skill from nothing: nobody told it the mod-4 trick — it fell out of playing itself. This is the same loop (self-play + value updates) that took TD-Gammon to superhuman backgammon in 1992 and AlphaZero to superhuman Go, chess and shogi — just with a neural network instead of a 21-row table.

EP 02

The ladder — every rung is a former you

How do you measure progress when there's no opponent but yourself? You freeze checkpoints and make today's agent fight yesterday's. This agent learns a harder game (Race to 40, add 1–4) slowly, on purpose. Each rung = train 20 games, freeze a copy, then play 150 matches against the previous checkpoint. ELO is computed live from those match results. The bars show the newest champion against every ancestor.

ELO from win rate w via 400·log₁₀(w/(1−w)), chained rung to rung, matches played with 8% move noise so mirror games aren't identical. Watch the curve flatten at the top — once the game is solved, beating your past self gets impossible.

Why it matters: this ladder is literally how AlphaZero was monitored — new network vs. frozen best network, promote only if it wins ≥55%. The plateau you just produced is real too: self-play ELO saturates when the game is mastered. In Go the ceiling was simply thousands of ELO further up than anyone expected.

EP 03

The automatic curriculum — why the fair fight teaches fastest

Self-play's superpower is that the opponent is always exactly at your level. Race three identical learners on Race to 21, same budget, different sparring partners: one trains against a random mover, one against a frozen master that never blunders, one against itself. The y-axis is the honest metric: % of positions where the learner's move is provably perfect.

The master-trained learner sits second seat against a perfect first mover, so it literally cannot win — every game returns −1 and there is nothing to learn from. The random-trained learner wins constantly but only meets blunders, so it learns slowly and sloppily. Same algorithm, same α and ε for all three.

Good case: self-play always serves a ~50% win rate opponent — maximum learning signal, difficulty rising automatically as you improve. Bad case: both fixed opponents starve the learner — too strong means no reward ever, too weak means no pressure. This is why AlphaStar built an entire league of past selves instead of training against one fixed bot.

EP 04

Chasing your own tail — the rock-paper-scissors trap

Race to 21 has a best strategy, so self-play converges. Rock-paper-scissors does not: rock beats scissors beats paper beats rock — every strategy has a counter. Watch two self-play agents adapt to each other's latest strategy (left): the mix orbits forever, never settling. Each dot is the agent's current strategy inside the triangle of all possible mixes. Right: the classic fix — best-respond to the average of all past selves — spirals into the unbeatable center.

Exploitability = how much a perfect adversary would score against the current strategy (0 = unbeatable). Left agent: smoothed best response to the opponent's latest mix. Right agent: fictitious play — best response to the opponent's historical average.

Bad case: in any game with non-transitive strategies (StarCraft build orders, poker styles, MOBA drafts), naive self-play chases its own tail — champion beats yesterday's champion yet loses to last week's. Good case: averaging over past selves (fictitious play, and its modern descendant CFR) converges to the Nash mix. OpenAI Five and AlphaStar both mixed in old checkpoints for exactly this reason.

EP 05

The champion who never met a stranger

A tiny bluffing game: cards 1–3, ante a chip, the bettor may bluff or check, the caller may call or fold. Two agents co-evolve by naive self-play. Against its own sibling the champion's earnings look rock-steady — but pit each day's champion against a simple fixed human it never trained against, and its edge swings wildly, collapsing to zero at the peaks of its own bluffing fashion cycle. The averaged agent (green) banks a guaranteed profit against the same human, forever.

The out-of-distribution human:

All EV numbers are exact expectations over the 6 possible deals, recomputed every step — no simulation noise. The Nash bluffing rate for this game is 1/3; watch the averaged agent find it while the naive pair orbits around it.

The scary part: measured against its training partner, the overfit champion looks identical to the robust one — the weakness is invisible from inside self-play. This is why poker bots (Libratus, Pluribus) play the average strategy from CFR, and why every serious self-play lab also evaluates against held-out opponents and humans: an agent can be superhuman against its own family and mediocre against a stranger who just always calls.