Overfitting to the Simulator, Playable

EP 01

Home field — flawless in the maze it grew up in

Meet Maze A and its resident agent. Train it — you'll see value spreading backwards from the goal, the way RL training converges — then run it. It takes the mathematically shortest path, every time. By every benchmark measured inside this maze, training is a total success.

Maze A. The gold square is the goal.

We compute the converged policy exactly (the arrows are the true optimal action in every cell) so you can see the end state of training without waiting through thousands of episodes. That's what "trained to convergence on one environment" means.

✓ Good case: inside its own training environment the agent is genuinely perfect — and that's the trap. A 100% score on the training simulator tells you nothing yet about whether it learned maze-solving or just this maze's answer key.

EP 02

One maze swap — the champion is lost

Same agent, same weights — new maze. During training, Maze A never changed, so the agent never needed to look at the walls: it just memorized "at position (3,4), go right." Drop it somewhere new and those memorized moves chain into walls, circles, and dead stares. Click repeatedly — every maze breaks it differently.

Its policy is a lookup from position to move — watch what that's worth here.

This models an agent whose observations were constant during training, so it effectively learned position → action. Deep RL agents do the visual equivalent: they key on incidental features (background, layout, timer position) that were accidentally constant in the simulator.

✗ Bad case: zero transfer — the skill was an illusion created by a frozen environment. ✓ Good case: put it back in Maze A and it's instantly perfect again, which is exactly why single-simulator scores keep fooling people: the overfit agent looks identical to a competent one until the world shifts by one wall.

EP 03

The diversity dial — train on N mazes, watch skill appear

Now train a smarter agent that senses its local surroundings (nearby walls + a goal compass) and learns which move works in each situation, pooled across N different training mazes. Drag the dial. With N = 1 its situation table is tiny and maze-specific. As N grows, situations repeat across mazes, maze-specific quirks get outvoted, and real maze-solving emerges — measured live on 30 mazes it has never seen.

training mazes N 1

Everything recomputes live as you drag: the agent is retrained on N procedurally generated mazes and re-tested on 30 held-out ones. "Situations recognized" is how often the test threw a local situation the agent had seen in training — when it hasn't, it can only guess. (The agent also prefers not to re-tread cells it has already visited — a standard trick that keeps any policy from dithering in circles.)

✗ Bad case: N = 1 — the table memorizes one maze and shrugs at everything else. ✓ Good case: crank N and generalization emerges from diversity alone — same agent, same learning rule, more varied worlds. This is the core finding behind procedural environments: you don't teach transfer, you make memorization impossible.

EP 04

The gap chart — recreating the CoinRun lesson

OpenAI's CoinRun study made this famous with one chart: score on seen levels vs score on unseen levels, as training levels increase. Agents trained on few levels aced them while failing new ones — a gap invisible if you only test where you train. Run the same experiment right here on our maze agent and watch both curves get computed point by point.

Six agents will be trained and tested on 40 held-out mazes each.

Live simulation, not a stock chart — numbers vary slightly run to run. In CoinRun (Cobbe et al., 2019) agents needed on the order of 10,000 levels before the seen/unseen gap closed; our toy closes faster because mazes are far simpler than platformer levels.

The lesson: the seen-level curve stays high no matter what — ✗ training performance cannot detect simulator overfitting. Only held-out environments reveal it. ✓ Good case: as diversity grows, the two curves converge — the honest definition of "it actually learned the task."

EP 05

The race — specialist vs generalist on a maze neither has seen

Final exam. The Specialist (red) trained to perfection on Maze A alone. The Generalist (green) trained on 40 random mazes and never scored a perfect run on any of them. Now both wake up in a brand-new maze. Race them — as many times as you like.

Red = Specialist (1 maze, perfect at home). Green = Generalist (40 mazes, perfect nowhere).

The generalist isn't guaranteed to win every maze — it fails sometimes, honestly. Run several races and keep score.

The punchline: the agent with the worse training scores wins in the real world. This is why robotics teams use domain randomization — thousands of randomized simulator variants — before touching real hardware: an agent that has never seen the same world twice can't memorize a world, so it's forced to learn the skill.