Grokking, Playable

EP 01

The phenomenon — a model that understands 4,000 steps late

The classic setup: teach a network 40% of the answers to (a + b) mod 97, hold out the rest. Press train. Training accuracy (white) hits 100% almost instantly — pure memorization. Test accuracy (gold) flatlines near 1%… and keeps flatlining… and then climbs a cliff. One catch: on a linear time axis you can't even see the story — hit the log button.

✗ bad case: judge at step 1,000 — "it's just memorizing, kill the run" (you'd have been wrong)✓ good case: patience + log axis — the click arrives, test 1% → 99%

—

Curve dynamics are a faithful simulation of the published grokking runs (Power et al. 2022) — the shapes, gap and suddenness match the paper; this page doesn't train a real transformer in your tab.

Grokking broke two rules at once: "zero training loss means learning is done" and "if validation is flat, stop." A model can be finished memorizing and not yet started understanding — and the gap between those two events can be 100× longer than the memorization itself.

EP 02

Swiss cheese → wallpaper — watch the table become a formula

What does the model know in each phase? This is the full answer table for (a + b) mod 23 — every cell colored by the model's answer. Drag the training-progress slider. Early: the cells it was taught are right (scattered swiss cheese) and everything else is noise. After the click: the whole table snaps into diagonal stripes — the periodic structure of modular addition. It didn't fill in more cells; it found the pattern that generates all cells.

✗ bad case: memorization phase — 40% right, arranged exactly like the training set (a lookup table)✓ good case: post-grok — 100% right including cells it never saw, because stripes don't need lookup

training progress early —

Green = correct answer, red = wrong. Which held-out cells are wrong during memorization is re-randomized per run; the striped end-state is the true structure of mod-23 addition.

Memorization and understanding produce the same training score and completely different objects — a phone book versus a formula. Interpretability work later confirmed it literally: post-grok networks implement a clean trigonometric algorithm for modular addition. The network didn't get better at remembering; it replaced remembering.

EP 03

The impatient teacher — weight decay sets the alarm clock

Grokking has a dial: weight decay, the steady tax on large weights. Memorization needs big, specific weights; the general formula is compact. Decay slowly bleeds the phone book while the formula survives. Drag the dial and run: more decay, earlier click. Zero decay — the silence outlasts your budget. Too much — the model can't even hold the training set.

✓ good case: moderate decay — grok arrives ~3,000 steps in✗ bad case ×2: zero decay — no click within 10× the budget · heavy decay — can't memorize, can't grok, flat zero

weight decay medium —

Time-to-grok vs decay follows the published relationship (stronger regularization → earlier, until collapse); each run adds a marker to the summary strip below the curves so you can map the dial's whole range.

Grokking isn't magic patience — it's a slow-motion competition that regularization referees. This is also the modern recipe: with the right weight decay and data, later work made models grok almost immediately. The mystery didn't vanish, but it acquired an engineering dial.

EP 04

Inside the click — two solutions, one body

The cleanest mental model: during the silence, two circuits share the network. The memorization circuit (red bar) is strong, heavy, and taxed hard by weight decay. The generalizing circuit (green bar) is weak but efficient, and it grows quietly because it explains the data more cheaply. Press run and watch the bars: test accuracy doesn't move until green overtakes red — but green's growth is visible long before the click. That's an early-warning signal accuracy can't give you.

✓ good case: watching circuit strengths — the crossover is predictable hundreds of steps ahead✗ bad case: watching only test accuracy — the click arrives with zero warning

training data fraction 40% —

Circuit strengths follow the efficiency-competition account of grokking (memorization pays rent per example; the general circuit pays once). More training data → memorization gets more expensive → earlier crossover. Both effects play out live.

This picture is why interpretability researchers love grokking: it's the one place where "the model suddenly changed" can be traced to a concrete internal event — one circuit dethroning another. Progress measures built on internals (not benchmarks) saw the transition coming; that idea now powers early-warning evals far beyond modular arithmetic.

EP 05

Spot the grokker — three flatlines walk into a bar

The expensive skill: at step 2,000, a pre-grok run, a hopeless overfit, and a too-cold learning rate all look like the same flat line. Each round shows three unlabeled train/test curve pairs cut off mid-run — click the one that will grok. Then the future is revealed and your streak updates. The tells are subtle but real: the grokker's train accuracy is already perfect while its weight-norm trace (thin line) is falling; the overfit's gap grows with rising norm; cold-lr hasn't even finished memorizing.

✓ good case: streak of 5 — you can bill for reading training runs now✗ bad case: picking the overfit — its flatline never ends, it just quietly rots

click the panel you think will grok

Round parameters (grok time, noise, which panel) are randomized every time; your click is judged against the generator's ground truth.

The meta-lesson of grokking for anyone who trains models: a flat validation curve is not one situation — it's at least three, with opposite correct actions (wait / stop / turn up the lr). The observable that disambiguates them is almost never the accuracy itself. Look underneath.