The Learning Rate, Playable

EP 01

Three fates — same data, same model, one dial

A line is being fit to 30 noisy points by gradient descent, right now, on this canvas. Two knobs are being learned (slope and intercept) — you only control the learning rate. Watch the gold line hunt for the data at 0.05, crawl at 0.001, and at 1.5 spin off the screen as the loss readout hits astronomical numbers.

✓ good case: lr 0.05 — line locks on in ~40 steps✗ bad case: lr 1.5 — the line leaves the observable universe

lr 0.050 —

Slider is log-scaled (like every lr plot in every paper). The fit is genuine 2-parameter gradient descent on mean squared error; the same 30 points every run.

The regimes are universal. Crawl / converge / explode shows up identically in a 2-parameter line fit and a 2-trillion-parameter LLM, because both are the same procedure: step size × slope. Only the landscape got bigger.

EP 02

The sweep — how the pros actually pick the number

Nobody derives the right learning rate — they sweep it. Press the button: 24 training runs launch at log-spaced rates from 0.0001 to 3, each dot is a finished run's final loss. The shape that appears is the most famous curve in practical deep learning: flat-ish valley, then a cliff. Click any dot to replay that run in EP 01's style.

✓ good case: the valley floor is wide — anything 0.02–0.2 works✗ bad case: the right edge is not a slope, it's a cliff — 0.6 fine, 1.2 dead

click a dot after sweeping to inspect that run

All 24 runs execute in your browser when you click — about 3,600 gradient steps total. The asymmetry (gentle left, cliff right) is why practitioners err small.

The cliff is the lesson. Undershooting the best lr by 10× costs you time; overshooting by 2× costs you the run. Every "lr finder" tool (fastai popularized it) is exactly this sweep, automated — and the rule of thumb "pick the value one notch left of the cliff" comes from this picture.

EP 03

Warmup — the first steps are the most dangerous

Fresh models start in terrible places: gradients are huge and momentum has no history to average over. Slam a big learning rate into that and the run dies in the first 20 steps — even though the same rate would be perfectly safe later. Warmup ramps the rate from ~0 to full over the first N steps. Drag the warmup slider from 0 and watch the red trace turn green.

✗ bad case: warmup 0 — peak lr hits the chaotic start, loss spikes and never recovers✓ good case: warmup 30 steps — the same peak lr sails through

warmup steps 0 —

The task: fit a small 8-parameter model with momentum 0.9 from a deliberately bad init. Divergence or survival is decided by the actual dynamics each run — rerun and the seed changes.

Every serious LLM run warms up — typically a few hundred to a few thousand steps. It costs almost nothing and removes the single most common cause of day-one loss explosions. If you've ever seen a loss curve spike at step 12 and flatline: this dial was at zero.

EP 04

Decay — you can't land a plane at cruising speed

With noisy (stochastic) gradients, a constant learning rate never truly converges — it orbits the optimum in a noise ball whose radius is proportional to the rate. To settle into the valley floor you must slow down late in training. Compare the three landing strategies and read the measured final-loss floor of each.

✓ good case: cosine decay — glides to a ~15× lower floor than constant✗ bad case: decay too early — the run freezes halfway down the hill, permanently undertrained

—

Same noisy quadratic task, 400 steps each. "Floor" = mean loss over the final 50 steps, measured per run. The schedule curve is drawn under the loss trace.

Schedules are landing gear. Cosine decay (or linear-to-zero) is the default in modern pretraining because the endgame of training is noise management: big steps to travel, small steps to park. And "decay too early" is a real, expensive mistake — it looks like convergence but is just paralysis.

EP 05

The dial is coupled — batch size moves the sweet spot

Change the batch size and yesterday's perfect learning rate is wrong. Press the button: a 6×6 grid of 36 complete training runs executes (lr × batch), each cell colored by its measured final loss. A diagonal band of green appears — that's the linear scaling rule: double the batch, double the workable rate.

✓ good case: batch 32 at lr 0.4 — same quality as batch 4 at lr 0.05✗ bad case: keep lr 0.4 but drop to batch 4 — noise × big steps = divergence

—

Every cell is a real 250-step stochastic run on the same task — about 9,000 steps total per click. Green = low final loss, red = diverged. The band you see was computed, not drawn.

Hyperparameters hunt in packs. The lr–batch coupling (Goyal et al.'s "linear scaling rule") is why scaling a run from 8 GPUs to 256 isn't just "same settings, more machines" — and why published hyperparameters never transfer verbatim to your setup. Tune the dial after you fix the batch.