The Loss Landscape, Playable

EP 01

Drop the ball — where it lands was decided before it moved

This curve is a loss landscape: left-right is one model parameter, height is how wrong the model is. Gradient descent is dumb and local — it only ever rolls downhill from wherever it starts. Click anywhere on the canvas to drop the ball and watch. There are two valleys; only one is the good one.

✓ good case: drop right of the hill → deep valley, loss 0.05✗ bad case: drop left → trapped in the shallow valley forever

click the curve to drop a ball

Every ball follows the exact same rule: x ← x − lr·slope(x). No ball ever climbs. The only difference between winners and losers is the starting point.

Initialization matters because gradient descent has no map and no undo. This is why training runs use carefully-designed random inits — and why two runs of the same code can land in different valleys.

EP 02

The step-size dial — creep, converge, explode

The learning rate is how far the ball moves per step. It is the single most-tuned number in deep learning, and it has three regimes. Drag the slider and press play — same valley, three completely different fates. The loss trace on the right is live.

✓ good case: lr 0.10 — smooth glide, converged in ~25 steps✗ bad case: lr 1.05 — each step overshoots more, loss → ∞

learning rate 0.100 —

The explosion is real math, not theatre: on this parabola any lr above 1.0 makes each step land farther from the bottom than the last.

Too small wastes compute, too large destroys the run. Real training schedules (warmup, cosine decay) are choreography for this one dial — start careful, speed up, then slow down to settle into the valley floor.

EP 03

The ravine — why plain gradient descent zigzags

Real landscapes aren't round bowls — they're ravines: cliff-steep in one direction, almost flat in another. The gradient points across the ravine, not along it, so the ball ping-pongs between the walls while barely crawling toward the exit. Crank the steepness, then switch on momentum and watch the same ball average out its own zigzag.

✗ bad case: steep ravine + plain GD — 200 steps of wall-bouncing✓ good case: momentum 0.9 — the zigzag cancels itself, ~5× fewer steps

ravine steepness 12× —

Contours are the true landscape; the gold trail is every step the optimizer actually took. Steps-to-converge is measured, not scripted.

This picture is why Momentum, RMSProp and Adam exist. Adam is essentially "remember which directions keep flip-flopping and shrink them, remember which direction is consistent and stretch it." Every modern LLM is trained by a descendant of this fix.

EP 04

Noise is a feature — let SGD shake the ball loose

Full-batch gradient descent computes the exact slope — and exactness is a trap: the ball settles into the first pit it finds, even a nasty narrow one. Stochastic gradient descent estimates the slope from a small random batch, so every step carries noise. Drag the batch-size slider down and watch the noise kick the ball out of the sharp pit and over into the wide valley.

✗ bad case: full batch — exact, deterministic, permanently stuck in the sharp pit✓ good case: small batch — noisy steps hop the wall into the wide valley

batch size full batch —

Smaller batch = larger gradient noise here, matching the 1/√batch scaling of real SGD noise. Escape or capture is decided by the actual simulation each run.

SGD's sloppiness is secretly regularization. The noise makes narrow pits uninhabitable — the ball can only stay somewhere wide. That bias toward wide valleys is one reason models trained with SGD generalize better than the math says they should.

EP 05

Sharp vs flat — the earthquake test

Two balls, both at the bottom of their valley: one in a sharp pit (lower training loss!), one in a flat basin. Now the earthquake: test data never matches training data exactly, which shifts the whole landscape sideways. Drag the shift slider and watch who survives. The sharp ball's loss rockets; the flat ball barely notices.

✓ good case: flat basin — landscape shifts, loss barely moves✗ bad case: sharp pit — a 0.3 shift multiplies its loss ~20×

train→test shift 0.00 —

Dashed curve = training landscape, solid = shifted test landscape. Both losses are read off the actual curves at the balls' positions.

Generalization has geometry. "Flat minimum" means: being slightly wrong about the world barely hurts. Sharp minima ace the training set and faceplant in production — this is why the winner of EP 04's noisy search (wide valleys) is exactly what you want, and why "lowest training loss" is never the goal.