Reading Loss Curves, Playable

EP 01

The healthy patient — and the linear-scale illusion

This is what good looks like: both curves dive, then grind, then flatten, with validation (green) riding slightly above train (white) — a small, stable gap. Now the trap: click the scale toggle. On a linear axis this healthy run looks finished by step 80 — you'd stop it, wasting the second half where the loss drops another 5×. Log scale is not a preference; it's the instrument.

✓ good case: log scale — the late progress is clearly visible✗ bad case: linear scale — "it's flat, kill the run" (it wasn't flat)

—

Each "patient" is a fresh real training run (polynomial regression, gradient descent) — the shapes vary but the anatomy doesn't.

Memorize the healthy silhouette before the sick ones. Dive, grind, flatten; val above train by a whisker. And read losses in log space — half of all "my model stopped learning" reports are axis bugs in the human.

EP 02

Ward round — the six classic diseases

Six beds, six patients, six signatures. Click any thumbnail to enlarge it with the tell-tale annotated. The two on the right of the top row are the famous look-alikes — underfit and cold-lr both crawl, and telling them apart is the differential every junior gets wrong (the tell: cold-lr is still improving steadily at the end; underfit has truly flattened).

✓ good case: you can name the disease from the silhouette alone✗ bad case: mistaking cold-lr for underfit — you'd redesign the model when one number needed changing

click a thumbnail to enlarge

All six are generated by actually inducing the pathology in a toy run: the overfit patient really has too much capacity, the hot-lr patient really oscillates, the leak patient really shares data between splits.

The six tells: val turns up = overfit · both plateau high = underfit · sawtooth/spikes = lr hot · straight slow slide = lr cold · val below train = leakage · periodic waves = data not shuffled. Screenshot this list; it's 80% of real-world curve debugging.

EP 03

The diagnosis game — new patient every round

Board exam time. Each round generates a fresh, unlabeled patient with random severity — same generator as the ward, new random parameters, so you can't memorize pixels, only signatures. Six buttons, one correct answer, streak counter. Get five in a row and you're cleared for real dashboards.

✓ good case: streak ≥ 5 — you read silhouettes, not pixels✗ bad case: the generator will keep serving you the underfit/cold-lr pair until you learn the tell

score 0 · streak 0

Severity, noise and duration are randomized each round — some cases are subtle on purpose. Your score only counts the first click per patient.

Diagnosis order in real life: check the axes, check where val sits relative to train, check the end-of-run slope, then look for rhythm. In that order — it resolves 5 of the 6 diseases before you ever open the config file.

EP 04

The leak — when "too good" is the disease

The only disease that looks like a promotion. Drag the slider to let a fraction of validation examples secretly appear in the training set — the standard accident (duplicate rows, near-duplicates, test answers in the pretraining crawl). Watch validation sink below train. Every instinct says celebrate; the X-ray says the eval is broken.

✓ good case: 0% leak — val above train, boring and trustworthy✗ bad case: 30% leak — val "beats" train; the model ships, then meets reality

leaked fraction 0% —

The leak is mechanical, not cosmetic: that fraction of the val set literally is training data, so the model has memorized those answers. The gap you see is measured on the contaminated split.

val < train is a fire alarm, not a victory lap. Ask any team that "beat SOTA" for a week. Benchmark contamination in LLMs is this same picture at civilization scale — which is why serious labs now decontaminate training data against their eval sets and report it.

EP 05

Spike autopsy — recover, or roll back?

Real pretraining logs aren't smooth — they have incidents. Below is a long run with its gradient-norm co-signal (the second vital sign). Inject incidents yourself: a spike (one bad batch / momentary instability) usually self-heals in a few dozen steps; a slow divergence never does. The autopsy rule: judge the 50 steps after the anomaly, not the anomaly itself.

✓ good case: spike + full recovery — note it in the logbook, keep training✗ bad case: grad-norm climbs and loss follows — every step past that point is wasted money; roll back

—

Top: loss (log). Bottom: gradient norm. The injected spike is a transient lr kick; the divergence is a genuine slow blow-up of the toy weights — both play out through the real dynamics.

Big runs are managed like patients in ICU: checkpoints every few hundred steps, and the on-call engineer's question is always the same — "did it recover?" Loss spikes that heal are folklore in every LLM lab; the ones that don't are why checkpoint-and-rollback is non-negotiable infrastructure.