Double Descent, Playable

EP 01

Three regimes — calm, possessed, calm again

Twenty noisy data points, one model whose size you control (number of random features). Visit the three regimes: small (P=8) — smooth but too simple; the interpolation threshold (P=20, exactly as many knobs as data points) — the fit threads every point and thrashes violently between them; heavily overparameterized (P=80) — still hits every point, yet the curve between them goes… calm. Same data. Bigger model. Better behavior.

✗ bad case: P = n — possessed spaghetti, the textbook's overfitting nightmare made flesh✓ good case: P = 4n — interpolates the same points politely

—

Model: linear regression on P random Fourier features, solved by (near-)minimum-norm least squares — the standard clean setting where double descent provably appears. Every curve you see is a fresh solve.

The middle picture is the one your intuition fears; the right picture is why deep learning works anyway. When knobs vastly outnumber data, infinitely many models fit perfectly — and the optimizer picks the flattest of them. Excess capacity isn't rope to hang yourself; it's room to be graceful.

EP 02

The full sweep — draw the curve that broke the textbook

Press the button: forty model sizes train back to back, and their measured train and test errors plot themselves. You'll watch the classical U form — then the spike at P=n — then the modern surprise: the second descent, often ending below the classical optimum. This is the chart from Belkin et al. and OpenAI's "Deep Double Descent," rebuilt from live fits.

✗ bad case: stopping the sweep at the peak — "see, bigger is worse!" (a decade of textbooks did)✓ good case: the far right — biggest model, best test error of the whole sweep

label noise 0.25 —

Test error is measured on 200 held-out points per model size. Crank the noise and re-sweep: the peak grows — the spike feeds on noise.

"Bigger overfits" wasn't wrong — it was a local truth. The U is real, it just isn't the whole road. Modern LLMs live far to the right of the peak, which is why "this model is absurdly overparameterized" and "this model generalizes beautifully" stopped being a contradiction.

EP 03

Autopsy of the peak — measure the contortion

Why does error explode exactly at P=n? Because there the model has no choice: exactly one solution threads all twenty noisy points, and hitting noise exactly requires violent coefficients. The gold curve below is the measured size (norm) of the model's weights at each P — watch it spike precisely at the threshold and relax after, when the model regains freedom to choose among many perfect fits.

✗ bad case: P=n with noisy labels — weight norm 40× baseline, wild output between points✓ good case: P≫n — many interpolating solutions exist; min-norm picks the calmest

label noise 0.25 —

Set noise to zero and re-measure: the spike nearly vanishes. The peak is the price of taking noise literally with zero slack — no noise, no contortion.

The peak is forced dishonesty. At the threshold the model must swear the noise is signal. This is also why a pinch of ridge regularization flattens the whole spike — slack, from any source, is the antidote.

EP 04

Descent over time — when "train longer" beats "stop early"

Double descent also happens along the time axis. Watch this run: test error falls, rises (the memorization bump — our overfitting page would tell you to stop right there), then falls again to a deeper minimum as slower, genuine structure finishes learning. Early stopping at the first dip would have kept the worse model. Crank the label noise to make the bump — and the dilemma — bigger.

✗ bad case: obey the early-stopping rule at step ~150 — you keep a model 2× worse than the late one✓ good case: patience through the bump — the second valley is the true optimum

label noise 0.3 —

Conceptual simulation of epoch-wise double descent (fast memorization + slow structure learning), honest about being a model of the dynamics rather than a real transformer. The tension with the early-stopping advice on our overfitting page is real and regime-dependent — that's the point.

Two pages of this site now disagree, and both are right: in the classical regime (small model, scarce data) stop early; in the overparameterized regime, training through the bump can reach a better basin — grokking is this phenomenon in its most extreme form. Knowing which regime you're in is the actual skill.

EP 05

The cruelest twist — more data can make things worse

Hold the model fixed at P=25 features and grow the dataset. Common sense says error only falls. Watch the measured curve: as n climbs toward 25, test error rises — you're dragging the interpolation threshold onto yourself — then falls once data safely outnumbers knobs. There is a documented regime where hiring more data hurts.

✓ good case: n = 12 or n = 60 — both comfortably away from the threshold, both fine✗ bad case: n = 25 = P — the worst test error on the whole curve, achieved by adding data

—

Each point: fresh data of size n, same P=25 model, test error on 200 held-out points, averaged over 8 seeds — the bump survives the averaging.

Sample-wise double descent (Nakkiran et al.) is the best argument that this is real structure, not a curiosity: capacity, time, and data all show the same threshold scar. Practical rule: never operate a model near its interpolation threshold — cross it decisively (more capacity, regularization, or a lot more data) or stay classical.