Overfitting vs Generalization, Playable

EP 01

The capacity dial — from too dumb to too eager

Twelve solid dots are the training set (a gentle curve plus noise). Six hollow dots are the test set — same truth, new samples, never shown to the model. Drag the degree slider: at 1 the model is too rigid to learn the shape (underfit); around 3 it captures the truth; past 9 it starts threading every noisy dot exactly — and the hollow dots drift away from the curve. Both error numbers are live.

✓ good case: degree 3 — train and test errors both small and close✗ bad case: degree 12 — train error ≈ 0, test error 30× worse

polynomial degree 1

—

Every drag re-solves the least-squares fit from scratch on the same 12 points. The wiggle you see at high degree is the model spending its extra capacity on noise.

Fitting the data is not the goal — fitting the process that generated it is. Train error measures the first; only held-out data measures the second. This one distinction is why every ML project on earth splits its data before doing anything else.

EP 02

The scissors — watch train and test split mid-training

Overfitting isn't only about model size — it happens over time. Here a flexible model trains step by step on scarce data while both losses are tracked. They fall together, then the scissors open: train keeps dropping, test turns around and climbs. The gold marker is the moment you should have stopped. Add more data and watch the scissors open later and less.

✓ good case: stop at the marker — best model you'll ever get from this data✗ bad case: train to the end — train loss 4× better, test loss 2× worse

training points 15 —

Genuine gradient descent on a degree-9 polynomial; the early-stopping marker is placed at the measured minimum of the test curve, after the run finishes.

Early stopping is the oldest free lunch in ML: monitor a validation set, keep the checkpoint where it bottomed out, ignore that training loss "could still improve." The gap between the two curves is memorization happening in real time.

EP 03

The smoking gun — make it memorize pure nonsense

Skeptical that "low training error" can be completely meaningless? Do the famous experiment (Zhang et al., 2017, in miniature): press the shuffle button to randomly reassign every training answer, destroying all real structure. Then fit. The flexible model still drives training error to nearly zero — it's memorizing pure noise — while test error sits at "guessing" level. Same model, same optimizer, same zero training error, opposite meaning.

✓ good case: real labels — train ≈ 0 AND test low: it learned the pattern✗ bad case: shuffled labels — train ≈ 0 and test terrible: it learned the answer key

—

The shuffle genuinely permutes the y-values among the same x-positions each click — there is nothing left to learn, and the fit happens anyway.

Capacity to memorize proves nothing about learning. Modern nets can memorize any labeling of their training data — the original paper showed ImageNet-sized nets hitting zero error on random labels. Generalization lives in the data's structure and the training recipe, not in the loss hitting zero. When a benchmark score looks too good, this experiment is why you ask about contamination.

EP 04

Drown the noise — data is the strongest regularizer

Keep the dangerous degree-10 model, but feed it more. Drag the slider from 10 points to 200 and watch the same over-eager polynomial straighten out: with 10 points the noise is the dataset; with 200 points the noise averages away and even a crammer is forced to learn the pattern. Test error is measured live at every position.

✗ bad case: degree 10 on 10 points — spaghetti through every noise spike✓ good case: degree 10 on 200 points — capacity tamed, test error near the noise floor

training points 10 —

Model capacity is frozen at degree 10 throughout — the only thing changing is evidence.

When in doubt, get more data. Doubling the dataset routinely beats a month of architecture cleverness — this slider is why. It's also the quiet engine of the LLM era: the models got big, but what made big safe was trillions of tokens doing exactly what this slider does.

EP 05

The pencil-weight penalty — and its own overdose

Can't shrink the model, can't buy data? Charge the model for complexity. Ridge regularization adds a fee proportional to the size of the coefficients, so wild wiggles have to pay rent. Drag λ up from zero: the curve calms, test error falls… then keep going and the fee gets so heavy the model can't afford the true pattern either. The U-curve on the right is measured from real fits at every λ.

✗ bad case ×2: λ=0 memorizes noise · λ=1000 too broke to learn anything✓ good case: the U's bottom — wiggle gone, pattern intact

λ (log) 0.0001

—

Degree fixed at 12, data fixed at 14 points. The right-hand U is recomputed by refitting at 40 λ values when the page loads — your slider position is the gold dot on it.

Every regularizer is a tax with a Laffer curve. Weight decay, dropout, early stopping, small batches — all charge the model for flexibility, all can be overdosed into underfitting. Tuning ML is mostly finding the bottom of this U — and it's why "just add more regularization" is as wrong as "just add more capacity."