Dropout, Playable

EP 01

The clique — click a neuron and watch the house of cards

This 8-neuron net was just trained (in your browser, on page load) without any dropout. It fits the data beautifully — but look how it did it: some neurons grew huge opposite weights that cancel each other out. That's co-adaptation: a fragile conspiracy, not eight independent detectors. Click any neuron to fire it and watch the prediction curve — and the measured error — react.

✓ good case: the full team — error tiny✗ bad case: fire one keystone neuron — the whole fit collapses

—

Neuron size = |output weight|. The net is genuinely retrained fresh each page load, so which neuron is the keystone varies — hunt for it.

Co-adaptation is technical debt inside the weights. Neurons stop learning features and start learning to patch each other's mistakes. It fits the training set fine — and shatters on anything new. This fragility is the disease dropout was invented to cure.

EP 02

Layoffs live — train through the flicker

Now train with dropout. Every step, each hidden neuron survives only with probability 1−p — the dead ones flash red and contribute nothing for that step. No neuron can rely on a partner that might vanish, so each is forced to be independently useful. Try p = 0.5, then push to 0.8 and watch the layoffs become so brutal the team can't learn at all.

✓ good case: p = 0.5 — noisy ride, healthy final fit✗ bad case: p = 0.8 — five of eight fired per step, permanent underfit

dropout p 0.5 —

Real training, ~40 steps/frame. The flashing mask is the actual mask used in that step's forward and backward pass — inverted dropout (activations scaled by 1/(1−p)) so no test-time correction is needed.

Dropout is a reliability tax collected during training. The flickering makes optimization slower and noisier — that's the price. What it buys: no neuron can be a specialist in patching another's bug, because its partner might be dead this step.

EP 03

The ablation exam — who handles losing a teammate

The fair fight. Press the button: two identical nets train from the same init — one plain, one with dropout 0.5. Then the exam: remove each of the 8 neurons in turn and measure how much the error jumps. The bars are the measured damage. Plain nets have keystones — one bar towers over the rest. Dropout nets took the layoffs in training, so no single neuron matters much.

✗ bad case: plain net — one ablation multiplies error ~10×✓ good case: dropout net — every ablation is survivable, damage spread flat

—

Both nets: 8 hidden units, same data, same steps, same seed for the init. Sixteen ablations measured per click. Numbers change run to run — the pattern doesn't.

This redundancy is the whole point. Dropout-trained representations are distributed: knowledge lives in many places at once. It's the same principle as RAID for disks or "no single point of failure" in infrastructure — enforced on neurons by random firing.

EP 04

The volume bug — everyone came back and now it's too loud

Classic real-world bug. Train with p = 0.5 the "naive" way (no scaling during training). At test time all neurons are back — so the hidden layer's total output is roughly 2× louder than anything the output layer saw during training. Toggle the fix and watch the prediction curve snap onto the data. The offset you see is measured, not drawn.

✗ bad case: forget the rescale — every prediction systematically inflated✓ good case: multiply activations by (1−p) at test — curve lands on the data

—

Modern frameworks use "inverted dropout" (scale by 1/(1−p) during training) precisely so nobody can forget this step. This machine trains the naive variant on purpose so the bug is visible.

The mean must match. Whatever noise you inject during training, test-time behavior has to preserve the expected signal the next layer calibrated to. Forgetting this is a rite of passage — and the reason your from-scratch implementation "trains fine but predicts garbage."

EP 05

The hidden ensemble — one net, many opinions

Deepest way to see dropout: every random mask defines a different sub-network, and training visits millions of them with shared weights. So after training you own an ensemble. Keep dropout ON at test time and sample: each thin gold line is one masked sub-net's opinion. Crank K and watch the white average settle — and notice the opinions fan out exactly where there's no data.

✗ bad case: trust a single sampled sub-net (K=1) — jagged and confidently wrong✓ good case: K=50 average — smooth fit, plus a free uncertainty band

samples K 1 —

This is Monte-Carlo dropout (Gal & Ghahramani): the spread of the gold lines is a usable uncertainty estimate. Data lives only in the middle of the range — watch the fan at the edges.

Dropout ≈ training 2⁸ sub-networks for the price of one (here; 2 to the billions in real nets), then averaging them. That's why it regularizes, and why leaving it on at inference gives you error bars almost for free — a trick still used in medical and robotics models today.