Backpropagation, Playable

EP 01

One neuron, one knob — the derivative is a prediction

Before chains and layers: one knob. The neuron computes y = w·x, the loss is how far y is from the target. The gradient claims: "nudge w by a tiny ε and the loss will change by gradient×ε." That's a testable prediction — press the wiggle button and we compare the claim against an actual measured nudge.

✓ good case: tiny nudge — prediction matches measurement to 4 decimals✗ bad case: giant nudge — the local slope lies about faraway terrain

weight w 0.60 nudge ε 0.010

—

x = 1.5, target = 3.0, loss = (y − target)². The "predicted" number uses the analytic gradient 2x(wx−t); the "measured" number actually recomputes the loss at w+ε.

A gradient is not magic — it's a local promise. "If you turn this knob a hair, wrongness changes by this much." Everything else in deep learning is bookkeeping to get that promise for every knob cheaply.

EP 02

The chain rule — blame is a product, and one zero kills it

Now put three blocks in a row: y = w₂ · ReLU(w₁·x). How much is w₁ to blame for the final loss? Multiply the local slopes along the path — that's the whole chain rule. Each box below shows its live local slope; the product is compared against a real measured wiggle of w₁. Then drag w₁ negative and watch the ReLU box flatline the entire chain.

✓ good case: chain product = measured wiggle, every time✗ bad case: ReLU asleep → local slope 0 → w₁ gets zero blame forever (dead neuron)

w₁ 0.80 w₂ 1.20

—

x = 1.0, target = 2.0. Chain shown: dLoss/dw₁ = dLoss/dy · dy/dh · dh/da · da/dw₁ — four local slopes, one multiplication.

Backprop is just this, repeated. One backward sweep reuses each local slope to price every knob in one pass — that's why training a billion parameters costs about 2× a forward pass, not a billion×. And the dead-ReLU you just made is a real production bug: a neuron whose blame is stuck at zero never learns again.

EP 03

Forward glow, backward blame — train a real net, live

This is an actual 2-2-1 neural network learning XOR in your browser — the classic problem a single neuron can't solve. Gold glow flowing right is the forward pass (activations); the numbers on each edge are that weight's current blame from the last backward pass. Press auto and watch the loss die. Then press the sabotage button.

✓ good case: random init — loss 0.25 → 0.002, XOR solved✗ bad case: all-zero init — perfect symmetry, both hidden neurons stay twins, loss frozen

loss: —

Real gradients, real training — sigmoid activations, learning rate 2.5, the four XOR examples each step. Nothing is scripted; the zero-init freeze is the genuine mathematics of symmetry.

Why nets start random: backprop assigns identical blame to identical neurons. Start twins, stay twins — the network can never use its width. One line of init code separates "learns XOR in 2 seconds" from "learns nothing, ever."

EP 04

The whisper chain — make the gradient vanish

Blame travels backwards as a product — and a sigmoid's local slope is at most 0.25. Stack layers and the product shrinks like 0.25 × 0.25 × … Drag the depth slider and read the bars: each bar is the actual measured gradient reaching that layer. At depth 15, layer 1 hears a whisper of 10⁻⁹ — it will effectively never learn. Then flip the activation to ReLU and watch the chain survive.

✗ bad case: 15 sigmoid layers — front layers get ~10⁻⁹ of the blame✓ good case: same depth with ReLU — the signal reaches the front intact

depth 6 layers —

Bars are on a log scale and computed by running real backprop through the chain (weights 1.0, input 0.5) — not a formula overlay.

This one picture froze deep learning for ~20 years. Pre-2010 nets deeper than a few layers simply didn't train. ReLU, residual connections and LayerNorm are all plumbing to keep this product from dying — a transformer is, among other things, a machine for keeping gradients alive across 100 layers.

EP 05

The avalanche — explode it, then clip it

Same chain, opposite disease. Make each layer's weight 1.8 and the backward product now grows exponentially: at depth 20 the front-layer gradient is in the tens of thousands. Take one training step with that and the weights leave orbit — the loss readout goes to infinity. Now switch on clipping, which caps the gradient's overall size before the step, and the same net trains calmly.

✗ bad case: weight 1.8, depth 20, no clip — one step, loss = ∞✓ good case: same net, clip at 1.0 — steps stay sane, loss falls

layer weight 1.80 depth 20 —

The gradient norm and post-step loss are computed from the actual chain each click. Linear layers used here to isolate the product effect.

Gradient clipping is the seatbelt of every big training run. RNNs made this famous; LLM logs still show "grad_norm" spikes getting clipped daily. The lesson of EP 04 + 05 together: backprop is a product, and products only know two speeds — dying and exploding. All of training stability is keeping it near 1.