Knowledge Distillation, Playable

EP 01

The teacher knows more than the answer

Ask a big classifier "what is this picture?" and you can keep either its final hard label ("cat.") or its full soft distribution ("78% cat, 12% fox, 9% dog…"). That second thing is what Hinton called dark knowledge — it encodes that cats resemble foxes and nothing resembles a truck. Pick an input, raise the temperature to magnify the whisper, then switch to hard labels and watch all of that structure die.

Temperature T = 1.0

At T = 1 the teacher is nearly certain — the similarity structure is there but almost invisible. Raise T to amplify it.

Why it matters: a hard label carries at most a couple of bits per example; the soft distribution tells the student how the teacher sees the world — which classes are neighbors, which are absurd. ✓ Good case: warm soft targets expose "husky is 40% wolf" for free. ✗ Bad case: the hard label (or an ice-cold T) throws that away, and the student must rediscover it from scratch.

EP 02

The race: same student, two diets

Two identical tiny students train live in your browser on the same 24 examples from a teacher whose knowledge is the two-blob map on the left. One student gets the teacher's soft probabilities; the other only gets the verdicts. Press start and watch the learning curves — the score is real agreement with the teacher, measured on 625 held-out points every few epochs.

soft-target student: — hard-label student: —

Real gradient descent on a 17-parameter radial-basis classifier, full-batch, 1,500 epochs. The "teacher" is a fixed smooth function standing in for a big trained network.

The pattern: with only 24 examples, verdicts alone under-determine the boundary — the hard-label student guesses wrong in the gaps. Soft targets say how far each point is from the boundary, so the same 24 points teach far more. This is exactly why DistilBERT and the Gemma/Llama "distilled" small models train on teacher logits, not just labels.

EP 03

The shrink test — how small can the student go?

Distillation's promise: a much smaller model keeps most of the quality. But "most" has a floor. This machine distills six students of shrinking size — 37 parameters down to 2 — from the same teacher, live, and plots what survives at each size.

six students, trained one by one

Each bar is a freshly trained student (1,500 epochs each). Quality = agreement with the teacher on a 625-point grid.

The economics: quality falls gently, then off a cliff. ✓ Good case: cutting parameters ~4× typically keeps ~95% of the biggest student's quality — this is why a distilled 8B model can feel close to its 70B teacher. ✗ Bad case: below the task's intrinsic complexity (here, "two blobs" needs enough basis functions to draw two blobs) the student can't hold the shape at all, no matter how good the targets are.

EP 04

The student inherits the teacher's scars

A distilled student's ceiling is its teacher. This teacher has a blind spot — a patch (dashed circle) where it is confidently wrong about the truth. Distill the student and watch the mistake get copied, faithfully, into the smaller brain. Then train the same student on the truth instead and see the scar vanish — proving the error came from the teacher, not from being small.

—

Left: ground truth. Middle: the teacher (spot the inverted patch). Right: your student — train it and compare.

The ceiling: a student trained purely on teacher outputs can approach the teacher but essentially never exceed it — ✓ good case: 98% agreement means distillation "worked". ✗ Bad case: that 98% includes the teacher's bugs, biases and hallucination habits, copied at full fidelity. Distill a flawed model and you ship its flaws in a smaller box.

EP 05

Temperature — the dark-knowledge dial, and how to break it

Before distilling, the teacher's outputs are softened by a temperature T. Too cold, and soft targets collapse back into hard labels — the dark knowledge is squashed out. Too hot, and every class melts toward 50/50 — there's no signal left to learn. Pick a T, distill a fresh student (it trains instantly, for real), and build the curve yourself.

Distillation T T = 1

Each point you add is a real student, freshly trained at that temperature (800 epochs), scored against the teacher.

The dial: ✓ good case — a moderate T keeps the teacher's hints alive and the student lands near its best. ✗ Bad case — crank T high and targets flatten toward a coin flip: the student learns almost nothing (watch the boundary score crater). Freeze T near zero and you're back to hard labels from EP 02. In practice distillation papers tune T like a first-class hyperparameter, because it literally sets how much dark knowledge gets through.