Interpretability Probes, Playable

EP 01

Open the skull — what an activation actually looks like

A network turns every sentence into columns of numbers — activations — one column per layer. Below, a toy 6-layer network reads sentence A and a sentence B of your choice. Left and middle: raw activations (8 neurons tall, 6 layers wide). Right: what actually moved between them.

Synthetic activations (structured signal + noise) built to mirror what probing studies measure in real transformers. Every number and accuracy on this page is computed live in your browser — nothing is pre-baked.

Two lessons at once: ✓ flip one property and a consistent pattern shifts — the information really is in there. ✗ but no single cell is "the tense neuron", and staring at raw heatmaps tells you almost nothing. You need an instrument. That instrument is a probe.

EP 02

Train a mind-reader in ninety gradient steps

A probe is a deliberately tiny classifier — here, one linear layer, 9 numbers total — trained to read the frozen network's layer-4 activations and predict a property of the input sentence. Pick a property and press train. The curve is real held-out accuracy, recomputed after every single gradient step.

Probe for

Why it matters: this is literally the method behind hundreds of "BERTology" papers: freeze the model, fit one linear layer on its activations, report accuracy. If a ruler this dumb can read a property, that property is sitting in the representation in plain sight.

EP 03

A depth map of where knowledge lives

Run the same probe at every layer and you get a depth profile of the network. Syntax-flavoured properties (tense, plurality) peak in early layers and fade; meaning-flavoured ones (sentiment) build up late. Drag the noise slider and watch the whole map degrade honestly — the 18 probes below retrain from scratch every time you move it.

Activation noise 0.40

Each point = one linear probe, freshly trained on 120 sentences and scored on 200 held-out ones. The wiggle between runs is real estimation noise — probing papers report it too.

The real version: exactly this experiment on BERT (Tenney et al., Hewitt & Manning) showed part-of-speech living early, parse structure in the middle, semantics late — "BERT rediscovers the classical NLP pipeline". Probes drew the first functional maps of transformers.

EP 04

The grammar nobody taught it

This toy model was trained on exactly one job: judge sentiment. Nobody ever gave it a grammar lesson. Probe its layer-2 activations for tense anyway — then run the sanity control every honest probing paper needs.

Run the three steps in order. Every bar is a freshly trained classifier, scored on held-out sentences.

The real version: language models trained only to predict the next word turn out to encode parse trees, coreference chains, even a full board state in Othello-GPT — none of it in the training objective. Probes are how researchers caught them doing it.

EP 05

The probe that lies — encoded is not used

At layer 4 a probe reads plurality with high accuracy. So the model must be using plurality to judge sentiment… right? There is exactly one way to find out: surgically delete that direction from the activations the model reads, and watch what happens to its behaviour.

Deleting a direction = projecting every activation onto the plane orthogonal to the probe's learned weight vector, then re-running the frozen model.

Why it matters: a probe proves information is present, never that it is causally used. This trap is exactly why modern interpretability moved from probing to interventions — ablation, activation patching, steering — where you edit the wire and watch the behaviour.

EP 06

Give the probe a brain and it starts hallucinating

These activations are pure static — random numbers containing zero information — and the "tense" labels are coin flips. A linear probe stays honest about that. Now crank the probe's capacity up into an MLP and watch it "discover" tense anyway. On its own training set, that is.

Probe capacity

The rule of the field: a powerful probe is a student who answers from their own knowledge, not the textbook's. That is why serious probing work keeps probes linear (or reports "selectivity", Hewitt & Liang) and always runs the shuffled-label control you pressed in EP 04.