Sparse Autoencoders, Playable

EP 01

Too many thoughts, too few neurons — superposition

This toy "model" stores 8 concepts in only 6 neurons. That's impossible if each concept gets its own neuron — so instead each concept is a direction across all 6, and directions overlap. Toggle concepts on and off and watch the neurons: every neuron responds to several concepts at once, and mixtures pile up into an unreadable smear. The left grid is the ground truth — how much each concept leans on each neuron.

The 8 concept directions are real vectors in 6-D, spread as far apart as geometry allows. This packing trick is exactly what "superposition" means in interpretability papers.

Why it matters: the good case is capacity — the network stores 8 things in 6 slots, and real LLMs store millions of concepts in thousands of neurons. The bad case is the price: no single neuron means anything. That's why "what does neuron 3,412 do?" was a dead end for interpretability — and why we need an unmixer.

EP 02

The unmixer — watch a dictionary learn itself

Now bolt on a sparse autoencoder: 16 candidate feature slots that must reconstruct the 6-D activation while keeping as few slots active as possible. It starts random — press Train and watch the bottom row organize: slots lock onto the true concepts (they take their color when their direction matches, cosine > 0.75), and each input lights up only the slots for the concepts actually in it. Untrained = dense grey mush. Trained = sparse, labeled, readable.

A real SAE, really training: every click runs gradient descent (reconstruction loss + L1 sparsity penalty, decoder columns kept unit-norm) in your browser. Only the data is a toy.

What just happened: nobody told the SAE what the 8 concepts were. Sparsity pressure alone forced it to rediscover them, because "few slots per input" is only achievable if slots align with the true ingredients. This is exactly how Anthropic pulled millions of human-readable features out of Claude's residual stream.

EP 03

How many slots? Merge, match, or shatter

The one hyperparameter everyone fights about: dictionary size. Drag the slider — each position trains a fresh SAE from scratch, live — and read the match matrix (rows = the 8 true concepts, columns = learned slots, bright cell = aligned). Too few slots and one slot is forced to merge several concepts. Around 8–16 you get a clean diagonal-ish match. Go big and slots die (never fire, marked ✕) or one concept splits into shards.

Dictionary slots 8

Every slider move trains a brand-new SAE for 3,500 gradient steps right here — the counters are measured from that run, not scripted.

The real-world echo: at 4 slots you get "bridge-slash-sadness" chimera features — merged, uninterpretable. At 32 you get dead slots (wasted compute) and feature splitting: "coffee" shatters into near-duplicates. Production SAEs face the same dial, just with 34 million slots instead of 32 — and splitting is why bigger dictionaries keep finding "new" features that are really shards of old ones.

EP 04

Grab a feature and steer — the Golden Gate trick

Interpretability without causality is stamp collecting. The test: take a feature the SAE recovered on its own, inject it into the activation, and see if the model's output changes in exactly that one way. The base activation below is "coffee + rain" (a cozy morning). Pick a recovered feature, crank the slider, and watch the word stream tilt. Then try steering a raw neuron instead — same strength, but it drags several unrelated concepts because neurons are cocktails.

Steer Strength 0.0

This is literally the Golden Gate Claude demo: Anthropic found a bridge feature with an SAE, clamped it high, and Claude worked the Golden Gate Bridge into every answer. Clean single-concept steering is the proof the features are real — and the overcrank collapse you can trigger here (strength ≳ 3) is why clamped-feature models also got weird and obsessive.

EP 05

The sparsity tax — clean features aren't free

One knob rules the SAE: the L1 penalty λ, the price charged per unit of activation. Drag it and a fresh SAE trains at that price; each run drops a dot on the chart (activity per input vs. reconstruction error). Cheap activations → the SAE reconstructs almost perfectly but smears every input over many slots: dense, unreadable, 0 concepts recovered. Expensive → beautifully sparse and interpretable, but reconstruction crumbles and slots start dying: the SAE is now lying about what the model computed.

Sparsity price λ 0.15

Every dot is a full 3,500-step training run at that λ, measured on 300 fresh probe inputs. Nothing is pre-baked.

The uncomfortable truth: an SAE is a lossy compression of the model's thoughts, and λ sets what gets thrown away. Too honest (low λ) and you learn nothing; too tidy (high λ) and your "explanation" no longer matches the computation. Every published SAE sits somewhere on this frontier — when you read "we explain 65% of the variance", this chart is what they're confessing to.