Feature Superposition, Playable

EP 01

The impossible closet — pack five ideas into two neurons

Two neurons give you a 2-D space. Each feature the network wants to store becomes an arrow (a direction) in that space. Two arrows can be perfectly perpendicular — after that, every new feature must squeeze in at an angle. Keep pressing add and watch the overlap grow: that overlap is interference, the tax on every readout.

Why it matters: real models face this exact squeeze at scale — millions of concepts, thousands of neurons per layer. Anthropic's toy-models work showed features under pressure snap into these same geometric arrangements (antipodal pairs, pentagons) all by themselves.

EP 02

The sparsity loophole — overlap is fine if nobody shows up

Five overlapping features, two neurons, thousands of inputs. Each feature fires on a random fraction of inputs — that fraction is the slider. The histogram shows every readout the decoder makes: gold = features that were truly on (should read 1), grey = features that were off (should read 0). Slide right and watch the two piles smear into each other.

Feature activity

Readout = dot product with the feature's direction, "detected" above 0.85. Real learned toy models use ReLU with a negative bias — same idea: small interference gets filtered, big collisions get through.

The key fact about the world: real features are sparse. Almost no sentence is about DNA, the Golden Gate Bridge, or 15th-century tax law — so their directions can overlap almost for free. Sparsity is why superposition works in LLMs, not just in toys.

EP 03

Why the network does it on purpose

The network has two honest options: dedicate its two neurons to two features and drop the rest, or superpose everything and eat the interference. Which is cheaper? Both strategies run live below across the whole sparsity range — errors per input, measured, not derived. Find the crossover.

Features to store

Each point: 350 simulated inputs. Dedicated = store 2 features perfectly, ignore the rest. Superposed = pack all of them and read through the interference.

The punchline: superposition is not sloppiness — left of the crossover it is the loss-minimizing choice, so gradient descent walks straight into it. Anthropic's toy models show it appearing as a sharp phase change exactly when features get sparse and numerous.

EP 04

The neuron that means everything — and nothing

Now look at the same system the way early interpretability did: one neuron at a time. Present each pure concept and watch the two neurons respond. Then try to finish the sentence "neuron 1 is the ______ detector". You can't — and that's the point.

Present a few different concepts and keep an eye on which neurons cross the firing threshold.

Why single-neuron interpretation died: in superposition, neurons are just the axes the hardware happens to use — concepts live in directions between them. Real LLM neurons famously fire for "academic citations, English dialogue, HTTP requests and Korean text" at once. The unit of meaning is the direction, not the neuron.

EP 05

Crash test — make the features collide

You are the input now. Toggle features on and off, and the two neurons store their sum; the decoder then tries to read every feature back. One feature at a time works perfectly. Then press cat + car — the interference adds up into a phantom "mug" the network never saw. Press cat + moon and both vanish entirely.

The real-world echo: interference under dense activation is a genuine failure mode of compressed representations — one proposed mechanism behind weird LLM slips. It is also the target of sparse autoencoders, which try to un-superpose these directions back into clean, readable features.