Attention (Queries, Keys, Values), Playable

EP 01

Click a word, watch it interview the sentence

Each word below carries a hand-crafted query vector (what it needs) and a key vector (what it offers). Click any token: we compute its query · key dot product against every other token, softmax the scores, and draw the resulting attention weights as arcs and bars — live, no fakery. Adjectives hunt for nouns, the verb hunts for its participants, articles hunt for the thing they introduce.

distance penalty (like learned position bias)

Query/key vectors here are hand-set for clarity; real transformers learn them from data. The math — dot products, softmax — is the real thing.

The mental model: attention is a soft database lookup that happens for every token in parallel. Click "chased" — a verb genuinely attends to both its animals at once. Now untick the distance penalty and click "hungry": with two similar nouns and no position signal, its attention nearly ties between cat and mouse. Real transformers face exactly this, which is why they learn positional biases.

EP 02

The atom of attention — drag two arrows, read one number

Everything in attention reduces to Q·K: multiply matching dimensions, add them up. Geometrically that's alignment × size. Drag the white query and the two keys below. The dashed line shows each key's projection onto the query — that projection times the query's length is the dot product. The bars show which key wins after softmax.

Drag any arrowhead. Green and red are two competing keys; white is the query.

The catch you just found: the dot product confuses pointing the right way with being loud. In the "magnitude hijack" preset, a key that points 70° away still wins purely by being huge. That failure is real — it's why transformers normalize activations (LayerNorm) and divide scores by √d before the softmax.

EP 03

The sharpness knob — from mush to tunnel vision

Raw scores aren't weights yet — softmax turns them into a probability-like mix, and a single scale factor decides how peaky it is. The query here is "it" in “the cat saw the mouse in the kitchen and suddenly it pounced”; each candidate token carries a color as its value. The output swatch is the actual weighted blend of those values — what "it" becomes after attention. Slide the sharpness and watch both the weights and the blend change.

output value mix

sharpness 1.00

Why √d matters: in high dimensions raw dot products get huge, which slams softmax into the "too sharp" regime you just saw — one-hot, brittle, gradient-dead. Dividing by √d keeps attention in the useful middle zone where it can hedge between "cat" and "mouse" instead of betting everything on one.

EP 04

The famous "it" — one pronoun, two possible bosses

“The animal didn't cross the street because it was too ___.” Who is "it"? If the sentence ends in tired, "it" means the animal; if wide, it means the street. Here the final adjective reshapes the pronoun's query vector, and the attention weights — computed live from Q·K — flip accordingly. This is the exact example the original transformer paper used to show off.

Pick an ending above.

Good case: with a clue like "tired" or "wide", attention snaps to the right noun — context literally rewires who looks at whom, something fixed pipelines never managed. Bad case: with a useless clue ("just there") the weights split almost 50/50; the model must guess, and this is precisely where real LLMs get pronouns wrong.

EP 05

Death by dilution — drown the signal in a long sentence

Softmax has a quiet weakness: every extra token takes a bite of the probability mass, even if its score is small. The pronoun "she" below is trying to attend back to "the architect" — but a distracting "the lawyer" sits in the middle, and you control how many filler words pad the gap. Watch the architect's attention share, computed live, melt as the sentence grows.

filler words in between 0

Bad case: nothing "broke" — every step of the math is correct, yet at 40 fillers the pronoun's value mix is mostly noise and the lawyer looks nearly as good as the architect. Good case: keep the gap short and attention is razor sharp. This dilution is a core reason long-context models lose facts "in the middle" of a document, and why retrieval and re-ranking still matter.