Self-Attention vs Cross-Attention, Playable

EP 01

Flip the plug — where do keys and values come from?

The bottom row is a French sentence asking questions (queries). The top row is wherever the keys and values are plugged in. In self mode it's the same sentence; in cross mode it's an English source. Click any bottom token and switch modes — every weight you see is a live Q·K softmax, and the entropy readout tells you how focused the lookup is. Then plug it into a document about the stock market and watch the lookup die.

The whole distinction in one sentence: self-attention builds understanding within a sequence; cross-attention imports it from another one. And as the "wrong document" mode shows, cross-attention is only as good as its source — near-flat weights mean the value mix is noise. Retrieval-augmented models live and die by this.

EP 02

Watch French words find their English parents

“the black cat drinks milk” → “le chat noir boit du lait”. Click any French word and see its cross-attention over the English source, computed live from shared meaning vectors. Notice chat/noir: French puts the adjective after the noun, so the two attention lines cross — attention handles reordering that word-by-word alignment never could.

Good case: content words (chat, noir, boit, lait) each lock onto their true counterpart — this is the alignment that made attention famous in 2014 translation models. Bad case: click “du” — English simply has no word for it, so its attention smears. Function words are where crisp alignment quietly breaks down.

EP 03

A decoder's double life — one eye on each sequence

A real translation decoder uses both sockets at every step: self-attention (green arcs) over what it has already written — for grammar, agreement, fluency — and cross-attention (gold lines) into the source — for meaning. Step through the French sentence being written and watch the division of labor: when writing “noir”, self-attention checks “chat” (masculine → noir, not noire) while cross-attention fetches “black”.

Division of labor: self-attention = coherence with your own past, cross-attention = faithfulness to the source. The bad case is “du”: English has no such word, so cross-attention smears and self-attention finds nothing useful in the prefix either — the decoder must supply French grammar from its trained weights alone. Drop either socket and you get fluent nonsense or faithful gibberish.

EP 04

The wrong “bank” — when the clue head grabs nothing

Translating English “bank” into French forces a choice: banque (money) or rive (riverside). This tiny decoder uses two cross-attention heads: a white word head that finds “bank” itself, and a gold clue head that hunts the sentence for disambiguating context. All weights and the final choice are computed live. Try all three sources — especially the one with no clue at all.

Bad case: with no clue in the source, the clue head's attention goes flat and the model falls back on its frequency prior — it confidently prints banque even if you meant the riverbank. Cross-attention can only surface evidence that exists; when the source is ambiguous, the model doesn't say “unsure”, it just guesses fluently. This is a real failure mode of machine translation to this day.

EP 05

Words onto pixels — cross-attention paints where you point

In diffusion models like Stable Diffusion, image patches send queries and your prompt's words offer keys and values — that's cross-attention between text and pixels. Click a word of the prompt to see its live attention heatmap over the image. Then lower the binding strength slider: color words start leaking onto the wrong object — the exact reason image models sometimes give you a blue cat on a red mat when you asked for the opposite.

binding strength 1.00

Good case: object words (“cat”, “mat”) localize cleanly at any binding strength — nouns are easy. Bad case: attribute words (“red”, “blue”) depend on the model binding the adjective to the right noun; weaken that and the heatmaps overlap, which shows up in real generations as color bleeding and attribute swapping. Research fixes like Attend-and-Excite literally edit these cross-attention maps.