Positional Encoding, Playable

EP 01

Prove it: attention can't see order

Self-attention pools word vectors from every position at once — the math is a weighted sum, and sums don't care about order. Below, two sentences are turned into pooled vectors (the stripe patterns). Toggle position encoding OFF and shuffle: the two stripes become bit-for-bit identical, no matter how scrambled sentence B gets.

Toy setup: each word gets a fixed 24-dim vector and the sentence vector is a pooled sum (what attention reduces to). With encoding ON, each word vector is bound to its slot's sinusoidal code before pooling — the same word-meets-position interaction attention's Q·K dot products exploit. Real transformers are bigger but share the same blindness.

Why it matters: the transformer paper's authors had to bolt position codes onto the input before layer one, because nothing downstream can recover order that was never encoded. Every GPT you've used starts with exactly this fix.

EP 02

Every position gets a wave barcode

The classic fix: give position p a vector of sines and cosines at different frequencies — fast waves on top distinguish neighbors, slow waves below distinguish paragraph-scale distances. Each column in the heatmap is one position's barcode. Now starve it: drag dimensions down and watch far-apart positions start to collide.

Position A 3 Position B 30 Dimensions 16

We use wavelength base 100 instead of the paper's 10,000 so the slow waves visibly change within 48 positions — same formula, zoomed in.

The design trick: it's a binary counter made smooth — like a clock with 8 hands ticking at different speeds. With all 16 dims, all 48 positions are unique; with 2 dims (one wave), the code repeats every wavelength and position 3 impersonates position 28. Multi-frequency is what makes the barcode collision-proof.

EP 03

The negation trap — same words, opposite meaning

Order blindness isn't a cosmetic bug. These sentence pairs use the exact same multiset of words, but mean the opposite — because "not" flips whatever comes next. Score them both ways and watch the bag-of-words model literally return the same number for both.

Real-world echo: pre-transformer bag-of-words sentiment models famously scored "not bad, actually good" and "not good, actually bad" identically. Position information is what lets a model bind "not" to its target — the difference between reading and keyword-matching.

EP 04

Scramble it — position codes carry the receipt

Once a position code is added into each token's vector, the token carries its slot number with it wherever attention shuffles the data. Shuffle the tokens below, then hit restore: with codes ON, each token's barcode is decoded back to a slot and the sentence snaps back perfectly. With codes OFF there is nothing to sort by — restoring is a blind guess.

Decoding here is nearest-neighbor: each token's carried barcode is matched against the 9 known position codes. That's a toy stand-in for what attention layers learn to do implicitly — but the information content is the real thing.

Why it matters: inside a transformer, tokens are processed as an unordered set at every layer. The only reason the output isn't word salad is that each vector smuggles its own address. Delete the address and this episode is what the model experiences permanently.

EP 05

Past the edge of the map — learned vs sinusoidal

There are two classic ways to get position codes: learn a lookup table (GPT-2 style, one row per position, trained up to some length) or compute sinusoids (a formula that works at any position). Drag the query past position 32 — the table's edge — and watch one scheme keep producing clean barcodes while the other returns garbage.

Query position 20

The "learned" table here is random-but-fixed vectors for positions 0–31, which is what an untrained embedding table is; past row 31 the index simply doesn't exist, so we show the undefined read as noise. Real GPT-2 hard-truncates at 1024 for exactly this reason.

The cliff is real: GPT-2 cannot process token #1025 — its position table ends. Sinusoids extend forever but absolute positions still drift out of the trained distribution. The modern answer is to encode relative position by rotation — that's RoPE, the next page in this saga.