One token, one dial — position becomes an angle
Take a 2D slice of a query vector. Its direction encodes content; RoPE encodes position m by rotating it an extra m·θ. Drag both sliders: content sets the white ghost arrow, position spins the gold one further around. The readouts are computed live from the rotated vector — notice what never changes.
θ is fixed at 15° per position in this slice. Real RoPE does this to every consecutive pair of dimensions in Q and K, each pair with its own θ — see EP 03.
The punchline: attention only feels m − n
Rotate a query at position m and a key at position n, then take their dot product — the score attention actually uses. Because rotations subtract inside a dot product, the score is cos(Δcontent + (m−n)·θ): it depends only on the gap m−n. Hit "shift both +7" and compare against the additive scheme running in parallel.
Move the sliders, then shift. The additive baseline adds a sinusoidal position vector to the same content vectors — the standard pre-RoPE recipe.
Eight clock hands — why one frequency isn't enough
A single rotating pair aliases: after one full turn, position 0 and position ~6 look identical. So RoPE splits the vector into pairs, each rotating at its own speed θi = 10000−i/8 — fast hands resolve neighbors, slow hands resolve chapters. Drag the position and toggle down to one hand to see the aliasing yourself.
The alias readout scans positions 0–300 live for the one whose fastest hand lands closest to the current angle. The full-code distance readout uses all 8 hands (16 dims).
Slide the whole window — the attention pattern that wouldn't move
Here a query attends over 12 keys. The white outline is the attention-score pattern at window start 0. Now drag the window deep into the document: with RoPE (top), every relative offset is unchanged, so the bars never leave the outline. The absolute-position model (bottom) computes different scores for the same phrase just because it moved.
Both models score the same 12 content vectors; only the position mechanism differs. Deviation = largest change of any bar versus its own start-0 baseline, recomputed live.
The cliff past the horizon — where RoPE still frays
Relative isn't a free lunch. Suppose training only ever showed gaps up to 32 tokens (shaded). The curve is the live attention score between two identical-content tokens as their gap grows. Past the shaded zone the slow hands enter rotation angles the model has never seen — the familiarity meter drops and the score wanders through never-trained territory. Then apply NTK-style rescaling and watch the fix.
Honesty label: the score curve is exact RoPE math, but "degradation" in a real model comes from its weights meeting unseen angles — our familiarity meter (fraction of the 8 hands sitting at an angle visited during the 0–32 training range) is a proxy for that, not a measured perplexity.