The KV Cache, Playable

EP 01

Attention re-reads everything — unless you keep notes

To pick word #10, attention must compare against a key and value vector for every one of the 9 words before it. A naive model recomputes all of them, every step. A cached model computes each token's K/V once, files it away, and only computes the newest one. Step through generation and watch what actually gets computed (gold = computed this step, grey = pulled from the cache).

Each square is one token's key/value pair. Real models compute one K/V pair per layer per head — same story, multiplied a few thousand times.

The trick in one line: K and V for token 7 never change once written — so computing them again at step 8, 9, 10… is pure waste. Caching them is why ChatGPT can answer your 50th message without silently re-processing messages 1–49 for every single word.

EP 02

The race — quadratic re-reader vs linear note-keeper

Give both models the same compute budget per second and let them race to generate N tokens. The no-cache model pays 1+2+3+…+N units (each token costs more than the last); the cached model pays 1 unit per token, flat. Watch the odometers — the gap is not a constant factor, it widens with every token.

Tokens to generate 19

At 19 tokens the cache is exactly 10× cheaper — drag the slider right and the ratio keeps climbing. Nothing is hardcoded: both odometers burn identical units/second.

Why it matters: without the cache, a 1,000-token reply costs 500,500 token-computations instead of 1,000 — a 500× tax. This single optimization is the difference between LLM chat being a product and being a demo. The bad case is simply turning it off.

EP 03

The memory meter — your chat physically lives in VRAM

The cache isn't free: every token's K/V vectors sit in GPU memory for the whole conversation. Per token that's 2 × layers × model-width × bytes. Pick a model, a GPU and a precision, then drag the context slider and watch the cache eat the memory left over after the weights — until it hits the wall.

Model GPU KV precision GQA (share KV across 8 heads)

Context length

The memory wall: a 7B model at fp16 stores ~0.5 MB of cache per token — a 128k-token chat is ~64 GB of pure notes, more than the weights themselves. This is exactly why GQA, MQA and KV quantization exist: they attack the cache, not the model. Good case: GQA + int8 fits 128k on one card. Bad case: naive fp16 OOMs long before that.

EP 04

Why long chats get slow — and long-context APIs get expensive

Two consequences of the cache living in VRAM. Speed: to write each token, the GPU must stream the weights plus your entire cache through its memory bus — bigger cache, slower tokens. Money: each user's cache claims exclusive memory, so longer contexts mean fewer users per GPU. Drag the context slider and watch users vanish from the grid while the price climbs.

GPU Context per chat

Back-of-envelope model: 7B weights at fp16 (14 GB), full-attention fp16 KV, decode assumed memory-bandwidth-bound. Real serving stacks add paging, GQA and batching tricks — the shape of the curve is what's real.

Connect the dots: this is why API pricing scales with context, why your 100k-token chat types noticeably slower than a fresh one, and why "1M context" headlines always come with an asterisk about cost. Good case: short chats pack dozens of users on one GPU for cents. Bad case: one 128k chat can hog the entire card.

EP 05

Evict the notes, forget the plot

When the cache won't fit, one fix is a sliding window: keep only the last W tokens' K/V and silently drop the rest. But attention can only look at entries that still exist. A secret door code is stated early in this chat. Keep chatting, shrink the window, then ask for the code — and watch what the model does when the answer has been evicted.

Cache window 512

The code “7391” was mentioned at token 60. Chat, evict, then ask.

Conceptual demo — the "model" here is a lookup, but the eviction rule is exactly how sliding-window attention and StreamingLLM-style caches behave.

The honest trade: eviction buys you infinite chat length at constant memory, and it works great while everything that matters is recent (good case). But evicted tokens aren't "remembered vaguely" — they are gone, and the model will often answer confidently anyway (bad case). When a long chat suddenly forgets your name, you may be watching a cache policy, not a dumb model.