Prompt Caching, Playable

EP 01

What the homework is — keys and values, not memories

When a model reads your prompt, it computes a K and V vector for every token at every layer — the "notes" attention will consult while generating. That work is the expensive part of reading. Press both buttons: the cold request computes notes for all 2,000 prefix tokens; the warm request loads them from storage and only computes the 20 new tokens of your question. The timers and token counters below are the whole story.

✗ bad case: cold — 2,020 tokens computed, full latency, full price✓ good case: warm — 20 computed, 2,000 loaded, ~10× cheaper on the prefix and visibly faster to first token

—

Real providers price this explicitly: cached input tokens cost ~10% (sometimes less) of fresh ones. The animation's compute/load split mirrors the real KV-cache reuse; latency ratios are illustrative but directionally faithful.

Prompt caching is memoization, not memory. The model learned nothing between your requests — the infrastructure just refused to redo identical work. Which is why the discount has strict, mechanical rules… next episode.

EP 02

Prefix or nothing — the cache can't skip around

The rule that surprises everyone: the cache only works on an identical prefix. Token 500's notes depend on tokens 1-499 (that's what attention means), so the moment the streams diverge, everything after the split must be recomputed — even if later parts match exactly. Drag the divergence point and watch the reusable region (gold) get truncated at the first difference, no matter how much matches afterward.

✓ good case: identical first 1,800 tokens → 1,800 reused, only the tail recomputed✗ bad case: one different token at position 10 → 10 reused, 1,990 recomputed; the 1,500 identical tokens later are worthless

first difference at token 1800 —

This is causality, not a lazy implementation: token i's KV genuinely encodes everything before it. A "smarter" cache that reused matching middles would return mathematically wrong attention states.

Order is money. The same information arranged differently has a different price. Everything stable goes first (system prompt, tools, documents); everything volatile goes last (user question, timestamp). Violate that once and your cache hit rate is a rounding error.

EP 03

One changed word — hunt the cache-killers

A game of spot-the-expensive-mistake. Below is a prompt template a team ships to production. It looks innocent. Click the parts you think invalidate the cache on every single request. There are three killers hiding in it — find all three and the meter shows your monthly bill dropping by 84%.

✗ bad case: a timestamp in the system prompt — every request differs at token ~15, cache hit rate 0%✓ good case: killers moved to the end or removed — 92% of tokens cached on every call

0/3 killers found — click the prompt lines

The three killers are real patterns from production postmortems: an inline timestamp, a random request-ID "for tracing", and user data interpolated above the static instructions.

Cache-killers are invisible in code review because they're semantically harmless — a timestamp changes nothing about the task. They only show up in the bill. Teams now lint prompts for exactly this: nothing dynamic above the fold.

EP 04

Design for the cache — rearrange a real workload

You run a support bot: 10,000 requests/day, each = instructions (2k tokens) + knowledge base (6k) + user message (~200). Three architectures, same content: naive (user message first — "personalization!"), cache-aware (static prefix, user last), and cache-aware + TTL awareness (traffic batched to keep entries warm). The bill is computed live from cache economics; drag the traffic slider to see how volume changes the math.

✗ bad case: user-first layout — $492/day, every token always cold✓ good case: static-first + warm traffic — $78/day, same model, same content, same answers

requests/day 10,000 —

Pricing model: $3/M fresh input tokens, $0.30/M cached, 5-minute cache TTL refreshed on hit. At low traffic entries expire between hits — watch the middle bar worsen as you drag left.

Prompt layout became a line item. The same product, re-ordered, costs 6× less — which is why "cache-friendly prompt architecture" is now a standard interview topic for AI infra roles, and why providers publish prefix-structuring guides that are really just this chart in prose.

EP 05

The agent flywheel — why agents are suddenly affordable

The killer app of caching isn't chat — it's agents. An agent loop appends: each turn is the entire previous conversation plus one new tool result. Without caching, turn N re-pays for all N−1 previous turns — quadratic total cost. With caching, each turn pays mostly for what's new — near-linear. Run a 30-turn agent both ways and watch the two bills separate into different universes.

✗ bad case: no cache, 30 turns — cumulative cost curves upward, $4.31 for one task✓ good case: cached, same 30 turns — $0.61, and turn 30 costs barely more than turn 3

Each turn appends 600-1,200 tokens (tool results vary). Cumulative curves are integrated live from the same pricing as EP 04. The quadratic-vs-linear gap widens with every turn — try imagining 300 turns.

Caching is the silent enabler of the agent era. Long-horizon agents that loop hundreds of times would be economically absurd if every turn re-read the whole transcript at full price. When you wonder how a coding agent can afford to think for an hour — this discount, compounding every single turn, is most of the answer.