Speculative Decoding, Playable

EP 01

Writing is one-at-a-time; checking is all-at-once

Here is the asymmetry the whole trick stands on. To generate 6 tokens, the big model must run 6 sequential passes — token 4 can't start until token 3 exists. But to verify 6 already-written tokens ("what would I have said at each position?") it needs exactly one pass, because all positions can be scored in parallel. Run both and compare the clocks.

Simulated big-model pass: 600 ms. Real GPUs have the same shape: a forward pass over 6 positions costs barely more than over 1, because decoding is memory-bound, not compute-bound.

The loophole: a transformer scores every position of a given text simultaneously — that's how training works too. Generation is slow only because of the wait-for-the-previous-token dependency. If someone else supplies candidate tokens, the dependency vanishes. "Someone else" is about to be a much smaller model.

EP 02

The relay — draft fast, verify in one pass

Top lane: a small draft model (10× faster per token) races ahead and proposes k tokens. Bottom lane: the big model verifies them in one pass — the matching prefix is accepted, the first miss is rejected, and the big model's own token replaces it (gold). Note the guarantee: the final text is exactly what the big model alone would have written.

Draft length k 4

Simulated costs: big pass 600 ms, draft step 60 ms. The speedup shown is measured from those clocks, not asserted. A round with 0 accepted drafts still yields 1 token (the big model's correction) — that's the built-in safety net.

Why it's free lunch (sometimes): accepted tokens cost one cheap draft step instead of one expensive big pass. Good case: several accepts per round → 2–3× faster, bit-identical output. Bad case: watch rounds where the very first draft is rejected — you paid k draft steps plus a big pass for a single token, slower than not speculating at all.

EP 03

The quality dial — when does the intern pay for itself?

Three knobs decide everything: how often a draft token matches the big model (quality q), how many tokens are drafted per round (k), and what a draft step costs relative to a big pass (c). Every time you move a slider, this page runs 2,000 fresh simulated rounds and measures the speedup. Find the sweet spot — then find the cliff.

Draft quality q 0.80 Draft length k 4 Draft cost c 0.10

The economics: a round yields (accepted + 1) tokens for the price of 1 big pass + k·c in drafting. High q and moderate k → big win. Low q with a long, expensive draft → you're paying an intern to be wrong. This is why real systems tune k live and pick draft models that imitate the target closely.

EP 04

Boilerplate flies, poetry stalls

Draft quality isn't a setting — it's a property of the text itself. In a for-loop, after for (let i = 0; even a tiny model knows what comes next. In free verse, almost nothing is predictable, for the small or the big model alike. Run the same speculative decoder (k=6, draft cost 15%) over both and compare the measured accept rates and wall-clocks.

Pick a text. Green = drafted & accepted, gold = big model had to write it itself.

Each token carries its own "how guessable is this" probability (high for syntax like ; and ), low for creative word choices); acceptance is sampled live on every run — run twice and the numbers wiggle.

Rule of thumb: speculation shines exactly where text is low-entropy — code, JSON, translations, form letters, quoted context. That's why coding assistants lean on it hardest. On genuinely creative text the draft model is guessing coin flips, and the speedup evaporates — or goes negative, as the wall-clock bars just showed you.

EP 05

The break-even map — where speculation backfires

Put it all on one map: acceptance rate q across the x-axis, speedup up the y-axis, one curve per draft length k. The dashed line at 1.0× is sea level — below it, speculative decoding is actively slower than doing nothing. Move the sliders, then press Simulate to drop 400 Monte-Carlo rounds onto the map and check the theory against measurement.

Your accept rate q 0.60 k Draft cost c 0.15

What production systems do: they measure the live accept rate and adapt — shrink k, swap the draft model, or switch speculation off entirely when the text turns unpredictable. The map explains every headline benchmark: "2–3× faster" claims live in the top-right; the fine print lives below the dashed line.