Top-k & Top-p Sampling, Playable

EP 01

Every prediction drags a tail of garbage

A language model finishing “The chef seasoned the soup with ___” doesn't return one word — it returns a probability for every word it knows. Below are 15 of them, sorted. The top is sensible. The bottom is “gravel”, “sadness” and “wifi” — each near-zero, but never exactly zero. Sample from the raw distribution enough times and the tail will bite. Try it.

Your samples will appear here.

Probabilities are illustrative but shaped like a real LLM head: a few strong candidates, then an exponentially thin tail over the rest of a 100k-word vocabulary.

The core problem: greedy decoding (always the top word) is repetitive and dull, but honest sampling occasionally serves gravel soup. Everything on this page is about keeping the randomness while deleting the garbage.

EP 02

Top-k: keep the best k words, execute the rest

Top-k is the blunt fix: sort the words, keep the first k, set every other probability to zero, renormalize, then sample. Drag the slider and watch the knife move. Then sample 100 times — at k=5 the nonsense words are not just unlikely, they are impossible.

k = 5

Set k, then sample.

Good case: k=5 here deletes the tail with zero nonsense in any number of samples. Bad case: slide to k=1 — that's greedy decoding, 100 identical answers. And notice the deeper flaw: k is fixed, while the number of genuinely good words changes with every prompt. That flaw is EP 04.

EP 03

Top-p: keep words until you've banked p worth of belief

Top-p (nucleus sampling) cuts by cumulative probability mass instead of by count: walk down the sorted list adding up probabilities, and stop once the running total passes p. Each bar below shows its running cumulative total — drag p and watch the cut point find itself.

p = 0.90

Set p, then sample.

Good case: p=0.9 keeps every plausible seasoning and bans the garbage — without you ever choosing a word count. Bad case: slide p to 1.0 and you've kept 100% of the mass, i.e. no truncation at all — gravel is back on the menu. Real defaults live around p=0.9–0.95 for exactly this reason.

EP 04

Flat vs peaked — the fight top-p won

Two prompts, two shapes. “The capital of France is ___” is peaked: one word owns almost all the mass. “My favorite color is ___” is flat: a dozen answers are all equally fine. Apply the same top-k to both, then the same top-p, and count the survivors on each side. One knife adapts. One doesn't.

k = 3

PEAKED · “The capital of France is ___”

FLAT · “My favorite color is ___”

Why it matters: a single fixed k cannot be right for both shapes — k=3 wastes nothing on the peaked prompt but strangles the flat one to a quarter of its valid answers. Top-p reads the shape and resizes the pool automatically. This adaptivity is why the nucleus-sampling paper (Holtzman et al., 2019) displaced top-k as the default in GPT-style APIs.

EP 05

The strategy race — 60 rolls, live scoreboard

Same soup prompt, four decoding strategies. Each run draws 60 samples and scores itself live on two axes that fight each other: nonsense served (tail leaks) and variety (distinct words). The perfect strategy has zero nonsense and healthy variety. See who wins.

Pick a strategy to run 60 samples.

The scoreboard says it all: raw sampling leaks nonsense, greedy has zero variety (the k=1 doom), and both truncation strategies keep variety while serving nothing inedible. This exact trade-off — run live in front of you — is why every production LLM ships with top-p on by default.