Perplexity, Playable

EP 01

Every word is a little shock — measure it live

This page contains a real bigram model trained on a tiny storybook corpus. Type a sentence: for each word it computes the probability it expected, converts it to surprise (−log₂ p, in bits), and averages. Perplexity is 2^{average surprise}. Green bars = "saw it coming", red bars = "did not expect that at all".

The model is a bigram counter over ~180 words of storybook text, smoothed so unseen words get a small floor probability instead of zero. GPT does the same accounting with a trillion-word corpus and much longer context.

The mental model: perplexity is compression in disguise — a word that costs 2 bits was a 1-in-4 guess; a word that costs 10 bits was a 1-in-1024 shock. "Training a language model" literally means "make the surprise bars shorter on real text".

EP 02

The gibberish detector — fluent beats scrambled

Three sentences made of perfectly ordinary ingredients: a natural one, the same words shuffled, and a fluent one written with rare words. Score them. Word order and word choice both show up in the number — the shuffled sentence has identical words but broken bigrams, so its perplexity explodes.

Lower perplexity = less surprised = more like the text the model grew up on.

Good case: this ranking ability is perplexity's day job — it picks the best output in speech recognition ("recognize speech" vs "wreck a nice beach"), flags corrupted text, and is the loss curve every LLM training run tries to push down. Same-words-different-order getting a wildly different score is the whole point: it measures structure, not vocabulary.

EP 03

What the number means — a die with N faces

Perplexity has a physical reading: a model with perplexity N is exactly as confused as someone forced to guess by rolling a fair N-sided die. Here is a model choosing among 8 next words. Drag the slider from "clueless" (uniform — an 8-sided die) to "certain" (one word gets everything — a 1-sided die) and watch the die shrink.

Confidence

Now the headlines make sense: when GPT-2 hit perplexity ~29 on web text, it meant "every word, the model is only as unsure as a 29-sided die over a 50,000-word vocabulary". Modern LLMs push that into the single digits — that shrinking die IS the progress of language modeling.

EP 04

The cliché wins — low perplexity is not good writing

Time to break the metric. Score these three: a memorized cliché straight from the training data, a false sentence stitched from familiar pieces, and Carl Sandburg's genuinely brilliant fog line. The masterpiece loses. Badly. Then try to write a great sentence that beats the cliché — you'll feel the bias yourself.

Perplexity only asks "have I seen word patterns like this before?" — it has no column for true, beautiful, or useful.

Bad case, twice: low perplexity ≠ true (the false sentence scores great because every bigram is familiar) and low perplexity ≠ good (novelty is literally penalized — surprise is the definition of the score). Optimize a writer for perplexity and you get confident clichés. Sound familiar?

EP 05

Surprise is in the eye of the model

Perplexity is not a property of text — it's a property of a text–model pair. This page has two models: the storybook model and a second one trained on tiny tech docs. Score the same sentence under both and watch the number swing by an order of magnitude. Neither model is "right"; they just grew up reading different things.

Good case: this relativity is useful — "which model is least surprised by my domain?" is a legitimate way to pick a model, and it's why AI-text detectors use perplexity (AI-ish text is suspiciously unsurprising to an AI). Bad case: the same relativity breaks naive comparisons — perplexities from different models, vocabularies or tokenizers are not comparable numbers, and a human writing plainly can look "AI-generated" to a detector for exactly the reasons you saw in EP 04.