Repetition Penalty, Playable

EP 01

First, meet the disease — the doom loop

This page hosts a tiny language model trained on a short lighthouse story. With greedy decoding (always pick the most likely next word) it takes a deterministic path through a finite vocabulary — so it must eventually revisit a state, and from there it repeats the same phrase forever. Run it and catch the loop; the repeated stretch gets underlined.

Press the button — 64 words of pure determinism.

The model is a real word-level trigram model (with a pinch of unigram smoothing) built live from a ~160-word story — a miniature of what GPT does with billions of parameters.

The disease: nothing is broken — the model is doing exactly what it was asked. “Most likely next word” is a rail, and rails run in circles. Every fix you'll meet below exists because of this one screen.

EP 02

Now the cure — slide the tax from 1.0 to 2.0

Same model, same greedy decoding, one change: before picking, we divide each candidate's probability by penalty^{times already used}. At 1.0 nothing changes. Around 1.2 the loop from EP 01 quietly dies while the text stays grammatical. Slide, run, and watch the two live meters: is there a loop, and how many distinct words survived?

Penalty 1.00

Set the knob, then generate.

Penalty 1.00 — no tax. This is plain greedy decoding: expect the exact loop from EP 01.

The magic number: real systems ship values like 1.1–1.3 for exactly the reason you just felt — a tiny tax is enough to make the second lap around a loop lose to a fresh continuation, without distorting anything else. This is the repetition_penalty / frequency_penalty field in every LLM API.

EP 03

See the bars bend — the mechanism in slow motion

Here is the model's actual next-word distribution after the word “the”. Click any candidate to pretend it already appeared in the text (click again for twice, three times…), and drag the penalty. Grey ticks show each word's original probability; the gold bars show what the penalty leaves behind. Probability doesn't vanish — it flows to the words you haven't used.

mark as already used:

Penalty 1.30

One sentence version: a repetition penalty is just an edit to the probability bars, applied at every step, keyed to your own history. The model isn't retrained and learns nothing — the decoder simply refuses to trust the same word twice at full price.

EP 04

Crank it to 2.0 — and watch “the” get banned

The tax can't tell a doom loop from grammar. English needs “the” dozens of times; a story needs its hero's name on every line. Generate at a gentle 1.15, then at a harsh 1.9. Words shown in red are places where the penalty overrode the model's natural choice — hover them to see which word was blocked. Watch articles disappear and sentences turn to telegraphese.

or dial it 1.50

Try gentle first, then harsh.

The honest failure: too-strong penalties are a classic production bug — outputs that dodge “the”, “a”, “is” and the user's own name, sounding like a robot with a thesaurus allergy. If a model ever answers you in weirdly article-free English, someone turned this exact knob too far.

EP 05

Lists and code — where repetition is the right answer

Sometimes repeating a token isn't a rut — it's the spec. A shopping list must repeat “buy”; a program must repeat “;” and “(”. Below, a tiny model writes each format while a live checker counts well-formed items. At penalty 1.0 the structure holds; push the tax up and the checker watches semicolons and “buy” get priced out of the language.

Format Penalty 1.00

Pick a format and let it write.

This model samples with a little randomness (T = 0.45), so run it a few times — the trend, not any single run, is the lesson.

Why your coding assistant cares: code, tables, poetry refrains and legal boilerplate are legitimately repetitive, which is why serious systems use mild penalties, scope them to recent context, or exempt structural tokens. A knob that saves chat can silently corrupt code — same tax, different neighborhood.