Constrained Decoding, Playable

EP 01

The mask — legal tokens live, illegal tokens die

Here's the whole trick in one picture. The model has produced a probability for each of 12 candidate next-tokens. A grammar (right now: "a number") marks each as legal or illegal. Constrained decoding sets every illegal probability to zero and renormalizes — so the model samples only from what's allowed, even if its favorite token was forbidden. Switch the active rule and watch a different set of tokens survive.

✓ good case: rule "number" — the model wanted "the" (28%), but only digits survive; output stays valid✗ bad case: no mask — the model's top pick "the" wins, and your JSON parser throws at character 1

—

The 12 candidate probabilities are fixed; the mask is what changes. "Renormalize" means the surviving probabilities are rescaled to sum to 1 — the model's relative preferences among legal tokens are preserved.

Constraint happens at the logits, not the prompt. The model still computes its full distribution — you just refuse to let it act on the illegal parts. This is why constrained decoding gives a hard 100% guarantee where prompting gives a hopeful 95%: one edits reality, the other edits the request.

EP 02

The JSON automaton — a bouncer with a checklist

Where does the mask come from? A state machine that tracks where you are in the grammar. After { it expects a key or }; after a key it demands :; after a value it wants , or }. Generate a JSON object token-by-token below: at each step the automaton lights up only the legal next-tokens. Try to force it into an invalid state — you can't, the illegal buttons are disabled.

✓ good case: every path through the machine emits parseable JSON, guaranteed by construction✗ bad case: try to type a value before a colon — the token literally isn't offered; invalid states are unreachable

state: START

This is a real (small) pushdown-style automaton for a JSON subset, running live — the enabled/disabled buttons are computed from the current state, and the output is guaranteed parseable at every step.

Any grammar you can write as a state machine, you can enforce for free. JSON schemas, SQL, regex, function-call signatures, even a specific enum of allowed answers — libraries like Outlines and grammformal-grammar backends compile your schema into exactly this automaton, then mask the vocabulary with it every step.

EP 03

Prayer vs lock — run 500 requests each

The empirical showdown. Send the same "give me JSON" task 500 times three ways: bare prompt, prompt with threats and examples ("ONLY JSON, no markdown, I will be fired"), and grammar-constrained. Read the valid-JSON rate. Prompting gets you most of the way — and "most of the way" is exactly the problem when a downstream parser needs 100%.

✓ good case: constrained — 500/500 valid, by construction, forever✗ bad case: best prompt — 96% valid, i.e. 20 pipeline failures per 500 calls, at random, in production

—

Failure modes sampled realistically: bare prompt leaks markdown fences, prose preambles ("Sure! Here's..."), trailing commas; the threatening prompt fixes most but not the rare ones. Constrained can't fail the parse — that's the definition.

95% and 100% are different kinds of number. A 95% valid rate isn't "pretty good" — it's a 1-in-20 landmine that detonates at 3am under load. Constrained decoding is how "structured outputs" / "JSON mode" in every major API deliver the guarantee that prompting mathematically cannot.

EP 04

The tokenizer trap — where naive masking corrupts the model

The subtle bug that makes constrained decoding hard to implement well. The grammar thinks in characters; the model thinks in tokens, and one token can straddle a boundary — the token " then true might only exist as a single merged token "true. Naive char-level masking forbids that merged token and forces the model down a rare, low-probability spelling path — quietly wrecking output quality. Toggle token-aware masking and watch the "model surprise" meter drop.

✓ good case: token-aware mask — allows the natural merged token, model stays on its high-probability path✗ bad case: char-only mask — legal but unnatural; the model is forced onto tokens it rates 8× less likely, degrading everything downstream

—

"Surprise" = average negative log-prob of the tokens the model was forced onto vs. the tokens it would have chosen freely (both still valid). The gap is the hidden cost of getting the token boundaries wrong.

The constraint must speak the model's language, not yours. This token-vs-character mismatch is why a correct constrained-decoding library is real engineering, not a regex — and why early naive implementations produced valid-but-dumb JSON. The lock has to fit the exact shape of the model's vocabulary.

EP 05

The reasoning tax — when the cage hurts the bird

The counterintuitive finding. If you force the model into JSON immediately, it can't do its usual trick — think out loud first, then answer. Compare three protocols on a set of word problems: free-text reasoning (best accuracy, unparseable), JSON-only from token one (parseable, dumber), and the fix — reason freely, then constrain only the final answer field. Drag the "how early do we lock" slider and watch accuracy fall as the cage closes sooner.

✓ good case: think first, constrain last — full reasoning accuracy AND valid JSON✗ bad case: lock from token 0 — valid JSON, but accuracy drops ~15 points; the model can't show its work

lock the format after N reasoning tokens 0 (immediate) —

Accuracy vs lock-point measured over 300 simulated word problems (reasoning tokens genuinely help until a plateau); the "valid JSON" guarantee holds at every setting — only accuracy moves.

Structure and reasoning fight, and the resolution is sequencing. Every mature structured-output stack now does the same dance: let the model ramble in a scratchpad, then clamp the grammar over just the answer. It's the general lesson of the whole topic — constrain the output, never the thinking.