The chopper — text becomes chunks before the model wakes up
A tokenizer is a compression scheme: it owns a fixed menu of ~100,000 chunks and greedily covers your text with the biggest ones it knows. Common words are on the menu whole (1 token). Rare words aren't — they get covered by scraps. Type below: every token costs the same to process, so the meter is a real cost meter.
Hand-built ~300-piece vocabulary that mimics how GPT-style BPE splits text; real tokenizers learn ~100k pieces from data. Cost uses $3 per million input tokens, a typical mid-tier API price.
The language tax — the same sentence, up to 3× the bill
The chunk menu was learned mostly from English text, so English gets the best compression. The same sentence in Chinese falls off the menu: common characters survive as 1 token, rarer ones shatter into raw UTF-8 byte fragments. German compound words shred too. Same meaning, different token bill — click each language and watch the meter.
Byte tokens are shown as ⟨hex⟩ pairs — that's literally what the model receives for characters missing from its menu. The gap is exaggerated here (~2–3×); measured gaps on real tokenizers are ~1.5–2× for Chinese, worse for Thai or Khmer.
The strawberry problem — letters are invisible inside tokens
Here's the famous one. You see letters; the model sees opaque token IDs. "strawberry" arrives as two sealed boxes — the model never receives an r, it receives #24807 and #19772. Ask it to count letters and it can only guess from what tokens usually contain. Try it, then hit "spell it out" to fix it the way prompt engineers actually do.
Why 9.11 beats 9.9 — numbers get sliced at weird places
Numbers aren't numbers to a tokenizer — they're digit chunks. "9.11" arrives as 9 · . · 11 and "9.9" as 9 · . · 9. A model reasoning over chunks compares "11" against "9" — and 11 is bigger, right? Pit two numbers against each other and watch where the token brain and real math disagree.
The "token brain" here compares the digit chunks left of the decimal numerically, then the chunks right of it as standalone integers — a simplified model of the shortcut LLMs are suspected to take. Real models fail this exact pair less reliably but measurably.
THE CAPS LOCK TAX — same words, different tokens, triple the bill
To a tokenizer, "the", "The" and "THE" are unrelated symbols. Common casings earned a menu slot; SHOUTING and aLtErNaTiNg case never got frequent enough, so they fall back to scraps. Edit the sentence, then compare the four casings of the exact same words — every count below is computed live.
Real tokenizers do keep separate tokens for a few frequent cased forms ("The", "THE" for some words) — but the tax on unusual casing is real, and it also quietly degrades quality, since rare token sequences got less training.