Multimodal Tokenizers, Playable

EP 01

Chop it up — an image is a paragraph of patches

Step one is brutally simple: cut the image into a grid of patches (ViT-style, usually 14×14 or 16×16 pixels each) and treat each patch as one "word", read left-to-right, top-to-bottom like text. Drag the patch size on the scene below and watch the token count change — and notice what happens to the tiny sign in the corner as patches grow.

✓ good case: small patches — the sign spans several tokens, its shape survives✗ bad case: big patches — the whole sign falls inside ONE token, averaged into mush before the model ever sees it

patch size 16 px —

The scene is generated (sky, mountain, house, and a small "SALE" sign); the patch grid and token count are computed from your slider. The sign-legibility readout measures how many patches the sign occupies.

The patch grid is the model's retina, and it has fixed acuity. Anything smaller than a patch is a rounding error — which is why models miss tiny text, small UI icons, distant faces. It didn't "fail to notice"; the information was destroyed in tokenization, before intelligence was ever involved.

EP 02

The codebook — a dictionary of visual syllables

For generation (VQ-style tokenizers), patches don't stay as raw numbers — each is snapped to its nearest entry in a learned codebook: a finite dictionary of visual syllables. Below, a real k-means-style codebook is built live from the scene's patches. Drag the codebook size: with 256 codes the reconstruction is faithful; with 8, every patch must pick from 8 clichés and the image becomes visual mad-libs.

✓ good case: 256 codes — reconstruction error tiny, the scene survives its own dictionary✗ bad case: 8 codes — sky, wall and mountain all snap to the same grey syllable; measured error 12× worse

codebook size 64 —

Real quantization running live: patches are clustered (k-means, 8 iterations) into your chosen number of codes, then each patch is replaced by its code's average — the reconstruction error is measured per pixel.

A codebook is BPE for pixels: a finite vocabulary that everything must be spelled with. Get it right and an image is 256 integers a language model can natively predict — which is exactly how autoregressive image generators (and modern any-to-any models) write pictures token by token.

EP 03

The compression tax — 588× smaller, what died?

Do the arithmetic nobody shows you: a 1024×1024 RGB image is ~3.1 million numbers. After tokenization it's 256-1024 tokens. That's a ~600× compression, and compression that violent must kill something. The autopsy below shows what: reconstruct the scene through the tokenizer and inspect the difference map — where the reds glow is what the model literally cannot know about your image.

✓ good case: smooth regions (sky, walls) — near-zero loss, cheap to compress✗ bad case: the sign's letters and the fence texture — the difference map burns red; fine structure paid the whole tax

—

Reconstruction = EP 02's quantizer at production-ish settings; the difference map is the actual per-pixel error, amplified for visibility. High-frequency detail dominates the loss — the universal signature of every visual tokenizer.

Vision-language models don't see your image — they see a summary. The tax explains a whole family of failures: misread serial numbers, missed watermark text, "describe the chart" answers that miss small labels. The fix in production is never a smarter model first — it's more tokens per image (higher resolution tiling), i.e., paying more tax up front.

EP 04

One vocabulary — text and pixels in the same sentence

The endgame: once images are tokens, a single model can read and write mixed sequences — text tokens and image tokens interleaved in one stream, one next-token loop for everything. Watch a unified model process a real mixed prompt below: the token stream shows text (white) and image tokens (gold) flowing through the same context window, and the same predict-next machinery producing both kinds.

✓ good case: "describe this image then draw a variation" — one model, one sequence, no glue code✗ bad case: the image eats 1,024 of your 4,096 context tokens — paste four screenshots and there's no room left to think

images attached 1 —

Context accounting is exact for the chosen setup (1,024 tokens per image, 4,096 window). The streaming animation shows why "how many tokens does an image cost" became a real line item in multimodal API pricing.

Tokenization is the great unifier: the reason one model can caption, answer, and generate images isn't a vision module bolted on — it's that everything got translated into the same currency. Audio and video are joining through the exact same door (audio codecs like SoundStream are codebooks too).

EP 05

Resolution economics — spend tokens where the pixels matter

Production dilemma: a document photo arrives. Tokenize it at low res (cheap, 256 tokens) and the fine print dies; at full res (4× tiles, 1,280 tokens) everything survives but costs 5× and crowds the context. Modern systems route: cheap global view + high-res tiles only where needed. Play the operator: choose a strategy for each of three jobs, the meters score your reading accuracy per token spent.

✓ good case: landscape photo at low res — 94% of the value at 1/5th the tokens✗ bad case: dense invoice at low res — the numbers are sub-patch noise; "read the total" becomes hallucination bait

resolution low (256 tok) —

Accuracy-per-job curves encode the structural truth: smooth content saturates at low res; text-dense content needs patches smaller than the glyphs. Token counts match common production tiling schemes.

"How should I tokenize this?" is the new "what resolution should I scan at." Every serious multimodal deployment now has an image router making this call per request — and when a vision model bill surprises someone, the answer is almost always here, in tokens-per-image, not in the model at all.