Autoregressive vs Diffusion, Playable

EP 01

The race — one brush vs a thousand hands

Same target image, two machines, and the honest currency of generation: model calls. Each tick, both sides get exactly one forward pass. The autoregressive painter spends it on one pixel (raster order, conditioned on all previous pixels). The diffusion sculptor spends it refining all 256 pixels at once, one notch less noisy. Press play and watch who crosses the line first.

Speed

Conceptual simulation — both “models” here secretly know the target. The thing being measured is real though: AR needs one sequential call per pixel; diffusion needs a fixed number of whole-canvas passes (32 here) no matter how many pixels there are.

✓ Diffusion's good case is parallelism — every pixel updated in every call. ✗ AR's bad case is the queue: pixel 256 cannot start until pixel 255 is done. This is why LLMs stream text word by word while Stable Diffusion pops out the whole picture at once.

EP 02

Parallelism math — why 256 calls turns into a million

The race gets uglier as images grow. AR's sequential bill scales with the number of pixels (N²); diffusion's stays a flat number of passes. Drag the resolution up. Then try AR's escape hatch — bigger “tokens” (predict a 2×2 or 4×4 patch per call) — and see how much it buys back.

Resolution 32×32 AR token size

This slider is why diffusion won images: at photo resolutions the AR queue explodes into the hundreds of thousands of sequential calls, while diffusion still charges ~30–50. It is also why every serious AR image model (VQGAN-style, Parti, GPT-4o image gen) predicts fat patch tokens, never raw pixels.

EP 03

Poison one pixel — error cascade vs self-repair

Now sabotage both machines. Force one wrong pixel into the AR painter mid-generation: every later pixel is conditioned on it, so wrongness breeds wrongness — the cascade fans out down-and-right. Do the same to diffusion by corrupting a whole 4×4 patch mid-denoising: the next passes simply denoise the damage away… unless you strike at the very last step.

AR: wrong pixel # 40 Diffusion: corrupt at

Toy corruption model: an AR pixel whose already-painted neighbors are wrong is only 55% likely to be right itself (vs 99.7% normally). Diffusion pulls every pixel 22% closer to the clean image per pass. Both numbers are made up; the asymmetry they produce is exactly real.

✗ AR's bad case: one early mistake and everything conditioned on it inherits the damage — the same exposure-bias cascade that derails an LLM after one bad token. ✓ Diffusion's good case: every pass revisits every pixel, so mid-course errors get sanded away. Strike at step 30, though, and even diffusion ships the scar.

EP 04

The text test — where pixel-by-pixel wins

Time for AR's revenge. Ask both machines to render a word — exact, discrete, thin-stroked structure. The AR painter commits to a hard 0-or-1 token per pixel and nails every stroke. Diffusion's output is a continuous field averaged over noise: crank its steps and it sharpens, but 1-pixel strokes stay smeared and the threshold keeps chewing holes in letters.

Diffusion steps 8

Conceptual model of a real pathology: diffusion regresses toward an average over many plausible images, which blurs high-frequency discrete detail; the toy mimics that with a step-dependent blur + residual noise before thresholding.

This is why image generators historically misspelled “RESTAURANT”: averaging over noise fights crisp discrete structure. AR-style decoders (DALL·E 1, Parti, GPT-4o image gen) treat pixels like vocabulary and commit — which is exactly what typography needs.

EP 05

Pick your engine — four jobs, two machines

Final exam. Pick a job, hit run: both engines generate it, and the scoreboard — fidelity and sequential calls — is computed live from the two outputs, not from opinions. Notice the verdict flips from job to job: there is no overall winner, only fit.

The job

Pick a job and run it.

The modern answer is “both”: hybrids like MaskGIT decode many tokens in parallel, diffusion LLMs write text by denoising, and GPT-4o-style AR image models brought exact text back. The frontier isn't AR vs diffusion — it's knowing which failure mode your job can't afford.