Quantization, Playable

EP 01

All the numbers a weight is allowed to be

A 16-bit float can be tens of thousands of different values between −1 and 1. Drop to INT4 and only 15 values exist; INT2 leaves 3. Every weight must snap to its nearest allowed value — that snap is the error. Drag the weight, switch precision, and watch the number line empty out underneath it.

weight

The whole trick in one picture: quantization replaces the real number line with a coarse grid (here: symmetric "absmax" grid, the standard scheme). ✓ Good case: at INT8 the worst possible snap is ~0.4% of the range — no network cares. ✗ Bad case: at INT2 a weight can move by up to a quarter of the whole range. Everything else on this page follows from this picture.

EP 02

Quantize a real network and watch its face

This is an actual 25-weight neural network living in this page. The grey curve is what it computes; the gold curve is the same network after its weights are rounded to the precision you pick. The error number is measured live across the whole input range. Find the bit-width where the face starts to melt.

A 1→8→1 tanh network with seeded random weights, quantized per-tensor with absmax scaling — the same recipe (in miniature) as real INT8/INT4 LLM quantization.

What you just felt: ✓ good case — INT8 and usually INT4 curves sit almost on top of the original: billions of tiny roundings mostly cancel. ✗ Bad case — at INT2 whole hidden units collapse onto identical values and the function changes shape, not just detail. This is why "W8" and "W4" builds of Llama are everywhere and "W2" is a research problem.

EP 03

Will it fit? The 70-billion-weight arithmetic

Why does anyone tolerate rounding errors at all? Memory. A model's weight file is simply parameter count × bytes per weight. Pick a model and a precision and see which real machines can hold it — the numbers update live, no magic.

model

Weights only — the KV cache and activations need extra room on top, so real headroom requirements are higher.

The motive for the crime: ✓ good case — 70B at INT4 is ~35 GB: a model that needed a multi-GPU server at FP16 suddenly fits on one 48 GB card or a beefy laptop, for a percent or two of quality. ✗ Bad case — the same model at FP32 is 280 GB; no consumer machine on earth can even load it. Memory pressure, not elegance, is why quantization won.

EP 04

The INT2 cliff — quality vs bits, measured live

EP 02 was one network. Here we quantize twelve seeded networks at every bit-width and average the surviving quality — a real sweep, computed in front of you. Two recipes race: one shared scale for all weights (crude) versus one scale per tensor (standard). Watch where each falls off.

—

Quality = 100% minus output error relative to each network's output range, averaged over 12 networks.

The shape to remember: a long flat shelf, then a cliff. ✓ Good case: 8→4 bits costs a few percent while cutting memory 2–4× — nearly free. ✗ Bad case: below ~3 bits there aren't enough values left to tell weights apart, and quality falls off vertically. Finer-grained scales (per-tensor here; per-channel or per-group in real LLMs) push the cliff lower — that's what schemes like GPTQ and AWQ are fighting for.

EP 05

One weight ruins everything — the outlier problem

Absmax scaling has an Achilles heel: the grid is sized by the largest weight. Real LLMs contain rare outlier weights 20–100× bigger than the rest. Inject one below and watch it stretch the grid so far that every normal weight snaps to zero. Then try the two standard rescues.

The punchline of modern quantization: ✗ bad case — one weight out of 48 destroys the other 47 under naive INT4; in real transformers exactly this killed early 8-bit attempts. ✓ Good case — clipping, or keeping the ~0.1% of outliers in high precision (the LLM.int8() trick), rescues nearly all the quality. Handling outliers is the difference between quantization papers that work and ones that don't.