Chinchilla Optimality, Playable

EP 01

Two labs, two answers — walk both paths and keep the receipts

2020, Kaplan et al. (OpenAI): as budgets grow, grow the model much faster than the data. 2022, Hoffmann et al. (DeepMind): no — grow them equally. Pick a path and click through three budget generations (each 10× the last). Both paths spend identical compute; the loss they buy diverges more with every generation. The famous casualty is marked: Gopher, 280B parameters fed only 300B tokens, beaten by Chinchilla — 4× smaller, same budget, 4× the data.

✗ bad case: Kaplan path at generation 3 — a giant that read almost nothing, measurably worse loss✓ good case: Chinchilla path — same three cheques, better model every generation

—

Losses computed from the published Chinchilla loss surface L = 1.69 + 406/N^0.34 + 411/D^0.28 with C = 6·N·D. The Gopher-vs-Chinchilla marker uses the real specs (280B/300B vs 70B/1.4T).

Why did smart people get it wrong for two years? Kaplan's fits reused a fixed learning-rate schedule across run lengths — a subtle methodology bug that biased the answer toward "bigger model." The lesson outlived the correction: a scaling law is only as good as the experiment design underneath it.

EP 02

Dig the isoFLOP bowls — find the optimum the way DeepMind did

How do you find the right ratio? The Chinchilla method: fix a compute budget, train many models of different sizes (each gets whatever data the budget allows), plot loss against size. The points form a bowl — too small underfits, too big is undertrained — and the bottom is the optimum for that budget. Run the sweep at three budgets and watch the bottoms line up into a law.

✓ good case: three bowls, bottoms connected — the exponent falls out of your own experiment✗ bad case: sweep only one budget — a bottom, but no law; you can't extrapolate a single bowl

—

Each sweep "trains" 12 model sizes at the fixed budget and measures loss from the Chinchilla surface — the same isoFLOP profiling as the paper's Approach 2, miniaturized.

This chart is why the result stuck: it isn't a theory, it's a measurement protocol anyone can rerun. Labs now dig isoFLOP bowls before every big run — the bowls for your architecture, your data mix, your tokenizer. The 20:1 number is the output of this ritual, not scripture.

EP 03

The 20-to-1 ruler — grade history's models

The rule of thumb the paper left behind: ~20 tokens of training data per parameter. The chart plots famous models by their actual ratio. GPT-3: 1.7 tokens per parameter — starved. Gopher: 1.1 — starved. Chinchilla: 20 — the point itself. LLaMA-7B: 140 — deliberately overfed (wait for EP 04). Drag the sliders to spec your own model and the ruler grades it live.

✗ bad case: 175B params on 300B tokens — a 2022 flagship, wasting most of its own size✓ good case: your spec lands near the line — every FLOP pulling its weight

your model (B params) 13B your data (B tokens) 260B —

Historical points use published specs (GPT-3 175B/300B, Gopher 280B/300B, Chinchilla 70B/1.4T, LLaMA-7B/1T+). Your model's "wasted capacity" readout compares its loss to the compute-matched optimal spec.

The ratio explains a strange-sounding headline from 2022-23: models got smaller and better at the same time. Nothing mystical — the industry stopped buying parameters it couldn't afford to feed. Parameter count quietly stopped being the bragging metric; training tokens took its place.

EP 04

The inference tax — why the recipe broke within a year

Chinchilla optimizes training compute — and silently assumes you'll never serve the model. But every token you generate in production costs compute proportional to model size, forever. Drag the slider from "research demo" to "planet-scale product" and watch the true optimum walk away from Chinchilla's point toward smaller, absurdly overtrained models. This is the LLaMA philosophy, and it's why the 20:1 rule died young.

✓ good case: research prototype, few users — Chinchilla-optimal is genuinely optimal✗ bad case: 1B requests/day on a Chinchilla-optimal giant — inference bill dwarfs training cost in weeks

lifetime inference volume demo (≈0) —

Total cost = 6·N·D (training) + 2·N·T (serving T lifetime tokens). For each slider position the optimizer re-solves for the cheapest (N, D) that hits a fixed quality target — the gold dot is your true optimum, the grey dot is Chinchilla's.

"Compute-optimal" was answering a question nobody in production asks. The production question is cost-per-quality over the model's lifetime — and its answer is train small models way past 20:1 (LLaMA-class models went 10× over). When you see a 8B model trained on 15T tokens, this slider is why.

EP 05

When the pantry empties — repetition and the quality lever

The recipe's other silent assumption: unlimited fresh tokens. Real corpora end. What then — repeat? Measured answer (Muennighoff et al.): repeating data works surprisingly well up to ~4 epochs, then each pass buys less and less until you're teaching the model to recite. Set your unique-data supply and crank the epochs; the effective-data curve and final loss are computed live.

✓ good case: 4 epochs over scarce data — nearly as good as 4× fresh tokens✗ bad case: 40 epochs — effective data flatlines while memorization keeps climbing

unique tokens 100B epochs 1 —

Effective data follows the published diminishing-returns fit (repeated tokens decay in value with each pass); the memorization-risk meter is schematic, the loss numbers come from the Chinchilla surface on effective tokens.

This is the bridge from Chinchilla to the 2025-26 agenda: once every lab was 10× over the ratio and the public web was exhausted, the game moved to data quality, synthetic data, and curricula — squeezing more effective tokens from the same pantry. The allocation question didn't go away; it moved inside the dataset.