Scaling Laws, Playable

EP 01

Log paper — the axis that turns despair into a law

Here are 12 training runs, each 3× more compute than the last, with realistic measurement noise. On linear axes they tell a depressing story: massive spending, progress flattening — "scaling is over." Click the toggle. On log-log axes the same 12 points snap into a straight line: every 10× of compute buys the same fixed fraction of loss, forever. Neither view is a lie; one is just the wrong instrument.

✗ bad case: linear axes — "diminishing returns, stop investing" (wrong read)✓ good case: log-log — a power law you can set a watch to

—

Points are generated from a Chinchilla-style power law L = E + B·C^−0.05 plus noise — a conceptual model with realistic exponents, not data from a real lab.

A power law means constant returns in ratio-space: each 10× of compute cuts reducible loss by the same percentage. It never "flattens" — it just needs exponentially more input for each equal step. Both the optimism and the cost of the AI era live in that one sentence.

EP 02

The bet — predict a run you can't afford

The reason labs trust the line: it extrapolates. You have cheap small runs; the question is what a 30×-bigger run will score, before you pay for it. Choose how many of the small runs to use, press fit — the gold dashed line is your prediction, the hollow circles are the big runs you "couldn't afford" (held out). More history, tighter bet.

✓ good case: fit 8 runs — the 30× run lands within a few % of prediction✗ bad case: fit only 3 noisy early runs — the extrapolation misses badly, $100M of wrong

runs used for the fit 3 —

The fit is real least-squares in log-log space, redone each click; held-out points come from the same law + noise. Prediction error is measured, not scripted.

This chart is how GPT-4-class budgets got approved. OpenAI reported predicting GPT-4's loss from runs using 10,000× less compute. Scaling laws turned "let's see what happens" into an engineering discipline with error bars — that's the real revolution, more than any single model.

EP 03

The allocation game — Chinchilla's trillion-dollar correction

Compute ≈ parameters × tokens. Same budget, one dial: make the model bigger, or feed it longer? Drag the slider and read the measured loss. There's a valley in the middle — and history's most expensive miss is marked on the chart: GPT-3-era models were far to the left (huge model, starved of data) until DeepMind's Chinchilla paper showed everyone had been over-building and under-feeding.

✗ bad case: the GPT-3 preset — same compute, visibly worse loss (over-parameterized, under-trained)✓ good case: the balance point — roughly 20 tokens per parameter

budget split — —

Loss surface uses Chinchilla-form L = E + A/N^0.34 + B/D^0.28 with illustrative constants; the valley location (~20 tokens/param) matches the published finding.

Scaling isn't just "more" — it's "more, in the right ratio." Chinchilla instantly made a generation of giant models obsolete at their own budgets, and it's why later models got smaller while getting better: same law, better allocation. When compute is the budget, allocation is the alpha.

EP 04

"Emergence" — miracle, or the ruler you chose?

Famous claim: abilities "suddenly emerge" at scale — nothing, nothing, nothing, then boom, the model can do arithmetic. Here's the whole controversy in one machine. The model's per-step skill improves smoothly with scale (top curve, always). But a task that needs k perfect steps in a row scored as all-or-nothing succeeds with probability p^k — crank k and watch a smooth curve turn into a cliff. Then switch the ruler to partial credit and the cliff melts away.

✗ bad case: all-or-nothing scoring, k=10 — "emergence!" headlines from a smooth capability✓ good case: partial-credit scoring — the same models, a boring gradual slope

steps required k 1 —

Accuracies are computed by simulating 500 test items per scale point with the shown per-step success rate — the cliff and its disappearance both come out of the simulation.

Some "emergent abilities" are real, some are metric artifacts (Schaeffer et al. made exactly this argument). The practical lesson cuts both ways: smooth internal progress can hide behind a cliff-shaped benchmark — so a model that "can't do X" may be one scale-notch away. Plan for cliffs, measure with slopes.

EP 05

The wall — where the straight line starts to lie

Every extrapolation dies somewhere. The law assumes you can always feed the bigger model proportionally more fresh data — but the internet is finite. Drag the data-cap slider: once runs need more tokens than exist, extra compute can only buy parameters, and the measured points peel away from the beautiful line. There's also a floor the line can never pass: the irreducible entropy of language itself.

✓ good case: data-rich regime — the law holds across three orders of magnitude✗ bad case: past the data wall — the line predicts 2.1, reality delivers 2.6, and the gap compounds

available data (tokens) unlimited —

The bend is computed: past the cap, D is frozen and only N grows, so the D-term of the Chinchilla formula stops improving. This is the "data-constrained scaling" regime studied by Muennighoff et al.

The data wall is why 2025-26 AI sounds the way it does: synthetic data, curriculum quality, multi-epoch training, licensing wars over text corpora, and the pivot to test-time compute — all of them are attempts to keep buying loss after the easy tokens run out. The law didn't break; its cheapest input did.