Batch Size, Playable

EP 01

The noisy compass — every batch is a poll, not a census

The white arrow is the true gradient — the average over the whole dataset. Nobody computes that in practice; they poll a batch and hope. Each gold arrow below is one real minibatch estimate at batch size B. Drag the slider: at B=1 some arrows point backwards; by B=64 the poll is tight. The measured scatter shrinks like 1/√B — check the readout against the prediction.

✗ bad case: B=1 — 1 in 6 sampled arrows points away from the goal✓ good case: B=64 — scatter within 12% of the true direction

batch size B 1 —

Slider is powers of two, 1→256. Each arrow really averages B per-example gradients drawn from a noisy dataset; the std readout is computed from the 40 samples, the "theory" number is σ/√B.

Batch size buys certainty at √ prices. 4× the batch, only 2× less noise. That square root is the economics under everything else on this page — certainty gets expensive fast, which is why nobody polls the whole dataset.

EP 02

One budget, three ways to spend it

Fix the real currency — 2,048 example-reads — and choose how to spend it: B=4 buys 512 noisy steps, B=32 buys 64 decent steps, B=256 buys just 8 immaculate ones. Run all three and compare final losses at the same compute. The x-axis is examples consumed, so the race is fair.

✓ good case: the middle — enough signal per step, enough steps to travel✗ bad case: B=256 — 8 perfect steps still leave you halfway up the hill

—

Same task, same lr, same budget of gradient evaluations — only the packaging differs. Rerun a few times; the ranking is robust, the exact numbers wander.

Past a point, cleaner steps stop paying. Once the batch is big enough that noise isn't the bottleneck, doubling it just halves your step count for nothing — that's the "critical batch size" idea (McCandlish et al.) that decides how many GPUs a training run can even use efficiently.

EP 03

The GPU wall — hardware votes for big batches

So why not tiny batches everywhere? Because hardware hates them. Every step pays a fixed overhead (kernel launches, memory moves) before any math happens, and a GPU multiplies thousands of numbers at once whether you use them or not. The top curve is measured throughput; the bottom is time-to-target — a U: too small starves the silicon, too big wastes reads on steps you didn't need.

✗ bad case: B=1 — the GPU spends 95% of its life waiting, not computing✓ good case: B near the parallel width — silicon saturated, steps still plentiful

per-step overhead 8 ms GPU parallel width 64 —

A simulation with honest structure: step time = overhead + ceil(B / width) × unit; statistical efficiency per example from EP 02's noise model. Both curves recompute as you drag.

The batch size you run is a peace treaty between statistics (wants small) and silicon (wants big). Gradient accumulation, data parallelism, and the mega-batches of LLM training all exist to move this treaty line — but the U never goes away.

EP 04

Sharp landings — what big batches cost you later

The landscape below has two homes: a sharp pit (right next to where training starts) and a wide valley (further, slightly lower). Batch noise is the only force that can bounce the ball out of the pit. Press run: 30 independent trainings execute at your chosen B, and the tally shows where they ended up. Then judge them like production models — the "test shift" wobbles the landscape ±0.25.

✓ good case: B=2 — most runs escape to the wide valley, survive the shift✗ bad case: B=128 — nearly every run parks in the pit; the shift wrecks them

batch B 1 —

Each dot is a finished run's final position. "Shifted loss" is measured by re-evaluating each endpoint on the wobbled landscape — flat-valley endpoints barely move, pit endpoints spike.

This is the "generalization gap of large-batch training" (Keskar et al.) that bit everyone who scaled up naively: same train loss, worse test accuracy. The cure in practice is EP 05's dial — put the noise back with a bigger learning rate.

EP 05

The thermostat — lr and batch are one dial wearing two hats

What actually shapes the trajectory isn't lr or B alone — it's their ratio, the noise temperature ≈ lr/B. Run the pairs below: (lr 0.04, B 8) and (lr 0.16, B 32) have the same temperature and land the same way. Then break the rule: scale B up 8× without touching lr, and watch the run go cold — straight into the nearest pit, exactly EP 04's disease.

✓ good case: 4× batch AND 4× lr — same temperature, same behavior, faster wall-clock✗ bad case: 8× batch, same lr — temperature ÷8, run freezes in the sharp pit

—

Each button runs 20 fresh trainings and reports the flat-valley landing rate plus the measured step-noise std — the "temperature" line in the readout is computed from the runs, not the formula.

The linear scaling rule, explained: "double the batch, double the lr" isn't folklore — it's temperature conservation. It's how a run moves from 8 GPUs to 512 without changing what it learns. And when it finally breaks down (very large B), you've found the critical batch size again, from the other side.