GPU Parallelism, Playable

EP 01

Do the memory math — training is 8× heavier than the model

Everyone knows a 70B model is "big." Few people do the actual arithmetic of training it: weights are only the beginning — you also hold gradients (same size), Adam's two momentum buffers (2× more), and activations for the backward pass. Pick a model size and precision; the stacked bar is the honest bill, drawn against one 80GB GPU.

✓ good case: 7B, int8, inference only — 7GB, fits on a gaming card✗ bad case: 70B, mixed-precision training with Adam — ~1.1TB, that's 14 GPUs of pure memory before you compute anything

model 70B —

Bill: weights (2B/param fp16, 1B int8) + gradients (2B) + Adam states (8B, kept in fp32) + activations (rule-of-thumb ~1.4B/param at practical batch/sequence, checkpointing on). Real numbers vary; the order of magnitude doesn't.

Training memory ≈ 16 bytes per parameter before activations — the number that makes multi-GPU non-optional. It's also why "ZeRO" and optimizer-sharding exist, and why inference (2 bytes or less per param) lives on completely different hardware economics than training.

EP 02

Data parallelism — clone the brain, split the food

The easy trick: put a full copy of the model on every GPU, feed each a different slice of the batch, then average everyone's gradients (all-reduce) each step. Drag the GPU count and read the measured speedup curve. Then flip the interconnect from NVLink to plain ethernet and watch the curve bend — averaging a 14GB gradient blob every step is a lot of network.

✓ good case: 8 GPUs on NVLink — 7.6× speedup, 95% efficiency✗ bad case: 64 GPUs on slow ethernet — 21× (33% efficiency), and every GPU still needs the FULL model in memory

GPUs 8 —

Cost model: step time = compute/N + all-reduce time (ring: 2·(N−1)/N × gradient bytes ÷ link bandwidth). The curve is computed from it live — the bend is bandwidth arithmetic, not decoration.

Data parallelism scales the batch, not the model. It's the workhorse of every cluster — and completely useless for the EP 01 problem, because each clone still needs the whole terabyte. For that you have to start cutting the brain itself.

EP 03

Tensor parallelism — slice every matrix, pay per layer

The surgical trick: cut each weight matrix into column strips — GPU 1 holds columns 1-1024, GPU 2 the next batch, and so on. Every GPU computes its strip of every layer, then they exchange partial results — at every single layer, every step. Watch the animation slice a matmul across 4 GPUs, then check the efficiency table: this only works when the GPUs share a fast local link.

✓ good case: TP=8 inside one NVLink node — memory per GPU ÷8, ~90% efficiency✗ bad case: TP=8 across ethernet nodes — the per-layer chatter makes it SLOWER than a single GPU

tensor slices 4 —

Efficiency computed from: per-layer compute ÷ slices + two all-gathers per layer over the chosen link. 80 layers per forward pass — the multiplier that makes bandwidth decisive.

Rule of the field: tensor parallel stays inside the box. The NVLink domain (8, lately 72 GPUs) is exactly the radius where slicing matrices is cheap — one reason NVIDIA sells servers, not just chips, and why "how many GPUs per NVLink domain" is a headline spec now.

EP 04

Pipeline parallelism — the assembly line and its bubble

The third cut: give each GPU a contiguous chunk of layers — GPU 1 owns layers 1-20, GPU 2 owns 21-40… Data flows through like an assembly line. The catch is the bubble: while GPU 1 chews the first batch, everyone downstream stands idle. The fix: chop the batch into micro-batches so the line stays busy. Drag both sliders and read the measured idle fraction off the Gantt chart.

✗ bad case: 8 stages, 1 micro-batch — 87% of all GPU-seconds are spent waiting✓ good case: 8 stages, 32 micro-batches — bubble squeezed to 18%

pipeline stages 4 micro-batches 1 —

The Gantt is drawn from the actual schedule (GPipe-style fill-drain); idle fraction (S−1)/(S−1+M) emerges from the drawing, and the readout measures it off the schedule rather than quoting the formula.

Pipeline parallel is the cheap-network option: stages only talk to neighbors, once per micro-batch — so it crosses slow links gracefully where tensor parallel dies. The price is the bubble and the bookkeeping; interleaved schedules (1F1B) exist purely to shave this idle triangle.

EP 05

The 3D recipe — assemble a real 70B training run

Now do the real job. You have 64 GPUs (8 nodes × 8 GPUs, NVLink inside a node, slower links between), and a 70B model that needs ~1.1TB for training. Choose your three numbers — tensor slices (TP), pipeline stages (PP), data-parallel clones (DP) — such that TP×PP×DP ≤ 64. The checker runs three audits: does it fit in memory? does TP stay inside a node? how bad is the bubble? Find a config that passes all three.

✓ good case: TP8 × PP4 × DP2 — fits, TP inside the node, 76% est. efficiency (the Megatron classic)✗ bad case ×2: DP64 (nothing fits) · TP16 (spills across nodes, efficiency craters)

TP 8 PP 4 DP 2 —

Audits: memory/GPU = 1.1TB ÷ (TP×PP) vs 80GB · TP ≤ 8 to stay in-node · bubble from EP 04's schedule at 16 micro-batches · DP all-reduce over inter-node links. Throughput estimate multiplies the three efficiencies.

Every frontier training run is a hand-tuned triple like this (plus sharded optimizers, sequence parallel, expert parallel for MoE...). When a lab says "we trained on 25,000 GPUs," the real flex is that someone made 25,000 × three-way-parallelism × weeks run without the whole thing deadlocking. The math you just did is that job's kernel.