LoRA, Playable

EP 01

Don't repaint the wall — the thin-band trick

This is one weight matrix W from a model — a 26×26 grid, 676 numbers. Full fine-tuning updates every cell. LoRA freezes W and trains two thin bands instead: B (26×r, left, green) and A (r×26, top, green). Their product B·A is the update — look at the stripy, structured pattern it paints. Every count below is computed from the grid you're looking at.

rank r 2

The bet LoRA makes: the change a fine-tune needs is low-rank — a few directions, not 676 independent ones. The stripes you see are that constraint made visible. It holds surprisingly often, which is why one consumer GPU can now fine-tune a 70B model.

EP 02

The napkin math — train 0.1%, keep the quality

Pick a model size and slide the rank. The calculator applies LoRA to the four attention matrices of every layer (the classic recipe) and computes — live — how many parameters you actually train, and how much optimizer memory you dodge. The LoRA bar in the chart is drawn to true scale. Squint.

rank r 8

Counting only the attention weights (4·d² per layer) that classic LoRA adapts — the MLP blocks roughly double a real model's total, which makes the trained fraction even smaller than shown. Optimizer memory assumes Adam at ~12 bytes per trainable parameter.

Why it matters: at rank 8 on a 7B-class model you train well under 1% of the attention weights, and the Adam optimizer state shrinks from tens of gigabytes to megabytes. That gap is the entire reason "fine-tune your own model tonight" became a hobby instead of a datacenter project.

EP 03

One brain, many hats — hot-swap the adapter

Because the base weights never change, a LoRA fine-tune is just a small file you add on top — so you can keep one base model in memory and swap personalities like hats. Click an adapter and watch the same sentence restyle instantly. The size readout is computed with the EP 02 formula (d=4096, 32 layers, r=8).

Styles are simulated with text banks — but the sizes are real math, and real serving stacks (e.g. multi-LoRA inference servers) swap adapters exactly this way, thousands per GPU.

Why it matters: a full fine-tune of a 7B model is a ~14 GB artifact you must load and serve separately; a LoRA adapter is a few dozen megabytes you can hot-attach per request. One GPU, one base model, a thousand customers, each with their own "hat".

EP 04

When the rank runs out — watch LoRA underfit

LoRA can only paint updates built from r directions. Here the left heatmap is the update a task actually needs (ΔW); the right is the best rank-r approximation your budget allows. The error is computed live from the matrices' true singular spectrum. A light style tweak needs 2 directions. A genuinely new skill needs 12. Slide the rank and watch which one survives.

LoRA rank r 4

The honest limit: "fine-tuning changed the vibe but the model still can't do the new task" is the classic symptom of rank starvation — common when teaching new languages, long reasoning chains, or domain skills far from pretraining. The fixes are exactly what you just did: raise r, or admit the task needs full fine-tuning.

EP 05

Adapter pile-up — why stacking styles turns to mush

If one adapter is great, are four adapters at once amazing? Each adapter pushes the model's behavior in its own direction (the colored arrows — real 2D vectors). Activating several sums the arrows; the white arrow is what the model actually receives. The fidelity score is the live cosine between each adapter's intent and that sum. Toggle adapters and watch both the math and the sentence degrade together.

Why it matters: naively adding LoRA weights is vector addition in a space the adapters never agreed to share — two adapters often coexist, four rarely do. This is why "merge LoRAs" tools expose mixing weights, and why serious multi-skill models are trained jointly instead of stapled together.