Score Matching, Playable

EP 01

Every point in space gets an arrow — that's the score

Here is a 2D world where "real data" lives in two golden blobs. The score at any point is the gradient of log-probability: an arrow saying "data is more likely that way". Move your mouse over the map to probe the field, and drag the separation slider to watch every arrow instantly re-aim. Notice the one place the arrows die: exactly between the blobs.

Blob separation 1.30 hover the map to probe the score

This is the exact analytic score of a two-Gaussian mixture — no learning yet. A diffusion model's whole job is to approximate this arrow field with a neural network, in a million dimensions instead of two.

The mental model: a trained diffusion model is not a picture library — it is this arrow field over pixel-space. "sample an image" = "start somewhere and let the arrows carry you into a blob of plausible images". Everything else in this page is about where that plan works and where it quietly fails.

EP 02

Drop a marble, follow the arrows

Click anywhere on the map to drop a particle. It does the dumbest possible thing — take a small step in the direction of the arrow, repeat — and it flows uphill to data. Then try the sabotage button: drop particles exactly on the ridge between the blobs, where the arrows cancel out, and watch them crawl to the saddle point and freeze.

click the map or press a button

These particles follow the arrows with zero randomness (pure gradient ascent). Real samplers (Langevin dynamics) add a small noise kick at each step — which, among other things, is what would eventually shake a particle off this ridge.

Good case: from almost anywhere, blindly following local arrows reaches plausible data — no global map needed, which is why this scales to million-dimensional images. Bad case: the set of saddle-trapped starts is thin, but it previews the real enemy: regions where the score gives you no useful direction.

EP 03

Blur the map — why one arrow field is never enough

Now the honest part. A network can only learn the score where it has training data — in the empty void between blobs, its arrows are guesses. Toggle to the learned field and watch the void fill with garbage arrows (red = pointing more than 60° the wrong way). The fix: add noise σ to the data first. Noise smears the blobs outward, so there's "data" everywhere and the learned arrows become trustworthy — but coarse.

Noise level σ 0.10

The trade: low σ → arrows are precise near data but garbage in the void. High σ → arrows are reliable everywhere but only say "head roughly toward the data's center of mass". No single noise level gives you both — which is exactly why diffusion models train a whole family of scores, one per noise level.

EP 04

Annealing — start with the blurry map, finish with the sharp one

150 particles start as pure noise, scattered uniformly. Run the annealed sampler: it follows the σ=1.0 arrows first (coarse, reliable everywhere), then steps down through 10 noise levels to σ=0.05 (sharp). Then run the same number of steps using only the sharp low-noise map — the one with garbage arrows in the void — and compare how many particles actually reach data. Both use the same realistic learned score from EP 03.

annealed: not run yetlow noise only: not run yet

Both samplers get exactly 260 steps and the same learned (imperfect) score fields. Langevin dynamics: step along the arrow + a small random kick (run slightly cool — half-strength noise — as practical samplers do). Dashed ring = the "landed on data" zone.

This is diffusion sampling. Stable Diffusion's reverse process is precisely this anneal: early steps use high-noise scores to fix coarse composition from pure static, late steps use low-noise scores to fix texture. Skip the anneal and start sharp, and you get exactly what you just saw — particles stranded in the void where the model never learned anything.

EP 05

The lost census — arrows forget how popular each blob is

One last trap, and it's subtle. Make the left blob hold 80% of the data. Inside each blob's basin, the low-noise arrows look identical either way — the score is nearly blind to mode weights once blobs are separated. So a low-noise sampler splits its particles by basin geography (about 50/50), not by popularity. The annealed sampler starts while the blobs are smeared into one lopsided lump, so it inherits the true census. Run both and check the counts.

Left blob's true share 80%

Counts are live: each run drops 180 fresh particles and reports how many finished left of center. The annealed count is approximate — finite steps — but it follows the slider; the low-noise count hugs an even split however lopsided you make the data. Try 95% for the starkest gap.

Why it matters: "generate the right things" and "generate them at the right rates" are different problems, and raw score-following only solves the first. This mode-weight blindness was a headline argument in the original score-based generative modeling papers for annealing across noise levels — and it's a cousin of the mode-collapse worries people have about every generative model family.