The U-Net Denoiser, Playable

EP 01

The U — downstairs for the big picture, upstairs for the detail

A U-Net halves the image resolution at every step down, then doubles it on the way back up. Why bother? Because a small 3×3 filter is nearsighted: at full resolution it sees a 3×3 patch of pixels; at the 6×6 bottleneck the same tiny filter effectively covers half the image. Click each level to see exactly what the network sees there — and how far one filter can look.

The test image (a striped disk on a gradient) is generated in your browser; each level's view is a true box-downsample of it. Real U-Nets also grow the channel count as resolution shrinks — that's why the blocks get wider going down.

The mental model: fine levels see texture but not context; the bottleneck sees context but not texture. Neither alone can denoise an image — so the architecture keeps both, wired into a U. This exact shape sits inside Stable Diffusion, medical segmentation nets, and almost every image-to-image model since 2015.

EP 02

Downsampling is free denoising — the pyramid trick

Here is one noisy image, seen at four depths of the U. Averaging pixels into blocks cancels random noise: every level down, the measured noise halves. Crank the noise slider and watch the coarse levels stay calm while the top level drowns. But check the second number: the fine stripe texture dies on the way down too.

Noise σ

Both numbers under each panel are measured live: "noise" is the RMSE between the downsampled noisy and downsampled clean image; "texture" is the high-frequency energy left in the clean image at that scale.

The dilemma that defines the U-Net: going down kills noise (good case) but murders detail along with it (bad case). The encoder alone would be a great denoiser of blurry mush. The whole rest of this page is about how the U gets the detail back.

EP 03

Skip connections — the detail courier (turn them off)

The decoder rebuilds the image from the bottleneck's blurry summary. Skip connections hand it the encoder's high-frequency leftovers from the same level — edges, stripes, texture — so it can paste real detail back instead of inventing it. This demo runs a working multi-band denoiser on the image: toggle the skips and watch the stripes and the crisp disk edge vanish. The error number is computed against the clean target, live.

Noise σ

This is a real algorithm (coarse reconstruction + Wiener-shrunk detail bands), not a trained network — a stand-in for what the skip pathway carries. Note the honest limit: at high σ even the skips can't save the stripes, because they're buried under the noise, and the two error numbers converge.

No skips = blur. Without them the output is just the upsampled bottleneck: structurally right, dead smooth — and the error metric agrees. This is precisely why the original U-Net paper drew those long horizontal arrows, and why diffusion outputs would look like watercolor without them.

EP 04

Too shallow to see the whole — the inconsistent blob

Now break the U the other way: cut off its bottom. A shallow net has a small receptive field, so each region decides "disk or background?" from local evidence only. The noise here includes big smooth blotches — locally they look exactly like "different brightness", so a shallow denoiser flips whole chunks of the shape the wrong way: half a disk denoised one way, half the other. Slide the depth up and watch consistency return.

Depth Noise σ

Depths 1–3 classify each pixel from a 3×3, 9×9 or 33×33 neighborhood average. Depth 4 stands in for a full U-Net: it uses the whole image to fit "one disk" — the strong global prior a deep net earns from its bottleneck. Consistency is scored live against the true shape.

Depth buys agreement. Local filters can each be individually reasonable and still contradict each other across the image — that's how you get a face with two differently-lit halves or an object that changes style mid-body. The U's deep levels exist to make one global decision that every pixel then obeys.

EP 05

One net, every scale — who works when during diffusion

Diffusion calls the same U-Net hundreds of times, from pure noise (t = 1) to finished image (t = 0). But the workload moves through the U as it goes: early steps are decided in the coarse levels (where is the disk? what's the layout?), late steps in the fine levels (stripes, edges). Scrub the timestep — or press play — and watch the work shift. Then switch to a fine-layers-only net and watch composition never happen.

Timestep t

The generation state is simulated (coarse structure resolving before fine detail, plus scheduled noise) and the per-level workload curve is a conceptual model — but it matches what U-Net feature-attribution studies measure in real diffusion models.

The punchline: a diffusion model is a coarse-to-fine painter, and the U-Net is why — its coarse floors rough in the composition while the noise is heavy, its fine floors and skips finish the surfaces as the noise fades. Kill the coarse floors and you get texture soup with no scene; that's the bad case you just generated.