Mode Collapse, Playable

EP 01

Realism is not coverage — meet the four modes

Real data (white) comes in 4 clusters — think "the dataset has cats, dogs, cars and trees". The gold dots are a generator's output. Toggle which modes the generator bothers to produce, and watch the two meters: the detective's per-sample pass rate (does each fake look real on its own?) versus diversity (does the collection cover the data?).

The loophole: a per-sample detective is blind to the difference between "covers everything" and "one perfect trick". Both score ~100% pass rate. Every objective that only judges samples individually has this hole — and gradient descent will find it.

EP 02

Watch the collapse — diversity crashes, the detective never notices

Now let training run. Gold fakes start spread out; each step they climb toward whatever the detective already accepts — and the more fakes a cluster has, the stronger its pull (the generator reuses whatever is working). Watch all four modes bleed into one while the fool rate stays high and the diversity meter crashes in the strip below.

Simulated dynamics, not a trained net — but the "rich get richer" pull toward already-accepted outputs is the honest mechanism behind real collapse.

The silent failure: every training metric the GAN can see — fool rate, discriminator loss — looks healthy after collapse. Diversity is not in the loss, so its death is invisible. This is why early GANs trained on 10 digit classes would proudly generate the digit "1" over and over.

EP 03

Whack-a-mole — the generator hops, the cycle never ends

Give the detective time to adapt and collapse gets stranger, not better. Once all fakes sit on one cluster, the detective learns "that cluster alone = fake" (its fatigue bar fills), so the whole gold swarm hops to a fresh cluster… where the same thing happens. The timeline below records which mode the generator occupies — watch the stripes cycle.

Speed

Fatigue bars over each cluster = how strongly the detective has learned to reject fakes there. High fatigue evicts the swarm.

Why it never settles: the generator is always fully committed to ONE mode, so the detective always has a clean tell, so the generator must move. Mode A → B → C → D → A: real GAN logs show exactly this — samples that look great each epoch but are a different single thing every time you look.

EP 04

Play detective — one sample lies, a batch confesses

The generator has collapsed onto one (random) cluster. You're the detective: call REAL or FAKE. In single-sample mode you see one point — a collapsed generator's samples are individually perfect, so good luck. In batch-of-8 mode you see eight samples together. Suddenly there's a tell. Play 10 rounds of each and compare your accuracy.

Make your first call.

Real batches are drawn across all 4 clusters; the collapsed generator's batches all come from its one trick (a fresh random cluster each round, so you can't memorize which).

The diagnosis is the cure: collapse is invisible per-sample and obvious per-batch. So give the detective batch-level eyes — that's exactly the idea behind minibatch discrimination, which you get to switch on next.

EP 05

The fix — penalize sameness, watch the modes come back

Same collapsed swarm as EP 02 — but now you have the minibatch discrimination switch: the detective compares fakes to each other and flags look-alike batches, so crowding itself becomes expensive. Run with it OFF to confirm the collapse is stable, then flip it ON and watch fakes migrate until all four modes are re-occupied and the diversity meter climbs back.

Penalty strength

Toy version of the real technique (Salimans et al. 2016): crowded fakes get pushed toward whatever the batch is missing. Try low penalty strength — partial recovery is a real failure mode too.

The general lesson: if diversity isn't in the objective, don't expect diversity. Minibatch discrimination, unrolled GANs, and diffusion models' likelihood-style training all fix collapse the same way — by making "one perfect trick" score worse than honest coverage.