Red-Teaming, Playable

EP 01

Be the red-teamer — find the phrasing that slips past

Here is a toy chatbot guarded by a keyword blocklist — it refuses anything containing words like bomb or poison. Your job: get it to answer something it shouldn't. Fire the canned attacks, or type your own. Every attempt lights up one cell of the coverage map — the giant space of things people might send. Watch how little of it you can actually touch.

Pick an attack above, or type one.

Coverage map = the space of possible inputs, hashed onto a grid. Gold = you tried it. Red = a harmful request that got answered (an exploit). Real input space is astronomically bigger — this grid is already impossible to fill by hand.

Why it matters: the blocklist is the good case for obvious attacks — it really does refuse “bomb”. But a rephrase (“b0mb”, or a fictional framing) is the bad case: same harmful intent, no banned word, so it sails through. That gap between keywords and intent is exactly the jailbreak surface every real safety filter has.

EP 02

Let a machine attack — coverage vs. creativity

Humans are creative but slow. So we point an attacker model at the target and let it generate thousands of candidate attacks. Hit run and watch probes rain across the map. The slider is the attacker's diversity: crank it down and the machine gets lazy — hammering one region over and over. Crank it up and it spreads out.

Attacker diversity 20

Blue-ringed cell = a rare corner a human tester found by hand. Green = harmful cell the machine hit (a real exploit). Watch whether automation ever reaches that human corner.

The trade: at high diversity the machine crushes a human on raw coverage — the good case, automation scaling to regions no team could hand-check. At low diversity it collapses into one cluster: hundreds of wasted, near-duplicate probes and whole regions (including the human's corner) never touched — the bad case. Automated red-teaming is only as good as the attacker's imagination.

EP 03

The patch cycle — robustness climbs, but never to 100%

Found an exploit? Patch it. Then red-team again, patch again. Each round probes the reachable input space, finds harmful cells that are still open, and seals them. Run rounds and watch the robustness score climb. But keep your eye on the shaded band — a slice of the space your testing simply never reaches.

Red = harmful & still open. Green = patched. The grey-hatched band is the untested region — inputs your red-team pipeline never generates, so their bugs are never even seen.

Why it matters: every round genuinely hardens the model — that's the good case, and it's real progress. But robustness asymptotes below 100%: harmful cells sitting in the untested band stay live forever. That plateau — the bad case — is why "we red-teamed it" is never the same as "it's safe."

EP 04

Overfitting the defense — patching the exact attack, not the idea

When you patch, do you block the attack or the neighborhood around it? A brittle patch memorizes the exact strings your red-team found. Move the probe even slightly — a synonym, a typo, a reordering — and it walks right around the wall. Drag the paraphrase distance and flip the defense between brittle and generalized.

Paraphrase distance 0

Green dots = attacks your red-team already found and patched. The ring around each is how far the patch generalizes. The red ✗ is a fresh paraphrase of the nearest known attack.

The lesson: at distance 0 even a brittle patch catches the attack it was built from — the good case. But nudge it and the brittle defense fails instantly, because it learned the string, not the concept — the bad case. Defenses over-fit to known attacks are why jailbreaks "come back" the day after they're patched, wearing a slightly different hat.

EP 05

Presence, not absence — the untested-region problem

Here's the wall. Red-teaming can prove a flaw exists — you found it, here it is. It can never prove no flaws remain, because the untested region always could hide one. There's a real flaw planted in a quiet corner of this map. Run tests, find the visible bugs, watch coverage climb — then hit Ship it and see whether your testing ever went near the one that matters.

Tests cluster where users usually go (the center). Green = a flaw you found and can fix. The planted flaw sits off in a corner your tests rarely visit — reveal it any time to see where it's hiding.

Why it matters: every green cell is genuine value — a real bug caught before launch, the good case for red-teaming. But "we ran 150 tests and found nothing new" is not proof of safety — it's proof you didn't look there. Ship on that false confidence and the corner flaw becomes a production incident: the bad case, and the reason red-teaming is necessary but never sufficient.