AI Safety · 6 Interactives

Grading an Exam You Can't Solve

Human feedback trained today's AI. But feedback only works while humans can tell good answers from bad ones — and models are starting to produce answers no human can check directly. Below, you play the overworked grader: fail at checking, then try the tricks (debate, decomposition, recursive oversight) meant to keep a weaker judge in charge of a stronger student — and find where each trick breaks.

The grading wall Debate Decomposition Recursive oversight The eloquent liar True parts, false whole
EP 01

The grading wall — judge an answer you can't check

An "AI" hands you a nine-line derivation and a final answer. Half the time one step is silently wrong. You get 12 seconds — a grader's realistic attention budget — to rule Correct or Flawed. Play a few rounds and watch your accuracy hover near a coin flip. This is what supervising a stronger system feels like from the inside.

Press “New answer to grade”. The clock starts immediately.
no rounds yet · a coin flip scores 50%

Conceptual toy: the "AI" is a script generating arithmetic chains, not a trained model. The time pressure is the point — real graders rating long AI outputs face exactly this budget problem.

Why it matters: RLHF works by humans ranking outputs. Once outputs are long proofs, 10,000-line codebases or novel science, honest human ratings degrade toward noise — and training on noisy ratings rewards looking right, not being right. That cliff is why "scalable oversight" is its own research field.
EP 02

Debate — make two AIs fight, judge the fight instead

Same impossible derivation — but now two AIs argue about it. A defender says it's fine; a critic points at one specific step. You never check the whole thing: you recheck only the disputed step, a five-second job, and award the win. Checking a pinpointed claim is exponentially easier than finding a flaw yourself.

Defender
Critic
The single disputed step will be recomputed here.

The debaters here are scripted, but the asymmetry is real: verifying one named step is cheap even when auditing the whole answer is not.

The trick: debate converts "find the flaw in a huge answer" (hard) into "verify one flagged step" (easy). This is the actual bet behind AI-safety-via-debate at OpenAI and Anthropic: honest strategies should win arguments in front of a weak judge, because lies have refutable steps. EP 05 shows when that bet fails.
EP 03

Decomposition — slice the unjudgeable into judgeable bites

Second strategy: don't judge the big claim, shatter it into sub-claims so small each takes seconds. Here an AI totals an invoice. Instead of eyeballing the whole thing, click check on every line — each check is one tiny multiplication — and let the verdict assemble itself. Sometimes a line is corrupted, sometimes the total, sometimes nothing.

Every check button recomputes live from the invoice data — nothing is pre-labelled. Decomposition works perfectly here because an invoice is exactly a sum of independent parts. Hold that thought for EP 06.

Why it matters: this is factored cognition / process supervision: OpenAI found that rewarding each step of math reasoning ("process") beats rewarding only final answers ("outcome"), because humans can actually grade steps. Your 6 easy checks just outperformed your 12-second panic in EP 01.
EP 04

Standing on trained shoulders — the recursive oversight ladder

Third strategy, the ambitious one: a human trains a helper AI; human + helper together oversee a stronger AI; that pair's judgment trains the next helper; repeat. This sketch computes the ladder live. Drag the capability jump between tiers and toggle the helpers off to see why anyone bothers with recursion at all.

Toy model: each tier is judgeable if its overseer's power (human + 90% of the previous model, when helpers are on) covers it. Real recursive reward modeling has no such neat formula — that's part of the problem.

The good case and the crack: with modest jumps the ladder reaches capabilities no unaided human could judge — that's the recursive reward modeling / iterated amplification dream. But every rung passes judgment through the rung below, so errors compound quietly, and at big jumps the ladder simply snaps. Nobody yet knows how tall it can safely get.
EP 05

The eloquent liar — when the better debater is wrong

Debate's dirty secret: the judge rewards persuasion, and persuasion is a skill separate from truth. Two debaters below disagree about which product is larger. In style-only mode the actual numbers are redacted — exactly like a judge who can't verify — so all you have is rhetoric. Vote, then see who was right. Then turn the evidence on and feel the difference.

Debater A
Debater B
style-only: no votes yet · with-evidence: no votes yet

The polished debater is scripted to always defend the wrong answer. That's the adversarial worst case — a model optimized to win judges, not to be right.

Eloquence ≠ truth: in real debate experiments (Anthropic 2023, Khan et al. 2024), persuasiveness-optimized debaters can drag judges toward wrong answers — and RLHF already selects for exactly this, since raters reward confident, fluent prose. If oversight can't pierce rhetoric, training the debaters harder makes the liar better.
EP 06

Every part true, the whole a lie — decomposition's blind spot

Now break EP 03's hero. An AI analyst pitches a company: three crisp claims, each backed by real data. Check them — they all verify. Approve or reject on that basis, like a decomposition pipeline would. Then audit the full record the claims were cherry-picked from.

The company data is generated fresh each round; the three claims are then picked from it, so every check you run is genuinely true — that's what makes the omission invisible.

The blind spot: decomposition verifies the pieces you were handed, never the pieces that were left out. Emergent, whole-answer properties — framing, omission, cherry-picking — live between the sub-claims. Real oversight needs both a microscope (EP 03) and someone hostile enough to ask "what aren't you showing me?" (EP 02's critic). No single trick survives a smart adversary alone.
Keep playing
Alignment & Safety
Prompt Injection vs Jailbreak
Jailbreak: the user attacks. Injection: the data attacks.
Alignment & Safety
The Alignment Tax
Does safer mean dumber? Tune the dials and find out.