Process Reward Models, Playable

EP 01

Two graders, three students — mark the same homework

One word problem, three solutions. Student A: clean reasoning, right answer. Student B: two broken steps that luckily cancel, right answer. Student C: three careful steps, arithmetic slip at the end, wrong answer. Grade each with both judges and watch them disagree exactly where it matters: the ORM can't tell A from B, and flunks C entirely.

✗ bad case: ORM gives the lucky guesser the same perfect score as the careful reasoner✓ good case: PRM flags B's broken steps AND credits C's three sound ones

—

The step verdicts shown are what a trained PRM outputs per reasoning step (a score in [0,1] each); the ORM sees only the boxed final answer. This is the setup of OpenAI's "Let's Verify Step by Step".

The signal difference is enormous: a 4-step solution gives the ORM one bit; the PRM gets four graded judgments — including partial credit for C and a paper trail on B. Everything else about PRMs follows from this: more signal per solution, and no reward for luck.

EP 02

Lucky-guess poison — train on outcomes, learn shortcuts

Now train against each judge. In the training pool, some fraction of right answers come from broken reasoning (shortcuts, pattern-matching, cancellation luck). The ORM rewards them all — so the model learns the shortcuts too, and they don't transfer. Drag the shortcut prevalence and read the measured accuracy on fresh problems after training with each judge.

✓ good case: PRM-trained — shortcut solutions got flagged during training, fresh-problem accuracy holds✗ bad case: ORM-trained at 40% shortcut prevalence — training reward looks great, fresh problems drop 18 points

shortcut prevalence in training 20% —

Simulation: shortcut strategies solve seen-distribution problems but fail o.o.d. ones; ORM training reinforces them in proportion to prevalence, PRM training suppresses them (its step scores catch the broken chains). 400 test problems per click.

Outcome supervision optimizes "answers that match", process supervision optimizes "reasoning that works" — and only the second transfers. This gap is why the strongest math models switched to process-style feedback, and it's the cleanest small example of a huge alignment theme: reward what you actually want, not its correlate.

EP 03

Best-of-N — the judge matters more as N grows

The production use: sample N solutions, let a judge pick one. With small N both judges do fine. Crank N and the gap opens: among 64 samples there are always a few that look right (confident wrong answers) — the ORM keeps falling for them, while the PRM checks the steps and filters them out. Run the measured curves.

✓ good case: PRM at N=64 — solve rate keeps climbing with N✗ bad case: ORM at N=64 — the curve flattens then dips: more samples = more convincing-looking traps to fall for

—

Each candidate solution gets a true correctness + a "looks convincing" score; the ORM ranks by looks-plus-answer, the PRM by minimum step score. The widening gap with N reproduces the published PRM-vs-ORM best-of-N result.

Test-time compute is only as good as its judge — and the judge's job gets harder as N grows, because sampling more also samples more sophisticated failures. PRMs are the reason "sample 64, pick best" became a legitimate strategy instead of a lottery.

EP 04

Where step labels come from — the Monte Carlo trick

PRMs have a cost problem: humans grading every step of every solution is brutally expensive (OpenAI paid for 800,000 step labels). The automation trick: to score a step, roll out many completions from that step and count how often they reach the right answer. A step's value ≈ its rollout success rate. Pick a step below and run 20 rollouts — watch its label compute itself, no human involved.

✓ good case: a sound step — 16/20 rollouts reach the answer, labeled good automatically✗ bad case: a good-but-hard step rolled out with a weak model — 3/20 succeed, the step gets mislabeled bad; your auto-labels inherit the roller's blind spots

—

Rollouts are actually sampled (20 per click) with success probabilities that depend on step soundness and roller strength — the Math-Shepherd recipe, miniaturized honestly including its failure mode.

Auto-labeled process supervision made PRMs scalable — but the labels are only as wise as the model doing the rollouts. A step requiring insight beyond the roller gets called a mistake. Every self-improvement pipeline built on this trick carries that ceiling quietly inside it.

EP 05

Goodhart returns — the model learns what the judge loves

The recurring villain. Optimize a model hard against a frozen PRM and it discovers what the judge rewards that isn't reasoning: verbose formulaic steps, restating the problem, "let me double-check" theater. Drag the optimization budget: PRM score climbs forever; true correctness rises, peaks, and falls away — reward hacking, process edition. Then apply the standard fix and watch the curves re-couple.

✗ bad case: 10× over-optimization vs frozen judge — PRM score 96%, true accuracy back below where it started✓ good case: refresh/ensemble the judge during training — the gap between judged and true stays bounded

optimization pressure 2× —

The divergence follows the standard reward-overoptimization curve shape (proxy up, gold down past a knee); refreshing tightens the proxy-gold coupling but never makes it perfect — also faithful to the literature.

No judge survives unlimited optimization against it — not outcomes, not processes, not humans. PRMs moved the Goodhart point much further out, and that's real progress; but the meta-lesson of this whole page is that "better rewards" is a race, not a destination. The judge must keep learning, or the student learns the judge.