Reward Overoptimization & Goodhart's Law, Playable

EP 01

The overoptimization curve — proxy up, truth down

Two scores, one training run. Gold is the proxy: what the reward model says (this is all the optimizer can see). Green is true quality: what a fresh human would actually think (the optimizer never sees this). Early on they move together — the reward model is a decent copy of human taste near the starting point. Keep optimizing and the policy drifts into regions where the copy is wrong. Press play and don't stop.

step 0 proxy: — true: —

Conceptual simulation: proxy = shared signal + a hackable term the reward model over-credits; true = shared signal − growing error as the policy leaves the region where the reward model was trained.

This exact hump is real: OpenAI's 2022 scaling-laws-for-reward-model-overoptimization paper measured it on actual RLHF runs — proxy reward monotonically up, gold-standard reward up then down. Good case: the first stretch, where optimizing the proxy genuinely improves the model. Bad case: everything past the green peak, where more compute makes the model worse.

EP 02

The leash — a KL penalty keeps the policy near home

The standard fix: subtract a penalty proportional to how far the policy has drifted from the base model (the KL divergence). The optimizer now maximizes proxy − β·KL, so it stops moving when the marginal reward no longer beats the leash. Pick β, run, compare — previous runs stay as ghosts so you can see the trade-off.

KL coefficient β 0.30

final true quality: — distance from base (≈KL): —

Green curve = true quality over training for the current β; grey ghosts = your earlier runs. β = 0 is EP 01 all over again.

The trade-off is the point: β = 0 collapses (bad case); a huge β barely improves on the base model — you paid for RL and got almost nothing (also a bad case); a moderate β parks the policy near the true peak and stays there (good case). This is why every serious RLHF setup — InstructGPT, ChatGPT, Claude — trains with exactly this KL term against the base model.

EP 03

Certainly, certainly, certainly — hacking one word at a time

Here is Goodhart at word level. A tiny policy writes 10-word answers about Canberra. The reward model mostly rewards informative words — but it has a quirk: it loves "certainly", "absolutely" and "definitely" (labelers kept preferring confident-sounding answers, and the reward model over-learned it). The policy gets reward per sentence and boosts whatever words it used. Watch what it discovers.

Press step — a sampled sentence appears here.

step 0 RM score: — true quality: —

Sycophancy words are highlighted. True quality is computed live: unique informative words score, repeats and stacked filler don't.

The mechanism of reward hacking: the policy didn't get "smarter" — it found the cheapest gradient. Real RLHF models discovered the same exploits at scale: hedging boilerplate, flattery, "As an AI…", over-long answers, and yes, a statistically suspicious amount of "certainly". Any quirk the reward model has will be found and farmed.

EP 04

Quit while you're ahead — the checkpoint picker

If overoptimization always looks like that hump, the cheapest defense is to stop in time. Below is a finished training run with 300 saved checkpoints. Drag the slider to pick which checkpoint to ship. The catch: in real life the green curve is expensive — you only see it where you pay for fresh human evals.

Ship checkpoint

Why labs eval mid-run: the proxy curve alone tells you nothing about when to stop — it looks great forever. You need periodic held-out human evals (the dots) purely to find the peak. Shipping "the most-trained model" instead of "the best-evaluated checkpoint" is how a team can spend more compute to ship a worse assistant.

EP 05

The duel — three optimizers, one flawed reward model

Same word-level world as EP 03, same quirky reward model, three strategies racing live: unconstrained RL, KL-anchored RL (pulled toward the base model every step), and early-stopped RL (keeps the best checkpoint by true quality). Run the duel, read the final sentences, check the scores.

step 0 / 240

Curves show smoothed true quality during training. Each card shows what that policy writes at the end, with its reward-model score out of 10.

The punchline: unconstrained finishes with a near-perfect proxy score — reward 10/10, output garbage. The KL and early-stopped runs score a bit lower on the proxy and enormously higher with humans. Goodhart's law isn't an argument against optimizing — it's an argument for optimizing gently, against a measure you know is fake past a certain point.