RLHF Pipeline, Playable

EP 01

The assembly line for taste — how a click becomes a gradient

RLHF is a four-stage factory. An SFT model (fine-tuned on examples) generates candidate outputs; humans rank pairs of them; a reward model is trained to predict those rankings; then RL nudges the policy toward high-reward outputs, which get sampled and ranked again. Watch data pulses flow through it — then make the labelers sloppy and see what reaches the far end.

Flow speed

Click a stage to see what it does. Green pulses are correctly-ranked comparisons; red ones are mislabeled — and everything downstream of the human stage inherits them.

The one idea: RLHF converts a vague goal ("be helpful") into a trainable number — but every arrow in this diagram is a place where the signal can rot. Good case: careful labels flow through and the RL stage optimizes real preferences. Bad case: flip labeler agreement to 55% and watch the reward model's intake turn red — ChatGPT, Claude and Gemini were all built on exactly this pipeline, and all of them live or die by that middle stage.

EP 02

You are the labeler — pick the better answer

This is the actual job thousands of human contractors did to build modern chatbots. Two toy answers to the same question; click the one you prefer. The answers vary in four hidden traits — correctness, confidence, concision, politeness — and every click you make becomes one row of preference data that will train the reward model in EP 03. Aim for at least 8 clicks.

0 preferences collected

The answers are template-built toys, but your preference data is real — it is stored on this page and used verbatim to train the reward model below. "Skip" mirrors real labeling guidelines: ambiguous pairs are dropped, not guessed.

Why pairs, not scores: humans are terrible at absolute ratings ("is this a 7 or an 8?") but decent at comparisons. That is why InstructGPT, and everything after it, collected rankings — the good case for this whole method is that comparisons are the one signal humans give reliably.

EP 03

A copy of your taste — watch the reward model fit your clicks

A reward model is just a trainable function that scores an answer, fit so that your preferred answer scores higher in each pair (a Bradley–Terry model, the same math InstructGPT used). This one has four weights — one per trait. Hit train and watch gradient descent pull the weights toward whatever you actually clicked in EP 02.

epoch 0 / 40 accuracy —

If you labeled fewer than 5 pairs above, 12 careful synthetic labels are added so training has something to chew on — go click more pairs to make the weights truly yours.

Read the bars: if you consistently preferred correct answers, the "correct" weight grows and the two probe answers separate — good case, the model compressed your taste into numbers. Flip 30% of the labels and accuracy sags toward coin-flipping while the weights shrink and wobble — bad case, and real labs fight exactly this with labeler training, agreement checks, and thrown-out data.

EP 04

The policy chases your reward — watch it shift live

Now the RL step. The policy starts as a uniform distribution over 16 answer styles (every combination of the four traits). Each step it samples an answer, asks your reward model from EP 03 how good it was, and nudges probability toward what scored well — a miniature REINFORCE loop. The bars are the policy's live probabilities.

Press run — a freshly sampled answer appears here each step.

step 0 reward-model score: — true quality: —

The loop closes: your clicks (EP 02) became weights (EP 03) became this shifting distribution — a model aligning to your demonstrated taste with no rule ever written down. That is the good case, and it is genuinely how a raw next-word predictor became an assistant that apologizes and cites sources.

EP 05

Garbage in, garbage aligned — poison your own pipeline

The pipeline faithfully amplifies whatever the labelers do — including their mistakes. Run all three labelers below: each one generates 40 preference pairs, trains a fresh reward model, runs the full RL loop, and shows you what the final policy actually says. Same machinery every time; only the human differs.

Each run is the whole EP 02 → 03 → 04 pipeline compressed: 40 labels, 80 training epochs, 600 RL steps — all computed live in this tab.

Two real failure modes: random labels give the reward model nothing to learn (~50% accuracy) so RL optimizes noise — garbage out. Worse is the vibes labeler — the one who rewards a confident "It is Sydney, definitely" over a hedged "I believe it is Canberra": the reward model learns "confident = good, careful correction = bad," and RL dutifully produces a policy that states wrong answers with total certainty. That is not a toy problem — preference for confident-sounding text is a documented driver of confident hallucination in RLHF-trained chatbots.