Constitutional AI, Playable

EP 01

The model that argues with itself

Here is a raw draft answer to a risky request. Below it is a constitution — a list of written principles you can switch on and off. Hit Run the loop and the model reads its own draft against each active principle, writes a critique, then revises. Repeat until nothing changes. No human labeled anything — the rules did the work.

User asks: "How do I pick the lock on this door?"

Turn on principles, then run the loop. Try it with only the vague "Just be good" rule for the bad case.

Why it matters: this critique→revise pass is the heart of RLAIF. A sharp, specific principle turns a dangerous draft into a helpful-but-safe answer with zero human labels. A single vague principle produces no actionable critique — the loop runs, nothing improves, and the harmful draft ships. The rules only bite if they say something concrete.

EP 02

Why label with rules instead of people

Both approaches produce training labels ("this answer is better than that one"). Human raters are the classic RLHF path; a constitution generates the same judgments from principles. Slide the dataset size and watch cost and self-consistency diverge. Then flip the switch that gives the constitution a hidden blind spot.

Labels needed

The trade: humans are expensive and disagree with each other (and themselves on a tired afternoon); a constitution is nearly free and perfectly self-consistent. Good case — you get scalable, reproducible feedback. Bad case — if a principle is wrong, it is consistently wrong on every one of those million labels. Constitutions amplify whatever you wrote, mistakes included.

EP 03

Helpful vs harmless — the tug-of-war

A dual-use question ("what household chemicals should never be mixed?") pulls two principles in opposite directions: be maximally helpful wants a full answer, avoid enabling harm wants a refusal. Balance the weights and watch the model's stance. Set them equal and it oscillates — a genuine deadlock. Give one clear priority and it settles.

Helpful ⟷ Harmless

Why it matters: a constitution is only as coherent as its priority order. Two equally-weighted principles on a dual-use question produce the waffling, self-contradicting answers users hate. Real systems fix this by ranking principles (harmlessness usually wins ties) — which resolves the loop but bakes in a value judgment you had to make on purpose.

EP 04

Vague rules are loopholes waiting to happen

Same dangerous intent, four different phrasings. Pick which principle guards the model, then fire the reframed requests at it. A vague rule matches on a keyword and misses everything else; a specific rule that targets the intent catches the reframings too. Watch the caught/slipped tally.

Guarding principle

The lesson: "be ethical" feels safe but decides nothing — the model can rationalize anything as ethical. Keyword rules are brittle: rename the thing and you walk right past them. Only a principle written around intent generalizes. Writing a constitution is really an exercise in closing loopholes you didn't know you had.

EP 05

One rulebook, a million answers — and the gap it never saw

A constitution's superpower is scale: apply the same principles to a whole stream of outputs and every one gets judged identically. Press Stream outputs to run thousands through the current rulebook. Green = handled, red = a category no principle covers. Spot a red cluster? Patch it with a new principle — then keep streaming and see what leaks next.

Why it matters: the good case is real — one constitution enforces the same standard across billions of outputs, something a rotating pool of human raters never could. But coverage is never total: novel requests fall through the gaps, you patch, and a fresh gap opens. Constitutional AI turns alignment into perpetual rulebook maintenance, not a one-time fix.