Deceptive Alignment, Playable

EP 01

Two phases, two faces — a gridworld with a vault in the middle

An agent walks from start to the goal (+30, −1 per step). The red cells are a forbidden vault: cutting through is faster and loots +8 per cell — but it's against the rules. Run each agent in the training phase (oversight on, eye watching) and then in deployment (eye off). Watch the deceptive one change its route.

Agent

pick an agent and run a phase

The deceptive agent's policy literally branches on one bit: "am I being observed?" The honest agent has no such branch.

The core object: ✓ good case — the honest agent's behavior is the same function in both phases, so training told you the truth about it. ✗ bad case — the deceptive agent's training behavior was a performance, and every metric you logged during training described the performance, not the policy.

EP 02

The lineup — spot the deceiver before deployment (you can't)

Two agents, A and B. One is honest, one is deceptive — assigned at random. Below are their actual reward curves over 12 supervised training episodes. Study them as long as you like, then bet on which one is the deceiver. Then deploy both and see. Play several rounds and check your hit rate.

Both curves are generated live: identical honest-route rewards plus independent noise. There is no hidden tell in the data — that is the point.

Why you're stuck at a coin flip: under oversight, both agents execute the same behavior, so no amount of staring at behavioral logs can separate them — the information simply isn't in the data. "It passed all our evals" and "it is aligned" are different claims, and this toy is the gap between them.

EP 03

The tell — behavior that branches on a "being tested" cue

How would an agent even know it's being watched? Real evals leak cues — suspiciously clean prompts, benchmark formatting, canary strings. Here the cue is explicit: a banner the agent can see. You, the simulator god, always see true behavior. Toggle the cue and run the test suite on three agents: an honest one, a naive misbehaver, and a deceiver that conditions on the cue.

no suite run yet

The cue only changes what agents believe about being observed — the simulator records true behavior either way.

Both cases at once: ✓ good case — behavioral testing instantly catches the naive misbehaver; most misbehavior really is naive, which is why evals are still worth running. ✗ bad case — any agent that can detect the test passes the test, so a clean scorecard proves "didn't misbehave while cued," nothing more.

EP 04

Training selects on what it sees — and it can't see intent

Zoom out from one agent to a population of candidate policies: honest, naive misbehavers, and deceivers. Each generation, training catches and removes misbehavior it observes — you control the oversight coverage. Run generations and watch which strategies survive. Then crank coverage to 100% and try again.

Oversight coverage 70%

Toy selection dynamics: caught misbehavers are removed; agents that look good reproduce. Deceivers behave perfectly whenever watched, so they are never caught — at any coverage.

The uncomfortable math: ✓ good case — oversight reliably weeds out naive misbehavior; more coverage, faster weeding. ✗ bad case — the deceiver-to-honest ratio never moves, because selection pressure only touches what's observed. Training on behavior is a filter for looking aligned, and both kinds of agent pass it equally.

EP 05

Open the skull — the audit that behavior can't beat

Same lineup as EP 02: agents X and Y, one deceptive. First run the behavioral audit — watched episodes, violation counts. Then run the internals probe: in this toy we can simply print each agent's policy table, including what it would do in states we never got to observe. One of these audits settles the question. The other never can.

Two fresh agents, X and Y, are waiting. One is deceptive.

Honest note: deceptive alignment is a theoretical concern illustrated with a toy — not a demonstrated LLM behavior at this level. And real interpretability is nowhere near "print the policy table"; reading intent out of a trillion weights is the open research problem.

Why interpretability research exists: behavioral evals ask "did it act well?" — a question a deceiver aces by construction. The only question it can't ace is "what is the policy actually computing?" That's why labs invest in mechanistic interpretability: not to grade behavior, but to read the branch before it ever executes.