POMDPs & Belief States, Playable

EP 01

Same maze, two kinds of sight — run it and watch it drown

One maze, one goal, one simple "walk toward the goal" policy. The only difference between the two buttons is what the agent is allowed to see. With the full map it plans around the U-shaped wall. With a 3×3 keyhole it can't know the pocket is a dead end — it walks straight in and thrashes. Nothing about its brain changed; only its observation did.

✓ good case: full sight → 14 steps, clean path✗ bad case: keyhole → trapped in the pocket

ready

Both runs use the same greedy "reduce distance to goal" instinct; the full-sight run gets a planner on top because planning needs the map. That is exactly the MDP → POMDP downgrade.

The definition, felt: an MDP gives your policy the true state; a POMDP gives it an observation that may hide what matters. A vacuum robot bumping the same chair leg forever isn't stupid — it's partially observed.

EP 02

Two forks that look identical — the aliasing trap

The corridor below has two fork tiles that look exactly the same to the agent — same walls, same color, same everything. But the treasure needs UP at the first fork, DOWN at the second. A memoryless policy maps observation → action, so identical observations force identical actions. It literally cannot be right twice. Watch the success rate over 200 live episodes.

✗ bad case: memoryless — same look, same move, 0% or a coin flip✓ good case: one bit of memory → 100%

success: —

"One bit of memory" here just counts forks seen so far (0 or 1). That single bit disambiguates the two identical observations.

Perceptual aliasing is the core disease of partial observability: different states, identical observations. Every "the robot does the right thing in room A and the same wrong thing in room B" bug is this picture.

EP 03

The belief bar — being lost, mathematically

You're dropped somewhere in a 20-cell hallway with doors at positions 4, 10 and 16. You can't see your position — only a noisy sensor that says "door" or "wall". The gold bars are your belief: a probability for every cell, updated by Bayes' rule each time you sense or move. Sense a few times, move, sense again — watch the fog condense into a spike.

✓ good case: asymmetric doors + 10% noise → belief collapses in ~4 actions✗ bad case: symmetric hallway or 45% noise → forever lost

Sensor noise 10%

entropy: — · top cell: —

This is a real Bayes filter running in your browser — the same math (as a particle/Kalman filter) that localizes every warehouse robot and self-driving car.

The POMDP trick: you can't know the state, but you can know exactly how unsure you are. The belief is itself a perfectly observable "state" — solve the MDP over beliefs and you've solved the POMDP. The catch: with symmetric evidence the belief stays two-humped, and no amount of sensing fixes it — you must move to break the symmetry.

EP 04

The tiger behind the door — paying for information

The most famous POMDP ever posed: two doors, a tiger behind one (−100), treasure behind the other (+10). You may open a door, or listen for −1: you'll hear the tiger on the correct side with the accuracy on the slider. The gold meter is your live belief; the numbers under each action are its expected value given that belief. Play a few rounds and watch your score.

✓ good case: 85% ears — listen twice, open, profit✗ bad case: 55% ears — information too weak to ever be worth its price

Ear accuracy 85% score: 0

Fresh round: tiger placed randomly. Listening updates the belief by Bayes' rule; the EVs update live.

Information has a price and a value. At 85% accuracy each listen is worth far more than its −1 cost; at 55% you'd need ~11 listens to reach 90% certainty — the math says open blind or walk away. Every "should the agent call a tool / run another search / ask the user?" decision in modern agent design is the tiger problem wearing a suit.

EP 05

The T-maze — memory is the cure (until it leaks)

A light flashes at the start of a corridor: gold means turn LEFT at the far junction, white means turn RIGHT. By the time the agent reaches the junction, the light is long gone — the observation there is identical either way. A reactive agent is stuck at 50%. An agent that carries one remembered bit scores 100%… unless its memory leaks a little every step. Stretch the corridor and watch leaky memory die with distance.

✓ good case: perfect memory — 100% at any length✗ bad case: 5% leak/step × long corridor → back to coin flip

Corridor length 7 Memory leak/step 0% success: —

The curve plots measured success vs corridor length for the current leak setting — every point is 300 fresh simulated episodes, not a formula.

This is why RNNs, LSTMs and long context windows exist. Memory turns a POMDP back into something solvable — and "leak × distance" is exactly the vanishing-memory problem that made LSTMs beat vanilla RNNs, and long-context transformers beat both. When an agent framework adds a scratchpad, it's installing this bit.