Hierarchical RL, Playable

EP 01

One brain, hundreds of tiny steps — the flat agent flails

The quest: grab the key (bottom-left), pass the locked door (middle), open the chest (right room). Reward arrives only at the chest — so a flat learner exploring with primitive moves gets zero signal until it stumbles through the entire sequence by chance. Run it and watch. Then switch to the short task and see the same agent look almost competent.

Task

This is raw exploration with primitive moves under a 300-step budget — exactly what a flat learner must get lucky with before its first reward ever arrives. The shading shows every cell it visited.

The horizon problem: success probability collapses roughly exponentially with task length. ✓ Short horizons (reach the key) are easy to stumble into; ✗ chaining three of them almost never happens by luck — and no reward means no learning. This is why sparse-reward, long-horizon tasks were unsolvable for flat RL.

EP 02

Hire a manager: three subgoals turn a maze into a checklist

Same world, same primitive moves — but now a manager issues subgoals one at a time (go to key → go to door → go to chest) and a worker policy handles the step-by-step navigation (with a 12% action-slip, so it's not a perfect robot). The 300-step horizon collapses into three short hops.

The worker is a learned-navigation stand-in (shortest-path with noise); the comparison numbers are all simulated live against the flat agent from EP 01 under the same 300-step budget.

Why it works: the manager reasons over ~3 decisions instead of ~300, and each worker leg is a short, dense, learnable problem. This is the idea behind options, feudal networks and manager–worker architectures: long horizons become tractable when credit only has to travel between subgoals, not between hundreds of raw steps.

EP 03

Your turn — you are the manager

Click subgoals and the worker executes them. You never steer a single step — you only decide what to want next. Sequence it right and you'll finish near the optimal step count. Sequence it wrong (try the chest first) and watch your worker waste its life bumping into a locked door.

The worker gives up on a subgoal after 10 futile bumps and reports back — a luxury EP 05's worker won't have.

Feel the division of labor: as manager you made ~3 decisions; the worker made every one of the primitive moves. That's the entire trick — and also the entire risk: the worker faithfully executes whatever you ask, including orders that can't currently be satisfied.

EP 04

Wrong abstraction: a manager with useless subgoals

Hierarchy only helps if the subgoals carve the task at its joints. Give the manager a “corner library” — four subgoals that visit the map's corners — and it commands with total confidence while achieving nothing, because no subgoal ever lands on the chest. Run the tournament: flat luck vs task-aligned manager vs corner manager, 20 live episodes each.

Watch one episode

Managers here pick randomly from their subgoal library until the 350-step budget runs out — isolating the effect of the library itself.

The honest lesson: a bad abstraction is often worse than no abstraction — the corner manager's confident subgoals systematically steer the agent away from the key cells that random flailing occasionally hits. Choosing the subgoal space is the hard, unsolved part of hierarchical RL; it's where most real systems live or die.

EP 05

Deadlock — the manager demands the impossible

The manager says “go to the chest”. The worker has no key. The door is locked. A naive worker never questions orders — it walks to the door and bumps into it, forever, while the step counter climbs. Then run the fixed version: the worker is allowed to report “infeasible”, and the manager replans.

Choose a manager below.

Both runs are live simulations of the same worker — the only difference is whether “this subgoal is impossible right now” can flow back up the hierarchy.

The deeper point: hierarchy adds a new failure surface — the interface between levels. Deadlocks like this appear in real HRL systems, in robot task planners, and in every org chart: a boss who can't hear “that's currently impossible” converts one bad decision into infinite wasted work. Feedback upward isn't a courtesy; it's a correctness requirement.