One brain, two printouts — the heatmap and the arrows
The same gridworld, shown twice. Left: the value function V(s) — how much future reward each square promises (brighter = better, red = the pit). Right: the policy — in each square, the arrow points to the best neighbor. The arrows are not stored anywhere: they are derived from the values by being greedy. Sweep value iteration and watch worth ripple out from the goal while the arrows organize themselves. Click any square to move the goal.
✓ Good case: once values converge, a perfect policy falls out for free — argmax over neighbors, no extra learning. ✗ Bad case: stop after 1–2 sweeps and look at squares far from the goal — the value signal hasn't reached them yet, so their arrows are still clueless.
Paint the values, break the policy
Now you are the value function. Click squares to raise or lower their value; the greedy arrows update instantly (each square points at its highest-valued neighbor). Then release the agent — it blindly follows your arrows. A smooth downhill-to-goal landscape works beautifully. But paint one square too bright and you create a local peak: the agent climbs onto it, its best neighbor points back, and it oscillates forever.
Greedy policy here = "step to the neighboring square with the highest value" (the real rule is argmax of reward + γ·V, same idea). The agent has no memory and no map of its own — only your numbers.
The report card — deriving values from a behavior
EP 01 went values → policy. Now the other direction: fix a behavior, and compute the values of following it — that's policy evaluation, the "critic". This is a cliff world: the bottom edge between S and G is a fatal drop. Policy A detours along the top; policy B hugs the cliff. Wind sometimes shoves the agent sideways (the slip slider). Evaluate both and watch the report card flip as the wind picks up.
Heatmap = value of each square under the displayed policy, computed live by iterative policy evaluation (γ = 0.97, small step cost).
Rock, paper, exploited — where "pick the best" dies
In a maze, being predictable is fine. Against an opponent who adapts, it's fatal. This bot keeps a value for rock, paper and scissors and can play anywhere from pure argmax (always the "best" move) to a pure mixed policy (uniformly random). The opponent watches your recent moves and plays the counter to your favorite. Run both extremes and compare the score curves.
Opponent = frequency counter: it looks at your last 12 moves and throws the counter to your most common one. Your bot updates its action-values from results — and that's exactly what makes it readable.
"How am I doing?" — the question a policy can't answer
Two finished policies, arrows only — no values attached. One threads the gap between the pits (short but slippery); one loops around the top (long but calmer). Watch single episodes: both usually reach the goal looking equally confident. A policy is just instructions — it carries no scoreboard. The only way to rank them is to derive values from the policies: roll them out many times and average the returns.
✗ Bad case: eyeballing single runs — both agents "look fine", and a policy alone can't tell you its own expected return. ✓ Good case: Monte Carlo rollouts turn a mute policy into numbers you can compare, improve, and ship.