Credit Assignment, Playable

EP 01

Thirty moves, one trophy — the last move grabs all the glory

The simplest accountant: whoever moved last gets the reward. Hidden truth of this game: exactly one move flips the win probability between 10% and 90%; every other move is a meaningless coin flip. Play episodes, watch credit flow (the glow), then reveal which move actually mattered. Try both game types — the naive scheme is right in exactly one of them.

game type

episodes: 0 prime suspect: —

Bottom bars = the learner's estimated importance per move: |avg credit when that move's choice was good − when it was bad|, computed live from the episodes you play.

Why it matters: “blame the last thing that happened” is how naive systems (and many humans) assign credit. It works for sudden-death games and slot machines — and fails silently whenever consequences are delayed, which is chess, robotics, dialogue, and life.

EP 02

Spread it backward — the discount dial

Better idea: give EVERY move a slice of the outcome, shrinking as you go back in time — move t gets γ^T−t × outcome. Same early-decisive game (move 3 rules). Play episodes, then drag γ and watch the importance bars morph live — the stored episodes are re-scored instantly. Can you find a γ that exposes move 3?

discount γ 0.90

episodes: 0

Play some episodes first, then sweep γ from 0.5 to 1.0 and watch move 3 sink and surface.

The tension: at γ=0.9 move 3's credit is scaled by 0.9²⁷ ≈ 0.06 — buried under the sampling noise of 29 irrelevant moves. γ→1 rescues it here, but in long stochastic games undiscounted credit has brutal variance. Discounting is not truth, it's a prior — “recent moves matter more” — and this game violates it.

EP 03

Eligibility traces — breadcrumbs that get paid all at once

Now the real machinery: TD(λ). An agent walks a corridor of 18 states toward a +1 reward. Every state it visits leaves a decaying trace (the white rings); whenever a surprise (TD error) occurs, ALL traced states get updated at once. The gold glow is the learned value V — literally credit as glow intensity. Run one episode with λ=0, then with λ=0.9, and compare how far the credit travels per episode.

trace decay λ 0.80

episodes: 0 V(start) = 0.000 corridor lit: 0%

Real TD(λ) with α=0.4, γ=0.98 — the values you see are genuinely learned, not staged. Set λ=0 and reset to feel the one-state-per-episode crawl.

Why it matters: λ interpolates between TD(0) (cheap, slow to propagate) and Monte Carlo (fast to propagate, noisy). Eligibility traces are why TD-Gammon could feel the consequences of early moves — and the same “decaying breadcrumb” idea lives on inside GAE, the advantage estimator used by PPO in modern RLHF pipelines.

EP 04

Move 3 decided the game. Which accountant finds it?

Head-to-head on identical episodes (early-decisive game, move 3 rules the outcome): last-move credit vs discounted credit (γ=0.85) vs uniform credit (γ=1). All three see exactly the same data. Each panel shows that method's importance estimate per move, scaled to its own maximum — the green dashed line marks the truth. Feed them episodes and watch who converges.

episodes: 0

Same episode stream for all three accountants. Verdicts update live.

The honest scoreboard: uniform credit finds move 3 in a few hundred episodes; γ=0.85 shrinks move 3's signal to 1.6×0.85²⁷ ≈ 0.02, below the noise floor of the moves near the end — it would need tens of thousands of episodes; last-move credit can never find it, at any sample size. This is why AlphaGo-style systems propagate whole-game outcomes back through every position rather than trusting recency.

EP 05

Spread too thin — when the signal dies on the way back

The failure mode that stalls real RL: sparse reward + long horizon. A wandering agent (75% forward, 25% back) gets +1 only at the far end of the corridor. Fix the training budget at 30 episodes and stretch the corridor. Watch V(start) — the agent's belief that starting is worth anything — die as the distance grows. Then flip on eligibility traces and see how much horizon they buy back.

corridor length 20 states eligibility traces (λ=0.9)

V(start) = — signal reached: —

Every run trains from scratch with TD(λ), α=0.3, γ=0.99 — results vary run to run because the walk is random. That variance is part of the lesson.

The wall every RL practitioner hits: when reward is one bit at the end of a long trajectory, credit per decision shrinks toward zero and learning stalls — this is why Montezuma's Revenge resisted DQN for years. The cures are all credit-assignment surgery: traces and n-step returns, reward shaping, subgoals, or a value network that moves the reward closer.