Discount Factor (γ), Playable

EP 01

The fading spotlight — γ decides how far you can see

A reward that arrives t steps from now gets multiplied by γ^t. Each bar below is one future timestep; its height is how much a reward there is worth today. Drag the slider and watch the spotlight of caring stretch and shrink. The green line marks the effective horizon — roughly where the future stops mattering.

γ 0.90

Why discount at all: in a task that never ends, an undiscounted sum of rewards is infinite — every plan scores ∞ and none can be compared. γ < 1 squashes an infinite future into one finite, comparable number (at most r_max/(1−γ)). That's the good case: γ makes "the long run" mathematically possible to want.

EP 02

Cookie now or cake later — one slider flips the plan

A gridworld with two exits: a +3 cookie three steps away and a +10 cake nine steps away. The arrows are the agent's optimal policy, recomputed by value iteration every time you move the slider. To a discounter, the cookie is worth 3·γ³ and the cake 10·γ⁹ — drag γ and find the exact moment the whole plan flips.

γ 0.70

The flip lives near γ ≈ 0.82 (where 3·γ³ = 10·γ⁹). Nothing about the world changed — only how much tomorrow is worth.

The point: γ is not a detail, it's the agent's personality. The same world, the same rewards, the same algorithm produce opposite lives at γ = 0.7 and γ = 0.9. When a trained robot "refuses" to do the long profitable thing, the first suspect is a discount factor that made the long thing literally not worth it.

EP 03

The horizon ruler — 1/(1−γ) in your hands

There's a beautiful rule of thumb: an agent with discount γ effectively plans about 1/(1−γ) steps ahead. γ = 0.9 → 10 steps. γ = 0.99 → 100. γ = 0.999 → 1000. Pick a task, slide γ, and check the marker against the green zone the task actually needs. This is how practitioners actually choose γ — horizon first, gamma second.

task γ

The healthy zone drawn on the ruler is roughly 0.5×–3× the task length: enough horizon to see the goal, not so much that credit smears over irrelevant futures.

Real numbers: DQN played Atari at γ = 0.99 (~100-step horizon — a Pong rally fits). Locomotion papers often use 0.99–0.995; games with hour-long strategy push 0.999+. Mismatch either way is a real bug: too low and the goal is invisible (✗); absurdly high and learning slows and destabilizes for nothing (✗, see EP 05).

EP 04

The myopic loop — a small treat, forever

Two agents race in identical corridors. A +1 cookie sits at cell 5 (it respawns), a +10 cake waits at the far end (eating it teleports you back to start, so the trip can repeat). Each agent follows the optimal policy for its own γ, computed live by value iteration. The top agent uses your slider; the bottom one is fixed at γ = 0.95. Run 60 steps and compare what they actually collect.

top agent's γ 0.50

The cookie respawns instantly — a simplification, but exactly the shape of real reward loops: notifications, snack breaks, farmable game points.

The trap is rational: the myopic agent isn't broken — pacing at the cookie truly is optimal under its γ, even though it collects half as much actual reward. This is the classic "reward-hacking-adjacent" failure: an agent that grinds a tiny farmable reward forever because the big payoff sits beyond its horizon. Raise γ and watch it walk right past the cookie.

EP 05

The edge of one — why γ→1 melts your value estimates

If patience is good, why not γ = 1? Watch a TD-learning agent estimate the value of a state in a never-ending loop of noisy +1 rewards. The true value is 1/(1−γ): at γ = 0.9 it's 10 — learned in seconds. At γ = 0.999 it's 1000 — the estimate crawls and wobbles. At γ = 1 there is no true value at all: the target is infinite and the estimate just climbs forever. Same code, four gammas.

γ 0.9

Real TD(0) updates on a 6-state ring, step size 0.08, rewards 1 ± noise. Nothing is faked — the divergence at γ = 1 is the actual math failing.

The engineering truth: value scale grows like 1/(1−γ), and so do variance and learning time — that's why nobody ships γ = 1 in a continuing task. When practitioners need true far-sightedness they reach for other tools (reward normalization, average-reward formulations, hierarchical policies) instead of pushing γ against the wall.