Test-Time Compute, Playable

EP 01

The second axis — a small model that thinks vs a giant that blurts

Two contestants on a 200-problem math exam: a big model that answers instantly, and a small model with a thinking budget you control. Drag the dial. On easy sets the giant's raw knowledge wins. Slide to the hard set and keep paying: around a few thousand thinking tokens, the small thinker walks past the giant. Every score is a live 200-problem simulation.

✓ good case: hard problems + 8k thinking tokens — the small model overtakes a model 20× its size✗ bad case: easy problems + 8k tokens — same score as instant answers, 50× the bill

thinking budget 0 tokens —

Simulation: each problem has a hidden difficulty; the thinker's solve probability grows log-linearly with budget (the published o1-style curve shape), the giant has fixed higher baseline skill. 200 fresh problems per drag.

Intelligence became a rental market. The train-time axis buys permanent capability; the test-time axis rents extra capability per query. The economic consequence is everywhere in 2025-26 pricing: "reasoning effort" tiers, thinking-token billing, and the return of the small-but-patient model.

EP 02

Best-of-N — sampling is cheap, judging is the bottleneck

The simplest way to spend test-time compute: generate N independent solutions and let a verifier pick the best. Two dials below: N, and how good your verifier actually is. With a strong verifier, accuracy climbs beautifully with N. Now degrade the verifier and watch the curve flatten — past a point, more samples just give the judge more chances to pick a confident-looking wrong answer.

✓ good case: verifier 95% + N=32 — solve rate 41% → 88%✗ bad case: verifier 65% + N=32 — stuck at 62%; you're paying for lottery tickets a blind judge can't read

samples N 4 verifier accuracy 95% —

Measured over 500 simulated problems per setting: each sample solves with p=0.12, the verifier scores each candidate and picks its favorite — verifier mistakes are actually sampled, not approximated.

Best-of-N moved the frontier from generation to verification. This is why process reward models, unit tests, and formal checkers became the hot infrastructure: with a perfect verifier (like code that either passes tests or doesn't), N scales gloriously — which is exactly why math and code fell to test-time scaling first.

EP 03

Trees vs retries — don't restart from zero

On a 4-step problem, independent retries are wasteful: fail at step 3 and the whole attempt is trash, including two perfectly good steps. Search keeps the tree: back up one node and try a different branch, reusing everything above it. Same token budget, both strategies, measured head-to-head. Then flip to a 1-step problem and watch the tree's bookkeeping become pure overhead.

✓ good case: 4-step problem, budget 60 steps — tree search 74% vs retries 31%✗ bad case: 1-step problem — retries win; the tree spends its budget managing a tree of one branch

problem depth 4 steps step budget 60 —

Each duel: same problem instance (per-step success 65%, verifiable steps), same budget. Retries restart on failure; the tree backtracks one level. 300 instances per click, wins tallied.

This is the AlphaGo lesson arriving in language: search amortizes partial progress. Modern reasoning models blend both — sampling for diversity, backtracking inside a chain — and the deeper the task (proofs, agent workflows, refactors), the more the tree pays.

EP 04

The router — a fixed budget and a mixed inbox

Real workloads mix trivia with monsters. You have a fixed total budget across 1,000 queries: spend it evenly, or route it — skim the easy ones, pour tokens on the hard ones. The catch: routing requires predicting difficulty before solving, and your router is imperfect. Drag its accuracy and watch adaptive routing beat uniform… then fall below it when the router starts sending monsters to the fast lane.

✓ good case: 85% router — adaptive beats uniform by 9 points at the same cost✗ bad case: 55% router (coin flip) — worse than not routing at all; hard problems get starved at random

router accuracy 85% —

Workload: 70% easy (solvable with 100 tokens), 30% hard (need 4k+). Both strategies spend exactly the same total tokens; only the allocation differs. Every bar is a fresh 1,000-query run.

"How long should I think?" is itself a prediction problem — and it's the one every serious deployment now solves: effort routers, "auto" reasoning modes, escalation ladders (cheap model → expensive model → human). The margin between good and bad routing is worth more than most model upgrades.

EP 05

The overthinking trap — thinking longer, getting wronger

The dirty secret of thinking budgets: the curve isn't monotone. On problems the model can already solve, long chains start second-guessing correct answers — talking themselves out of the right one, drifting into invented complications. Drag the budget across the three problem types and find each one's peak. Past it, you're paying extra to be wrong slower. The latency meter runs too, because your user is still waiting.

✓ good case: each problem type at its knee — max accuracy, tolerable latency✗ bad case: 32k tokens on an easy question — accuracy DOWN 6 points from its peak, 40× the cost, 30 seconds of spinner

thinking budget 512 tokens —

The inverted-U on easy problems reflects the documented "overthinking" phenomenon in reasoning models (chains that revise correct answers away). Accuracy at each budget is measured over 400 simulated problems; latency assumes 60 tokens/s.

Thinking is a drug with a dose-response curve. The skill of the reasoning era isn't "more compute" — it's dosing: verifiers to stop early when the answer is checkable, routers to skip thinking when it isn't needed, and caps because the tail of the distribution is where the money and the accuracy both die.