Chain-of-Thought, Playable

EP 01

The leap and the ladder — same problem, two ways to answer

Here is a four-step word problem. Try answering it in one leap (blurt the final number — like a model forced to answer immediately) and then on the ladder (write each small step). Every attempt is simulated with realistic error rates: one giant mental jump is fragile, each tiny step is ~97% safe. The tallies accumulate across your attempts — run each button a few times.

Pick a strategy above.

Simulated solver, not a real LLM — but the error rates (one fragile leap vs many ~97%-safe steps) mirror how transformer accuracy behaves on multi-step arithmetic.

The key move: writing a step turns a hard computation into an easy one plus a memory. Each generated token literally becomes input for the next forward pass — the page is the model's scratch paper. This is the entire mechanism behind "let's think step by step" and today's reasoning models.

EP 02

The depth curve — where the leap dies and the ladder survives

Make it quantitative. Slide the problem depth (how many chained steps the answer needs) and the per-step reliability, and watch both survival curves, computed live: the one-leap curve collapses almost immediately, while the step-by-step curve decays gently as reliability^depth. But look closely at the deep end — the ladder is not immortal either.

Problem depth 6 Per-step reliability 0.97

Why benchmarks flipped overnight: on shallow tasks, prompting style barely matters; past a few steps of depth, chain-of-thought is the difference between ~10% and ~80%. And the gentle decay of the ladder is why even reasoning models still fail very long proofs — 0.97 to the 50th power is 22%.

EP 03

Confidently wrong — one bad step poisons everything after it

The dark side of the ladder: steps are sequential, so an early slip doesn't stay contained — every later step does flawless arithmetic on a wrong number. Run chains and catch a cascade: the slip is marked in red, and notice how fluent and confident everything after it still looks. That's what makes wrong chains dangerous — they read exactly like right ones.

Per-step reliability 0.90 Depth 7

Run a few chains — sooner or later one slips.

Read the steps, not the tone: a chain-of-thought is an argument, and arguments inherit the truth of their premises. This is why real LLM deployments add step verifiers, unit tests, or a second model checking each line — confidence and correctness are uncorrelated after the first slip.

EP 04

The token tax — overthinking "2 + 3"

Chain-of-thought is not free: every reasoning token costs money and latency. Ask a trivial question both ways and watch the meter. Direct answering nails it instantly; "reasoning mode" burns dozens of tokens to arrive at the same place — and at maximum verbosity it occasionally talks itself out of a correct answer.

Question Verbosity 2×

Choose a strategy.

Token counts and latency are measured live from the text actually generated in this demo. The occasional self-sabotage at max verbosity mirrors a real documented failure: long reasoning chains can overturn correct first instincts on easy questions.

Reasoning is a budget, not a virtue: this is exactly why modern APIs expose a "reasoning effort" dial and why routers send easy queries to fast non-thinking models. Spend chain-of-thought where EP 02's curve says the leap fails — nowhere else.

EP 05

The jury trick — sample many chains and let them vote

If one chain is right 69% of the time, run several independent chains and take the majority answer — wrong chains scatter, right chains agree. Slide the jury size and watch the voting accuracy (estimated live from 1,500 simulated elections). Then switch to the trick question, where most chains share the same tempting mistake — and watch the jury make things worse, unanimously.

Jury size 7

Run an election to see the ballots.

This is "self-consistency," a real technique that added double-digit accuracy on math benchmarks — and its real limit: voting only fixes independent errors. When the model has a systematic blind spot, ten samples produce ten agreeing wrong answers, with extra confidence on top. More samples never fix a shared bias.