Logits & Softmax, Playable

EP 01

Grab the logits — raw scores become probabilities under your fingers

The left panel shows five logits — raw, unbounded scores (they can be negative!) the model assigns to candidate next words. The right panel is what softmax makes of them: numbers that are all positive and sum to exactly 100%. Drag any bar on the left up or down and watch the probabilities react instantly.

Softmax: p_i = e^z_i / Σ e^z_j. Note what dragging never does: change the order. Softmax is monotonic — the #1 logit is always the #1 probability.

✓ Why this design shines: logits let the network compute freely (any real number, no constraints), and softmax converts to a legal probability distribution at the very end while preserving the ranking. Every LLM you've used runs this exact function over ~100,000 logits on every single token.

EP 02

The exponential lever — +2 logits ≈ 7× the probability

Two candidate words, one number that matters: the logit gap Δ between them. Because softmax exponentiates, the probability ratio is exactly e^Δ — no matter what the absolute scores are. Slide the gap and watch a linear nudge become an exponential landslide.

logit gap Δ

The rule of thumb to keep: Δ=1 → 2.7×, Δ=2 → 7.4×, Δ=3 → 20×, Δ=5 → 148×. Good case: a genuinely better answer only needs a few logits of headroom to dominate cleanly. Bad case: the same lever amplifies a tiny scoring error into near-certainty — a 0.5-logit mistake already skews the odds 1.65 to 1.

EP 03

Temperature lives INSIDE the formula — it divides the logits

The temperature knob isn't bolted on after the fact — it's a division inside softmax: p_i = e^z_i/T / Σ e^z_j/T. Dividing all logits by T shrinks or stretches every gap at once. The live table below shows each step of the computation as you slide T.

Temperature 1.00 make top-2 nearly tied (3.20 vs 3.05)

One division, two personalities: T<1 stretches gaps → sharper, more deterministic; T>1 shrinks gaps → flatter, more adventurous. ✗ The trap: tick the "nearly tied" box and set T=0.1 — softmax turns a 51/49 judgment call into a 95%+ verdict. Low temperature doesn't add knowledge; it just hides the model's uncertainty.

EP 04

The 70/30 illusion — confident numbers from a coin-flip brain

A gap of just 0.85 logits — barely different scores — comes out of softmax as a decisive-looking 70% vs 30%. But scores that close are inside the model's own noise: rephrase the question slightly and the scores wobble by about ±1 logit. Press the button to re-ask 60 paraphrases and see how often the "70% answer" actually survives.

true logit gap

Each square = one paraphrase of the same question, with ±1 logit of score noise (a realistic wobble). Mustard = answer A won, grey = answer B won.

✗ The bad case in one line: softmax reports 70/30, reality behaves like 60/40 — the displayed probability looks like measured confidence but is just an exponentiated guess. ✓ Push the gap past ~2.5 logits and the answer survives every paraphrase: large logit gaps are trustworthy, small ones only look it. This is one reason token probabilities are a shaky hallucination detector.

EP 05

Saturation — where the megaphone jams at full volume

Crank one logit sky-high and softmax pins its probability to ~100% — and then stops responding. The sensitivity of the output, p(1−p), collapses toward zero: new evidence for a competitor barely dents the verdict, and in training, gradients through a saturated softmax vanish. Also try shifting ALL logits by +100: absolutely nothing changes, because softmax only ever sees the gaps.

logit of token A

Two lessons hiding in one curve: ✓ in the moderate zone the model stays responsive — evidence moves probabilities, gradients flow, learning works. ✗ in the saturated zone it can no longer express doubt or absorb corrections. And the +100 shift explains a real trick: implementations subtract the max logit before exponentiating (the log-sum-exp trick) — it's free precisely because softmax is shift-invariant.