Models don't read words — they read tokens
Before a model sees your text, it gets chopped into tokens — frequent chunks that may be whole words, pieces of words, or punctuation. Type anything below and watch it happen. Notice: common words survive whole, rare words get split.
Simplified subword tokenizer for illustration — real ones (BPE) learn their chunks from data, but split text the same way in spirit.
Watch it guess — a tiny LLM living in this page
This is a real (tiny!) language model trained right here in your browser on a few paragraphs of text. At every step it looks at recent words, computes a probability for each possible next word, and samples one. The bars are its actual internal probabilities — GPT does the same thing with 100,000+ options instead of a handful.
Temperature 0.8 — balanced: mostly picks likely words, occasionally surprises.
Why context is everything
Same machine, one change: how many recent words it's allowed to see before guessing. Watch the predictions for the sentence below sharpen as you give the model more memory. This is the intuition behind “context windows” — and why models with amnesia ramble.
Fluency is not truth — make it hallucinate
The model can only remix what it was trained on — but it never says "I don't know" by default. Ask it about something inside its tiny world, then about something it has never seen. Watch the confidence stay high either way.
The doom loop — why greedy decoding repeats itself
Set temperature to zero and the model always picks the single most likely word. Sounds sensible — but a deterministic path through a finite vocabulary must eventually revisit a state, and from there it loops forever. Run it and catch the loop; then let a little randomness break the spell.