Base vs Instruct Models, Playable

EP 01

Same prompt, two different animals — autocomplete vs answer

Type-identical input, opposite behavior. A base model was trained on raw web text, so it asks: "what document does this look like, and what comes next in it?" An instruct model was further trained on (question → answer) pairs, so it asks: "what does this person want?" Pick a prompt, flip the switch, generate.

Pick a model and generate.

Simulated with curated continuation banks (no live model in this page) — but the split is faithful: these are the document types base models really produce for bare questions.

The core idea: the base model is not being dumb — a bare question on the internet is usually followed by more quiz, not by an answer. GPT-3 (base) did exactly this in 2020; ChatGPT is GPT plus the answering reflex.

EP 02

Blame the training data — the web rarely answers its own questions

Here is a miniature "web crawl": 20 snippets that contain a bare question. Draw pages at random and watch what actually follows the question in each. The bar chart below is computed live from this corpus — a base model's "weird" behavior is just these statistics, learned perfectly.

Draw a page to inspect it.

Why it matters: pretraining optimizes "predict the next token of the web", and on the web an answer is a minority outcome after a question. Good case: this objective teaches grammar, facts and reasoning for free. Bad case: it never teaches helpfulness — that has to be bolted on afterwards.

EP 03

The transcript trick — coaxing a base model into playing assistant

Before instruction tuning existed, people discovered a hack: don't ask the base model a question — show it a transcript of a conversation and let it continue the document. Suddenly the parrot impersonates a butler. Or a pirate. Toggle the persona, complete the transcript — then keep sampling and watch the hack's ugly seam.

The grey text is the prompt we wrote; the model only ever "continues the document".

Good case: framing turns raw continuation into any persona you want — this is few-shot prompting, and it's why base models are prized for flexibility. Bad case: the model doesn't know the conversation should stop: it happily hallucinates your next "User:" line too, because a transcript document just… keeps going. Instruct tuning is what installed the brakes.

EP 04

"Reply in JSON" — measuring format obedience

Instruction tuning's superpower is following the format you asked for. Pick a format request, run 12 samples through each model, and let an actual checker (a real JSON.parse, a real line counter) grade every sample. The compliance percentages are computed live from the graded runs.

Run a batch to grade it.

instruct: —base: —

Even instruct models slip: watch for samples that add a chatty preamble — a strict parser fails them. That is exactly why real APIs ship a dedicated "JSON mode".

Why it matters: every tool-using agent, every structured-output pipeline, every "extract fields from this email" product lives on this obedience. Base models ignore the request (bad case); instruct models mostly comply (good case) — and the residual failures are why constrained decoding exists.

EP 05

The alignment tax — instruct tuning makes it samey

Instruction tuning narrows the output distribution — great for reliability, terrible for variety. Sample 8 continuations of the same story stem from each model. The variety metrics below are computed live from your actual samples: distinct outcomes and Shannon entropy (max 3.0 bits for 8 draws).

"The lighthouse keeper opened the door and ___"

Sample from a model.

The instruct bank is deliberately peaked ("Sure! Here's a continuation…") — this mirrors measured mode collapse in RLHF'd models, where a few safe completions absorb most of the probability mass.

The honest trade: the same tuning that makes a model answer reliably also collapses its imagination — researchers measure exactly this drop in output diversity after RLHF. That's why fiction writers and synthetic-data teams still reach for base models, and why "creative mode" is a product feature, not a joke.