Lost in the Middle, Playable

EP 01

Slide the needle — recall is a valley, not a plateau

The haystack below is a 120-sentence report. One sentence is the needle: “The vault code is 7-4-1-9.” Drag the slider to bury it anywhere. The curve is the model's probability of actually retrieving that fact when asked — measured behavior from long-context evals, not what the spec sheet promises.

needle position

The mental model: attention isn't a flat spotlight over the prompt. Primacy (the start anchors the task) and recency (the end is closest to the question) both get privileged treatment; the middle competes with everything and loses. Same shape as human serial-position memory curves — rediscovered inside transformers.

EP 02

Don't trust the curve — run 30 trials yourself

Pick a needle position, then ask the simulated model “What is the vault code?” thirty times. Each dot is one attempt: green = it retrieved 7-4-1-9, red = it answered from vibes (a garbled code or “I don't see one”). The hit rate is counted live from the trials, not read off the curve.

position 50%

Simulation: each trial succeeds with the position's recall probability. Real needle-in-a-haystack evals do exactly this — many trials per position — because single runs lie.

Why this matters for you: “it worked when I tested it” usually means the fact sat near the end of the prompt (right where you pasted it last). Move that same fact into the middle of a long document dump and your success rate quietly drops — per query, at scale, invisibly.

EP 03

The valley deepens — stretch the context and watch

Keep the needle pinned at the exact middle and grow the haystack from 8 sentences to 1,024. Ghost curves show where you've been. The readout is the live middle-of-context recall — then verify it with 20 sampled trials at the current length.

context length

The trap: “1M-token context window” is a capacity claim, not a recall claim. A model can accept a million tokens and still fish reliably only near the edges. Effective context — the region where retrieval stays sharp — is smaller than advertised context, and the gap grows with length.

EP 04

Run the benchmark — the heatmap every lab publishes

This is the classic needle-in-a-haystack grid: needle position across, context length down, one cell per condition. Hit Run live benchmark to fill every cell with 12 sampled trials each (960 questions total) — then click any cell to inspect its live numbers. Green corners, red belly.

click a cell after running

Each cell = 12 independent simulated retrievals; colors come from those counts, so your grid is a little different every run — just like real evals.

Read real model cards this way: when Anthropic, OpenAI or Google publish long-context evals, it's this exact grid. ✓ Good case: top rows and both edge columns stay green — short contexts and edge placements are safe. ✗ Bad case: bottom-middle cells — long context, buried fact — are where production RAG systems silently lose documents.

EP 05

Route around the hole — reorder, or retrieve

Two production fixes, tested live against the naive baseline at 256 sentences: edge reordering (place the key fact at the start or end on purpose) and retrieve-then-read (search for the needle's chunk first, hand the model a 12-sentence context). Each bar = 40 fresh simulated questions. Then tick “paraphrased needle” and watch which fix survives.

needle is paraphrased (hurts the retriever)

Simulation: retrieve-then-read finds the right chunk 94% of the time normally, 62% when the needle is worded differently from the question; recall inside a 12-sentence context uses the same U-curve as everywhere else.

The honest summary: reordering (LangChain ships it as “long-context reorder”) only works when you already know which fact matters — often you don't. Retrieval keeps contexts short, which is why RAG beats prompt-stuffing at scale, but it inherits the retriever's misses: a paraphrased needle fails before the model ever sees it. There is no free lunch — only shorter contexts.