Data Contamination, Playable

EP 01

Inflate a score with your own hands

One model, fixed real skill (it genuinely solves 42% of problems it has never seen). You control one thing: what fraction of the official benchmark's questions leaked into its training data. Drag the slider and watch two bars diverge: the benchmark score climbs toward 100% while the fresh-exam score — same difficulty, questions written yesterday — doesn't move a millimeter.

✓ good case: 0% leak — benchmark 42%, fresh exam 42%; the number means something✗ bad case: 60% leak — benchmark 77%, fresh exam still 42%; a headline built on memory

benchmark leaked into training 0% —

Both scores are measured live over 400 simulated questions each: leaked questions are answered from memory (~97% recall), unleaked ones from true skill (42%). That's the entire mechanism of contamination — no conspiracy required.

Contamination doesn't need cheaters, just a big scrape. GSM8K, MMLU, HumanEval — all sit on GitHub and blogs that feed every crawl. The gap you just created (benchmark minus fresh-exam) is exactly what researchers find when they rewrite benchmark questions and watch scores drop 10-30 points.

EP 02

N-gram forensics — catch the verbatim, miss the disguise

The standard audit: slide a window of n consecutive words over the training corpus and flag benchmark questions that share a long-enough run. Below, a benchmark question is checked against two training documents — one contains it verbatim, one contains a paraphrase (same problem, reworded). Drag n and watch the highlighter: the verbatim copy lights up like a crime scene; the paraphrase never triggers at any n.

✓ good case: verbatim leak vs 13-gram check — caught, 9 matching runs highlighted✗ bad case: paraphrased leak — zero n-gram matches at every window size, sails through the audit

n-gram window 13 —

The matching is real string comparison running on the two documents as you drag. 13 is the window GPT-4's contamination report actually used — now you know both why, and what it can't see.

N-gram checks are a lock that only stops honest people. They catch accidental verbatim leaks (most of them!) and miss anything reworded, translated, or reformatted — which is also exactly how a benchmark spreads across the real internet: quoted, paraphrased, translated into Chinese study guides.

EP 03

The canary probe — make the model confess

Sharper audit, no corpus access needed: show the model the first half of a benchmark question and ask it to continue. A clean model produces something plausible but different; a contaminated model completes it word for word — because it isn't predicting, it's remembering. Probe the two models below ten times each and read the confession rate.

✓ good case: clean model — 0/10 verbatim completions, plausible variety every time✗ bad case: contaminated model — 8/10 exact continuations, including the typo the benchmark itself contains

—

The tell inside the tell: benchmark files often contain small typos or odd formatting. A model that reproduces the typo didn't learn the math — it learned the file. Real audits (and "canary strings" planted in eval sets) use exactly this trick.

Memorization leaves fingerprints that reasoning doesn't. Completion probes, canary GUIDs, and typo-reproduction turned contamination from an unfalsifiable accusation into a measurable property — which is why eval sets now ship with planted canaries, and why serious labs report probe results alongside scores.

EP 04

The leaderboard illusion — rank by memory, call it intelligence

Eight models. By construction, their true skill is identical — 45%, every one of them. The only difference is how much of the benchmark leaked into each one's training scrape. Look at the leaderboard the public sees, then hit the reveal button. The ranking you were about to tweet about is a ranking of crawl dates.

✗ bad case: the public view — "Model-G crushes Model-B by 21 points!" (they are the same model)✓ good case: fresh-exam column revealed — eight statistical ties, one boring truth

—

Each model's benchmark score is measured by the EP 01 mechanism with its own random contamination level; the fresh-exam scores differ only by sampling noise, live-computed.

A leaderboard is only as meaningful as its least-contaminated row. This is why private held-out test sets, freshly-written eval suites (and paying humans to write new problems every quarter) became a business — and why a 2-point gap between models on a public benchmark is closer to astrology than measurement.

EP 05

Play the decontaminator — every filter leaks something

Now sit on the other side of the desk. Your training scrape contains three kinds of leaks — verbatim copies, paraphrases, and translations — plus a mountain of legitimate data. Toggle your filter stack and read the scoreboard: how much contamination survives, and how much good data your filters shredded by accident. Try to win. You can't, quite.

✓ good case: exact-dedup + 13-gram — cheap, catches 70% of leaks, near-zero collateral✗ bad case: add aggressive semantic filtering — catches translations at last, but deletes 6% of your legitimate math data with it

—

Toggle any combination — residual contamination and collateral damage are recomputed from a simulated 10,000-document scrape (65% verbatim / 25% paraphrase / 10% translated leaks) with published-ballpark catch rates per filter.

Decontamination is a tax with diminishing returns, and translation is the loophole that survives. The endgame isn't a perfect filter — it's changing the exam: private eval sets, continuously rewritten questions, and post-cutoff benchmarks like fresh contest problems. When the textbook can't be trusted, rotate the exam.