Retrieval-Augmented Generation, Playable

EP 01

Watch retrieval think — every chunk gets a score

This is the whole corpus: the 8-fact internal wiki of a fictional company, Northwind Robotics. Type any question and the retriever scores every chunk against it, live — matched terms light up gold, and the best chunk gets the gold border. This scoring step happens before the model writes a single word.

Scoring here is honest keyword overlap (with light stemming), normalized by chunk length — the classic BM25 family in miniature. Production systems usually score with embeddings instead; EP 03 shows why.

The mental model: RAG is search glued to generation. If the right fact gets a high score here, everything downstream can work; if it doesn't, no amount of model intelligence can recover it. Retrieval quality is the ceiling of the whole system.

EP 02

Answers with receipts — and the courage to say “I don't know”

Now the full pipeline: retrieve the best chunk, answer from it, cite it. Ask about the founding — clean, cited answer. Then ask about the stock price, which is in no document. With the confidence gate off, the pipeline grabs the least-bad chunk and “answers” anyway. Toggle the gate and watch it learn honesty.

refuse below 60% confidence

Ask a question above.

Confidence = fraction of your question's content words found in the retrieved chunk. Real systems use a reranker score, but the failure mode is identical.

Why it matters: retrieval always returns something — “most relevant” is not “relevant”. Every serious RAG deployment has this gate (a minimum score below which it says “not in my documents”), and tuning it is the difference between an assistant that admits gaps and one that improvises answers about your stock price.

EP 03

The paraphrase miss — same question, zero matches

The wiki says “Marta Voss has been the CEO…”. Ask “Who is the CEO?” and retrieval nails it. Ask the same thing as “Who is in charge of the company?” and keyword retrieval finds… the battery chunk — because “charge” matches “charges in 40 minutes”. Flip on semantic expansion to fix it.

semantic expansion

Ask one of the three questions.

Semantic expansion here is a hand-built synonym table (charge→ceo, firm→northwind…). In real systems, embedding vectors play this role — matching by meaning instead of spelling. That's why EP 01's honest keyword scorer is rarely used alone.

The #1 RAG failure in the wild: your users never phrase questions the way your documents phrase answers. Retrieval that matches surface strings misses — or worse, matches the wrong string confidently. This is exactly the problem vector embeddings were brought into RAG to solve.

EP 04

Garbage in, gospel out — poison one fact and ask again

RAG makes the model trust the documents more than itself — that's the feature. So what happens when a document is wrong? Ask about revenue on the clean wiki. Then press the poison button (revenue quietly becomes 100× bigger), ask again, and compare the confidence readouts.

The wiki is clean. Ask the question.

The uncomfortable truth: retrieval scores measure relevance, never truth — the poisoned chunk scores identically to the real one. Grounding eliminates made-up answers but faithfully amplifies whatever is in the index: stale wiki pages, a vendor's marketing claims, or a deliberately planted document. “With citations” means auditable, not correct.

EP 05

The top-k dial — starve it or drown it

How many chunks should you hand the model? The question below needs two facts (battery and price). Drag k: at 1 the answer is missing half; at 2–3 it's complete and lean; at 8 you're paying for support-ticket trivia the model must wade through. The coverage checklist and token bill update live.

“What is the Otter-3's battery life and price?”

retrieve top k = 1

Why it matters: k is a real dial in every RAG stack (and “context stuffing” is its failure mode). Too small → incomplete answers on multi-fact questions. Too big → higher cost, slower answers, and the well-documented “lost in the middle” effect where models overlook facts buried among irrelevant chunks. Most production systems live at k = 3–10 plus a reranker, for exactly the reasons you just felt.