Lesson 1 of 5
Why RAG Exists
"Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it." — Samuel Johnson, in Boswell's Life of Johnson (1775)
A language model has only the first kind. Everything it can tell you was compressed into its weights during training, from a corpus that was assembled once and then frozen. It has no index, no shelf to walk over to, no way to look something up. When you ask about your company's refund policy, the model does not consult anything — it produces the most plausible continuation of a question about refund policies, drawing on every refund policy it ever read, none of which were yours.
This is not a defect to be trained away. It is what the architecture is. And it produces a specific, predictable set of limitations, each of which matters more in production than in a demo: the model's knowledge stops at a cutoff date; it never contained your private data; it cannot distinguish what it knows well from what it half-remembers; and it cannot cite anything, because there is nothing to cite. Part I Chapter 5 established these as facts about the technology. This part is about what to do with them.
Retrieval-augmented generation is the answer that turned out to work, and its central idea is almost embarrassingly simple: if the model does not know something, find the relevant text and put it in the prompt. The model is then no longer recalling — it is reading. That single change converts a question about the model's memory into a question about your search quality, which is enormously better, because search quality is something you can inspect, measure, and fix.
The consequences are larger than they first appear. A model reading provided text can cite it, so an answer becomes checkable. Content updates the moment the source updates, with no retraining. Access control becomes possible, because you can decide what a given user's retrieval is allowed to return. And when something goes wrong you can ask a question you could never ask of a bare model: was the right information available, and did the model use it? That question has two answers, and telling them apart is most of the work in this part.
If context windows keep growing, why not simply put everything in the prompt and skip retrieval entirely? The answer is not only that it would be expensive — it is that adding more context past a point makes answers worse, and understanding why explains what retrieval is really for.
Four ways to give a model knowledge
There are only four, and choosing badly between them is one of the more expensive mistakes in this field.
Run to execute the tests…
The fifth test names a distinction people collapse constantly: fine-tune for form, retrieve for facts. Fine-tuning is genuinely good at teaching a house style, a rigid output shape or a domain register. It is unreliable at teaching facts, cannot cite them, and cannot update them without another training run.
Consider two lawyers. One has memorised a great deal of law and answers from memory, fluently, without opening anything. The other keeps the statute book on the desk, finds the relevant section, and reads it to you. On a familiar question they perform about equally. On an unusual one, on a law amended last month, or on any question where you need to check the answer yourself, the second is obviously better — and notice that the second lawyer is not more knowledgeable. They are better organised. Notice too what happens if the statute book is missing a volume: the second lawyer can tell you they cannot find it. The first will simply answer, and sound exactly as confident as always.
Why not just put everything in the prompt
The obvious objection to retrieval is that context windows keep growing, so eventually you can skip the search step. The cost argument answers this on its own, but the more interesting reason is that it does not actually work.
Run to execute the tests…
The third and fourth tests carry the real argument. Irrelevant context does not merely waste tokens — it lowers accuracy, and contradictory context lowers it sharply. Material buried in the middle of a very long prompt is used less reliably than material near either end. So retrieval is not a cost optimisation you can abandon when windows get cheap enough. Filtering is part of the quality.
The term comes from a 2020 paper by Lewis and colleagues at Facebook AI Research, which combined a neural retriever with a generator and trained them together. That original architecture is not what most people mean by RAG today — the modern version is looser and simpler, usually an off-the-shelf embedding model, a vector store, and a frozen generation model that never sees a gradient. What survived was the idea rather than the implementation: separate knowing from finding, and let each be improved independently. It is worth appreciating how much older the finding half is. Everything Part IV covers about chunking, ranking, term weighting and evaluation comes from information retrieval, a field with fifty years of results, its own evaluation traditions, and a well-earned scepticism about systems that look impressive on demonstrations. A recurring pattern in AI engineering is a team discovering, several months into a RAG project, that their central problem is a search problem and that the search literature solved it in about 1994.
The minimal pipeline
RAG has four stages. Everything in the rest of this part is a refinement of one of them.
The two dotted edges are the reason this diagram is worth drawing. They are different failures with different fixes, and a system that cannot distinguish them cannot be debugged.
Run to execute the tests…
The fourth test is the branch that matters most. Empty retrieval means "I do not have this", and that decision belongs in code, before a model is called. Sending an empty context and hoping the model refuses is how invention gets into a system that was supposed to be grounded.
The third is the security shape of the whole part: filter by permission before retrieval. If you filter after generation, the restricted content was already in the context — and in your logs, and possibly in a cache.
Which stage failed
When a RAG system gives a bad answer, there are exactly two possibilities, and conflating them is the most common way teams waste weeks.
Run to execute the tests…
The fifth test states the constraint that reorders most RAG roadmaps: retrieval recall is a hard ceiling on answer accuracy. If the right chunk reaches the model 80% of the time, no prompt in the world gets you past 80%. Teams who skip this measurement spend their effort on the generation stage, where it is visible and satisfying, while the actual limit sits upstream.
The third test names the trap in the other direction. A correct answer produced without the evidence looks like a pass and is not one — the model answered from its priors while your retrieval was broken, and you will find out on the first question where the prior is wrong.
Citations, and checking them
Grounding is only worth something if you verify it.
Run to execute the tests…
The fifth test draws a line worth remembering: citation validity proves the chunk existed and was retrieved. It says nothing about whether the sentence attached to it reflects what the chunk says. That is a harder check, and Chapter 5 is where we attempt it.
The first thing I build on any RAG project now is not the pipeline. It is a script that takes a question and prints the retrieved chunks with their scores, and nothing else. It takes twenty minutes and it changes how the whole project goes, because the first time you run it on real questions you discover that retrieval is much worse than you assumed — the right chunk sitting at position eleven, or split across two chunks so that neither scores well, or missing because the document uses "reimbursement" and the user typed "refund". Every one of those is invisible from the far end of the pipeline, where you see only a bad answer and reach for the prompt. I have watched a team spend three weeks on prompt engineering for a system whose retrieval recall was 0.55, which meant nearly half their questions were unanswerable before the model was invoked. Nobody had measured it, because measuring it requires deciding which chunk should have been retrieved, and that felt like admin rather than engineering. It is the highest-value admin in this part.
A Nigerian insurance company built an internal assistant over policy documents, and it demonstrated well enough to be rolled out to the claims team. Within a fortnight, complaints: the assistant kept giving answers that were subtly out of date. The team's assumption was a caching problem, and they spent several days on invalidation before someone checked the actual retrieved chunks for a failing question. The index contained both the current policy document and an archived 2023 version, and the archived one scored higher on most queries because it was longer and repeated key terms more often. Nothing was cached, nothing was stale, and the retrieval was working exactly as specified — it was returning the best keyword match, which happened to be the wrong document. The fix took an afternoon: a metadata filter excluding archived documents, plus a rule that every chunk carries its document's effective date and the prompt states it. The deeper lesson they took was about instrumentation rather than about archives. They had built a system whose most likely failure mode was invisible from the outside, and it took two weeks and a lucky guess to look in the right place.
Imagine your RAG system answers a question perfectly. Now ask: how would you know whether it was grounded? The answer contains the right fact and cites a chunk that was genuinely retrieved — but the model also happens to know this fact from training, so the same answer would have appeared with an empty context. Most RAG systems cannot distinguish those two situations, and the difference matters enormously, because one of them keeps working when your corpus changes and the other silently does not. The experiment that separates them is cheap and almost nobody runs it: take your eval set, replace the retrieved context with an unrelated document, and re-score. Every case that still passes was never using retrieval. Whatever number that produces is the fraction of your system that is a language model wearing a search engine's clothes.
Common Mistakes
Fine-tuning to teach facts. It is unreliable at facts, cannot cite them, and cannot update them without another training run. Fine-tune for form, retrieve for facts.
Assuming a bigger context window removes the need for retrieval. Irrelevant context lowers accuracy and contradictory context lowers it sharply. Filtering is part of the quality, not only a cost optimisation.
Padding to k with low-scoring chunks. A chunk scoring zero is not a near miss, and filling the context with them actively degrades the answer.
Building retrieval for eight documents. For a small stable corpus, put it in the prompt. Unnecessary infrastructure is its own failure mode.
Filtering permissions after generation. By then the restricted content has been in the context, the logs, and possibly a cache. Filter before retrieval.
Sending an empty context and hoping the model refuses. Empty retrieval means "I do not have this", and that branch belongs in code.
Measuring only end-to-end accuracy. You need retrieval recall and answer accuracy. One number cannot tell you which stage to work on.
Not recording gold chunk ids in eval cases. Without knowing which chunk should have been retrieved, failure attribution is not computable — and reconstructing it later is miserable.
Doing prompt engineering on a system with bad retrieval. Recall is a hard ceiling on accuracy. At 0.8 recall, no prompt gets you past 80%.
Counting a correct-but-ungrounded answer as a pass. The model answered from its priors while retrieval was broken. It looks like success until the prior is wrong.
Trusting citations without validating them. A fabricated citation survives a casual glance because nobody clicks every reference.
Treating a valid citation as a correct answer. Validity proves the chunk was retrieved, not that the sentence reflects it.
Storing chunks without their origin. A citation the user cannot open is decoration, and provenance is usually impossible to reconstruct afterwards.
Indexing archived and current documents together. The archived one is often longer and repeats key terms, so it wins on score while being wrong.
Using retrieval for live state. Order status and balances have an API. A vector store full of stale snapshots is a tooling problem misdiagnosed as a retrieval one.
Interview Corner
Build the whole minimal pipeline for a corpus you actually have: index(docs), retrieve(query, { k, roles }) filtering by permission inside the search rather than after it, augment(question, chunks) returning mustRefuse when retrieval is empty, and validateCitations(answer, retrievedIds). Then build the diagnostic that matters most: diagnose(evalCase, retrievedIds, answer) returning retrieval, generation, unsupported or ok, and report retrieval recall and answer accuracy as two separate numbers. Finally run the ungrounded test — re-score your eval set with unrelated context substituted — and find out what fraction of your answers never needed retrieval at all.
Run to execute the tests…
Summary
- A model recalls; it cannot look anything up. Its knowledge is frozen at a cutoff, never contained your private data, and cannot be cited.
- Four ways to supply knowledge: pretraining, fine-tuning, stuffing the prompt, and retrieval. Fine-tune for form, retrieve for facts — fine-tuning cannot cite, cannot update without retraining, and cannot filter per user.
- Bigger context windows do not remove the need for retrieval. Irrelevant context lowers accuracy, contradictory context lowers it sharply, and material buried mid-prompt is used less reliably. Filtering is part of the quality.
- The pipeline is four stages: chunk and index offline; retrieve, augment and generate per request. The rest of this part refines one stage each.
- Filter permissions before retrieval. After generation is too late — the content was in the context and the logs.
- Empty retrieval is a refusal, decided in code before the model is called.
- Two failure classes: retrieval (the chunk never arrived — no prompt fixes it) and generation (it arrived and was ignored). Systems that cannot tell them apart cannot be debugged.
- Retrieval recall is a hard ceiling on answer accuracy. Measure both, separately, with gold chunk ids recorded in the eval set.
- A correct answer without the evidence is not a pass. The model used its priors while retrieval was broken.
- Validate citations against what was actually retrieved — and remember that a valid citation proves the chunk existed, not that the claim reflects it.
In the next chapter, Chunking Strategies, we deal with the stage that quietly determines everything downstream: how a document becomes retrievable units. Chunk too large and precision collapses; too small and the answer is split across pieces that individually mean nothing. It is the least glamorous decision in RAG and the one most likely to be the reason your recall is 0.55.
What strikes me about RAG is how thoroughly it reframes the problem. Before it, "the model does not know our data" felt like a limitation of the model, and the natural response was to make the model know more — a training problem, requiring expertise most teams do not have and budgets most cannot justify. RAG's contribution was to say the model does not need to know it, it needs to be able to read it, and that turns an intractable machine-learning problem into a tractable search problem. I find this genuinely elegant, and I also notice how often it gets described as if it were a way of making models smarter. It is not. It is a way of making them better informed, which is a different and more achievable thing, and the distinction shows up the moment retrieval fails — because then you have exactly the model you always had, answering from priors, with all the confidence and none of the evidence. Johnson's two kinds of knowledge were never equally good. Knowing where to find something is only useful if the finding actually works, and most of the difficulty in this part comes from that unglamorous half.
Q&A · 0
Enrol to ask questions and join the discussion.
No questions yet — be the first to ask.