RAG in plain English — and the four places it goes wrong

If you’ve been anywhere near an AI project in the last two years, someone has said “RAG” at you. It’s worth understanding properly, because it’s the pattern behind most systems that are actually useful inside a business — and because the difference between a good implementation and a bad one is almost invisible from the outside until you’re relying on it.

The plain version

A language model knows a great deal about the world in general and nothing whatsoever about your business. It has never seen your pricing, your contracts, your support history or the policy you revised last March.

Retrieval-augmented generation solves that in the most direct way available. When someone asks a question, you first go and find the passages from your own documents that bear on it. Then you hand the model those passages along with the question and say: answer using this.

That’s it. Search, then answer. The “augmented” part is just the passages you pasted in.

The reason it matters is that the model stops guessing. It’s summarising text it can see rather than reaching for a plausible-sounding answer from training data. And because you know which passages you supplied, you can show the user exactly where the answer came from.

Where it goes wrong

Retrieval quality sets the ceiling. This is the big one. If the search step returns the wrong three paragraphs, no amount of prompt engineering rescues the answer — the model is being asked to answer from material that doesn’t contain the answer. Teams spend weeks tuning prompts when the actual failure happened one step earlier. Measure retrieval separately: for a set of real questions, does the right document come back in the top few results? Fix that number first.

Chunking destroys meaning. Documents get split into pieces before they’re indexed. Split naively — every 500 characters, say — and you cut tables in half, separate a clause from the definition it depends on, and orphan a heading from the section beneath it. A policy document, a contract and a support thread all want different splitting strategies. Using one setting for all of them is the most common silent quality killer we find.

Permissions get bolted on afterwards. If your retrieval layer can see every document in the company, and access control is applied by filtering the answer at the end, you have built a system that can leak. The salary review, the unannounced restructure, the client contract another team shouldn’t see. Permissions belong at retrieval — the search should never return what this user couldn’t already open.

Nothing is measured. Most implementations are evaluated by someone typing five questions and being pleased. Then a change is made, five different questions are tried, and nobody can say whether it got better. You need a fixed set of real questions with known-good answers, run on every change. Fifty is enough to start. Without it you’re not improving a system, you’re redecorating it.

What good looks like

A working retrieval layer has hybrid search — keyword and semantic together, because pure semantic search is oddly bad at exact terms like product codes and clause numbers. It has re-ranking, a second pass that reorders candidates by actual relevance. It has chunking tuned per document type. It enforces permissions at the retrieval step. It cites its sources so a human can check. And it has an evaluation set that runs automatically.

None of that is exotic. It’s about two weeks of additional work on top of a naive implementation, and it’s the difference between a demo and something your team will still be using next year.

Why it’s usually the right first build

A retrieval layer isn’t a product on its own. It’s infrastructure — and once it exists, every AI project after it gets cheaper.

The support agent needs it. The internal copilot needs it. The sales assistant that answers questions about your own services needs it. Build it once, properly, with permissions and evaluation in place, and each of those becomes a fortnight of work instead of a quarter.

Build it badly and you’ll rebuild it, usually after the first time someone gets an answer that was confidently, citably wrong.