Retrieval-augmented generation (RAG) is the pattern where an AI tool searches a body of documents before it answers, then writes its answer out of the passages it found instead of out of what the model absorbed during training. Question comes in, a search step pulls back a handful of passages it scores as relevant, those passages are pasted into the model’s prompt, the model answers over them. It matters to buyers because RAG is the machinery underneath nearly every vendor claim about being “grounded,” “citing its sources,” or “trained on your data” — and which of those three the vendor actually means changes what you have to test before you sign.
RAG is not training. Your documents never enter the model’s weights; they are read at question time and dropped afterward. A vendor saying “the model learns your contracts” is either describing RAG loosely or doing something that deserves a direct question about where your data ends up. RAG is also not a hallucination fix, and it is not a property of the model at all — it is a pipeline wrapped around a model that is otherwise unchanged. Two products running the identical model behave completely differently depending on how well that pipeline finds things.
The two halves, and only one of them is in the demo
Every RAG system has a retrieval half — index the corpus, cut it into chunks, search, rank what comes back — and a generation half, where the model writes prose from whatever the first half handed it.
Vendor demos exercise the generation half. That is the part that looks impressive, and it is not where the errors start. When the retrieval step misses, the model does not know it missed: it writes a fluent, well-cited answer from the wrong three passages and reports no problem. The model cannot tell you that the passage it needed was never handed to it. That single fact should reshape how you run a pilot, because it means testing the output quality of questions the tool answers well tells you almost nothing.
”Grounded” is a claim about plumbing, not about accuracy
The strongest public evidence here comes from law, where the claims were loudest and someone measured them. Stanford’s RegLab ran a preregistered evaluation of the RAG-based legal research tools sold by LexisNexis and Thomson Reuters — Lexis+ AI, Westlaw AI-Assisted Research, and Ask Practical Law AI — and found they hallucinate between 17% and 33% of the time. LexisNexis had marketed “hallucination-free” linked citations; Thomson Reuters said its tools avoid hallucinations by relying on trusted content. Both claims were overstated. The honest reading is not that RAG failed: those tools did beat general-purpose GPT-4 on the same questions. The reading is that retrieval over a curated corpus moves the error rate down and never to zero, so “grounded” describes an architecture, not a guarantee.
Two failure shapes come out of that, and they are not equally dangerous. One is a plainly wrong answer, which a domain expert catches. The other is a misgrounded answer — the substance is wrong but a real citation is attached to it, pointing at a real document that does not actually say what the answer claims. That one survives review, because the citation is exactly the signal a busy reviewer uses to decide the answer was already checked.
Diagnostic questions for a vendor
These are ordered so the first two do most of the work.
- Show me a question the tool gets wrong, and tell me whether retrieval or generation caused it. A vendor who has never characterized their own retrieval failures has not evaluated the system, only the demo.
- Does a citation land on a passage or on a document? Document-level citations are unverifiable at any real volume — nobody reads a 60-page MSA to confirm one sentence. Passage-level or clause-level citation is what makes review cheap enough to actually happen.
- What does it do when the answer is not in the corpus? The two behaviors are “I don’t have that” and a quiet fall back to the base model’s general knowledge. The second is where confident nonsense enters, and it is rarely disclosed on a pricing page.
- Are permissions applied at retrieval or after generation? The correct answer is retrieval: the model is only ever shown documents the asking user is already entitled to see. Glean built its indexing around permission-trimmed retrieval for exactly this reason. Filtering the answer after the model has read a restricted document is not access control, it is redaction with a leak.
- How long after a document changes is the new version the one being answered from? Minutes, hourly, or nightly are all defensible. Not knowing is not.
- Can I see what was retrieved, not just what was written? Without that panel, you cannot debug a wrong answer and neither can their support team.
Does a bigger context window make RAG unnecessary?
No, and the vendors selling million-token context windows are not claiming it does. Three things keep retrieval in the architecture: a corpus of tens of thousands of documents does not fit in any context window on the market; input cost scales with tokens, so pushing an entire knowledge base through the model on every question is priced per question; and a corpus that changes hourly is cheaper to re-index than to re-send.
What changed by 2026 is the mix. The working pattern is retrieve-then-reason — pull a generous slice of relevant material rather than the tightest possible three chunks, then let a long-context model work across all of it. That relaxes the precision demanded of the retrieval step without removing it. Treat any vendor pitch built on “our context window is so large we don’t need retrieval” as a claim about corpus size they have not thought through.
Common pitfalls
Running the pilot on the vendor’s corpus. A demo tuned over months on a sample document set tells you how good their sample is.
Guard: Supply your own documents, and write 30-50 real questions before the first demo. Include 5 whose answers are deliberately absent from the corpus, and score refusal behavior on those as a pass/fail line item.
Reading a citation as verification. The citation is the thing that makes a misgrounded answer look reviewed.
Guard: During the pilot, open the cited passage on a fixed sample — 20 answers per week is enough — and track the misgrounded rate separately from the wrong-answer rate. A tool at 5% wrong and 15% misgrounded is worse for your team than one at 15% wrong and 2% misgrounded.
Assuming permissions carry over from the source systems. Connector-based indexing copies content; whether it copied the access control list with it is a separate question with a separate answer per connector.
Guard: During the pilot, run the same three sensitive queries as two users at different permission levels and diff the results. Do this per connector, not once for the product.
Letting the corpus rot after go-live. Retrieval quality decays as superseded policies, expired contracts, and old playbooks accumulate, because the search step has no way to know which version is current.
Guard: Name an owner for the corpus before launch and give them a deletion mandate, not just an upload one. Re-test quarterly with a document edited that morning and a document that should have been removed.
Related
- MCP server explained — the other way an AI tool reaches your systems: calling them live rather than searching a copied index
- AI agent for ops — where retrieval sits inside an agent that also takes actions, and the autonomy tests that go with it
- AI agent vs RPA — the adjacent buying decision, and why agent failures are invisible to uptime monitoring
- Glean and Hebbia — horizontal enterprise search versus structured extraction across a large corpus, two different answers to the same retrieval problem
- Legal knowledge management — the corpus-ownership discipline that decides whether any of this works in a legal team