Your assistant answers a refund question with last year's policy. You try a larger model, rewrite the prompt, and add a warning to use the supplied context. The answer is still wrong.
The current policy was never retrieved. This guide shows how to measure RAG retrieval quality, step by step.
That is a retrieval failure, not evidence that the language model needs another instruction. Before changing the LLM in a retrieval-augmented generation system, inspect whether the right evidence reaches it, in the right order, with the right permissions and enough context to answer.
This guide gives engineering teams a practical way to evaluate that stage. It separates document coverage, retrieval ranking, context assembly, and answer quality so a poor final answer does not send the team toward the wrong fix.
Table of Contents
Separate the pipeline into observable stages
A typical RAG pipeline ingests source material, extracts text, creates chunks, adds metadata, indexes content, retrieves candidates, optionally reranks them, and passes selected context to a generator.
Microsoft's RAG architecture guide treats these as distinct design and evaluation decisions. That distinction matters because a correct document can disappear at several points.
It might never be ingested. Its table might be parsed incorrectly. The relevant sentence might be separated from its heading. A filter might exclude it. A reranker might push it down. Context truncation might remove it after retrieval succeeded.
Log enough information to tell those cases apart. For each test query, record the corpus snapshot, index and model versions, filters, retrieved IDs and ranks, reranked results, and the exact context sent to the generator. Keep these logs under the same access controls as the underlying content.
1. Build an evaluation set from real information needs
Start with questions that represent the product's actual work. Do not rely only on questions generated from documents: they often reuse the source wording and make retrieval easier than a real user's request.
Include exact identifiers, paraphrases, acronyms, recent policy changes, multi-document questions, ambiguous requests, and questions with no answer in the corpus. Add permission-sensitive cases under test identities rather than exposing restricted material to unauthorised users.
For each answerable question, record the authoritative source and the evidence required to answer it. Some questions need one clause. Others need two documents. Preserve that distinction in the labels.
A useful record contains a query ID, question, user role, expected evidence IDs, relevance labels, and notes about date or authority. A reference answer can help, but it does not replace evidence labels when you need to diagnose document retrieval.
Keep a development set for tuning and a separate holdout for the final comparison. Otherwise, repeated prompt and retrieval changes can overfit the same familiar questions.
2. Verify corpus coverage before measuring ranking
Ask whether the answer exists in the searchable corpus at all. An index cannot retrieve a policy that was never ingested.
Trace missing evidence back to its source. Check ingestion success, parsing, chunk creation, metadata, version dates, and access rules. Confirm that the current document replaced or clearly superseded the old one where the workflow requires it.
For the refund example, inspect the exact policy clause in the index. If a PDF parser omitted the table containing the exception, changing embeddings will not restore the missing text.
Track coverage failures separately from ranking failures. Otherwise, a retriever can be penalised for evidence absent from its input, while an ingestion bug remains hidden behind an average relevance score.
3. Pick RAG retrieval quality metrics that answer different questions
Classical retrieval metrics require relevance judgments. Define the unit first: document, chunk, or evidence item. Do not compare a document-level baseline with a chunk-level experiment and call the difference an improvement.
| Metric | What it helps answer | Main limitation |
|---|---|---|
| Recall@k | Did the top k include the known relevant items? | Incomplete relevance labels make the result provisional |
| Precision@k | How much of the returned set was relevant? | Does not tell you whether omitted evidence was essential |
| MRR@k | How early did the first relevant result appear? | Can hide missing evidence for multi-source questions |
| nDCG@k | Were more useful results ranked nearer the top? | Requires a consistent graded-relevance rubric |
| Evidence completeness | Did context contain every required fact? | Usually needs task-specific labels or review |
Use more than one metric. MRR can look excellent when the first useful chunk is ranked first but the second required policy is missing. Precision can rise when a system returns very little, while recall falls.
The BEIR research benchmark evaluates retrieval across varied tasks and domains. Its broader lesson for a product team is to test beyond one easy query style. Public benchmark performance is not a substitute for evaluation on your own corpus.
4. Work through a small example
Suppose a question requires two labelled evidence chunks: A and B. The retriever returns five distinct chunks in this order: A, X, Y, Z, B. Only A and B are relevant.
Recall@5 is 2/2, or 1. Precision@5 is 2/5, or 0.4. The reciprocal rank of the first relevant result is 1, because A appears first.
Now imagine the context builder keeps only the first four chunks. The retriever found both pieces of evidence, but the generator sees only one. A retrieval-only dashboard could report success while the product still fails.
This is an illustrative calculation, not a benchmark result. It shows why you should evaluate both the candidate list and the final context. If the answer requires a policy and an exception, retrieving only the policy is not enough.
5. Use LLM-based evaluation without confusing it with ground truth
When relevance labels are scarce, an LLM judge can estimate whether retrieved context helps answer a query. It can accelerate triage, but its score depends on the prompt, model, and evidence it sees.
Microsoft distinguishes document retrieval with relevance labels from LLM-assisted retrieval evaluation without those labels. They answer related questions using different evidence. Do not merge their scores into a single unnamed accuracy number.
Ragas context precision evaluates whether useful chunks appear ahead of irrelevant ones. Its context recall documentation describes reference-based measurement, including whether claims in a reference answer are supported by retrieved context. That claim-based measure is not identical to counting retrieved document IDs.
Name the exact metric variant and evaluator version in reports. Calibrate judge decisions against a human-reviewed sample, especially for contradictions, technical language, and unanswerable questions. A confident judge is still another model output to assess.
6. Compare simple baselines before adding complexity
Keep the corpus and evaluation set fixed while testing retrieval changes. A useful sequence is lexical search, dense retrieval, a hybrid combination, and then reranking where appropriate.
Lexical retrieval can be a useful baseline for exact codes and uncommon identifiers. Dense retrieval can help with paraphrases. Neither wins by definition. BEIR's published results show why lexical baselines should not be dismissed merely because a neural method is newer.
Change one major variable at a time: chunking, metadata filters, embedding model, candidate count, or reranker. Record quality, latency, and cost together. A reranker that improves relevance but exceeds the product's response budget may not be a practical improvement.
The BEIR implementation provides a reference framework for retrieval evaluation. Use its concepts or tooling where appropriate, but preserve your application's relevance definitions and permission constraints.
7. Inspect chunking and context assembly separately
Look at the text the model receives, not only the result IDs. Check whether headings, table headers, dates, and qualifying sentences survive extraction and chunking.
A chunk containing a price without its billing interval is incomplete evidence. A policy clause without the section specifying eligible customers can support the wrong answer. Larger chunks may restore context, but they can also dilute relevance and consume more tokens.
Inspect duplicate chunks, repeated boilerplate, and near-identical document versions. They can fill the context window without increasing evidence coverage. Deduplication should not remove genuinely different exceptions merely because the wording is similar.
After reranking and truncation, recompute evidence completeness on the exact selected context. Treat context assembly as a tested component, not a formatting step assumed to be harmless.
8. Test permissions, freshness, and abstention
A system can retrieve relevant evidence and still be unsafe. Test whether each identity can retrieve only authorised content. Include access removal and changed group membership, not only initial access.
Test questions whose correct answer changed recently. The expected result should point to the current authoritative version, with old material handled according to the application's source rules.
For unanswerable questions, check whether the system recognises insufficient evidence rather than forcing a plausible answer. A relevance metric on answerable queries does not establish safe abstention.
These are also buying criteria for enterprise AI search platforms, but the evaluation here goes further: verify behavior against test identities and known evidence rather than accepting feature descriptions alone.
9. Match the failure to the next experiment
| Observed failure | Investigate first |
|---|---|
| Required source absent from the index | Ingestion, parsing, connector coverage |
| Source indexed but never retrieved | Query handling, filters, embeddings, lexical baseline |
| Relevant evidence retrieved too low | Candidate recall, reranking, duplicate results |
| Evidence found but omitted from prompt | Context budget and assembly rules |
| Complete correct evidence reaches model, answer still wrong | Generation prompt, model behavior, answer evaluation |
| Restricted evidence reaches an unauthorised user | Permission enforcement and identity propagation |
This table is a diagnostic starting point, not proof of a single cause. Inspect failed examples before changing a component. Several failures can coexist in one query.
10. Make evaluation a release check
Record the baseline configuration and preserve failed examples. After each change, compare aggregate scores and slices such as exact identifiers, recent documents, languages, and permission-sensitive requests.
Do not accept a higher average if a high-risk slice regresses. Set release thresholds from the product's consequences and latency budget, rather than copying a universal score from a tutorial. Use a regression set and a holdout set for different purposes.
Keep answer quality evaluation separate but connected. Groundedness asks whether a response stays within its evidence; completeness asks whether it includes what the question requires. A grounded answer can still be incomplete, and complete-looking text can still be unsupported.
Fix the evidence path before changing the model
A larger LLM cannot reliably recover a document it never receives. First prove that the corpus contains the answer, the retriever surfaces it, and context assembly preserves it under the user's permissions.
Then evaluate generation against that known-good evidence. The point of retrieval measurement is not another dashboard. It is a shorter path from a wrong answer to the component that actually needs fixing.



