RAG, part 2: advanced methods and evaluation
The basic RAG pipeline fails on vague questions, questions that need several documents, and questions about a whole collection. This session covers the fixes (query rewriting, HyDE, multi-query with rank fusion, multi-hop retrieval, GraphRAG, Self-RAG and CRAG) and then shows how to measure retrieval and generation separately, so you know which part to fix.
What you will learn
- Why vocabulary mismatch hurts retrieval, and how query rewriting, HyDE and multi-query with reciprocal rank fusion fix it
- How to answer multi-hop questions by interleaving retrieval and reasoning (IRCoT)
- How GraphRAG builds an entity graph, finds communities and answers global questions by map-reduce over community summaries
- How Self-RAG’s reflection tokens and CRAG’s retrieval evaluator decide when to retrieve and what to trust
- How to compute recall@k, precision@k, MRR and nDCG for the retriever
- How to measure faithfulness, answer relevance and context relevance for the generator, and why faithfulness is not correctness
- How to diagnose the four main failure modes: missing content, wrong chunk, right chunk ignored, conflicting sources
Why this matters
Session 152 built the basic pipeline: split documents into chunks, embed them, search, rerank, put the best chunks in the prompt, and ask the model to answer with citations. That pipeline works for easy questions. Real users ask hard ones: vague questions, questions that use other words than the documents, questions that need two or three facts from different places, and questions about the whole collection (“what are the main themes in these 500 reports?”). This session covers the methods built for those cases, and then the part most teams skip: measuring whether the system works.
The main lesson of this session is simple. A RAG system has two parts that fail in different ways: the retriever (did the right chunk reach the prompt?) and the generator (did the model use it correctly?). If you only look at the final answer, you cannot tell which part to fix. So we measure them separately: retrieval with recall@k, MRR and nDCG; generation with faithfulness and answer relevance. Then we look at the common failure modes one by one.
The plan: first, change the query (rewriting, HyDE, multi-query with rank fusion). Second, retrieve in several steps for multi-hop questions. Third, change the index (GraphRAG). Fourth, let the model decide and criticize its own retrieval (Self-RAG, and the related CRAG). Fifth, evaluate retrieval and generation. Last, a field guide to failure modes: wrong chunk, right chunk ignored, conflicting sources, and missing content.
Changing the query: rewriting, HyDE and multi-query
The retriever compares the query with the chunks. Users write short, vague, conversational queries (“it won’t turn on after the update”); documents use formal words (“device fails to boot after a firmware upgrade”). This gap between the words of the question and the words of the answer is called vocabulary mismatch. Dense embeddings (session 150) reduce it but do not remove it, and keyword search such as BM25 suffers from it fully. The fix is cheap: before searching, use an LLM to change the query.
Query rewriting
Query rewriting asks an LLM to turn the user’s message into a better search query. In a chat, it also makes the query standalone: “and what about the Pro plan?” becomes “refund window for the Pro plan”, using the conversation history \(h\):
Here \(P_{\text{rewrite}}\) is the instruction (“rewrite the last user message as a standalone search query”), \(h\) the chat history and \(q\) the last message. A related trick is step-back prompting (Zheng et al., 2023): ask a more general question first (“what are the refund rules?”) and retrieve for both. [Certain]
HyDE: search with a fake answer
HyDE (Hypothetical Document Embeddings, Gao et al., 2022) goes further. Instead of searching with the question, ask an LLM to write a passage that answers it, then embed that passage and search with it. The passage may contain wrong facts, but it is written in the style and vocabulary of a real answer, so it lands near real answers in embedding space. The paper samples several fake documents \(\hat{d}_1, \dots, \hat{d}_N\) and also keeps the query itself as one more “hypothesis”: [Certain]
- Read the symbols
- \(\hat{d}_k\)
- the \(k\)-th hypothetical document, written by an instruction-following LLM (“write a passage that answers the question”)
- \(f\)
- the document encoder (Contriever in the paper); it maps text to a vector
- \(q\)
- the original question
- \(\hat{\mathbf{v}}\)
- the final search vector; the real documents with the highest inner product with it are returned
The encoder is the important filter: it keeps the general meaning and drops much of the made-up detail, and the search only returns real documents, so the fake facts never reach the answer directly. In the paper, HyDE with the unsupervised Contriever encoder raised nDCG@10 on TREC DL19 from 44.5 to 61.3, above BM25 (50.6) and close to a Contriever fine-tuned on MS MARCO labels (62.1). [Certain] The cost is one LLM call before every search, which adds latency. Query2doc (Wang, Yang and Wei, 2023) is a close cousin: it appends the generated passage to the query for BM25, and reports BM25 gains of 3% to 15% on MS MARCO and TREC DL. [Certain]
Multi-query and reciprocal rank fusion
Multi-query retrieval asks the LLM for several different phrasings of the question (for example three), runs a search for each, and merges the ranked lists. Each phrasing finds some relevant chunks; together they cover more. The standard way to merge ranked lists is reciprocal rank fusion (RRF) (Cormack, Clarke and Büttcher, 2009). It uses only the ranks, not the scores, so it can merge lists whose scores are not comparable (for example BM25 scores and cosine similarities, as in hybrid search in Part 1):
- Read the symbols
- \(m\)
- the number of ranked lists (one per sub-query, or one per retriever)
- \(\mathrm{rank}_i(d)\)
- the position of document \(d\) in list \(i\), starting at 1; if \(d\) is not in list \(i\), that term is 0
- \(k\)
- a constant that limits how much the very top ranks dominate; the paper uses \(k = 60\) [Certain]
With \(k = 60\), rank 1 gives \(1/61 = 0.0164\) and rank 2 gives \(1/62 = 0.0161\): almost the same. So RRF mostly rewards documents that appear in many lists, and only a little those that are first in one. Figure 1 compares the four strategies on a small IT help-desk collection.
How to read Figure 1
With the plain query, D1 has score 0: it shares no word with the question (“boot”, “firmware upgrade” vs “turn on”, “update”), so it is never found, and recall@3 is 0.5. Each change of the query fixes this in a different way. The rewrite adds “fails to boot”, the words of D1; HyDE’s fake answer talks about the power button and safe mode, which are in D1 and D2; multi-query runs three searches (shown with their top lists) and fuses them with RRF. All three reach \(\mathrm{Recall@}3 = 1.0\). Only multi-query puts a relevant chunk first (reciprocal rank 1.0). Its top two are an exact tie (0.0325 each), broken by document number. Notice that D3, the near miss, stays in the top 3 almost everywhere: query changes raise recall, and the reranker from Part 1 is still needed for precision.
The three sub-queries return these top lists (best first): plain query [D3, D2, D7, D5]; “computer does not boot after a firmware upgrade” [D1, D8]; “black screen after a system update” [D2, D3, D6]. With \(k = 60\):
- D2: rank 2 in list 1 and rank 1 in list 3: \(1/62 + 1/61 = 0.0325\)
- D3: rank 1 in list 1 and rank 2 in list 3: \(1/61 + 1/62 = 0.0325\)
- D1: rank 1 in list 2 only: \(1/61 = 0.0164\); D8: \(1/62 = 0.0161\); D6 and D7: \(1/63 = 0.0159\)
D2 and D3 tie because each is first in one list and second in another. D1, the best answer, is only third, because just one sub-query found it. RRF is a vote: being found by several searches counts more than being first in one.
- input: user message \(q\), chat history \(h\), retriever Search(text, top), number of sub-queries \(m\), keep \(k\)
- \(\hat{q} \leftarrow\) LLM(“rewrite as a standalone search query”, \(h\), \(q\)) // formula (153.1)
- if HyDE: \(\hat{d} \leftarrow\) LLM(“write a short passage that answers:”, \(\hat{q}\)); \(\mathbf{v} \leftarrow (f(\hat{d}) + f(\hat{q}))/2\); return Search(\(\mathbf{v}\), \(k\)) // formula (153.2), \(N = 1\)
- subs \(\leftarrow [\hat{q}]\) + LLM(“give \(m-1\) different phrasings of”, \(\hat{q}\)) // multi-query
- lists \(\leftarrow\) [Search(\(s\), top = 20) for \(s\) in subs] // can run in parallel
- score\([d] \leftarrow \sum_i 1/(60 + \text{rank}_i(d))\) over the lists that contain \(d\) // formula (153.3)
- return the \(k\) documents with the highest score, then rerank them with a cross-encoder // precision step from Part 1
Multi-hop questions: retrieve, reason, retrieve again
Some questions need facts from several documents that do not mention each other: “What is the population of the town where the inventor of the Fenwick lamp was born?” One document says who invented the lamp; a second says where that person was born; a third gives the town’s population. This is a multi-hop question: each hop needs the answer of the previous one to know what to search for. Benchmarks such as HotpotQA (Yang et al., 2018), 2WikiMultihopQA and MuSiQue are built from such questions. [Certain]
One search with the whole question fails, for a clear reason: the second and third documents contain words (the inventor’s name, the town’s name) that are not in the question. They can only be found after reading the first document. There are two families of fixes:
- Decompose first: ask the LLM to split the question into sub-questions (“who invented the Fenwick lamp?”, “where was that person born?”, “what is that town’s population?”), then answer them in order, filling the blanks.
- Interleave retrieval and reasoning: retrieve, write one reasoning sentence, use that sentence as the next query, and repeat. IRCoT (Trivedi et al., 2022) does exactly this with chain-of-thought sentences; with GPT-3 it improved retrieval by up to 21 points and QA by up to 15 points on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC. [Certain]
Written as a loop, with \(c_t\) the reasoning sentence written at step \(t\):
- Read the symbols
- \(q\)
- the original question; \(q_1 = q\)
- \(\mathcal{D}_t\)
- the passages retrieved at step \(t\) (new ones only)
- \(c_t\)
- the reasoning sentence written at step \(t\); it becomes the next query
The loop stops when the model writes “so the answer is …” or after a maximum number of steps. Figure 2 runs it on the lamp question.
How to read Figure 2
Step 1: the single search keeps P1 (0.369), P4 and P6 (0.272 each) and P2 (0.258). P3, which holds the answer, is fifth (0.154). Recall is 2/3, and the context contains the trap “Harlow grew to 90,000”: a model forced to answer may say 90,000. Steps 2 to 5: the first search keeps only 2 passages; the model writes “The Fenwick lamp was invented by Mara Ilves”, which finds P2; then “Mara Ilves was born in Tolby”, which finds P3. Recall reaches 3/3 with the same 4 passages. The price is three retrieval calls and three LLM calls instead of one of each.
- input: question \(q\), retriever TopK, passages per step \(k\), maximum steps \(T\)
- \(\mathcal{D} \leftarrow\) TopK(\(q\), \(k\)); reasoning \(\leftarrow\) [ ]
- for \(t = 1 .. T\):
- \(c \leftarrow\) LLM(“write the next reasoning sentence”, \(q\), \(\mathcal{D}\), reasoning) // formula (153.4)
- append \(c\) to reasoning
- if \(c\) contains “the answer is”: break
- \(\mathcal{D} \leftarrow \mathcal{D} \,\cup\) TopK(\(c\), \(k\), excluding \(\mathcal{D}\)) // the sentence names the next entity
- return LLM(“answer using only these passages, cite them”, \(q\), \(\mathcal{D}\), reasoning)
Multi-hop retrieval is a small step toward an agent: a model that decides, step by step, what to look up next (session 154). Its risks are the same: errors early in the chain carry forward (if hop 1 picks the wrong inventor, every later hop is wrong), and the number of steps must be capped.
GraphRAG: index the relations, not only the chunks
Chunk retrieval answers local questions: the answer sits in one or a few chunks. It fails at global questions about the whole collection: “What are the main themes in these podcast transcripts?” No single chunk contains the answer, and top-\(k\) search returns \(k\) random-looking pieces. GraphRAG (Edge et al., 2024, Microsoft) builds a different index for such questions: a knowledge graph of the entities in the text and their relations, grouped into communities, each with a summary written in advance. [Certain]
The indexing pipeline in the paper:
- Split the documents into chunks (600 tokens with 100 tokens of overlap in the paper’s experiments). [Certain]
- For each chunk, an LLM extracts entities (people, places, organizations, concepts), relationships between them and short descriptions. A “gleaning” step asks the model again for entities it missed. [Certain]
- Merge the extractions into one graph: entities are nodes; a relationship found many times becomes an edge with a larger weight.
- Split the graph into communities, groups of nodes that are densely connected to each other and loosely to the rest, with the Leiden algorithm, applied again inside each community to make a hierarchy (levels C0, the root, down to C3). [Certain]
- Ask an LLM to write a report-style summary of each community, from the bottom level up.
Leiden looks for a split that makes the modularity high: more edge weight inside communities than a random graph with the same node degrees would have. [Likely] (Leiden can also optimize other quality functions; GraphRAG’s default implementation uses modularity as far as we know.)
- Read the symbols
- \(A_{ij}\)
- edge weight between entities \(i\) and \(j\) (how often the relation was extracted); 0 if no edge
- \(w_i\)
- the total edge weight of node \(i\) (its weighted degree); \(W\) is the total edge weight of the graph
- \(w_i w_j / 2W\)
- the expected weight between \(i\) and \(j\) if edges were placed at random with the same degrees
- \(c_i\)
- the community of node \(i\); \(\mathbb{I}[\cdot]\) is 1 when the two nodes are in the same community
Two ways to answer:
- Global search uses map-reduce, the same idea as long-document summarization (session 147). Shuffle the community summaries of one level into context-sized groups; map: each group produces a partial answer and a helpfulness score from 0 to 100; drop the zeros; reduce: combine the best partial answers, in score order, into one answer. [Certain]
- Local search starts from the entities mentioned in the question, and collects their relations, neighbours, community summaries and source chunks into the context. [Likely] (This mode is described in the GraphRAG software, not in the main experiments of the paper.)
In the paper, on about 1 million tokens of podcast transcripts and 1.7 million tokens of news, GPT-4 judges preferred GraphRAG answers over plain vector RAG for comprehensiveness 72% to 83% of the time on podcasts and 72% to 80% on news, and for diversity 75% to 82% and 62% to 71%. [Certain] Answering from the root level C0 used about 26,657 context tokens on the podcast data, against 1,014,611 for map-reduce over the raw text: about 38 times fewer. [Certain] Note what was not claimed: these are win rates judged by an LLM on broad “sensemaking” questions, not accuracy on fact questions. For “what is the refund window?”, plain chunk retrieval is cheaper and at least as good. [Likely]
The cost is at indexing time: one or more LLM calls per chunk for extraction, plus one per community summary, and the whole graph must be rebuilt or patched when documents change. RAPTOR (Sarthi et al., 2024) is a lighter relative: it clusters chunk embeddings, summarizes each cluster, and repeats, building a tree of summaries that is searched together with the chunks; with GPT-4 it improved the best reported accuracy on the QuALITY benchmark by 20 points (absolute). [Certain]
- index: chunks \(\leftarrow\) split(docs, 600 tokens, overlap 100)
- for each chunk: (entities, relations) \(\leftarrow\) LLM(“extract entities and relations”, chunk); glean again for missed ones
- \(G \leftarrow\) merge all extractions; edge weight = number of times a relation was found
- communities \(\leftarrow\) hierarchical Leiden(\(G\)) // formula (153.5)
- for each community, bottom level first: summary \(\leftarrow\) LLM(“write a report on this community”, its entities, relations, child summaries)
- global query \(q\): groups \(\leftarrow\) pack shuffled summaries of one level into context-sized groups
- partial \(\leftarrow\) [LLM(“answer \(q\) from these summaries; rate helpfulness 0-100”, \(g\)) for \(g\) in groups] // map, parallel
- return LLM(“combine into one answer”, partial answers with score \(\gt 0\), best first) // reduce
Self-RAG and CRAG: let the model judge its retrieval
The basic pipeline always retrieves, and always trusts what it retrieved. Both are wrong sometimes. “Write a poem about autumn” needs no retrieval; extra passages only distract. And when the retrieved passages are off-topic, the model should notice and not use them. Two methods add these decisions.
Self-RAG
Self-RAG (Asai et al., 2023) trains one model to generate special reflection tokens alongside its answer. There are four kinds: [Certain]
| Token | Question it answers | Possible values |
|---|---|---|
| Retrieve | Should I retrieve now? | yes, no, continue (keep using the current passage) |
| IsRel | Is this passage relevant to the question? | relevant, irrelevant |
| IsSup | Is my segment supported by the passage? | fully supported, partially supported, no support |
| IsUse | Is my answer useful overall? | 5, 4, 3, 2, 1 |
How it is trained: GPT-4 labeled a few thousand examples per token type (4k to 20k each); a Llama 2 7B critic model learned to predict the tokens from those labels; the critic then inserted reflection tokens offline into 150k instruction–answer pairs, and the generator (Llama 2 7B or 13B) was trained on that data with the ordinary next-token loss. [Certain] At inference the generator itself emits the tokens; no separate critic is needed.
At inference, the model first predicts Retrieve. It retrieves only if the normalized probability of “yes” is above a threshold \(\delta\) (0.2 for most tasks in the paper): [Certain]
If it retrieves, it processes several passages in parallel, writes one candidate segment per passage, and ranks the candidates by their own probability plus a weighted sum of critique scores:
- Read the symbols
- \(y_t\)
- the candidate segment (for example one sentence) at step \(t\); \(x\) is the input, \(d\) the passage used
- \(\hat{r}^{G}\)
- the most desirable value of token type \(G\) (relevant; fully supported; 5)
- \(s_t^{G}\)
- the probability of that best value, divided by the sum over all \(N^G\) values of type \(G\): a number from 0 to 1
- \(w^{G}\)
- weights set at inference time, without retraining; the paper’s defaults are 1.0 for IsRel, 1.0 for IsSup and 0.5 for IsUse [Certain]
The weights are a dial between “fluent” and “grounded”. Raise \(w^{\text{IsSup}}\) and the model prefers segments the passage really supports, even if they read less smoothly. The paper keeps the best 2 candidates at each segment step (beam width 2). [Certain]
How to read Figure 4
With the paper’s default weights (1, 1, 0.5), d1 wins with 2.250, ahead of d3 (1.680) and d2 (1.350). Set all three weights to 0: only the segment probability counts, and d2, the most fluent text, wins (0.55), although its passage is irrelevant. Put weight only on IsUse (for example 1.0, others 0) and d2 wins again, with 1.150: “useful-sounding” is not “supported”. Raise \(\delta\) above 0.62 and the model does not retrieve at all; it answers from its own memory, with no citation.
d1 has segment probability 0.40; IsRel probabilities (relevant 0.9, irrelevant 0.1); IsSup (fully 0.7, partially 0.2, none 0.1); IsUse (5: 0.5, 4: 0.3, 3: 0.1, 2: 0.05, 1: 0.05). Each list sums to 1, so \(s^{\text{IsRel}} = 0.9\), \(s^{\text{IsSup}} = 0.7\), \(s^{\text{IsUse}} = 0.5\). With weights (1, 1, 0.5):
\(f = 0.40 + 1.0 \times 0.9 + 1.0 \times 0.7 + 0.5 \times 0.5 = 2.25\).
d2 (marketing page): \(0.55 + 0.3 + 0.2 + 0.5 \times 0.6 = 1.35\). d3 (old forum post): \(0.48 + 0.8 + 0.3 + 0.5 \times 0.2 = 1.68\). The critique terms, not the fluency term, decide the winner.
CRAG: correct the retrieval before generating
Corrective RAG (CRAG) (Yan et al., 2024) needs no change to the generator. A small retrieval evaluator (a fine-tuned T5-large, about 0.77B parameters) scores each retrieved document for the question, and two thresholds pick one of three actions: [Certain]
- Correct (at least one document scores above the upper threshold): keep the documents, but split them into small “knowledge strips”, drop irrelevant strips and recombine the rest.
- Incorrect (all documents below the lower threshold): discard them; rewrite the question into keywords and run a web search instead.
- Ambiguous (anything else): use both the refined documents and the web results.
Self-RAG and CRAG share one idea: retrieval quality should be checked before the answer depends on it. In a production system, the same idea is often a cheap rule: if the best reranker score is below a threshold, say “I could not find this in the documents” instead of answering. [Likely]
Evaluating retrieval: recall@k, MRR and nDCG
Now the second half of the session. To evaluate the retriever, you need, for each test question, the set of chunks that contain the answer (the relevant set). You can label them by hand, or generate questions from chunks with an LLM (then the source chunk is relevant by construction; this is cheap but makes questions that are easier than real ones). [Likely] Given the ranked list the retriever returns, the standard metrics are:
- Read the symbols
- \(\mathrm{Rel}\)
- the set of relevant chunks for the question; \(\mathrm{Top}_k\) the first \(k\) chunks returned
- \(\mathrm{rank}_q\)
- the position of the first relevant chunk for question \(q\); \(1/\mathrm{rank}_q = 0\) if none is returned. \(1/\mathrm{rank}_q\) for one question is the reciprocal rank (RR)
- \(g_i\)
- the relevance grade of the chunk at position \(i\) (0 or 1 for binary labels, so \(2^{g_i} - 1\) is also 0 or 1)
- \(\mathrm{IDCG@}k\)
- the DCG of the ideal ranking (all relevant chunks first); it makes nDCG lie between 0 and 1
Which one matters for RAG? Recall@k is the most important: the generator can only use what is in the prompt, so a relevant chunk at rank 7 is lost if you keep 5. Precision@k matters less, but low precision fills the prompt with distractors. MRR and nDCG reward putting relevant chunks early, which matters because models use the start of the context better than the middle (“lost in the middle”, Liu et al., 2023). [Certain] A useful habit: plot \(\mathrm{Recall@}k\) against \(k\). Where the curve flattens is how many chunks to keep. Figure 5 lets you build a ranked list and see the four numbers.
How to read Figure 5
The default has relevant results at ranks 2, 5 and 9, and 4 relevant chunks in the corpus (one was never returned). At \(k = 5\): recall 0.5, precision 0.4, RR 0.5, nDCG 0.397. Move \(k\) to 10: recall rises to 0.75 but precision falls to 0.3. Now click rank 1 to make it relevant: RR jumps to 1 and nDCG rises, but recall@5 rises only by 0.25. Raise \(\lvert\mathrm{Rel}\rvert\) to 8: recall drops, because recall is measured against everything relevant, not only what was found.
\(\mathrm{DCG@}5 = 1/\log_2 3 + 1/\log_2 6 = 0.631 + 0.387 = 1.018\). With 4 relevant chunks in the corpus, the ideal list puts them at ranks 1 to 4: \(\mathrm{IDCG@}5 = 1 + 0.631 + 0.5 + 0.431 = 2.562\). So \(\mathrm{nDCG@}5 = 1.018 / 2.562 = 0.397\). \(\mathrm{Recall@}5 = 2/4 = 0.5\), \(\mathrm{Precision@}5 = 2/5 = 0.4\), \(\mathrm{RR} = 1/2\). Over three test questions with first relevant ranks 2, 1 and 4, \(\mathrm{MRR} = (1/2 + 1 + 1/4)/3 = 0.583\).
Evaluating generation: faithfulness and answer relevance
Good retrieval does not guarantee a good answer. For the generator we ask two questions, and they are different:
- Faithfulness (also called groundedness): is every claim in the answer supported by the retrieved context? This is the RAG version of the summary faithfulness check in session 147.
- Answer relevance: does the answer address the question? A faithful answer can still dodge the question.
Neither needs a human-written reference answer, which is why they are popular. RAGAS (Es et al., 2023) defines them with LLM calls: [Certain]
where an LLM splits the answer into short statements \(S\) and then checks each against the context; \(V\) is the set of statements it judges supported. For answer relevance, the LLM writes \(n\) questions that the answer would answer, and we measure how close they are to the real question:
with \(\mathbf{e}\) an embedding model. An answer that drifts off topic produces generated questions unlike the original, so AR drops. RAGAS also scores the context:
- Read the symbols
- \(S, V\)
- the statements extracted from the answer, and those of them supported by the context
- \(q_i\)
- the \(i\)-th question generated from the answer; \(q\) is the user’s question
- \(\mathrm{CR}\)
- context relevance: a precision of the context, measured by the LLM picking the sentences it needs
On their WikiEval data (50 Wikipedia pages), RAGAS agreed with human pairwise judgments 95% of the time for faithfulness, 78% for answer relevance and 70% for context relevance. [Certain] So faithfulness is the most reliable of the three automatic scores, and context relevance the least. Later versions of the RAGAS library added reference-based metrics such as context recall, which do need a reference answer. [Likely]
Two more points matter in practice. First, faithfulness is not correctness: if the retriever returned an outdated policy, a faithful answer repeats the outdated policy. To measure correctness you need reference answers for at least a test set. Second, LLM judges make mistakes. ARES (Saad-Falcon et al., 2023) trains small judges on synthetic data and uses a few hundred human labels with prediction-powered inference to correct the judges’ bias and give confidence intervals. [Certain] Whatever you use, check the judge against a small human-labeled set first (session 155).
| What you measure | Metric | Needs labels? | Tells you to fix |
|---|---|---|---|
| Did the right chunks come back? | \(\mathrm{Recall@}k\), \(\mathrm{hit@}k\) | Relevant chunk ids per question | Chunking, embeddings, hybrid search, query rewriting |
| Were they near the top? | MRR, \(\mathrm{nDCG@}k\) | Same (graded labels for nDCG) | Reranker, fusion weights |
| Is the context focused? | \(\mathrm{Precision@}k\), context relevance | No (LLM judge) or yes | \(k\), reranker threshold, chunk size |
| Is the answer grounded? | Faithfulness | No (LLM or NLI judge) | Prompt, generator, citations, context order |
| Does it answer the question? | Answer relevance | No | Prompt, refusal behavior |
| Is it right? | Answer correctness, exact match, LLM judge vs reference | Reference answers | Everything, including stale or conflicting data |
- input: test set of (question \(q\), relevant chunk ids \(\mathrm{Rel}\), reference answer \(a^\ast\) if available), RAG system
- for each test item:
- ranked \(\leftarrow\) retriever(\(q\)); record \(\mathrm{Recall@}k\), \(\mathrm{RR}\), \(\mathrm{nDCG@}k\) against \(\mathrm{Rel}\) // formulas (153.8)-(153.10)
- answer \(\leftarrow\) generator(\(q\), ranked[:\(k\)])
- statements \(\leftarrow\) LLM(“split into short statements”, answer); verdicts \(\leftarrow\) judge(statements, context) // faithfulness, (153.11)
- record answer relevance (153.12) and, if \(a^\ast\) exists, correctness
- report every metric as a mean with a confidence interval, and slice by question type // multi-hop, recent docs, ...
- read 20 to 50 failures by hand and label each with a failure mode (next section)
Failure modes: a field guide
Barnett et al. (2024) collected failure points from three real RAG projects (research, education and biomedical): missing content, missed top-ranked documents, relevant documents dropped during context assembly, answer not extracted from the context, wrong format, wrong level of detail, and incomplete answers. [Certain] For debugging, they reduce to four questions asked in order:
- Is the answer in the corpus at all? If not: missing content. The right behavior is to say so. Fix: an “I don’t know” instruction and a score threshold; add the content.
- Did the right chunk reach the prompt? If not (context recall below 1): wrong chunk, a retrieval failure. Fix: query rewriting, hybrid search, better chunking, more candidates before reranking.
- Did the model use it? If the chunk is there but the answer is unsupported or contradicts it: right chunk ignored, a generation failure. Common causes: the chunk sat in the middle of a long context, too many distractors, or the model trusted its own memory. Fix: rerank so the best chunk is first, keep fewer chunks, ask for quotes before the answer.
- Is the answer faithful but still wrong? Then the sources disagree or are stale: conflicting sources. The model faithfully repeated the wrong one. Fix: store dates and versions as metadata, filter replaced documents, and tell the model to report conflicts and prefer the newest source.
Figure 6 runs this diagnosis on five cases.
How to read Figure 6
Compare “Wrong chunk” and “Conflicting sources”. Both give the wrong answer “14 days”. In the first, context recall is 0 and faithfulness is 0: the retriever failed, and the model also guessed. In the second, context recall is 1 and faithfulness is 1: the system faithfully repeated an outdated 2023 policy. Same wrong answer, opposite fixes. “Chunk ignored” has recall 1 but faithfulness 0.5: the gold chunk was third of five, and the model took 14 days from the Basic plan chunk. “Missing content” has no relevant chunk anywhere; the fix is to abstain, which turns correctness to 1 with no claims to check.
Putting it together in PyTorch
The code below runs with only PyTorch and no downloads. A unit-length word-count vector stands in for a dense encoder, so the numbers match Figure 1. It implements HyDE (formula 153.2), multi-query with RRF (153.3), recall@k, reciprocal rank and nDCG (153.8 to 153.10), the Self-RAG segment score (153.7) and faithfulness (153.11). It is deterministic.
import re, math, torch STOP = set("was were has had a an the of to in on for and or is are be it its my i me " "you your we if then than that this with by at as from after before when while " "can will won t not do does did should what which who how why where there their " "them they use used up out into so but every each".split()) def toks(s): return [w for w in re.findall(r"[a-z0-9]+", s.lower()) if w not in STOP] DOCS = ["If a device fails to boot after a firmware upgrade, hold the power button for 30 seconds to reset the controller, then start it again.", "A notebook that shows a black screen after a system update can be recovered by starting in safe mode and rolling back the update.", "To install a laptop update, open Settings and choose Software Update; the laptop restarts when the update is done.", "Batteries last about three years; replace the battery if the device turns off at 40 percent charge.", "Request a new laptop through the IT portal; delivery takes five working days.", "VPN errors after an update are usually fixed by reinstalling the VPN client.", "Printers that do not turn on should be checked for a loose power cable.", "The firmware upgrade calendar for servers is published every month by the infrastructure team."] RELEVANT = {0, 1} # D1 and D2 answer the question VOCAB = sorted({w for d in DOCS for w in toks(d)}) IDX = {w: i for i, w in enumerate(VOCAB)} def embed(text): """Stand-in for a dense encoder: unit-length word-count vector.""" v = torch.zeros(len(VOCAB)) for w in toks(text): if w in IDX: v[IDX[w]] += 1 return v / v.norm().clamp_min(1e-12) E_DOCS = torch.stack([embed(d) for d in DOCS]) # (8, |V|) def search(qvec, top=None): s = E_DOCS @ qvec order = sorted(range(len(DOCS)), key=lambda i: (-float(s[i]), i)) order = [i for i in order if s[i] > 0] # zero score = not retrieved return order[:top] if top else order def hyde_vector(query, fake_docs): """Formula (153.2): average the fake documents and the query.""" vs = [embed(d) for d in fake_docs] + [embed(query)] return torch.stack(vs).mean(0) def rrf(lists, k=60): """Formula (153.3): reciprocal rank fusion.""" score = {} for l in lists: for r, d in enumerate(l, start=1): score[d] = score.get(d, 0.0) + 1.0 / (k + r) return sorted(score, key=lambda d: (-score[d], d)) def recall_at_k(ranked, rel, k): return len(set(ranked[:k]) & rel) / len(rel) def rr(ranked, rel): return next((1 / (r + 1) for r, d in enumerate(ranked) if d in rel), 0.0) def ndcg_at_k(ranked, rel, k): dcg = sum(1 / math.log2(r + 2) for r, d in enumerate(ranked[:k]) if d in rel) idcg = sum(1 / math.log2(r + 2) for r in range(min(len(rel), k))) return dcg / idcg Q = "My laptop won't turn on after the update" FAKE = ["If your laptop fails to boot after an update, hold the power button for 30 seconds " "to reset it and start it again. If the screen stays black, start in safe mode and " "roll back the system update."] # what an LLM might write; may contain errors SUBQ = [Q, "computer does not boot after a firmware upgrade", "black screen after a system update"] runs = {"plain": search(embed(Q)), "HyDE": search(hyde_vector(Q, FAKE)), "multi-query+RRF": rrf([search(embed(q), top=4) for q in SUBQ])} for name, ranked in runs.items(): print(f"{name:16s} top3={[d + 1 for d in ranked[:3]]} " f"R@3={recall_at_k(ranked, RELEVANT, 3):.2f} RR={rr(ranked, RELEVANT):.2f} " f"nDCG@3={ndcg_at_k(ranked, RELEVANT, 3):.3f}") # Self-RAG segment score, formulas (153.6)-(153.7): choose among candidates cands = {"d1": (0.40, [0.9, 0.1], [0.7, 0.2, 0.1], [0.5, 0.3, 0.1, 0.05, 0.05]), "d2": (0.55, [0.3, 0.7], [0.2, 0.3, 0.5], [0.6, 0.3, 0.05, 0.03, 0.02]), "d3": (0.48, [0.8, 0.2], [0.3, 0.5, 0.2], [0.2, 0.4, 0.2, 0.1, 0.1])} w = torch.tensor([1.0, 1.0, 0.5]) # weights for IsRel, IsSup, IsUse for name, (p, rel, sup, use) in cands.items(): s = torch.tensor([t[0] / sum(t) for t in (rel, sup, use)]) # best value / sum print(name, round(p + float(w @ s), 3)) # faithfulness, formula (153.11): fraction of answer claims supported by the context verdicts = ["supported", "contradicted"] # from an NLI model or an LLM judge print("faithfulness", sum(v == "supported" for v in verdicts) / len(verdicts))
Output:
plain top3=[3, 2, 7] R@3=0.50 RR=0.50 nDCG@3=0.387 HyDE top3=[3, 2, 1] R@3=1.00 RR=0.50 nDCG@3=0.693 multi-query+RRF top3=[2, 3, 1] R@3=1.00 RR=1.00 nDCG@3=0.920 d1 2.25 d2 1.35 d3 1.68 faithfulness 0.5
With a real system, replace embed with a sentence-embedding model, the fake document and sub-queries with LLM calls, and the hand-written verdicts with an LLM or NLI judge. The metric code stays the same.
Where this is used
| Application | Advanced method that helps | What to measure |
|---|---|---|
| Chat assistants over company documents | Standalone query rewriting from chat history; abstain below a score threshold | Recall@k on real user questions; faithfulness; abstention rate on unanswerable questions |
| Customer support search | HyDE or multi-query to bridge user words and manual words | Hit@k, MRR; ticket deflection with human review |
| Research and due-diligence assistants | Multi-hop retrieval (IRCoT, decomposition); citations per claim | Recall of all needed passages; claim-level faithfulness |
| “What are the themes?” over large archives | GraphRAG or RAPTOR summaries; map-reduce answers | Pairwise LLM-judge win rates with human spot checks; cost per query |
| Policy, legal and medical QA | Version and date metadata; conflict reporting; CRAG-style checks | Correctness against references; conflict detection rate |
| Video-RAG and long-video QA | Multi-query over captions, speech transcripts and frame embeddings, fused with RRF; multi-hop across clips (“what did the speaker show after the demo?”) (session 210) | Recall@k of the right time segment; faithfulness to the transcript |
| Text-to-video retrieval and key-moment detection | HyDE-style expansion: write a caption of the moment you want, search clip embeddings with it (session 208) | Recall@k, MRR on moment ids; temporal overlap of returned segments |
| Multimodal document QA (slides, PDFs) | Page-image retrieval (for example ColPali) plus text chunks, merged with RRF | Recall@k of the right page; faithfulness to the page |
Common mistakes
- Judging the system only by the final answer. You cannot tell a retrieval failure from a generation failure. Always log what was retrieved, and measure recall@k separately.
- Treating faithfulness as correctness. A faithful answer to an outdated document is still wrong (Figure 6, conflicting sources). Keep a set of questions with reference answers.
- Building the test set from the same chunks the system indexes, with an LLM, and nothing else. Generated questions copy the chunk’s words, so retrieval looks better than it is with real users. Mix in real, messy questions. [Likely]
- Adding HyDE or multi-query without measuring latency and recall. Each adds at least one LLM call before search. Use them where the recall gain is real, often for short or vague queries only.
- Keeping too many chunks “to be safe”. Higher \(k\) raises recall but adds distractors and pushes the right chunk into the middle; measure answer quality at several \(k\).
- Using GraphRAG for simple fact lookup. Its indexing cost is high and it was built for global, sensemaking questions; plain chunk retrieval is usually better for “what is X?”.
- No path for “not found”. Without an abstain rule, every missing-content question becomes a hallucination.
- Trusting an LLM judge without checking it. Compare it with a few hundred human labels before using its numbers for decisions.
Interview questions
Your RAG system gives wrong answers. How do you debug it?core
What is HyDE, and why do the hallucinated details not hurt?core
Explain reciprocal rank fusion. Why use ranks instead of scores?basic
How would you answer a question that needs facts from two or three documents?core
When would you choose GraphRAG over normal chunk retrieval?advanced
What are Self-RAG’s reflection tokens and how are they used at inference?advanced
Define recall@k, MRR and nDCG. Which matters most for RAG?basic
How do you evaluate a RAG system without human-written reference answers?core
Faithfulness is 1.0 but users say the answer is wrong. What happened?core
The relevant chunk is in the prompt but the model ignores it. Why, and what do you do?advanced
How is CRAG different from Self-RAG?advanced
Exercises
- Three lists: A = [d1, d2, d3], B = [d3, d1], C = [d2, d4]. Compute RRF scores with \(k = 60\) and the fused order.
- A retriever returns 10 chunks; relevant ones are at ranks 1 and 6; the corpus has 3 relevant chunks. Compute recall@5, precision@5, RR and nDCG@5. Check with Figure 5.
- In Figure 4, which candidate wins with weights \(w^{\text{IsRel}} = 0\), \(w^{\text{IsSup}} = 1\), \(w^{\text{IsUse}} = 0\)? Give its score.
- An answer has 5 statements; 3 are supported by the context, 1 contradicts it, 1 is not in it. What is the faithfulness? Is the answer correct?
- HyDE with \(N = 4\) fake documents: what weight does the original query get in formula (153.2)? What happens to that weight as \(N\) grows?
- In Figure 2, why does the single search with the full question rank P3 fifth, and why does the hop query “Mara Ilves was born in Tolby” find it?
Answerscheck
Key takeaways
- Change the query before you change the index: rewriting (153.1), HyDE (153.2) and multi-query with reciprocal rank fusion (153.3) all attack vocabulary mismatch, at the cost of extra LLM calls before search.
- Multi-hop questions need several retrieval steps, because later hops need names found in earlier ones; interleave retrieval and reasoning (153.4) and cap the steps.
- GraphRAG builds an entity graph, finds communities (153.5) and pre-writes their summaries; it is for global questions, not fact lookup. Self-RAG (153.6, 153.7) and CRAG check retrieval quality before trusting it.
- Evaluate retrieval and generation separately: recall@k, MRR and nDCG (153.8 to 153.10) for the retriever; faithfulness and answer relevance (153.11, 153.12) for the generator; correctness needs references.
- Debug in order: missing content, wrong chunk, right chunk ignored, conflicting sources. The same wrong answer can need opposite fixes.
Key formulas
References for this session
- Gao et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels
- Edge et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey
- Gao et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels
- Edge et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Wang, Yang and Wei (2023). Query2doc: Query Expansion with Large Language Models
- Zheng et al. (2023). Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
- Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
- Trivedi et al. (2022). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Yang et al. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- Sarthi et al. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
- Yan et al. (2024). Corrective Retrieval Augmented Generation
- Es et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation
- Saad-Falcon et al. (2023). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
- Barnett et al. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System
- Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
- Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning