LLM & VLM courseBlog
153Track 6: LLM applications · session 11 of 20

RAG, part 2: advanced methods and evaluation

The basic RAG pipeline fails on vague questions, questions that need several documents, and questions about a whole collection. This session covers the fixes (query rewriting, HyDE, multi-query with rank fusion, multi-hop retrieval, GraphRAG, Self-RAG and CRAG) and then shows how to measure retrieval and generation separately, so you know which part to fix.

Time: 3 hours (reading 2 hours, exercises 1 hour)Level: IntermediateBefore this: Session 152 (RAG, part 1); Sessions 150 and 151 help

What you will learn

  • Why vocabulary mismatch hurts retrieval, and how query rewriting, HyDE and multi-query with reciprocal rank fusion fix it
  • How to answer multi-hop questions by interleaving retrieval and reasoning (IRCoT)
  • How GraphRAG builds an entity graph, finds communities and answers global questions by map-reduce over community summaries
  • How Self-RAG’s reflection tokens and CRAG’s retrieval evaluator decide when to retrieve and what to trust
  • How to compute recall@k, precision@k, MRR and nDCG for the retriever
  • How to measure faithfulness, answer relevance and context relevance for the generator, and why faithfulness is not correctness
  • How to diagnose the four main failure modes: missing content, wrong chunk, right chunk ignored, conflicting sources

Why this matters

Session 152 built the basic pipeline: split documents into chunks, embed them, search, rerank, put the best chunks in the prompt, and ask the model to answer with citations. That pipeline works for easy questions. Real users ask hard ones: vague questions, questions that use other words than the documents, questions that need two or three facts from different places, and questions about the whole collection (“what are the main themes in these 500 reports?”). This session covers the methods built for those cases, and then the part most teams skip: measuring whether the system works.

The main lesson of this session is simple. A RAG system has two parts that fail in different ways: the retriever (did the right chunk reach the prompt?) and the generator (did the model use it correctly?). If you only look at the final answer, you cannot tell which part to fix. So we measure them separately: retrieval with recall@k, MRR and nDCG; generation with faithfulness and answer relevance. Then we look at the common failure modes one by one.

The plan: first, change the query (rewriting, HyDE, multi-query with rank fusion). Second, retrieve in several steps for multi-hop questions. Third, change the index (GraphRAG). Fourth, let the model decide and criticize its own retrieval (Self-RAG, and the related CRAG). Fifth, evaluate retrieval and generation. Last, a field guide to failure modes: wrong chunk, right chunk ignored, conflicting sources, and missing content.

Changing the query: rewriting, HyDE and multi-query

The retriever compares the query with the chunks. Users write short, vague, conversational queries (“it won’t turn on after the update”); documents use formal words (“device fails to boot after a firmware upgrade”). This gap between the words of the question and the words of the answer is called vocabulary mismatch. Dense embeddings (session 150) reduce it but do not remove it, and keyword search such as BM25 suffers from it fully. The fix is cheap: before searching, use an LLM to change the query.

Query rewriting

Query rewriting asks an LLM to turn the user’s message into a better search query. In a chat, it also makes the query standalone: “and what about the Pro plan?” becomes “refund window for the Pro plan”, using the conversation history \(h\):

\[ \hat{q} = \mathrm{LLM}\big(P_{\text{rewrite}},\ h,\ q\big) \tag{153.1} \]

Here \(P_{\text{rewrite}}\) is the instruction (“rewrite the last user message as a standalone search query”), \(h\) the chat history and \(q\) the last message. A related trick is step-back prompting (Zheng et al., 2023): ask a more general question first (“what are the refund rules?”) and retrieve for both. [Certain]

HyDE: search with a fake answer

HyDE (Hypothetical Document Embeddings, Gao et al., 2022) goes further. Instead of searching with the question, ask an LLM to write a passage that answers it, then embed that passage and search with it. The passage may contain wrong facts, but it is written in the style and vocabulary of a real answer, so it lands near real answers in embedding space. The paper samples several fake documents \(\hat{d}_1, \dots, \hat{d}_N\) and also keeps the query itself as one more “hypothesis”: [Certain]

\[ \hat{\mathbf{v}} = \frac{1}{N+1}\left[\ \sum_{k=1}^{N} f(\hat{d}_k) \;+\; f(q)\ \right] \tag{153.2} \]
Read the symbols
\(\hat{d}_k\)
the \(k\)-th hypothetical document, written by an instruction-following LLM (“write a passage that answers the question”)
\(f\)
the document encoder (Contriever in the paper); it maps text to a vector
\(q\)
the original question
\(\hat{\mathbf{v}}\)
the final search vector; the real documents with the highest inner product with it are returned

The encoder is the important filter: it keeps the general meaning and drops much of the made-up detail, and the search only returns real documents, so the fake facts never reach the answer directly. In the paper, HyDE with the unsupervised Contriever encoder raised nDCG@10 on TREC DL19 from 44.5 to 61.3, above BM25 (50.6) and close to a Contriever fine-tuned on MS MARCO labels (62.1). [Certain] The cost is one LLM call before every search, which adds latency. Query2doc (Wang, Yang and Wei, 2023) is a close cousin: it appends the generated passage to the query for BM25, and reports BM25 gains of 3% to 15% on MS MARCO and TREC DL. [Certain]

Multi-query and reciprocal rank fusion

Multi-query retrieval asks the LLM for several different phrasings of the question (for example three), runs a search for each, and merges the ranked lists. Each phrasing finds some relevant chunks; together they cover more. The standard way to merge ranked lists is reciprocal rank fusion (RRF) (Cormack, Clarke and Büttcher, 2009). It uses only the ranks, not the scores, so it can merge lists whose scores are not comparable (for example BM25 scores and cosine similarities, as in hybrid search in Part 1):

\[ \mathrm{RRF}(d) = \sum_{i=1}^{m} \frac{1}{k + \mathrm{rank}_i(d)} \tag{153.3} \]
Read the symbols
\(m\)
the number of ranked lists (one per sub-query, or one per retriever)
\(\mathrm{rank}_i(d)\)
the position of document \(d\) in list \(i\), starting at 1; if \(d\) is not in list \(i\), that term is 0
\(k\)
a constant that limits how much the very top ranks dominate; the paper uses \(k = 60\) [Certain]

With \(k = 60\), rank 1 gives \(1/61 = 0.0164\) and rank 2 gives \(1/62 = 0.0161\): almost the same. So RRF mostly rewards documents that appear in many lists, and only a little those that are first in one. Figure 1 compares the four strategies on a small IT help-desk collection.

    Figure 1. Four ways to search for “My laptop won't turn on after the update” in eight help-desk chunks. D1 and D2 (green border) answer the question; D3 is about installing updates, a near miss that shares many words. The retriever is a word-count cosine similarity, a stand-in for a dense encoder that makes vocabulary mismatch easy to see. Kept chunks are highlighted; scores are on the right.
    How to read Figure 1

    With the plain query, D1 has score 0: it shares no word with the question (“boot”, “firmware upgrade” vs “turn on”, “update”), so it is never found, and recall@3 is 0.5. Each change of the query fixes this in a different way. The rewrite adds “fails to boot”, the words of D1; HyDE’s fake answer talks about the power button and safe mode, which are in D1 and D2; multi-query runs three searches (shown with their top lists) and fuses them with RRF. All three reach \(\mathrm{Recall@}3 = 1.0\). Only multi-query puts a relevant chunk first (reciprocal rank 1.0). Its top two are an exact tie (0.0325 each), broken by document number. Notice that D3, the near miss, stays in the top 3 almost everywhere: query changes raise recall, and the reranker from Part 1 is still needed for precision.

    Worked example: fusing three lists with RRF

    The three sub-queries return these top lists (best first): plain query [D3, D2, D7, D5]; “computer does not boot after a firmware upgrade” [D1, D8]; “black screen after a system update” [D2, D3, D6]. With \(k = 60\):

    • D2: rank 2 in list 1 and rank 1 in list 3: \(1/62 + 1/61 = 0.0325\)
    • D3: rank 1 in list 1 and rank 2 in list 3: \(1/61 + 1/62 = 0.0325\)
    • D1: rank 1 in list 2 only: \(1/61 = 0.0164\); D8: \(1/62 = 0.0161\); D6 and D7: \(1/63 = 0.0159\)

    D2 and D3 tie because each is first in one list and second in another. D1, the best answer, is only third, because just one sub-query found it. RRF is a vote: being found by several searches counts more than being first in one.

    Algorithm 1: query transformation before retrieval
    1. input: user message \(q\), chat history \(h\), retriever Search(text, top), number of sub-queries \(m\), keep \(k\)
    2. \(\hat{q} \leftarrow\) LLM(“rewrite as a standalone search query”, \(h\), \(q\)) // formula (153.1)
    3. if HyDE: \(\hat{d} \leftarrow\) LLM(“write a short passage that answers:”, \(\hat{q}\)); \(\mathbf{v} \leftarrow (f(\hat{d}) + f(\hat{q}))/2\); return Search(\(\mathbf{v}\), \(k\)) // formula (153.2), \(N = 1\)
    4. subs \(\leftarrow [\hat{q}]\) + LLM(“give \(m-1\) different phrasings of”, \(\hat{q}\)) // multi-query
    5. lists \(\leftarrow\) [Search(\(s\), top = 20) for \(s\) in subs] // can run in parallel
    6. score\([d] \leftarrow \sum_i 1/(60 + \text{rank}_i(d))\) over the lists that contain \(d\) // formula (153.3)
    7. return the \(k\) documents with the highest score, then rerank them with a cross-encoder // precision step from Part 1

    Multi-hop questions: retrieve, reason, retrieve again

    Some questions need facts from several documents that do not mention each other: “What is the population of the town where the inventor of the Fenwick lamp was born?” One document says who invented the lamp; a second says where that person was born; a third gives the town’s population. This is a multi-hop question: each hop needs the answer of the previous one to know what to search for. Benchmarks such as HotpotQA (Yang et al., 2018), 2WikiMultihopQA and MuSiQue are built from such questions. [Certain]

    One search with the whole question fails, for a clear reason: the second and third documents contain words (the inventor’s name, the town’s name) that are not in the question. They can only be found after reading the first document. There are two families of fixes:

    • Decompose first: ask the LLM to split the question into sub-questions (“who invented the Fenwick lamp?”, “where was that person born?”, “what is that town’s population?”), then answer them in order, filling the blanks.
    • Interleave retrieval and reasoning: retrieve, write one reasoning sentence, use that sentence as the next query, and repeat. IRCoT (Trivedi et al., 2022) does exactly this with chain-of-thought sentences; with GPT-3 it improved retrieval by up to 21 points and QA by up to 15 points on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC. [Certain]

    Written as a loop, with \(c_t\) the reasoning sentence written at step \(t\):

    \[ \mathcal{D}_t = \mathrm{TopK}\big(q_t\big), \qquad c_t = \mathrm{LLM}\big(q,\ \mathcal{D}_1 \cup \dots \cup \mathcal{D}_t,\ c_1, \dots, c_{t-1}\big), \qquad q_{t+1} = c_t \tag{153.4} \]
    Read the symbols
    \(q\)
    the original question; \(q_1 = q\)
    \(\mathcal{D}_t\)
    the passages retrieved at step \(t\) (new ones only)
    \(c_t\)
    the reasoning sentence written at step \(t\); it becomes the next query

    The loop stops when the model writes “so the answer is …” or after a maximum number of steps. Figure 2 runs it on the lamp question.

      Figure 2. Single-shot retrieval vs interleaved retrieval (IRCoT style) on a three-hop question over six passages. P1, P2 and P3 (green border) are the three needed passages; P6 is a trap about another town’s population. Step 1 shows one search with the full question and 4 passages kept. Steps 2 to 5 use the same budget of 4 passages, but spread over three searches. The scores are word-count cosine similarities of each passage with the current query.
      How to read Figure 2

      Step 1: the single search keeps P1 (0.369), P4 and P6 (0.272 each) and P2 (0.258). P3, which holds the answer, is fifth (0.154). Recall is 2/3, and the context contains the trap “Harlow grew to 90,000”: a model forced to answer may say 90,000. Steps 2 to 5: the first search keeps only 2 passages; the model writes “The Fenwick lamp was invented by Mara Ilves”, which finds P2; then “Mara Ilves was born in Tolby”, which finds P3. Recall reaches 3/3 with the same 4 passages. The price is three retrieval calls and three LLM calls instead of one of each.

      Algorithm 2: interleaved multi-hop retrieval (IRCoT style)
      1. input: question \(q\), retriever TopK, passages per step \(k\), maximum steps \(T\)
      2. \(\mathcal{D} \leftarrow\) TopK(\(q\), \(k\)); reasoning \(\leftarrow\) [ ]
      3. for \(t = 1 .. T\):
      4.    \(c \leftarrow\) LLM(“write the next reasoning sentence”, \(q\), \(\mathcal{D}\), reasoning) // formula (153.4)
      5.    append \(c\) to reasoning
      6.    if \(c\) contains “the answer is”: break
      7.    \(\mathcal{D} \leftarrow \mathcal{D} \,\cup\) TopK(\(c\), \(k\), excluding \(\mathcal{D}\)) // the sentence names the next entity
      8. return LLM(“answer using only these passages, cite them”, \(q\), \(\mathcal{D}\), reasoning)

      Multi-hop retrieval is a small step toward an agent: a model that decides, step by step, what to look up next (session 154). Its risks are the same: errors early in the chain carry forward (if hop 1 picks the wrong inventor, every later hop is wrong), and the number of steps must be capped.

      GraphRAG: index the relations, not only the chunks

      Chunk retrieval answers local questions: the answer sits in one or a few chunks. It fails at global questions about the whole collection: “What are the main themes in these podcast transcripts?” No single chunk contains the answer, and top-\(k\) search returns \(k\) random-looking pieces. GraphRAG (Edge et al., 2024, Microsoft) builds a different index for such questions: a knowledge graph of the entities in the text and their relations, grouped into communities, each with a summary written in advance. [Certain]

      The indexing pipeline in the paper:

      1. Split the documents into chunks (600 tokens with 100 tokens of overlap in the paper’s experiments). [Certain]
      2. For each chunk, an LLM extracts entities (people, places, organizations, concepts), relationships between them and short descriptions. A “gleaning” step asks the model again for entities it missed. [Certain]
      3. Merge the extractions into one graph: entities are nodes; a relationship found many times becomes an edge with a larger weight.
      4. Split the graph into communities, groups of nodes that are densely connected to each other and loosely to the rest, with the Leiden algorithm, applied again inside each community to make a hierarchy (levels C0, the root, down to C3). [Certain]
      5. Ask an LLM to write a report-style summary of each community, from the bottom level up.

      Leiden looks for a split that makes the modularity high: more edge weight inside communities than a random graph with the same node degrees would have. [Likely] (Leiden can also optimize other quality functions; GraphRAG’s default implementation uses modularity as far as we know.)

      \[ Q = \frac{1}{2W} \sum_{i,j} \left[ A_{ij} - \frac{w_i\, w_j}{2W} \right] \mathbb{I}\big[c_i = c_j\big] \tag{153.5} \]
      Read the symbols
      \(A_{ij}\)
      edge weight between entities \(i\) and \(j\) (how often the relation was extracted); 0 if no edge
      \(w_i\)
      the total edge weight of node \(i\) (its weighted degree); \(W\) is the total edge weight of the graph
      \(w_i w_j / 2W\)
      the expected weight between \(i\) and \(j\) if edges were placed at random with the same degrees
      \(c_i\)
      the community of node \(i\); \(\mathbb{I}[\cdot]\) is 1 when the two nodes are in the same community
      Figure 3. GraphRAG’s index on a toy collection of support documents. Entities (circles) extracted from chunks are linked by relations (lines, thicker = found more often). Leiden groups them into communities (shaded areas). Each community gets a summary written at indexing time. A global question is answered by map-reduce over these summaries; a local question starts from the entities it names and their neighbours.

      Two ways to answer:

      • Global search uses map-reduce, the same idea as long-document summarization (session 147). Shuffle the community summaries of one level into context-sized groups; map: each group produces a partial answer and a helpfulness score from 0 to 100; drop the zeros; reduce: combine the best partial answers, in score order, into one answer. [Certain]
      • Local search starts from the entities mentioned in the question, and collects their relations, neighbours, community summaries and source chunks into the context. [Likely] (This mode is described in the GraphRAG software, not in the main experiments of the paper.)

      In the paper, on about 1 million tokens of podcast transcripts and 1.7 million tokens of news, GPT-4 judges preferred GraphRAG answers over plain vector RAG for comprehensiveness 72% to 83% of the time on podcasts and 72% to 80% on news, and for diversity 75% to 82% and 62% to 71%. [Certain] Answering from the root level C0 used about 26,657 context tokens on the podcast data, against 1,014,611 for map-reduce over the raw text: about 38 times fewer. [Certain] Note what was not claimed: these are win rates judged by an LLM on broad “sensemaking” questions, not accuracy on fact questions. For “what is the refund window?”, plain chunk retrieval is cheaper and at least as good. [Likely]

      The cost is at indexing time: one or more LLM calls per chunk for extraction, plus one per community summary, and the whole graph must be rebuilt or patched when documents change. RAPTOR (Sarthi et al., 2024) is a lighter relative: it clusters chunk embeddings, summarizes each cluster, and repeats, building a tree of summaries that is searched together with the chunks; with GPT-4 it improved the best reported accuracy on the QuALITY benchmark by 20 points (absolute). [Certain]

      Algorithm 3: GraphRAG indexing and global search
      1. index: chunks \(\leftarrow\) split(docs, 600 tokens, overlap 100)
      2.    for each chunk: (entities, relations) \(\leftarrow\) LLM(“extract entities and relations”, chunk); glean again for missed ones
      3.    \(G \leftarrow\) merge all extractions; edge weight = number of times a relation was found
      4.    communities \(\leftarrow\) hierarchical Leiden(\(G\)) // formula (153.5)
      5.    for each community, bottom level first: summary \(\leftarrow\) LLM(“write a report on this community”, its entities, relations, child summaries)
      6. global query \(q\): groups \(\leftarrow\) pack shuffled summaries of one level into context-sized groups
      7.    partial \(\leftarrow\) [LLM(“answer \(q\) from these summaries; rate helpfulness 0-100”, \(g\)) for \(g\) in groups] // map, parallel
      8.    return LLM(“combine into one answer”, partial answers with score \(\gt 0\), best first) // reduce

      Self-RAG and CRAG: let the model judge its retrieval

      The basic pipeline always retrieves, and always trusts what it retrieved. Both are wrong sometimes. “Write a poem about autumn” needs no retrieval; extra passages only distract. And when the retrieved passages are off-topic, the model should notice and not use them. Two methods add these decisions.

      Self-RAG

      Self-RAG (Asai et al., 2023) trains one model to generate special reflection tokens alongside its answer. There are four kinds: [Certain]

      TokenQuestion it answersPossible values
      RetrieveShould I retrieve now?yes, no, continue (keep using the current passage)
      IsRelIs this passage relevant to the question?relevant, irrelevant
      IsSupIs my segment supported by the passage?fully supported, partially supported, no support
      IsUseIs my answer useful overall?5, 4, 3, 2, 1

      How it is trained: GPT-4 labeled a few thousand examples per token type (4k to 20k each); a Llama 2 7B critic model learned to predict the tokens from those labels; the critic then inserted reflection tokens offline into 150k instruction–answer pairs, and the generator (Llama 2 7B or 13B) was trained on that data with the ordinary next-token loss. [Certain] At inference the generator itself emits the tokens; no separate critic is needed.

      At inference, the model first predicts Retrieve. It retrieves only if the normalized probability of “yes” is above a threshold \(\delta\) (0.2 for most tasks in the paper): [Certain]

      \[ \text{retrieve} \iff \frac{p(\texttt{Retrieve}=\text{yes})}{p(\texttt{Retrieve}=\text{yes}) + p(\texttt{Retrieve}=\text{no})} \gt \delta \tag{153.6} \]

      If it retrieves, it processes several passages in parallel, writes one candidate segment per passage, and ranks the candidates by their own probability plus a weighted sum of critique scores:

      \[ f(y_t, d) = p\big(y_t \mid x, d, y_{\lt t}\big) + \sum_{G \in \{\text{IsRel},\,\text{IsSup},\,\text{IsUse}\}} w^{G}\, s_t^{G}, \qquad s_t^{G} = \frac{p_t(\hat{r}^{G})}{\sum_{i=1}^{N^G} p_t(r^{G}_i)} \tag{153.7} \]
      Read the symbols
      \(y_t\)
      the candidate segment (for example one sentence) at step \(t\); \(x\) is the input, \(d\) the passage used
      \(\hat{r}^{G}\)
      the most desirable value of token type \(G\) (relevant; fully supported; 5)
      \(s_t^{G}\)
      the probability of that best value, divided by the sum over all \(N^G\) values of type \(G\): a number from 0 to 1
      \(w^{G}\)
      weights set at inference time, without retraining; the paper’s defaults are 1.0 for IsRel, 1.0 for IsSup and 0.5 for IsUse [Certain]

      The weights are a dial between “fluent” and “grounded”. Raise \(w^{\text{IsSup}}\) and the model prefers segments the passage really supports, even if they read less smoothly. The paper keeps the best 2 candidates at each segment step (beam width 2). [Certain]

      Figure 4. Self-RAG’s segment ranking (formula 153.7) on toy numbers. Question: “How long is the refund window for the Pro plan?” The model’s normalized \(p(\texttt{Retrieve}=\text{yes})\) is 0.62. Three passages give three candidates: d1 is the refund policy, d2 a marketing page, d3 an old forum post. Each bar stacks the segment probability and the three weighted critique scores; the highest total wins.
      How to read Figure 4

      With the paper’s default weights (1, 1, 0.5), d1 wins with 2.250, ahead of d3 (1.680) and d2 (1.350). Set all three weights to 0: only the segment probability counts, and d2, the most fluent text, wins (0.55), although its passage is irrelevant. Put weight only on IsUse (for example 1.0, others 0) and d2 wins again, with 1.150: “useful-sounding” is not “supported”. Raise \(\delta\) above 0.62 and the model does not retrieve at all; it answers from its own memory, with no citation.

      Worked example: the score of candidate d1

      d1 has segment probability 0.40; IsRel probabilities (relevant 0.9, irrelevant 0.1); IsSup (fully 0.7, partially 0.2, none 0.1); IsUse (5: 0.5, 4: 0.3, 3: 0.1, 2: 0.05, 1: 0.05). Each list sums to 1, so \(s^{\text{IsRel}} = 0.9\), \(s^{\text{IsSup}} = 0.7\), \(s^{\text{IsUse}} = 0.5\). With weights (1, 1, 0.5):

      \(f = 0.40 + 1.0 \times 0.9 + 1.0 \times 0.7 + 0.5 \times 0.5 = 2.25\).

      d2 (marketing page): \(0.55 + 0.3 + 0.2 + 0.5 \times 0.6 = 1.35\). d3 (old forum post): \(0.48 + 0.8 + 0.3 + 0.5 \times 0.2 = 1.68\). The critique terms, not the fluency term, decide the winner.

      CRAG: correct the retrieval before generating

      Corrective RAG (CRAG) (Yan et al., 2024) needs no change to the generator. A small retrieval evaluator (a fine-tuned T5-large, about 0.77B parameters) scores each retrieved document for the question, and two thresholds pick one of three actions: [Certain]

      • Correct (at least one document scores above the upper threshold): keep the documents, but split them into small “knowledge strips”, drop irrelevant strips and recombine the rest.
      • Incorrect (all documents below the lower threshold): discard them; rewrite the question into keywords and run a web search instead.
      • Ambiguous (anything else): use both the refined documents and the web results.

      Self-RAG and CRAG share one idea: retrieval quality should be checked before the answer depends on it. In a production system, the same idea is often a cheap rule: if the best reranker score is below a threshold, say “I could not find this in the documents” instead of answering. [Likely]

      Evaluating retrieval: recall@k, MRR and nDCG

      Now the second half of the session. To evaluate the retriever, you need, for each test question, the set of chunks that contain the answer (the relevant set). You can label them by hand, or generate questions from chunks with an LLM (then the source chunk is relevant by construction; this is cheap but makes questions that are easier than real ones). [Likely] Given the ranked list the retriever returns, the standard metrics are:

      \[ \mathrm{Recall@}k = \frac{\lvert \mathrm{Rel} \cap \mathrm{Top}_k \rvert}{\lvert \mathrm{Rel} \rvert}, \qquad \mathrm{Precision@}k = \frac{\lvert \mathrm{Rel} \cap \mathrm{Top}_k \rvert}{k} \tag{153.8} \]
      \[ \mathrm{MRR} = \frac{1}{\lvert Q \rvert} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q} \tag{153.9} \]
      \[ \mathrm{DCG@}k = \sum_{i=1}^{k} \frac{2^{g_i} - 1}{\log_2(i+1)}, \qquad \mathrm{nDCG@}k = \frac{\mathrm{DCG@}k}{\mathrm{IDCG@}k} \tag{153.10} \]
      Read the symbols
      \(\mathrm{Rel}\)
      the set of relevant chunks for the question; \(\mathrm{Top}_k\) the first \(k\) chunks returned
      \(\mathrm{rank}_q\)
      the position of the first relevant chunk for question \(q\); \(1/\mathrm{rank}_q = 0\) if none is returned. \(1/\mathrm{rank}_q\) for one question is the reciprocal rank (RR)
      \(g_i\)
      the relevance grade of the chunk at position \(i\) (0 or 1 for binary labels, so \(2^{g_i} - 1\) is also 0 or 1)
      \(\mathrm{IDCG@}k\)
      the DCG of the ideal ranking (all relevant chunks first); it makes nDCG lie between 0 and 1

      Which one matters for RAG? Recall@k is the most important: the generator can only use what is in the prompt, so a relevant chunk at rank 7 is lost if you keep 5. Precision@k matters less, but low precision fills the prompt with distractors. MRR and nDCG reward putting relevant chunks early, which matters because models use the start of the context better than the middle (“lost in the middle”, Liu et al., 2023). [Certain] A useful habit: plot \(\mathrm{Recall@}k\) against \(k\). Where the curve flattens is how many chunks to keep. Figure 5 lets you build a ranked list and see the four numbers.

      Figure 5. Retrieval metrics for one question. The ten buttons are the retriever’s ranked results; green means relevant. Click a result to mark it relevant or not. Results after the cut-off \(k\) are faded. \(\lvert\mathrm{Rel}\rvert\) counts all relevant chunks in the corpus, including ones the retriever never returned; it cannot be lower than the number you marked.
      How to read Figure 5

      The default has relevant results at ranks 2, 5 and 9, and 4 relevant chunks in the corpus (one was never returned). At \(k = 5\): recall 0.5, precision 0.4, RR 0.5, nDCG 0.397. Move \(k\) to 10: recall rises to 0.75 but precision falls to 0.3. Now click rank 1 to make it relevant: RR jumps to 1 and nDCG rises, but recall@5 rises only by 0.25. Raise \(\lvert\mathrm{Rel}\rvert\) to 8: recall drops, because recall is measured against everything relevant, not only what was found.

      Worked example: nDCG@5 for relevant results at ranks 2 and 5

      \(\mathrm{DCG@}5 = 1/\log_2 3 + 1/\log_2 6 = 0.631 + 0.387 = 1.018\). With 4 relevant chunks in the corpus, the ideal list puts them at ranks 1 to 4: \(\mathrm{IDCG@}5 = 1 + 0.631 + 0.5 + 0.431 = 2.562\). So \(\mathrm{nDCG@}5 = 1.018 / 2.562 = 0.397\). \(\mathrm{Recall@}5 = 2/4 = 0.5\), \(\mathrm{Precision@}5 = 2/5 = 0.4\), \(\mathrm{RR} = 1/2\). Over three test questions with first relevant ranks 2, 1 and 4, \(\mathrm{MRR} = (1/2 + 1 + 1/4)/3 = 0.583\).

      Evaluating generation: faithfulness and answer relevance

      Good retrieval does not guarantee a good answer. For the generator we ask two questions, and they are different:

      • Faithfulness (also called groundedness): is every claim in the answer supported by the retrieved context? This is the RAG version of the summary faithfulness check in session 147.
      • Answer relevance: does the answer address the question? A faithful answer can still dodge the question.

      Neither needs a human-written reference answer, which is why they are popular. RAGAS (Es et al., 2023) defines them with LLM calls: [Certain]

      \[ F = \frac{\lvert V \rvert}{\lvert S \rvert} \tag{153.11} \]

      where an LLM splits the answer into short statements \(S\) and then checks each against the context; \(V\) is the set of statements it judges supported. For answer relevance, the LLM writes \(n\) questions that the answer would answer, and we measure how close they are to the real question:

      \[ \mathrm{AR} = \frac{1}{n} \sum_{i=1}^{n} \cos\big(\mathbf{e}(q_i),\ \mathbf{e}(q)\big) \tag{153.12} \]

      with \(\mathbf{e}\) an embedding model. An answer that drifts off topic produces generated questions unlike the original, so AR drops. RAGAS also scores the context:

      \[ \mathrm{CR} = \frac{\text{number of context sentences needed to answer } q}{\text{total number of context sentences}} \tag{153.13} \]
      Read the symbols
      \(S, V\)
      the statements extracted from the answer, and those of them supported by the context
      \(q_i\)
      the \(i\)-th question generated from the answer; \(q\) is the user’s question
      \(\mathrm{CR}\)
      context relevance: a precision of the context, measured by the LLM picking the sentences it needs

      On their WikiEval data (50 Wikipedia pages), RAGAS agreed with human pairwise judgments 95% of the time for faithfulness, 78% for answer relevance and 70% for context relevance. [Certain] So faithfulness is the most reliable of the three automatic scores, and context relevance the least. Later versions of the RAGAS library added reference-based metrics such as context recall, which do need a reference answer. [Likely]

      Two more points matter in practice. First, faithfulness is not correctness: if the retriever returned an outdated policy, a faithful answer repeats the outdated policy. To measure correctness you need reference answers for at least a test set. Second, LLM judges make mistakes. ARES (Saad-Falcon et al., 2023) trains small judges on synthetic data and uses a few hundred human labels with prediction-powered inference to correct the judges’ bias and give confidence intervals. [Certain] Whatever you use, check the judge against a small human-labeled set first (session 155).

      What you measureMetricNeeds labels?Tells you to fix
      Did the right chunks come back?\(\mathrm{Recall@}k\), \(\mathrm{hit@}k\)Relevant chunk ids per questionChunking, embeddings, hybrid search, query rewriting
      Were they near the top?MRR, \(\mathrm{nDCG@}k\)Same (graded labels for nDCG)Reranker, fusion weights
      Is the context focused?\(\mathrm{Precision@}k\), context relevanceNo (LLM judge) or yes\(k\), reranker threshold, chunk size
      Is the answer grounded?FaithfulnessNo (LLM or NLI judge)Prompt, generator, citations, context order
      Does it answer the question?Answer relevanceNoPrompt, refusal behavior
      Is it right?Answer correctness, exact match, LLM judge vs referenceReference answersEverything, including stale or conflicting data
      Algorithm 4: component-wise RAG evaluation
      1. input: test set of (question \(q\), relevant chunk ids \(\mathrm{Rel}\), reference answer \(a^\ast\) if available), RAG system
      2. for each test item:
      3.    ranked \(\leftarrow\) retriever(\(q\)); record \(\mathrm{Recall@}k\), \(\mathrm{RR}\), \(\mathrm{nDCG@}k\) against \(\mathrm{Rel}\) // formulas (153.8)-(153.10)
      4.    answer \(\leftarrow\) generator(\(q\), ranked[:\(k\)])
      5.    statements \(\leftarrow\) LLM(“split into short statements”, answer); verdicts \(\leftarrow\) judge(statements, context) // faithfulness, (153.11)
      6.    record answer relevance (153.12) and, if \(a^\ast\) exists, correctness
      7. report every metric as a mean with a confidence interval, and slice by question type // multi-hop, recent docs, ...
      8. read 20 to 50 failures by hand and label each with a failure mode (next section)

      Failure modes: a field guide

      Barnett et al. (2024) collected failure points from three real RAG projects (research, education and biomedical): missing content, missed top-ranked documents, relevant documents dropped during context assembly, answer not extracted from the context, wrong format, wrong level of detail, and incomplete answers. [Certain] For debugging, they reduce to four questions asked in order:

      1. Is the answer in the corpus at all? If not: missing content. The right behavior is to say so. Fix: an “I don’t know” instruction and a score threshold; add the content.
      2. Did the right chunk reach the prompt? If not (context recall below 1): wrong chunk, a retrieval failure. Fix: query rewriting, hybrid search, better chunking, more candidates before reranking.
      3. Did the model use it? If the chunk is there but the answer is unsupported or contradicts it: right chunk ignored, a generation failure. Common causes: the chunk sat in the middle of a long context, too many distractors, or the model trusted its own memory. Fix: rerank so the best chunk is first, keep fewer chunks, ask for quotes before the answer.
      4. Is the answer faithful but still wrong? Then the sources disagree or are stale: conflicting sources. The model faithfully repeated the wrong one. Fix: store dates and versions as metadata, filter replaced documents, and tell the model to report conflicts and prefer the newest source.

      Figure 6 runs this diagnosis on five cases.

          Figure 6. Five RAG cases for questions about a product’s refund policy. Retrieved chunks are listed in prompt order (green border = contains the answer). The answer’s claims are marked supported, contradicted or not in the context. The bars show context recall, context precision, faithfulness (formula 153.11) and correctness against the reference. The box applies the four diagnosis questions in order. Tick “Apply the fix” to see the case after the suggested change.
          How to read Figure 6

          Compare “Wrong chunk” and “Conflicting sources”. Both give the wrong answer “14 days”. In the first, context recall is 0 and faithfulness is 0: the retriever failed, and the model also guessed. In the second, context recall is 1 and faithfulness is 1: the system faithfully repeated an outdated 2023 policy. Same wrong answer, opposite fixes. “Chunk ignored” has recall 1 but faithfulness 0.5: the gold chunk was third of five, and the model took 14 days from the Basic plan chunk. “Missing content” has no relevant chunk anywhere; the fix is to abstain, which turns correctness to 1 with no claims to check.

          Putting it together in PyTorch

          The code below runs with only PyTorch and no downloads. A unit-length word-count vector stands in for a dense encoder, so the numbers match Figure 1. It implements HyDE (formula 153.2), multi-query with RRF (153.3), recall@k, reciprocal rank and nDCG (153.8 to 153.10), the Self-RAG segment score (153.7) and faithfulness (153.11). It is deterministic.

          import re, math, torch
          
          STOP = set("was were has had a an the of to in on for and or is are be it its my i me "
                     "you your we if then than that this with by at as from after before when while "
                     "can will won t not do does did should what which who how why where there their "
                     "them they use used up out into so but every each".split())
          def toks(s): return [w for w in re.findall(r"[a-z0-9]+", s.lower()) if w not in STOP]
          
          DOCS = ["If a device fails to boot after a firmware upgrade, hold the power button for 30 seconds to reset the controller, then start it again.",
                  "A notebook that shows a black screen after a system update can be recovered by starting in safe mode and rolling back the update.",
                  "To install a laptop update, open Settings and choose Software Update; the laptop restarts when the update is done.",
                  "Batteries last about three years; replace the battery if the device turns off at 40 percent charge.",
                  "Request a new laptop through the IT portal; delivery takes five working days.",
                  "VPN errors after an update are usually fixed by reinstalling the VPN client.",
                  "Printers that do not turn on should be checked for a loose power cable.",
                  "The firmware upgrade calendar for servers is published every month by the infrastructure team."]
          RELEVANT = {0, 1}                                  # D1 and D2 answer the question
          VOCAB = sorted({w for d in DOCS for w in toks(d)})
          IDX = {w: i for i, w in enumerate(VOCAB)}
          
          def embed(text):
              """Stand-in for a dense encoder: unit-length word-count vector."""
              v = torch.zeros(len(VOCAB))
              for w in toks(text):
                  if w in IDX: v[IDX[w]] += 1
              return v / v.norm().clamp_min(1e-12)
          
          E_DOCS = torch.stack([embed(d) for d in DOCS])     # (8, |V|)
          
          def search(qvec, top=None):
              s = E_DOCS @ qvec
              order = sorted(range(len(DOCS)), key=lambda i: (-float(s[i]), i))
              order = [i for i in order if s[i] > 0]          # zero score = not retrieved
              return order[:top] if top else order
          
          def hyde_vector(query, fake_docs):
              """Formula (153.2): average the fake documents and the query."""
              vs = [embed(d) for d in fake_docs] + [embed(query)]
              return torch.stack(vs).mean(0)
          
          def rrf(lists, k=60):
              """Formula (153.3): reciprocal rank fusion."""
              score = {}
              for l in lists:
                  for r, d in enumerate(l, start=1):
                      score[d] = score.get(d, 0.0) + 1.0 / (k + r)
              return sorted(score, key=lambda d: (-score[d], d))
          
          def recall_at_k(ranked, rel, k): return len(set(ranked[:k]) & rel) / len(rel)
          def rr(ranked, rel): return next((1 / (r + 1) for r, d in enumerate(ranked) if d in rel), 0.0)
          def ndcg_at_k(ranked, rel, k):
              dcg = sum(1 / math.log2(r + 2) for r, d in enumerate(ranked[:k]) if d in rel)
              idcg = sum(1 / math.log2(r + 2) for r in range(min(len(rel), k)))
              return dcg / idcg
          
          Q = "My laptop won't turn on after the update"
          FAKE = ["If your laptop fails to boot after an update, hold the power button for 30 seconds "
                  "to reset it and start it again. If the screen stays black, start in safe mode and "
                  "roll back the system update."]          # what an LLM might write; may contain errors
          SUBQ = [Q, "computer does not boot after a firmware upgrade", "black screen after a system update"]
          
          runs = {"plain": search(embed(Q)),
                  "HyDE": search(hyde_vector(Q, FAKE)),
                  "multi-query+RRF": rrf([search(embed(q), top=4) for q in SUBQ])}
          for name, ranked in runs.items():
              print(f"{name:16s} top3={[d + 1 for d in ranked[:3]]} "
                    f"R@3={recall_at_k(ranked, RELEVANT, 3):.2f} RR={rr(ranked, RELEVANT):.2f} "
                    f"nDCG@3={ndcg_at_k(ranked, RELEVANT, 3):.3f}")
          
          # Self-RAG segment score, formulas (153.6)-(153.7): choose among candidates
          cands = {"d1": (0.40, [0.9, 0.1], [0.7, 0.2, 0.1], [0.5, 0.3, 0.1, 0.05, 0.05]),
                   "d2": (0.55, [0.3, 0.7], [0.2, 0.3, 0.5], [0.6, 0.3, 0.05, 0.03, 0.02]),
                   "d3": (0.48, [0.8, 0.2], [0.3, 0.5, 0.2], [0.2, 0.4, 0.2, 0.1, 0.1])}
          w = torch.tensor([1.0, 1.0, 0.5])                 # weights for IsRel, IsSup, IsUse
          for name, (p, rel, sup, use) in cands.items():
              s = torch.tensor([t[0] / sum(t) for t in (rel, sup, use)])   # best value / sum
              print(name, round(p + float(w @ s), 3))
          
          # faithfulness, formula (153.11): fraction of answer claims supported by the context
          verdicts = ["supported", "contradicted"]          # from an NLI model or an LLM judge
          print("faithfulness", sum(v == "supported" for v in verdicts) / len(verdicts))

          Output:

          plain            top3=[3, 2, 7] R@3=0.50 RR=0.50 nDCG@3=0.387
          HyDE             top3=[3, 2, 1] R@3=1.00 RR=0.50 nDCG@3=0.693
          multi-query+RRF  top3=[2, 3, 1] R@3=1.00 RR=1.00 nDCG@3=0.920
          d1 2.25
          d2 1.35
          d3 1.68
          faithfulness 0.5

          With a real system, replace embed with a sentence-embedding model, the fake document and sub-queries with LLM calls, and the hand-written verdicts with an LLM or NLI judge. The metric code stays the same.

          Where this is used

          ApplicationAdvanced method that helpsWhat to measure
          Chat assistants over company documentsStandalone query rewriting from chat history; abstain below a score thresholdRecall@k on real user questions; faithfulness; abstention rate on unanswerable questions
          Customer support searchHyDE or multi-query to bridge user words and manual wordsHit@k, MRR; ticket deflection with human review
          Research and due-diligence assistantsMulti-hop retrieval (IRCoT, decomposition); citations per claimRecall of all needed passages; claim-level faithfulness
          “What are the themes?” over large archivesGraphRAG or RAPTOR summaries; map-reduce answersPairwise LLM-judge win rates with human spot checks; cost per query
          Policy, legal and medical QAVersion and date metadata; conflict reporting; CRAG-style checksCorrectness against references; conflict detection rate
          Video-RAG and long-video QAMulti-query over captions, speech transcripts and frame embeddings, fused with RRF; multi-hop across clips (“what did the speaker show after the demo?”) (session 210)Recall@k of the right time segment; faithfulness to the transcript
          Text-to-video retrieval and key-moment detectionHyDE-style expansion: write a caption of the moment you want, search clip embeddings with it (session 208)Recall@k, MRR on moment ids; temporal overlap of returned segments
          Multimodal document QA (slides, PDFs)Page-image retrieval (for example ColPali) plus text chunks, merged with RRFRecall@k of the right page; faithfulness to the page

          Common mistakes

          • Judging the system only by the final answer. You cannot tell a retrieval failure from a generation failure. Always log what was retrieved, and measure recall@k separately.
          • Treating faithfulness as correctness. A faithful answer to an outdated document is still wrong (Figure 6, conflicting sources). Keep a set of questions with reference answers.
          • Building the test set from the same chunks the system indexes, with an LLM, and nothing else. Generated questions copy the chunk’s words, so retrieval looks better than it is with real users. Mix in real, messy questions. [Likely]
          • Adding HyDE or multi-query without measuring latency and recall. Each adds at least one LLM call before search. Use them where the recall gain is real, often for short or vague queries only.
          • Keeping too many chunks “to be safe”. Higher \(k\) raises recall but adds distractors and pushes the right chunk into the middle; measure answer quality at several \(k\).
          • Using GraphRAG for simple fact lookup. Its indexing cost is high and it was built for global, sensemaking questions; plain chunk retrieval is usually better for “what is X?”.
          • No path for “not found”. Without an abstain rule, every missing-content question becomes a hallucination.
          • Trusting an LLM judge without checking it. Compare it with a few hundred human labels before using its numbers for decisions.

          Interview questions

          Your RAG system gives wrong answers. How do you debug it?core
          Separate the two components. Take a sample of failures and, for each, check in order: (1) is the answer in the corpus at all (missing content → abstain or add data); (2) did a relevant chunk reach the prompt (context recall; if not, a retrieval problem: query rewriting, hybrid search, chunking, more candidates before reranking); (3) if it was there, is the answer faithful to it (if not, a generation problem: reorder so the best chunk is first, fewer chunks, ask for quotes); (4) if faithful but wrong, the sources conflict or are stale (dates and versions as metadata). Then build a test set with relevant chunk ids and reference answers so the fix can be measured.
          What is HyDE, and why do the hallucinated details not hurt?core
          HyDE asks an LLM to write a passage that answers the question, embeds it, and searches with that vector (averaged with the query vector in the paper). The fake passage uses the vocabulary and style of real answers, so it lands near them. The made-up details matter less because the encoder compresses the passage to its general meaning, and only real documents are returned: the fake text never goes into the final prompt. It costs one LLM call per query. In the paper it raised Contriever’s nDCG@10 on TREC DL19 from 44.5 to 61.3.
          Explain reciprocal rank fusion. Why use ranks instead of scores?basic
          RRF gives each document \(\sum_i 1/(k + \mathrm{rank}_i)\) over the lists that contain it, with \(k = 60\) in the original paper. Scores from different retrievers (BM25, cosine, different sub-queries) are on different scales and cannot be added directly; ranks are always comparable. With \(k = 60\), the difference between rank 1 and rank 2 is tiny, so documents found by several lists win: it behaves like a vote. It is used for hybrid search and for multi-query retrieval.
          How would you answer a question that needs facts from two or three documents?core
          One search with the full question often misses the later hops, because their documents contain names that are not in the question. Either decompose the question into sub-questions and answer them in order, or interleave retrieval with reasoning (IRCoT): retrieve, write one reasoning sentence, search with that sentence, and repeat until the model writes the answer, with a step limit. Measure recall of all needed passages, not just one. Errors in early hops carry forward, so show the chain and its citations.
          When would you choose GraphRAG over normal chunk retrieval?advanced
          For global questions about a whole collection (“main themes”, “how do these groups relate”), where no single chunk has the answer. GraphRAG extracts entities and relations with an LLM, groups them into communities with Leiden, pre-writes community summaries, and answers by map-reduce over the summaries. In the paper, LLM judges preferred it to vector RAG on comprehensiveness about 72-83% of the time on podcast transcripts. The costs are many LLM calls at indexing, rebuild work when data changes, and no advantage for simple fact lookups. A lighter alternative is a RAPTOR-style summary tree.
          What are Self-RAG’s reflection tokens and how are they used at inference?advanced
          Retrieve (yes/no/continue), IsRel (relevant/irrelevant), IsSup (fully/partially/no support) and IsUse (1-5). The generator is trained to emit them, using labels from a critic distilled from GPT-4. At inference it retrieves only when the normalized \(p(\texttt{Retrieve}=\text{yes})\) exceeds a threshold (0.2 by default). It writes one candidate segment per retrieved passage, and ranks them by segment probability plus a weighted sum of the normalized probabilities of the best IsRel, IsSup and IsUse values (default weights 1, 1, 0.5), keeping a beam of 2. The weights can be changed without retraining to trade fluency for grounding.
          Define recall@k, MRR and nDCG. Which matters most for RAG?basic
          Recall@k: the fraction of all relevant chunks that are in the top \(k\). MRR: the mean over questions of \(1/\mathrm{rank}_q\), with \(\mathrm{rank}_q\) the rank of the first relevant chunk. nDCG@k: a sum of relevance discounted by \(\log_2(\text{rank}+1)\), divided by the ideal value. For RAG, recall@k at the \(k\) you actually put in the prompt matters most, because the generator cannot use what is not there; MRR and nDCG matter next, because models use the start of the context best.
          How do you evaluate a RAG system without human-written reference answers?core
          Use reference-free metrics with an LLM judge: faithfulness (split the answer into statements, check each against the context, \(F = |V|/|S|\)), answer relevance (generate questions from the answer and compare their embeddings with the question) and context relevance (fraction of context sentences needed). For retrieval, generate questions from chunks so the source chunk is a known relevant item, but add real user questions too. Calibrate the judge on a small human-labeled set; ARES uses a few hundred human labels with prediction-powered inference for this. Correctness still needs references for at least part of the test set.
          Faithfulness is 1.0 but users say the answer is wrong. What happened?core
          The answer matches the retrieved context, so the generator did its job; the context itself was wrong for this question. Usually: an outdated or superseded document, two documents that conflict (old and new policy), or a chunk about a different product or region that looks similar. Fixes: date and version metadata with filtering, deduplication of superseded documents, an instruction to report conflicts and prefer the newest source, and correctness tests with reference answers.
          The relevant chunk is in the prompt but the model ignores it. Why, and what do you do?advanced
          Causes: the chunk sits in the middle of a long context (models use the start and end better, “lost in the middle”); many similar distractor chunks; the model’s own prior knowledge conflicts with the chunk; or the chunk is badly formatted (a table split in half). Fixes: rerank and put the best chunk first (or first and last), keep fewer chunks, remove near-duplicates, ask the model to quote the supporting sentence before answering, and fine-tune with distractor passages if you control the model. Measure with faithfulness on cases where context recall is 1.
          How is CRAG different from Self-RAG?advanced
          Self-RAG changes the generator: it is trained to emit reflection tokens and uses them to decide when to retrieve and which segment to keep. CRAG leaves the generator alone and adds a small retrieval evaluator (a T5-large) that scores the retrieved documents; with two thresholds it picks Correct (refine the documents into relevant strips), Incorrect (discard them and do a web search) or Ambiguous (use both). CRAG is a plug-in for any RAG pipeline; Self-RAG needs a specially trained model.

          Exercises

          1. Three lists: A = [d1, d2, d3], B = [d3, d1], C = [d2, d4]. Compute RRF scores with \(k = 60\) and the fused order.
          2. A retriever returns 10 chunks; relevant ones are at ranks 1 and 6; the corpus has 3 relevant chunks. Compute recall@5, precision@5, RR and nDCG@5. Check with Figure 5.
          3. In Figure 4, which candidate wins with weights \(w^{\text{IsRel}} = 0\), \(w^{\text{IsSup}} = 1\), \(w^{\text{IsUse}} = 0\)? Give its score.
          4. An answer has 5 statements; 3 are supported by the context, 1 contradicts it, 1 is not in it. What is the faithfulness? Is the answer correct?
          5. HyDE with \(N = 4\) fake documents: what weight does the original query get in formula (153.2)? What happens to that weight as \(N\) grows?
          6. In Figure 2, why does the single search with the full question rank P3 fifth, and why does the hop query “Mara Ilves was born in Tolby” find it?
          Answerscheck
          1. d1: \(1/61 + 1/62 = 0.0325\); d3: \(1/63 + 1/61 = 0.0323\); d2: \(1/62 + 1/61 = 0.0325\); d4: \(1/62 = 0.0161\). d1 and d2 tie exactly (each has one rank 1 and one rank 2), then d3, then d4. 2. \(\mathrm{Recall@}5 = 1/3 = 0.333\); \(\mathrm{Precision@}5 = 1/5 = 0.2\); \(\mathrm{RR} = 1\); \(\mathrm{DCG@}5 = 1\), \(\mathrm{IDCG@}5 = 1 + 0.631 + 0.5 = 2.131\), \(\mathrm{nDCG@}5 = 0.469\). 3. d1: \(0.40 + 0.7 = 1.10\), ahead of d3 (0.78) and d2 (0.75). 4. \(F = 3/5 = 0.6\). Not necessarily: faithfulness measures support by the context, not truth; and one contradicted statement means at least part is wrong if the context is right. 5. \(1/(N+1) = 1/5 = 0.2\); as \(N\) grows the query’s weight goes to 0 and the fake documents dominate. 6. The question’s words are “population, town, inventor, Fenwick, lamp, born”; P3 shares only “population”, while P4 and P6 share “Fenwick lamp” or “population” and “town”. The hop query contains “Tolby”, which only P2 and P3 have, and P2 is already retrieved, so P3 is the best new passage.

          Key takeaways

          • Change the query before you change the index: rewriting (153.1), HyDE (153.2) and multi-query with reciprocal rank fusion (153.3) all attack vocabulary mismatch, at the cost of extra LLM calls before search.
          • Multi-hop questions need several retrieval steps, because later hops need names found in earlier ones; interleave retrieval and reasoning (153.4) and cap the steps.
          • GraphRAG builds an entity graph, finds communities (153.5) and pre-writes their summaries; it is for global questions, not fact lookup. Self-RAG (153.6, 153.7) and CRAG check retrieval quality before trusting it.
          • Evaluate retrieval and generation separately: recall@k, MRR and nDCG (153.8 to 153.10) for the retriever; faithfulness and answer relevance (153.11, 153.12) for the generator; correctness needs references.
          • Debug in order: missing content, wrong chunk, right chunk ignored, conflicting sources. The same wrong answer can need opposite fixes.

          Key formulas

          \[ \hat{\mathbf{v}} = \frac{1}{N+1}\Big[\sum_{k} f(\hat{d}_k) + f(q)\Big], \qquad \mathrm{RRF}(d) = \sum_i \frac{1}{60 + \mathrm{rank}_i(d)} \]
          \[ f(y_t, d) = p(y_t \mid x, d, y_{\lt t}) + \sum_G w^G s_t^G, \qquad \mathrm{Recall@}k = \frac{|\mathrm{Rel} \cap \mathrm{Top}_k|}{|\mathrm{Rel}|}, \qquad F = \frac{|V|}{|S|} \]

          References for this session

          1. Gao et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels
          2. Edge et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization
          3. Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
          4. Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey
          5. Gao et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels
          6. Edge et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization
          7. Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
          8. Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey
          9. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
          10. Wang, Yang and Wei (2023). Query2doc: Query Expansion with Large Language Models
          11. Zheng et al. (2023). Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
          12. Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
          13. Trivedi et al. (2022). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
          14. Yang et al. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
          15. Sarthi et al. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
          16. Yan et al. (2024). Corrective Retrieval Augmented Generation
          17. Es et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation
          18. Saad-Falcon et al. (2023). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
          19. Barnett et al. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System
          20. Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
          21. Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning