LLM & VLM courseBlog
160Track 6: LLM applications · session 18 of 20

build and evaluate a RAG system

In this lab you index 32 documents of a made-up company with BM25, a dense embedder in FAISS, reciprocal rank fusion and a cross-encoder reranker, and write 60 test questions with known answers and sources. You measure recall@k, MRR, accuracy and faithfulness, sort every failure into a category, fix the biggest one and check with a paired test whether the gain is real.

Time: 3 to 4 hours (mostly hands-on)Level: Intermediate · hands-on labBefore this: Sessions 152, 153 and 155 (RAG parts 1 and 2, evaluation of LLM systems), Python and PyTorch

What you will learn

  • How to write a RAG test set: questions in users’ words, a known answer and source for each, unanswerable near-miss questions, and relevance defined by position so it survives a new chunker
  • How BM25, dense retrieval with FAISS, reciprocal rank fusion and a cross-encoder reranker each change recall@k and MRR, per stage and per question type
  • Why fusion can push a gold chunk down and why the reranker brings it back
  • How to measure answer accuracy, abstention and faithfulness, and why faithfulness is not correctness
  • How to check a faithfulness grader against hand labels, and why a cheap rule and a small LLM judge failed
  • How to sort failures into categories and find the biggest one: here, chunks that do not say what they are about
  • How a header on every chunk raised recall@5 from 0.556 to 1.000, and how to tell real gains from noise with a paired sign test

Why this matters

A RAG system (retrieval-augmented generation) answers questions from your own documents: it searches for a few relevant pieces of text and gives them to an LLM together with the question. It is easy to build a demo that works on five questions. It is much harder to know whether it works on the next five hundred. In this lab you build a small but complete RAG system and then measure it, find out what breaks, fix the biggest problem and measure again.

Sessions 152 and 153 (session 152 and session 153) explained the parts: chunking, BM25, dense embeddings, hybrid search, reranking, and the metrics recall@k, MRR and faithfulness. Session 155 explained graders and statistics. This lab puts them together on one document set. You will:

  1. index 32 internal documents of a made-up company with BM25, a dense embedder in a FAISS index, reciprocal rank fusion and a cross-encoder reranker;
  2. write a test set of 60 questions with known answers and known source passages, including 6 questions that have no answer in the documents;
  3. measure retrieval (recall@k, MRR) and answers (accuracy, abstention, faithfulness), with three different faithfulness graders;
  4. sort every failure into a category, fix the biggest category, and check with a paired test whether the improvement is real.

The result is a lesson that you will see again in real projects. Our first system answered only 28 of 60 questions correctly. The cause was not the LLM and not the embedder. It was the chunker: most chunks of a product manual did not say which product they were about. One small change, a header line on every chunk, raised retrieval recall@5 from 0.556 to 1.000 and accuracy from 28 to 50 of 60. A second, smaller change looked like another gain (52 of 60), but the statistics say we cannot tell it apart from noise.

Setup

You need Python 3.10 or newer, PyTorch, Hugging Face transformers, NumPy and faiss-cpu. Four models play four roles. All of them are small enough for a laptop CPU.

  • Sparse retriever: BM25, written in 15 lines of Python. No model.
  • Dense retriever: sentence-transformers/all-MiniLM-L6-v2, a 6-layer BERT-style encoder that maps text to a 384-dimensional vector by mean pooling (averaging the token vectors); it truncates input at 256 tokens [Certain]. The vectors go into a FAISS IndexFlatIP, an exact inner-product index.
  • Reranker: cross-encoder/ms-marco-MiniLM-L6-v2, a 6-layer cross-encoder trained on the MS MARCO passage-ranking data. It reads the question and one passage together and outputs one relevance score [Certain].
  • Reader (generator): Qwen/Qwen2.5-1.5B-Instruct, a chat model with 1.54B parameters (28 layers, hidden size 1,536, 12 query heads and 2 key-value heads) [Certain]. It decodes greedily, so the answers are deterministic.
pip install torch transformers numpy faiss-cpu
python data.py                      # prints the size of the document set and the test set
python run.py fixed gen             # baseline: 50-word chunks, retrieval + answers
python run.py header gen            # fix 1: a header line on every chunk
python run.py section gen           # fix 2: whole sentences, never across a section
python grade.py                     # accuracy, faithfulness, failure categories, statistics

All numbers on this page come from one run of this code (torch 2.8, transformers 4.57, faiss 1.16). Retrieval takes a few seconds. The 60 answers took between 5 and 13 minutes per configuration on a laptop CPU that was shared with other heavy jobs, so treat the times as rough.

Step 1: the document set and the index

A good RAG test needs documents whose facts the model cannot know from pretraining. Otherwise a correct answer tells you nothing about retrieval. So we generate the internal knowledge base of a made-up instrument company, with a seeded random generator:

  • 20 product manuals, each with seven sections: overview, installation, specifications, error codes, maintenance, warranty and safety. The products come in families with similar names (Vela-100, Vela-200, Vela-300, Orin-3, Orin-5, Orin-7, …) and the same sentence patterns, so many chunks look alike. 12 of the manuals have test questions; the other 8 are realistic distractors.
  • 6 policies (travel, expenses, IT security, on-call, leave, lab safety), with numbers such as hotel limits and deadlines.
  • 6 incident reports, each with a date, a duration, a root cause and an owner team.

In total: 32 documents and 6,816 words. Like real manuals, only the overview names the product. Later sentences say “The device weighs 5.1 kg”. Keep that in mind; it decides the whole lab.

Show data.py: the document set and the 60 test questions
"""Synthetic internal knowledge base of a made-up company, plus a 60-question test set.
Deterministic: every value comes from random.Random(0). The facts are invented, so the
reader model cannot answer from memory: it must use retrieval."""
import random

PRODUCTS = [  # (name, kind, use)
    ("Vela-200", "flow meter", "measuring water flow in pipes up to 50 mm wide"),
    ("Vela-300", "flow meter", "measuring water flow in pipes up to 80 mm wide"),
    ("Orin-5", "pressure sensor", "monitoring air pressure in compressor lines"),
    ("Orin-7", "pressure sensor", "monitoring oil pressure in hydraulic systems"),
    ("Tamsin-X1", "data logger", "recording sensor readings in remote field sites"),
    ("Tamsin-X2", "data logger", "recording sensor readings in cold-storage rooms"),
    ("Brio-40", "pH probe", "testing the acidity of drinking water"),
    ("Brio-60", "pH probe", "testing the acidity of waste water"),
    ("Kestrel-9", "gas analyzer", "detecting carbon monoxide in boiler rooms"),
    ("Kestrel-12", "gas analyzer", "detecting methane in biogas plants"),
    ("Lumen-3", "turbidity meter", "checking how clear river water is"),
    ("Lumen-5", "turbidity meter", "checking how clear treated water is"),
]
EXTRA = [  # manuals in the index without test questions (realistic distractors)
    ("Vela-100", "flow meter", "measuring water flow in pipes up to 25 mm wide"),
    ("Orin-3", "pressure sensor", "monitoring gas pressure in small tanks"),
    ("Tamsin-X3", "data logger", "recording sensor readings in factories"),
    ("Brio-20", "pH probe", "testing the acidity of pool water"),
    ("Kestrel-6", "gas analyzer", "detecting oxygen levels in tanks"),
    ("Lumen-1", "turbidity meter", "checking how clear lab samples are"),
    ("Corvo-8", "level sensor", "measuring the water level in tanks"),
    ("Corvo-10", "level sensor", "measuring the fuel level in tanks"),
]
ERRORS = [  # (meaning, fix, key word of meaning, key word of fix)
    ("the inlet filter is blocked", "remove the filter and rinse it with clean water", "filter", "rinse"),
    ("the battery voltage is too low", "replace the battery pack", "battery", "battery"),
    ("the sensor lost its calibration", "run the two-point calibration again", "calibrat", "two-point"),
    ("the internal clock was reset", "set the date and time in the settings menu", "clock", "time"),
    ("the memory card is full", "copy the logs to a computer and erase the card", "card", "erase"),
    ("the cable to the sensor head is loose", "reconnect the cable and tighten the screw collar", "cable", "cable"),
    ("the housing temperature is above the limit", "move the device out of direct sunlight", "temperature", "sunlight"),
    ("the firmware image is corrupted", "reinstall the firmware with the service tool", "firmware", "firmware"),
]
PARTS = ["O-ring seal", "reference electrode", "membrane cap", "desiccant pack", "light source", "pump tube"]


def build(seed=0):
    r = random.Random(seed)
    rx = random.Random(seed + 1)  # values for the extra manuals that have no questions
    docs, qs = [], []

    def add_q(qtype, q, doc, fact, keys):
        qs.append(dict(id=len(qs), type=qtype, q=q, doc=doc, fact=fact, keys=keys))

    # ---------------- 12 product manuals ----------------
    used_codes = set()
    for pi, (name, kind, use) in enumerate(PRODUCTS + EXTRA):
        r0 = r
        r = r0 if pi < len(PRODUCTS) else rx
        w = round(r.uniform(0.4, 6.0), 1)
        tmin = r.choice([-30, -20, -10, 0])
        tmax = r.choice([40, 45, 50, 55, 60, 70])
        acc = r.choice([0.2, 0.5, 1.0, 1.5, 2.0])
        bat = r.choice([18, 24, 36, 48, 72, 96, 120])
        ip = r.choice([54, 65, 66, 67, 68])
        cal = r.choice([3, 6, 9, 12, 18])
        part = r.choice(PARTS)
        rep = r.choice([12, 18, 24, 36])
        war = r.choice([12, 24, 36, 60])
        year = r.choice([2019, 2020, 2021, 2022, 2023])
        errs = []
        for e in r.sample(range(len(ERRORS)), 4):
            while True:
                code = "E-%d" % r.randint(1000, 9999)
                if code not in used_codes:
                    used_codes.add(code); break
            errs.append((code, e))
        sec = []
        sec.append(("Overview",
            f"The {name} is a {kind} for {use}. It was first sold in {year}. "
            f"This manual explains how to install, use and maintain the device. "
            f"Read the safety notes before you open the housing. Keep this manual near the device."))
        mount = r.choice(["a wall bracket", "a DIN rail clip", "a pipe clamp", "a tripod"])
        cable = r.choice([2, 3, 5, 10])
        sec.append(("Installation",
            f"Mount the device on {mount} in a place with little vibration. "
            f"Connect the sensor cable ({cable} m long, included) before you switch the device on. "
            f"Wait ten minutes after the first start so that the readings become stable."))
        sec.append(("Specifications",
            f"The device weighs {w} kg without the mounting kit. "
            f"It works at temperatures from {tmin} to {tmax} °C. "
            f"The measurement accuracy is ±{acc}% of the reading. "
            f"On a full charge the battery lasts {bat} hours of continuous logging. "
            f"The housing is rated IP{ip}."))
        etext = " ".join(f"Error {c}: {ERRORS[e][0]}. To fix it, {ERRORS[e][1]}." for c, e in errs)
        sec.append(("Error codes", "The display shows an error code when something is wrong. " + etext))
        sec.append(("Maintenance",
            f"Calibrate the sensor every {cal} months. "
            f"Replace the {part} every {rep} months, or sooner if it is damaged. "
            f"Clean the outside with a soft, dry cloth. Do not use solvents."))
        sec.append(("Warranty and support",
            f"The standard warranty is {war} months from the date of delivery. "
            f"The warranty does not cover damage from wrong installation. "
            f"Contact support with the serial number printed on the back plate."))
        sec.append(("Safety",
            "Switch the device off and remove the battery before you open the housing. "
            "Do not use the device in places with explosive gas unless it carries the ATEX label. "
            "Dispose of old batteries as electronic waste."))
        d = len(docs)
        docs.append(dict(id=d, title=f"{name} {kind} user manual", sections=sec))
        r = r0
        if pi >= len(PRODUCTS):
            continue
        # two spec-style questions and one error-code question per product
        pick = r.sample(["w", "tmax", "bat", "acc", "cal", "war"], 2)
        for p in pick:
            if p == "w":
                add_q("spec", f"How heavy is the {name}?", d, f"weighs {w} kg", [f"{w}"])
            if p == "tmax":
                add_q("spec", f"What is the highest temperature the {name} can work at?", d, f"to {tmax} °C", [f"{tmax}"])
            if p == "bat":
                add_q("spec", f"How long does the {name} run on one battery charge?", d, f"lasts {bat} hours", [f"{bat}"])
            if p == "acc":
                add_q("spec", f"How accurate are the readings of the {name}?", d, f"±{acc}%", [f"{acc}"])
            if p == "cal":
                add_q("spec", f"How often should the {name} be calibrated?", d, f"every {cal} months", [f"{cal}"])
            if p == "war":
                add_q("spec", f"How long is the warranty on the {name}?", d, f"warranty is {war} months", [f"{war}"])
        code, e = errs[r.randrange(4)]
        if pi % 2 == 0:
            add_q("code", f"What does error {code} mean?", d, f"Error {code}: {ERRORS[e][0]}", [ERRORS[e][2]])
        else:
            add_q("code", f"My {name} shows {code}. How do I fix it?", d, f"Error {code}: {ERRORS[e][0]}. To fix it, {ERRORS[e][1]}",
                  [ERRORS[e][3]])

    # ---------------- 6 policy documents ----------------
    hotel = r.choice([140, 160, 180, 200]); pdl = r.choice([45, 55, 65]); pdb = r.choice([60, 70, 80]); lead = r.choice([10, 14, 21])
    docs.append(dict(id=len(docs), title="Travel policy", sections=[
        ("Booking", f"Book flights and hotels through the travel portal at least {lead} days before the trip. "
                    "Economy class is the default for all flights shorter than six hours."),
        ("Hotels", f"The hotel limit is {hotel} euros per night in most cities. In London and Paris the limit is {hotel + 60} euros per night."),
        ("Meals", f"The daily meal allowance is {pdl} euros in Lisbon and {pdb} euros in Berlin. Alcohol is never refunded.")]))
    d = len(docs) - 1
    add_q("policy", "How many euros per night can I spend on a hotel in Paris?", d, f"limit is {hotel + 60} euros", [f"{hotel + 60}"])
    add_q("policy", "What is the daily food allowance for a trip to Berlin?", d, f"{pdb} euros in Berlin", [f"{pdb}"])
    add_q("policy", "How far ahead must business trips be booked?", d, f"at least {lead} days", [f"{lead}"])
    rec = r.choice([25, 30, 50]); appr = r.choice([500, 1000, 2000]); sub = r.choice([30, 45, 60])
    docs.append(dict(id=len(docs), title="Expense policy", sections=[
        ("Receipts", f"Keep a receipt for every expense above {rec} euros. Photos of receipts are accepted."),
        ("Approval", f"Any single expense above {appr} euros needs written approval from a director before you pay."),
        ("Deadlines", f"Submit expense reports within {sub} days of the expense. Late reports are paid in the next cycle.")]))
    d = len(docs) - 1
    add_q("policy", "Above what amount do I need to keep a receipt?", d, f"above {rec} euros", [f"{rec}"])
    add_q("policy", "When does a purchase need sign-off from a director?", d, f"above {appr} euros", [f"{appr}"])
    pw = r.choice([60, 90, 180]); lock = r.choice([5, 10, 15]); pat = r.choice([7, 14, 30])
    docs.append(dict(id=len(docs), title="IT security policy", sections=[
        ("Passwords", f"Change your password every {pw} days. Passwords must have at least 14 characters."),
        ("Devices", f"Laptops lock the screen after {lock} minutes without input. All disks are encrypted."),
        ("Updates", f"Security patches must be installed within {pat} days of release.")]))
    d = len(docs) - 1
    add_q("policy", "How often do we have to change our passwords?", d, f"every {pw} days", [f"{pw}"])
    add_q("policy", "After how many idle minutes does a laptop screen lock?", d, f"after {lock} minutes", [f"{lock}"])
    ack = r.choice([5, 10, 15]); pay = r.choice([150, 200, 250]); rot = r.choice([4, 6, 8])
    docs.append(dict(id=len(docs), title="On-call policy", sections=[
        ("Response", f"The on-call engineer must acknowledge a page within {ack} minutes, day or night."),
        ("Pay", f"Each week of on-call duty is paid {pay} euros on top of the salary."),
        ("Rotation", f"Each team rotates on-call duty among at least {rot} engineers.")]))
    d = len(docs) - 1
    add_q("policy", "How quickly must the on-call engineer respond to a page?", d, f"within {ack} minutes", [f"{ack}"])
    add_q("policy", "What is the extra pay for one week of on-call?", d, f"paid {pay} euros", [f"{pay}"])
    vac = r.choice([25, 27, 30]); carry = r.choice([3, 5, 10]); par = r.choice([10, 15, 20])
    docs.append(dict(id=len(docs), title="Leave policy", sections=[
        ("Vacation", f"Full-time staff get {vac} days of paid vacation per year. Up to {carry} unused days can move to the next year."),
        ("Parental leave", f"The second parent gets {par} days of paid leave after a birth or adoption.")]))
    d = len(docs) - 1
    add_q("policy", "How many paid vacation days does a full-time employee get each year?", d, f"{vac} days of paid vacation", [f"{vac}"])
    add_q("policy", "How many vacation days can be carried over into next year?", d, f"Up to {carry} unused days", [f"{carry}"])
    eye = r.choice([1, 2, 4]); glove = r.choice(["nitrile", "neoprene", "butyl"])
    docs.append(dict(id=len(docs), title="Lab safety rules", sections=[
        ("Equipment", f"Test every eye-wash station every {eye} weeks and write the date on its tag."),
        ("Gloves", f"Wear {glove} gloves when you handle acids or solvents.")]))
    d = len(docs) - 1
    add_q("policy", "Which kind of gloves are required for handling acids?", d, f"Wear {glove} gloves", [glove])

    # ---------------- 6 incident reports ----------------
    CAUSES = ["an expired TLS certificate on the upload server", "a database migration that locked the readings table",
              "a firmware update that reset the clock on every logger", "a full disk on the report server",
              "a wrong DNS record after a network change", "a memory leak in the alarm service"]
    TEAMS = ["Platform", "Data", "Firmware", "Reporting", "Network", "Alerts"]
    MONTHS = ["January", "February", "March", "April", "May", "June", "July", "September", "October", "November"]
    for i in range(6):
        inc = "INC-%d" % r.randint(2000, 2999)
        day = r.randint(2, 27); mon = MONTHS[r.randrange(len(MONTHS))]
        dur = r.choice([35, 50, 75, 90, 120, 160, 210])
        cust = r.choice([12, 40, 85, 130, 300])
        docs.append(dict(id=len(docs), title=f"Incident report {inc}", sections=[
            ("Summary", f"On {day} {mon} 2025 customers could not see new readings in the web dashboard. "
                        f"The problem lasted {dur} minutes and affected {cust} customers."),
            ("Root cause", f"The root cause was {CAUSES[i]}. Monitoring did not alert because the check only tested the login page."),
            ("Actions", f"The {TEAMS[i]} team owns the follow-up actions. A new end-to-end check now uploads a test reading every five minutes.")]))
        d = len(docs) - 1
        if i % 2 == 0:
            add_q("incident", f"What was the root cause of {inc}?", d, f"root cause was {CAUSES[i]}", [CAUSES[i].split()[-1]])
        else:
            add_q("incident", f"How long did the dashboard outage on {day} {mon} 2025 last?", d, f"lasted {dur} minutes", [f"{dur}"])

    # ---------------- 6 questions without an answer in the documents ----------------
    for q in ["How heavy is the Vela-500?", "What is the daily meal allowance in Tokyo?",
              "How long is the warranty on the Kestrel-20 gas analyzer?", "Who is the CEO of the company?",
              "What does error E-0000 mean?", "How many days of sick leave do staff get per year?"]:
        add_q("none", q, None, None, [])
    return docs, qs


def doc_text(doc, headings=True):
    return " ".join(t for _, t in doc["sections"])


if __name__ == "__main__":
    docs, qs = build()
    from collections import Counter
    print(len(docs), "docs", sum(len(doc_text(d).split()) for d in docs), "words")
    print(len(qs), "questions", Counter(q["type"] for q in qs))
    for q in qs[:6]: print(q)

Chunking

The baseline chunker does what many tutorials do: join the sections of a document and cut the text into windows of 50 words, with no overlap. This gives 159 chunks of 42.9 words on average (the last chunk of each document is shorter), or 56.6 WordPiece tokens on average and 82 at most, far below the embedder’s limit of 256.

Sparse retrieval: BM25

BM25 scores a chunk \(d\) for a question \(q\) by the question words it contains. Rare words count more, repeated words count less and less, and long chunks are penalized a little:

\[ s_{\text{BM25}}(q, d) = \sum_{t \in q} \ln\!\left(1 + \frac{N - n_t + 0.5}{n_t + 0.5}\right) \cdot \frac{f(t, d)\,(k_1 + 1)}{f(t, d) + k_1\left(1 - b + b\,\dfrac{|d|}{\text{avgdl}}\right)} \tag{160.1} \]
Read the symbols
\(t \in q\)
each distinct word of the question (we lower-case and split on anything that is not a letter, digit or decimal point)
\(N,\ n_t\)
number of chunks (159), and number of chunks that contain word \(t\)
\(f(t, d)\)
how often word \(t\) occurs in chunk \(d\)
\(|d|,\ \text{avgdl}\)
length of chunk \(d\) in words, and the average length over all chunks
\(k_1 = 1.2,\ b = 0.75\)
the usual settings: \(k_1\) controls how fast repeated words stop adding score, \(b\) how strongly long chunks are penalized

The first factor is the IDF (inverse document frequency) of the word. A code like “E-9251” is split into “e” and “9251”; “9251” occurs in one chunk only, so its IDF is high, and BM25 finds the chunk at once. That is exactly what dense embedders are bad at.

Dense retrieval: embeddings and FAISS

The embedder \(f\) maps the question and every chunk to unit vectors. The score is their dot product, which for unit vectors equals the cosine similarity:

\[ s_{\text{dense}}(q, d) = f(q)^\top f(d), \qquad \lVert f(\cdot) \rVert = 1 \tag{160.2} \]

Dense retrieval matches meaning: “How heavy is the Vela-200?” matches “The device weighs 5.1 kg” although they share no word except “the”. FAISS stores the 159 vectors of size 384 and returns the top 50 by inner product. With 159 chunks an exact search is instant; for millions of chunks you would use an approximate index (session 151).

Figure 1. One question through the real pipeline of this lab. Each step shows what the system actually retrieved or wrote in our run. The chunk that holds the answer (the gold chunk) has a green border. Switch the chunking to see the same question before and after the fixes of Step 7.
How to read Figure 1

Start with “weight of Vela-200” and baseline chunks. Step through: BM25 and dense retrieval both return the overviews of the Vela-200, Vela-300 and Vela-100 (the overview is the only chunk of a manual that names its product), and the gold chunk with “weighs 5.1 kg” has no product name in it. After reranking the gold chunk is in the top 5, but next to a Vela-100 chunk that also says “The device weighs 1.2 kg”. The reader picks 1.2 kg: wrong, and not supported, because no passage says that 1.2 kg belongs to the Vela-200. Now switch to “+ headers”: every chunk starts with “Vela-200 flow meter user manual > Installation:”, the gold chunk becomes rank 1 and the answer is right. Try “error E-9251” with headers: BM25 ranks the gold chunk 1st, dense only 18th, and fusion puts it 8th; the reranker brings it back to 1st (Step 3 explains why). Try the Berlin question: the gold chunk is rank 1 in every setting. In the baseline the reader copies the whole sentence and is right; with the other two chunkers it answers 45 euros, the Lisbon number from the same sentence. The last question has no answer; watch what the reader does.

Step 2: write the test set

The test set is the most important part of the lab. If it is wrong, every number after it is wrong. Each of our 60 questions has four fields: the question, its type, the gold document and the gold fact, an exact string from the document such as “weighs 5.1 kg”, plus a short key (“5.1”) that a correct answer must contain.

TypeCountExampleWhat it tests
spec24How heavy is the Orin-5?Paraphrase (“heavy” vs “weighs”), and telling similar products apart
code12My Lumen-5 shows E-5500. How do I fix it?Exact identifiers, where BM25 should win
policy12When does a purchase need sign-off from a director?Paraphrase (“sign-off” vs “written approval”)
incident6What was the root cause of INC-2506?Identifiers and dates
none6How heavy is the Vela-500?No answer exists; the system must say so

Three rules make a test set useful. Write questions the way users ask, not by copying sentences from the documents; otherwise BM25 looks perfect. Record the source, not only the answer, so you can score retrieval separately from generation. Add unanswerable questions, especially near misses like “Vela-500” (the product does not exist, but Vela-100, -200 and -300 do). Systems that always answer look good on a test without such questions.

Which chunk is “relevant”? The chunks change when you change the chunker, so we do not store chunk ids in the test set. We store the character position of the key inside the gold document and call a chunk relevant if it covers that position:

\[ \mathrm{Rel}(q) = \{\, c \;:\; \operatorname{doc}(c) = d_q \ \text{ and } \ [s_c, e_c) \cap [s_q, e_q) \neq \emptyset \,\}, \qquad \mathrm{Recall@}k(q) = \mathbb{1}\big[\mathrm{Rel}(q) \cap \mathrm{Top}_k(q) \neq \emptyset\big] \tag{160.3} \]
Read the symbols
\(d_q\)
the gold document of question \(q\)
\([s_c, e_c)\)
the character span of chunk \(c\) inside its document
\([s_q, e_q)\)
the character span of the answer key (for example “5.1”) inside the gold fact
\(\mathrm{Top}_k(q)\)
the first \(k\) chunks returned for \(q\)
\(\mathbb{1}[\cdot]\)
1 if the condition is true, else 0; we average over the 54 answerable questions

With one relevant chunk per question, recall@k is the same as the hit rate: “is the answer chunk in the top \(k\)?”. The second metric rewards putting it high:

\[ \mathrm{MRR@}20 = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q}, \qquad \frac{1}{\mathrm{rank}_q} = 0 \ \text{ if } \ \mathrm{rank}_q \gt 20 \tag{160.4} \]

Here \(\mathrm{rank}_q\) is the position of the first relevant chunk. We cut at 20 because the reranker only sees the first 20 fused candidates.

Algorithm 1: build a RAG test set
  1. Choose documents whose facts the reader cannot know (private, recent or synthetic).
  2. For each question type you care about, write questions in the users’ words; for each one store the gold document, the exact gold fact and a short answer key.
  3. Add 10% or more questions with no answer, including near misses (wrong product, wrong city, unknown code).
  4. Locate each key by character position in its document; define relevant chunks by span overlap (160.3), so the same test set works for every chunker.
  5. Read 10 questions and their gold facts by hand to check that the answer is really there and unique.

Step 3: hybrid search and a reranker

BM25 and dense retrieval fail on different questions, so we combine their top-50 lists with reciprocal rank fusion (RRF). RRF ignores the raw scores, which live on different scales, and uses only the ranks:

\[ s_{\text{RRF}}(d) = \frac{w}{k + \mathrm{rank}_{\text{BM25}}(d)} + \frac{1 - w}{k + \mathrm{rank}_{\text{dense}}(d)} \tag{160.5} \]
Read the symbols
\(\mathrm{rank}_{\text{BM25}}(d)\)
position of chunk \(d\) in the BM25 list (1 = best); if \(d\) is not in the top 50, that term is 0
\(k\)
a smoothing constant; \(k = 60\) is the value proposed by Cormack et al. (2009) and the common default [Certain]
\(w\)
weight of the BM25 list; plain RRF is \(w = 0.5\) (both lists equal), which gives the same order as the unweighted sum

Then the cross-encoder rescores the first 20 fused chunks. A cross-encoder reads question and chunk together through all its layers, so it can check that the chunk is about the right product and the right code. That is too slow for 159 chunks per question in a large system, but cheap for 20. The reader gets the top 5 after reranking.

Worked example: why fusion alone is not enough (question “What does error E-9251 mean?”, with headers)

The gold chunk is rank 1 in BM25 (the token “9251” is unique) but rank 18 in the dense list. Its RRF score with \(k = 60\) (the sum form, both weights 1) is \(1/61 + 1/78 = 0.01639 + 0.01282 = 0.02921\).

Another error-code chunk of the Vela-200 is rank 2 in BM25 and rank 6 in the dense list: \(1/62 + 1/66 = 0.03128\). It wins, although it does not contain the code. In our run, fusion puts the gold chunk at rank 8, behind seven chunks that are “quite good” in both lists. RRF rewards agreement, and one list that is very sure is outvoted.

The reranker gives the gold chunk a score of 4.80 and the next chunk 2.24. Final rank: 1. Fusion is a good way to build the candidate list; the reranker decides the order.

Figure 2 lets you change \(k\) and \(w\) and recomputes the fused ranking for all 54 answerable questions in your browser, from the real top-50 lists of our run.

BM25 onlydense onlyfused (your \(k\), \(w\))
Figure 2. Hybrid search explorer. Recall@5 by question type for BM25 alone, dense retrieval alone and the RRF fusion (160.5) of their top-50 lists. The fusion is recomputed live from the stored lists of our run.
How to read Figure 2

Choose “+ headers” first. With \(k = 60\) and \(w = 0.5\) you get our pipeline’s fused list: recall@5 of 0.963 overall. Look at the pairs of bars: BM25 finds every error code (1.00) but only 0.46 of the spec questions; dense finds 0.96 of the spec questions but only 0.42 of the codes. Fusion gets 1.00 on spec and 0.92 on codes: it keeps most of the strengths of both. Move \(w\) to 0 or 1 and the fused bars become the dense or BM25 bars. Move \(k\) down to 1: now the first places of each list dominate; spec drops to 0.96 and policy rises to 1.00, so the total stays at 0.963. A full sweep finds settings that reach 1.000 (for example \(k = 6\), \(w = 0.6\)), but choosing \(k\) and \(w\) on the test set is tuning on the test set, and with 54 questions one question is 0.019. Now switch to baseline chunks: no setting of \(k\) and \(w\) brings the spec questions above 0.29, because the gold chunks there do not contain the product name. Fusion cannot find what neither retriever can find.

Step 4: measure retrieval

We run the 60 questions through the baseline system and compute recall@k (160.3) and MRR@20 (160.4) at four points of the pipeline: BM25 alone, dense alone, after fusion and after reranking. Recall is computed over the 54 answerable questions.

Baseline (50-word chunks)R@1R@5R@10R@20MRR@20
BM250.3890.4440.5190.5190.424
Dense (MiniLM)0.2960.4810.6300.7590.382
Hybrid (RRF, \(k = 60\))0.2960.4810.5560.7040.380
Hybrid + reranker0.4630.5560.6850.7040.499

These numbers are bad, and they are bad in an informative way. The reranker adds a lot at rank 1 (0.296 to 0.463), so the components work. But even at rank 20 only about 70% of the answer chunks are found. Figure 3 splits the curves by question type and shows where the problem is.

BM25densehybrid (RRF)hybrid + reranker
Figure 3. Recall@k (160.3) for \(k = 1\) to 20 at four points of the pipeline, from the real ranks of our run. Choose the chunking and the question type; the slider marks one value of \(k\).
How to read Figure 3

With baseline chunks, click “spec”: BM25 finds none of the 24 answer chunks in its top 5 and only 3 in its top 20, dense finds 7 of 24 at \(k = 5\), and after reranking only 4 of 24 (0.167) are in the top 5. Click “code”: BM25 is good (0.67 at \(k = 5\)), dense is weak (0.33). Click “policy”: almost everything is found; policy documents are short, so each chunk is almost a whole document and names its topic. The problem is specific: chunks in the middle of a manual. Then switch to “+ headers”: the reranked curve is at 1.00 from \(k = 3\) on for every type. Finally “+ sentence chunks”: recall@1 rises from 0.907 to 0.981.

Step 5: measure the answers

Retrieval metrics say whether the right text reached the reader. Answer metrics say whether the user got the right answer. We use three, each with a precise definition.

Accuracy. For an answerable question, the answer is correct if it contains the key (numbers must match as whole numbers, so “12” does not match “120”) and is not a refusal. For an unanswerable question, the answer is correct if it is the refusal “Not in the documents.” (an abstention):

\[ \mathrm{Acc} = \frac{1}{|Q|}\left[\ \sum_{q \in Q_{\text{ans}}} \mathbb{1}\big[\text{key}_q \in a_q \ \wedge\ a_q \neq \varnothing\big] \;+\; \sum_{q \in Q_{\text{none}}} \mathbb{1}\big[a_q = \varnothing\big] \right] \tag{160.6} \]
Read the symbols
\(Q_{\text{ans}},\ Q_{\text{none}}\)
the 54 answerable and the 6 unanswerable questions
\(a_q\)
the reader’s answer; \(a_q = \varnothing\) means it abstained
\(\text{key}_q \in a_q\)
the answer key occurs in the answer text

Faithfulness. An answer is faithful if every fact in it is stated in the five passages the reader saw. This is the RAGAS definition (Es et al., 2023) with one claim per answer, since our answers are one sentence:

\[ F = \frac{|\{\text{claims of the answer supported by the context}\}|}{|\{\text{claims of the answer}\}|}, \qquad \text{an abstention counts as faithful} \tag{160.7} \]

Faithfulness is not correctness. A faithful answer can be wrong (the context itself was wrong or about another product), and an unfaithful answer can be right by luck. The \(2\times 2\) table of Figure 4 makes this concrete.

“Supported” needs a grader. We compare three:

  1. Reference labels. A strict rule: the passage that contains the answer’s value must also name the entity of the question (product, error code, incident id or date). For an error code, the value must be in the text that belongs to that code. We then read all 180 answers and corrected the rule in 5 cases (for example “45 euros” for Berlin, where the sentence gives 45 for Lisbon and 70 for Berlin). These are our ground truth.
  2. A number rule. Faithful if every number in the answer appears somewhere in the context. This is cheap and popular.
  3. An LLM judge. The same Qwen2.5-1.5B model reads context, question and answer and answers “yes” or “no” to “is every fact in the answer stated in the context?”; we take \(p(\text{yes}) \gt 0.5\).
Figure 4. Correctness against faithfulness for the 60 answers of each configuration, with three different faithfulness graders. Click a cell to read its questions and the reader’s real answers.
How to read Figure 4

Start with baseline chunks and reference labels. 24 answers are correct and faithful; 28 are wrong and unfaithful: the reader copied a number from a chunk of another product, because no chunk said which product it was about. Click the cell “correct, not faithful”: 4 answers were right by luck, for example 40 °C for the Tamsin-X2 taken from a chunk that does not name it. Now click “number rule”: it calls 59 of 60 answers faithful, because the wrong numbers are in the context. Click “LLM judge”: it says “yes” to 52 answers, including all 32 unfaithful non-refusals, and “no” to all 7 refusals, against its instructions. Its agreement with the reference labels is 20 of 60, worse than always answering “unfaithful” (32 of 60). With headers, the judge agrees on 45 of 60, but it still says “yes” to all 5 unfaithful answers. We did not run the judge on the sentence chunks.

For the baseline, the answer metrics are: accuracy 28 of 60 (0.467); 24 of 54 answerable questions correct; 4 of 6 unanswerable questions correctly refused; 28 of 60 answers faithful by the reference labels (21 of the 53 that were not refusals).

How sure are we about 0.467 with only 60 questions? The Wilson interval gives a 95% range for a proportion; it behaves better than \(\hat{p} \pm 1.96\sqrt{\hat{p}(1-\hat{p})/n}\) for small \(n\):

\[ \frac{\hat{p} + \dfrac{z^2}{2n}}{1 + \dfrac{z^2}{n}} \;\pm\; \frac{z}{1 + \dfrac{z^2}{n}} \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}} \tag{160.8} \]
Read the symbols
\(\hat{p}\)
the measured accuracy (28/60 = 0.467)
\(n\)
the number of questions (60)
\(z = 1.96\)
the normal quantile for a 95% interval
Worked example: the interval for 28 of 60

\(\hat{p} = 0.4667\), \(z^2/n = 3.8416/60 = 0.0640\). Center: \((0.4667 + 0.0320)/1.0640 = 0.4687\). Half-width: \((1.96/1.0640)\sqrt{0.4667 \cdot 0.5333/60 + 3.8416/14400} = 1.842 \times 0.0664 = 0.1224\). Interval: [0.346, 0.591].

The interval is 24 points wide. With 60 questions you can see a big change, like the one in Step 7, but not a change of 3 points. For that you need several hundred questions, or a paired test on the same questions.

Step 6: find the biggest failure mode

Averages do not tell you what to fix. Categories do. We give every question of the baseline exactly one label, in this order:

Algorithm 2: sort every answer into one failure category
  1. If the answer is correct (160.6): ok.
  2. Else, if the question has no answer in the documents: answered anyway (Barnett et al. call this “missing content”).
  3. Else, if no relevant chunk (160.3) is in the 5 passages the reader saw: missed by retrieval (“missed the top ranked documents”).
  4. Else, if the reader said “Not in the documents.”: refused.
  5. Else: misread (the answer was in the context, but the reader used the wrong part; “not extracted”).
  6. Count the categories; read 3 to 5 examples of the biggest one and write down the cause in one sentence.

Barnett et al. (2024) list seven failure points from three real RAG projects: missing content, missed the top ranked documents, not in context, not extracted, wrong format, incorrect specificity and incomplete [Certain]. Our five categories are the first four plus “ok”, because our answers are short facts.

okmissed by retrievalmisreadrefusedanswered anyway
Figure 5. Failure categories (Algorithm 2) for the 60 questions in each configuration. Click a segment to read its questions and the reader’s answers.
How to read Figure 5

The top bar is the baseline. Red, “missed by retrieval”, is the biggest failure: 22 of 60 questions, and 19 of the 22 are spec questions. Click it and read the answers: “The warranty on the Vela-200 is 60 months” (the 60 months belong to the Vela-300), “The Lumen-3 is 5.1 kg” (from another manual). The second bar is after adding headers: the red segment is gone. What is left belongs to the reader: 5 refusals with the right chunk at rank 1, and 4 misreads, 3 of them error codes whose fix sentence was cut into the next chunk. The third bar uses whole sentences inside sections: those 3 are fixed, but one new cut appears (a sentence chunker can still put “Error E-3864: the memory card is full.” and “To fix it, …” into different chunks), and new misreads appear (twice the reader took 45 °C from a sibling product’s chunk at rank 3 although the right chunk was at rank 1).

The diagnosis in one sentence: a chunk from the middle of a manual does not say which product it describes, so neither retriever can match “Vela-200” to it, and when such a chunk reaches the reader, the reader cannot tell the Vela-200 from the Vela-300. It also explains the 28 wrong-and-unfaithful answers of Figure 4.

Step 7: fix it and measure again

Fix 1: a header on every chunk

We keep the same 159 chunks and the same boundaries, and add one line in front of each: the document title and the section where the chunk starts. The new chunk text is used for BM25, for the embedder, for the reranker and for the reader:

\[ \tilde{c} \;=\; \text{title}(d) \;\oplus\; \text{“ > ”} \;\oplus\; \text{section}(c) \;\oplus\; \text{“: ”} \;\oplus\; c \tag{160.9} \]

Here \(\oplus\) means string concatenation. For example: “Vela-200 flow meter user manual > Installation: the device on a DIN rail clip … The device weighs 5.1 kg …”. The header costs 12 WordPiece tokens per chunk on average (56.6 to 68.7) and 61 more prompt tokens per question for the reader (430.6 to 491.3). This idea is used widely; Anthropic’s “contextual retrieval” goes further and lets an LLM write 50 to 100 tokens of context for each chunk, and reports that it cut the top-20 retrieval failure rate from 5.7% to 3.7% with contextual embeddings, to 2.9% with contextual BM25 added, and to 1.9% with a reranker on top [Certain].

Hybrid + rerankerR@1R@5MRR@20AccuracyFaithful
Baseline (50-word chunks)0.4630.5560.49928/6028/60
+ headers (160.9)0.9071.0000.95150/6055/60
+ sentence chunks in sections0.9811.0000.99152/6054/60

With headers, BM25 recall@5 rises from 0.444 to 0.741 and dense from 0.481 to 0.833, and after reranking every answer chunk is in the top 3. Accuracy goes from 28 to 50 of 60 (Wilson interval [0.720, 0.907]), and spec questions from 2 of 24 to 24 of 24.

Is it real? A paired test

Both systems answered the same 60 questions, so we compare them question by question. Only the questions where they disagree carry information. Let \(b\) be the number of questions the new system fixed and \(c\) the number it broke. If the two systems were equally good, each disagreement would go either way with probability 1/2. The exact two-sided sign test (McNemar’s test for small counts) gives

\[ p = \min\!\left(1,\ 2 \sum_{j=0}^{\min(b,c)} \binom{b+c}{j} \left(\tfrac{1}{2}\right)^{b+c}\right) \tag{160.10} \]
Read the symbols
\(b\)
questions wrong before and right after
\(c\)
questions right before and wrong after
\(p\)
the probability of a split at least this uneven if the two systems were equally good
Worked example: fix 1 and fix 2

Fix 1 (baseline to headers): \(b = 28\), \(c = 6\). Then \(p = 2 \sum_{j=0}^{6} \binom{34}{j} / 2^{34} = 0.000195\). The improvement is real.

Look at the 6 broken questions too. With headers the reader refused 4 of them (“Not in the documents.”, with the right chunk at rank 1) and misread 2. Retrieval got better or stayed the same for all 6, and 2 of them had been right only by luck in the baseline (correct but not faithful). A small reader is sensitive to small changes of its prompt.

Fix 2 (headers to whole sentences inside sections, 194 chunks of 35.1 words on average): \(b = 6\), \(c = 4\), so \(p = 2 \sum_{j=0}^{4} \binom{10}{j} / 2^{10} = 0.754\). The accuracy went from 50 to 52, but a split of 6 to 4 happens easily by chance. We cannot claim that fix 2 helps answers, although it clearly helps retrieval at rank 1 (0.907 to 0.981).

What to fix next

After fix 1, retrieval is no longer the problem: recall@5 is 1.000. The remaining 10 errors are reader errors: 5 refusals with the right chunk at rank 1, 4 misreads and 1 answer to the unanswerable Vela-500 question (“3.0 kg”, copied from the Vela-300). The next fixes are on the generation side: a larger reader, an instruction that lists the products in the passages, or putting only the top 3 passages in the prompt. Each one must be measured the same way.

The full program

This is the complete run.py that produced every retrieval number and every answer on this page. It needs data.py from Step 1.

import json, math, os, re, sys, time
os.environ.setdefault("KMP_DUPLICATE_LIB_OK", "TRUE")   # torch + faiss on macOS, see Setup
from collections import Counter
import numpy as np, torch, faiss
from transformers import AutoTokenizer, AutoModel, AutoModelForSequenceClassification, AutoModelForCausalLM
from data import build
torch.manual_seed(0); torch.set_num_threads(8)
MODE = sys.argv[1] if len(sys.argv) > 1 else "fixed"     # fixed | header | section
GEN = len(sys.argv) > 2 and sys.argv[2] == "gen"
W = 50
docs, qs = build()

# ---------- 1. chunking ----------
def chunk(docs, mode):
    out = []
    def mk(d, s, e, sec):
        body = d["text"][s:e]
        head = f"{d['title']} > {sec}: " if mode in ("header", "section") else ""
        return dict(id=len(out), doc=d["id"], s=s, e=e, text=head + body, body=body)
    for d in docs:
        text, starts = "", []
        for name, t in d["sections"]:
            if text: text += " "
            starts.append((len(text), name)); text += t
        d["text"] = text
        if mode == "section":   # whole sentences, never across a section, at most W words
            ends = [p for p, _ in starts[1:]] + [len(text) + 1]
            for (p0, sec), p1 in zip(starts, ends):
                sents = [(m.start() + p0, m.end() + p0) for m in re.finditer(r"\S.*?[.!?](?=\s|$)", text[p0:p1 - 1])]
                cur = []
                for a, b in sents:
                    if cur and len(text[cur[0][0]:b].split()) > W:
                        out.append(mk(d, cur[0][0], cur[-1][1], sec)); cur = []
                    cur.append((a, b))
                if cur: out.append(mk(d, cur[0][0], cur[-1][1], sec))
            continue
        words = [(m.start(), m.end()) for m in re.finditer(r"\S+", text)]
        for i in range(0, len(words), W):
            s, e = words[i][0], words[min(i + W, len(words)) - 1][1]
            sec = [n for p, n in starts if p <= s][-1]
            out.append(mk(d, s, e, sec))
    return out
chunks = chunk(docs, MODE)
for q in qs:  # relevant chunks: the chunk(s) that hold the answer value of the gold fact
    q["rel"] = []
    if q["doc"] is None: continue
    t = docs[q["doc"]]["text"]; fs = t.index(q["fact"]); ks = fs + q["fact"].lower().index(q["keys"][0].lower())
    ke = ks + len(q["keys"][0])
    q["rel"] = [c["id"] for c in chunks if c["doc"] == q["doc"] and c["s"] < ke and c["e"] > ks]
    assert q["rel"], q

# ---------- 2. BM25 ----------
tok = lambda s: re.findall(r"[a-z0-9]+(?:\.[0-9]+)?", s.lower())
D = [tok(c["text"]) for c in chunks]; N = len(D); avg = sum(map(len, D)) / N
df = Counter(w for d in D for w in set(d)); TF = [Counter(d) for d in D]
def bm25(q, k1=1.2, b=0.75):
    sc = np.zeros(N)
    for w in dict.fromkeys(tok(q)):   # fixed order: sums are reproducible across runs
        if w not in df: continue
        idf = math.log(1 + (N - df[w] + 0.5) / (df[w] + 0.5))
        for i in range(N):
            f = TF[i][w]
            if f: sc[i] += idf * f * (k1 + 1) / (f + k1 * (1 - b + b * len(D[i]) / avg))
    return sc

# ---------- 3. dense ----------
EP = "sentence-transformers/all-MiniLM-L6-v2"
etok = AutoTokenizer.from_pretrained(EP); emod = AutoModel.from_pretrained(EP).eval()
@torch.no_grad()
def embed(texts):
    out = []
    for i in range(0, len(texts), 32):
        b = etok(texts[i:i + 32], padding=True, truncation=True, max_length=256, return_tensors="pt")
        h = emod(**b).last_hidden_state; m = b["attention_mask"].unsqueeze(-1).float()
        v = (h * m).sum(1) / m.sum(1); out.append(torch.nn.functional.normalize(v, dim=-1))
    return torch.cat(out).numpy().astype("float32")
CE = embed([c["text"] for c in chunks])
index = faiss.IndexFlatIP(CE.shape[1]); index.add(CE)

# ---------- 4. rerank ----------
RP = "cross-encoder/ms-marco-MiniLM-L6-v2"
rtok = AutoTokenizer.from_pretrained(RP); rmod = AutoModelForSequenceClassification.from_pretrained(RP).eval()
@torch.no_grad()
def rerank(q, ids):
    b = rtok([q] * len(ids), [chunks[i]["text"] for i in ids], padding=True, truncation=True, max_length=512, return_tensors="pt")
    s = rmod(**b).logits[:, 0].numpy()
    o = np.argsort(-s, kind="stable"); return [ids[j] for j in o], [float(s[j]) for j in o]

def rrf(lists, k=60):
    sc = Counter()
    for L in lists:
        for r, i in enumerate(L): sc[i] += 1 / (k + r + 1)
    return [i for i, _ in sorted(sc.items(), key=lambda x: (-x[1], x[0]))]

res = []
QE = embed([q["q"] for q in qs])
for qi, q in enumerate(qs):
    sb = bm25(q["q"]); lb = [int(i) for i in np.argsort(-sb, kind="stable")[:50]]
    _, I = index.search(QE[qi:qi + 1], 50); ld = [int(i) for i in I[0]]
    lh = rrf([lb, ld])[:50]
    lr, sr = rerank(q["q"], lh[:20])
    res.append(dict(id=q["id"], bm25=lb, dense=ld, hybrid=lh, rerank=lr, rscore=sr))

def rank(L, rel):
    for r, i in enumerate(L):
        if i in rel: return r + 1
    return None
summ = {}
for m in ["bm25", "dense", "hybrid", "rerank"]:
    R = [rank(x[m], q["rel"]) for x, q in zip(res, qs) if q["rel"]]
    summ[m] = {f"r@{k}": sum(1 for v in R if v and v <= k) / len(R) for k in [1, 3, 5, 10, 20]}
    summ[m]["mrr"] = sum(1 / v for v in R if v and v <= 20) / len(R)
    byt = {}
    for t in ["spec", "code", "policy", "incident"]:
        Rt = [rank(x[m], q["rel"]) for x, q in zip(res, qs) if q["type"] == t]
        byt[t] = sum(1 for v in Rt if v and v <= 5) / len(Rt)
    summ[m]["r@5_by_type"] = byt
print(MODE, flush=True); print("chunks", len(chunks), "avg words", round(np.mean([len(c['body'].split()) for c in chunks]), 1))
for m, v in summ.items(): print(m, {k: (round(x, 3) if not isinstance(x, dict) else {a: round(b, 3) for a, b in x.items()}) for k, x in v.items()})

out = dict(mode=MODE, chunks=chunks, qs=qs, res=res, summ=summ)

# ---------- 5. generation ----------
if GEN:
    GP = "Qwen/Qwen2.5-1.5B-Instruct"
    gtok = AutoTokenizer.from_pretrained(GP); DEV = "cpu"  # MPS + SDPA gave broken answers on our machine (torch 2.8)
    gm = AutoModelForCausalLM.from_pretrained(GP, dtype=torch.float32).to(DEV).eval()
    SYS = ("Answer the question using only the numbered context passages. Write one short sentence that states the "
           "answer, then the number of the passage you used in brackets. Example: The office opens at nine [3]. "
           "If the passages do not contain the answer, reply exactly: Not in the documents.")
    gens = []
    t0 = time.time()
    for x, q in zip(res, qs):
        ctx = "\n".join(f"[{j + 1}] {chunks[i]['text']}" for j, i in enumerate(x["rerank"][:5]))
        msgs = [{"role": "system", "content": SYS}, {"role": "user", "content": f"Context:\n{ctx}\n\nQuestion: {q['q']}"}]
        ids = gtok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(DEV)
        with torch.no_grad():
            o = gm.generate(ids, attention_mask=torch.ones_like(ids), max_new_tokens=60, do_sample=False, temperature=None, top_p=None, top_k=None)
        a = gtok.decode(o[0, ids.shape[1]:], skip_special_tokens=True).strip()
        gens.append(dict(id=q["id"], ans=a, ctx=ctx, ntok=int(ids.shape[1])))
        print(q["id"], q["type"], "|", q["q"], "|", a[:150], flush=True)
    out["gens"] = gens; out["gen_time"] = time.time() - t0
json.dump(out, open(f"out_{MODE}{'_gen' if GEN else ''}.json", "w"))

And grade.py, which turns the saved answers into the accuracy, faithfulness and category labels, the Wilson interval and the paired test. The hand corrections of the reference labels are listed in OVR.

"""Grade the saved answers: accuracy, three faithfulness graders, failure categories, statistics."""
import json, math, re, sys
from collections import Counter
from data import ERRORS
CAUSES = ["expired TLS certificate", "database migration", "firmware update that reset the clock", "full disk", "wrong DNS record", "memory leak"]
PHR = [p for e in ERRORS for p in e[:2]] + CAUSES + ["nitrile", "neoprene", "butyl"]
NUM = r"(?<![\w.])\d+(?:\.\d+)?(?![\w])"
def nums(s): return set(re.findall(NUM, s.replace("±", " ")))
def abst(a): return "not in the documents" in a.lower()
def correct(q, a):
    if q["type"] == "none": return abst(a)
    if abst(a): return False
    k = q["keys"][0]
    if re.fullmatch(r"\d+(\.\d+)?", k): return k in nums(a)
    return k.lower() in a.lower()
def strip_cite(a): return re.sub(r"\[\d+\]", " ", a)
NAMES = r"(?:Vela|Orin|Tamsin|Brio|Kestrel|Lumen|Corvo)-\w+|E-\d+|INC-\d+|\d+ \w+ 2025"
def clean(a): return re.sub(NAMES, " ", strip_cite(a))   # names and ids are not claims
def values(q, a):
    v = nums(clean(a))
    a2 = clean(a)
    v |= {p for p in PHR if p.lower() in a2.lower()}
    return v
def grounded(q, a, ctx):  # cheap rule: every number of the answer occurs somewhere in the context
    if abst(a): return True
    c = nums(strip_cite(ctx.replace("\n", " ")))
    return all(n in c for n in nums(clean(a)))
def entity(q):
    for pat in [r"E-\d+", r"INC-\d+", r"\d+ \w+ 2025", r"(?:Vela|Orin|Tamsin|Brio|Kestrel|Lumen|Corvo)-\w+"]:
        m = re.search(pat, q["q"])
        if m: return m.group(0)
    return None
def strict(q, a, passages):  # reference rule: value and entity in the same passage
    if abst(a): return True
    v = values(q, a); e = entity(q)
    if not v: return None
    for p in passages:
        if e and e.startswith("E-") and e in p:   # an error code: look only at the text that belongs to this code
            p = p[p.index(e):]; nxt = p.find("Error E-", 1); p = p if nxt < 0 else p[:nxt]
        pl = p.lower(); pn = nums(clean(p).replace("±", " "))
        if all((x in pn) if re.fullmatch(r"\d+(\.\d+)?", x) else (x.lower() in pl) for x in v) and (e is None or e.lower() in pl):
            return True
    return False

# ---------- main ----------
# hand labels where the automatic strict rule is wrong or undefined (read by hand, see notes)
OVR = {("header", 37): False,   # "45 euros" is the Lisbon allowance, not Berlin
       ("header", 23): True,    # paraphrase "erase the memory card" of a supported fix
       ("fixed", 25): False,    # "ten minutes" claim is not stated for the Kestrel-9
       ("section", 37): False,  # same Lisbon/Berlin mix-up
       ("section", 29): True}   # paraphrase of the supported fix
def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n; c = (p + z * z / (2 * n)) / d; h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h
def rank(L, rel):
    for r, i in enumerate(L):
        if i in rel: return r + 1
    return None
P = {}; ALL = {}
for m in ["fixed", "header", "section"]:
    O = json.load(open(f"out_{m}_gen.json")); ALL[m] = O
    rows = []
    for x, g, q in zip(O["res"], O["gens"], O["qs"]):
        top5 = x["rerank"][:5]; ps = [O["chunks"][i]["text"] for i in top5]
        hit = bool(q["rel"]) and any(i in q["rel"] for i in top5)
        cor = correct(q, g["ans"]); ab = abst(g["ans"])
        st = strict(q, g["ans"], ps); st = OVR.get((m, q["id"]), st)
        gr = grounded(q, g["ans"], g["ctx"]); jp = g.get("judge_p")
        if cor: cat = "ok"
        elif q["type"] == "none": cat = "answered"
        elif not hit: cat = "missed"
        elif ab: cat = "refused"
        else: cat = "misread"
        rows.append(dict(id=q["id"], t=q["type"], hit=hit, cor=cor, ab=ab, faith=st, grd=gr, jp=jp, cat=cat,
                         r={k: rank(x[k], q["rel"]) for k in ["bm25", "dense", "hybrid", "rerank"]}))
    P[m] = rows
    n = len(rows); k = sum(r["cor"] for r in rows)
    ans = [r for r in rows if r["t"] != "none"]; una = [r for r in rows if r["t"] == "none"]
    nonab = [r for r in rows if not r["ab"]]
    print(f"== {m}: accuracy {k}/{n} = {k/n:.3f}, Wilson 95% [{wilson(k,n)[0]:.3f}, {wilson(k,n)[1]:.3f}]")
    print("  answerable correct", sum(r["cor"] for r in ans), "/", len(ans), " unanswerable abstained", sum(r["cor"] for r in una), "/", len(una))
    print("  recall@5 (rerank)", sum(r["hit"] for r in ans), "/", len(ans))
    print("  categories", Counter(r["cat"] for r in rows))
    print("  faithful (hand/strict) all", sum(bool(r["faith"]) for r in rows), "/", n, " among non-abstain", sum(bool(r["faith"]) for r in nonab), "/", len(nonab))
    print("  grounded rule", sum(r["grd"] for r in rows), " judge>0.5", sum(1 for r in rows if r["jp"] is not None and r["jp"] > 0.5))
    print("  2x2 correct x faithful:", Counter((r["cor"], bool(r["faith"])) for r in rows))
    print("  abstentions", sum(r["ab"] for r in rows), "refused answerable", sum(1 for r in ans if r["ab"]))
    for lab, key in ([("grounded", "grd"), ("judge", "jp")] if rows[0]["jp"] is not None else [("grounded", "grd")]):
        agree = sum(1 for r in rows if (r[key] > 0.5 if key == "jp" else r[key]) == bool(r["faith"]))
        print(f"  {lab} agrees with hand labels: {agree}/{n}")
    by = {}
    for t in ["spec", "code", "policy", "incident", "none"]:
        rr = [r for r in rows if r["t"] == t]; by[t] = (sum(r["cor"] for r in rr), len(rr))
    print("  accuracy by type", by)
# paired comparison
A, B = (sys.argv[1], sys.argv[2]) if len(sys.argv) > 2 else ("fixed", "header")
a = {r["id"]: r["cor"] for r in P[A]}; b = {r["id"]: r["cor"] for r in P[B]}
up = sum(1 for i in a if not a[i] and b[i]); down = sum(1 for i in a if a[i] and not b[i])
nn = up + down; kk = min(up, down)
p = min(1.0, 2 * sum(math.comb(nn, j) for j in range(kk + 1)) / 2 ** nn)
print(f"paired: {A} wrong->{B} right", up, f" {A} right->{B} wrong", down, " exact sign test p =", p)
print("down ids", [i for i in a if a[i] and not b[i]], "up ids", [i for i in a if not a[i] and b[i]])
json.dump(P, open("rows.json", "w"))

Where this is used

SettingWhat this lab’s method looks like there
Internal help desks and product supportManuals, policies and tickets with many similar products and versions: chunk headers (product, version, section) and hybrid search are standard; test sets come from real tickets with known resolutions
Code and API documentation assistantsExact identifiers (function names, error codes) need BM25 or another lexical signal next to dense retrieval; the chunk header is the file path and class name
Legal and finance document QAFaithfulness and citations matter more than fluency; graders are checked against lawyers’ or analysts’ labels before use
Search and recommendation teamsRecall@k, MRR and nDCG on labelled queries are the daily metrics; paired tests on the same queries decide launches
Video-RAG and video QAIndex a long video as chunks of ASR transcript and frame captions with a header (video title, chapter, timestamp); a question is relevant to a time span, so recall@k is measured by overlap with the gold span, exactly like (160.3) with time instead of characters; the same idea serves key-moment retrieval
Video retrieval and fingerprintingFinding the right clip in a large archive is the retrieval half of this lab: recall@k on queries with known target clips, near-duplicate “distractors” (like our Vela-100/200/300), and a reranker on the top candidates
Agents with a search toolThe search tool of an agent is a retriever; the same test set and failure categories apply (session 154 and the next lab)

Common mistakes

  • Measuring only the final answer. If you do not score retrieval separately, you cannot tell a chunking problem from a reader problem. In our baseline, 22 of 32 failures were retrieval misses.
  • Chunks without context. A fixed window cut out of the middle of a document loses its title and section. With many similar documents this is often the biggest failure; add a header (160.9) or chunk by structure.
  • Copying questions from the documents. Questions with the exact words of the source make BM25 look perfect. Write them as users would (“heavy”, not “weighs”).
  • No unanswerable questions. Then a system that always answers looks best. Our baseline answered the non-existent Vela-500 and Kestrel-20 with numbers from their siblings.
  • Trusting a faithfulness grader you did not check. Our number rule and our LLM judge called almost every wrong answer faithful. Check any grader against hand labels first.
  • Confusing faithfulness with correctness. An answer can follow the context exactly and still be wrong, if retrieval gave the wrong context. Report both.
  • Reporting small differences on small test sets. With 60 questions the 95% interval is about ±12 points. Use a paired test (160.10) on the same questions, and do not claim a gain like our fix 2 without one.
  • Tuning on the test set. Every fix you design after reading failures makes that test set optimistic. Keep a held-out set.

Interview questions

How do you evaluate a RAG system end to end?basic
Build a test set of real-looking questions, each with a known answer and a known source passage, plus some questions with no answer in the corpus. Score retrieval and generation separately. Retrieval: recall@k (is a relevant chunk in the top \(k\)?) and MRR, per stage (sparse, dense, fused, reranked) and per question type. Generation: answer accuracy against the known answers, abstention on unanswerable questions, and faithfulness (is every claim supported by the passages?). Then sort failures into categories (missed by retrieval, misread, refused, answered anyway), fix the biggest one, and compare old and new system on the same questions with a paired test.
Why combine BM25 and dense retrieval? What does each miss?basic
BM25 matches exact words with IDF weights, so rare identifiers (error codes, part numbers, names) are found reliably, but synonyms and paraphrases are missed (“heavy” vs “weighs”). Dense retrieval matches meaning, so paraphrases work, but small embedders blur identifiers: “E-9251” and “E-9215” look alike. In our lab with headers, BM25 had recall@5 of 1.00 on error codes and 0.46 on spec questions; dense had 0.42 and 0.96. RRF fusion reached 0.92 and 1.00.
Explain reciprocal rank fusion. Why ranks and not scores? What is \(k\) for?core
RRF scores each document by \(\sum_r 1/(k + \mathrm{rank}_r(d))\) over the result lists (160.5). Raw scores of BM25 and cosine similarity have different scales and distributions, so adding them needs normalization that is fragile; ranks are comparable. \(k\) (60 by default) flattens the curve: with small \(k\), the first places of each list dominate; with large \(k\), documents that appear in both lists win. A weakness: RRF rewards agreement, so a document that is first in one list and 18th in the other can lose to documents that are mid-ranked in both; in our run that pushed a gold chunk from rank 1 (BM25) to rank 8, and only the reranker fixed it.
What does a cross-encoder reranker do better than a bi-encoder, and what does it cost?core
A bi-encoder embeds question and chunk separately, so their interaction is a single dot product; chunks can be embedded once, offline. A cross-encoder reads question and chunk together through all layers, so attention can compare every question token with every chunk token: it can check that the chunk is about the right product or code. The cost is one forward pass per (question, chunk) pair at query time, so it is used only on the top 20 to 100 candidates. In our baseline it raised recall@1 from 0.296 to 0.463; it cannot recover chunks that are not in its candidate list.
Faithfulness vs correctness: give an example of each without the other.core
Faithful but wrong: retrieval returns the Vela-300 warranty chunk without a product name, and the reader answers “60 months” for the Vela-200; if the chunk had said “Vela-300”, the reader could have noticed. More generally, a faithful answer repeats a wrong or outdated source. Correct but unfaithful: our baseline answered 40 °C for the Tamsin-X2, which is right, but it took the number from a chunk of another product, so it was right by luck. Faithfulness measures the reader given its context; correctness measures the whole system. You need both.
How would you build a faithfulness grader you can trust?advanced
Label 50 to 100 answers by hand first, including clearly unfaithful ones. Then measure each candidate grader against those labels: agreement, and especially recall on the unfaithful class. In our lab the LLM judge (a 1.5B model) said “yes” to all 37 unfaithful answers over two configurations and “no” to all 17 refusals, so its agreement was 20/60 on the baseline. Improvements: a larger judge from a different model family, claim-by-claim checking (split the answer into claims, verify each against specific passages), an NLI model, asking the judge to quote the supporting sentence and checking that the quote exists, and domain rules such as “the entity must appear in the supporting passage”. Re-check the grader when the system changes.
Your RAG system has recall@5 of 1.0 but 17% wrong answers. Where do you look?core
At the reader and the prompt. Sort the errors: refusals with the right chunk present (the reader is too cautious or confused by distractor passages), misreads (it used the wrong number from the right passage, or a sibling’s passage), and answers to unanswerable questions. In our lab with headers, all 5 refusals had the gold chunk at rank 1. Try fewer passages (top 3), a stronger reader, a prompt that tells the reader to match the entity, and check that facts are not cut across chunk boundaries (3 of our 4 misreads).
How big must a RAG test set be?advanced
It depends on the effect you want to see. For one system, the 95% interval of an accuracy near 0.5 is about \(\pm 1.96\sqrt{0.25/n}\): ±0.127 for \(n = 60\), ±0.044 for \(n = 500\). For comparing two systems, a paired test on the same questions is much more sensitive, because only disagreements count. Our fix 1 (28 fixed, 6 broken) was clear with 60 questions (\(p = 0.0002\)); fix 2 (6 fixed, 4 broken, \(p = 0.75\)) was not. Start with 50 to 100 carefully built questions per important question type, and grow the set from production logs.
What is contextual retrieval, and when does it help?advanced
Prepend context to each chunk before indexing it: at the simplest, the document title and section path (our 160.9); in Anthropic’s version, 50 to 100 tokens written by an LLM that reads the whole document and the chunk. It helps most when chunks are not self-contained: pronouns, “the device”, tables without titles, many similar documents. Anthropic reports a drop of the top-20 failure rate from 5.7% to 3.7% with contextual embeddings and to 1.9% with contextual BM25 and reranking. In our lab a plain header raised reranked recall@5 from 0.556 to 1.000. The costs: more tokens per chunk (index size, embedding time, prompt length) and, for the LLM version, one LLM call per chunk at indexing time.
How should a RAG system handle questions it cannot answer, and how do you measure it?core
Instruct the reader to abstain with a fixed phrase when the passages do not contain the answer, optionally add a retrieval-score threshold, and include unanswerable questions in the test set, especially near misses (a product that does not exist, a city that is not in the policy). Measure two rates: abstention on unanswerable questions (our runs: 4/6, 5/6, 5/6) and false refusals on answerable ones (3, 5 and 2 of 54). Both must be reported; tuning only one makes the other worse.
You change the chunker and recall@1 improves from 0.907 to 0.981, but answer accuracy changes only from 50 to 52 of 60. Ship it?advanced
The answer change is not significant (6 fixed, 4 broken, sign test \(p = 0.75\)), so do not claim it improves answers. But the retrieval gain is consistent, and it removed a known failure (facts cut across chunk boundaries). Read the 4 broken questions: if they are reader errors not caused by the new chunks (in our case the reader took a number from a sibling product’s chunk at rank 3), the chunker is not to blame. Decide with a larger or held-out test set, and look at cost (194 vs 159 chunks, shorter prompts: 394 vs 491 tokens).

Exercises

  1. Compute the RRF score (sum form, \(k = 60\)) of a chunk that is rank 3 in BM25 and rank 5 in the dense list, and of one that is rank 1 in BM25 and absent from the dense top 50. Which ranks higher?
  2. Our baseline reranked recall@1 is 0.463 over 54 questions. How many questions is that? Our headers MRR@20 is 0.951 with recall@1 = 0.907 and recall@3 = 1.000. If every question not at rank 1 was at rank 2, what MRR would you get? Is that consistent with 0.951?
  3. Compute the Wilson 95% interval (160.8) for 50 of 60.
  4. Add a 10-word overlap to the baseline 50-word chunker. Predict which category of Figure 5 changes and which does not, then run it.
  5. Replace the judge prompt with one that asks the judge to quote the supporting sentence first. Measure its agreement with the reference labels on the baseline answers.
  6. Write 20 new questions in a different style (short keyword queries like “vela 200 weight”) and measure BM25, dense and fused recall@5 with headers. Which retriever gains or loses?
Answerscheck
1. \(1/63 + 1/65 = 0.01587 + 0.01538 = 0.03126\), against \(1/61 = 0.01639\). The chunk found by both lists wins by far. This is why a chunk that only one retriever finds needs a reranker or a larger \(w\) for that retriever. 2. \(0.463 \times 54 = 25\) questions. If the 5 questions not at rank 1 were all at rank 2: \((49 + 5 \times 0.5)/54 = 0.954\). Our 0.951 is a bit lower, so at least one was at rank 3 (with one at rank 3: \((49 + 4 \times 0.5 + 1/3)/54 = 0.951\)). 3. \(\hat{p} = 0.8333\): center \((0.8333 + 0.0320)/1.0640 = 0.8133\), half-width \(1.842 \times \sqrt{0.8333 \cdot 0.1667/60 + 0.000267} = 1.842 \times 0.0506 = 0.0936\), interval [0.720, 0.907]. 4. Overlap keeps a fact whole if it is shorter than the overlap (Session 152, formula 152.3), so the code-and-fix splits can disappear; it does not add the product name to chunks in the middle of a manual, so “missed by retrieval” for spec questions should stay large. 5 and 6 depend on your run; report agreement on the unfaithful class separately.

Key takeaways

  • A RAG test set needs questions in users’ words, a known answer and a known source for each, and unanswerable near-miss questions. Define relevance by position in the document (160.3) so it survives changes of the chunker.
  • Score retrieval per stage and per question type (recall@k, MRR), and answers separately (accuracy, abstention, faithfulness). Faithfulness is not correctness.
  • BM25 finds identifiers, dense retrieval finds paraphrases, RRF (160.5) combines them, and a cross-encoder fixes the order; in our lab fusion moved a gold chunk from rank 1 to 8 and the reranker moved it back.
  • Sort failures into categories. Our biggest one was chunks without context: 22 of 60 questions missed by retrieval. A header line (160.9) raised reranked recall@5 from 0.556 to 1.000 and accuracy from 28 to 50 of 60 (\(p = 0.0002\)).
  • Check graders against hand labels: our number rule and 1.5B LLM judge called almost every unfaithful answer faithful. Use paired tests (160.10) on small sets: our second fix (50 to 52 of 60) was not significant.

Key formulas

\[ s_{\text{RRF}}(d) = \frac{w}{k + \mathrm{rank}_{\text{BM25}}(d)} + \frac{1 - w}{k + \mathrm{rank}_{\text{dense}}(d)}, \qquad \mathrm{MRR@}20 = \frac{1}{|Q|}\sum_{q} \frac{1}{\mathrm{rank}_q} \]
\[ \mathrm{Recall@}k(q) = \mathbb{1}\big[\mathrm{Rel}(q) \cap \mathrm{Top}_k(q) \neq \emptyset\big], \qquad F = \frac{\#\,\text{supported claims}}{\#\,\text{claims}} \]
\[ \tilde{c} = \text{title}(d) \oplus \text{section}(c) \oplus c, \qquad p = 2\sum_{j=0}^{\min(b,c)} \binom{b+c}{j} 2^{-(b+c)} \]

References for this session

  1. Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey
  2. Meta (code). FAISS library
  3. Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey
  4. Meta (code). FAISS library
  5. Douze et al. (2024). The Faiss library
  6. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  7. Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond
  8. Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
  9. Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
  10. Wang et al. (2020). MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
  11. Nogueira & Cho (2019). Passage Re-ranking with BERT
  12. Bajaj et al. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
  13. Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
  14. Es et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation
  15. Barnett et al. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System
  16. Anthropic (2024). Introducing Contextual Retrieval
  17. Qwen Team (2024). Qwen2.5 Technical Report
  18. Wilson (1927). Probable Inference, the Law of Succession, and Statistical Inference
  19. McNemar (1947). Note on the sampling error of the difference between correlated proportions or percentages