RAG as input compression: FLARE, REPLUG, Self-RAG
Putting a whole document collection into the prompt is slow and costs a lot of GPU memory. Retrieval-augmented generation (RAG) puts only the few relevant pieces into the prompt, so it is also a way to compress the input. This session does the cost math for long context against retrieval, then studies three ways to make retrieval smarter: FLARE (retrieve only when the model is unsure), REPLUG (train the retriever with a frozen LLM as the teacher) and Self-RAG (the model decides and checks its own retrieval with special tokens).
What you will learn
- Why RAG is a form of input compression: the prompt holds k chunks instead of the whole corpus
- The cost formulas for prefill FLOPs and KV-cache memory, and how to compare long context with retrieval for a real model
- FLARE: draft the next sentence, retrieve only when a token is below a confidence threshold, and build the query by masking unsure tokens
- REPLUG: an ensemble over retrieved documents for a black-box LLM, and LSR training that makes the retriever agree with the LLM
- Self-RAG: reflection tokens that decide when to retrieve and score relevance, support and usefulness
- The hidden serving costs of each method: re-prefill after each retrieval, k parallel sequences, extra decode tokens
- When a long context is the better choice, and how to apply the same idea to video (retrieve clips, not all frames)
Why this matters
A company has 100,000 tokens of manuals. A user asks one question. You can put all 100,000 tokens in the prompt, or you can search the manuals first and put in only the five best paragraphs, about 1,300 tokens. For Llama 3.1 8B the first choice needs about 200 times more prefill compute and about 70 times more KV-cache memory. This session shows where those numbers come from.
Retrieval-augmented generation (RAG) means: before the model answers, a retriever (a search system) finds a few relevant pieces of text in a large collection, and we add only those pieces to the prompt. RAG was introduced to give models fresh and private knowledge and to reduce made-up facts; session 152 covers the basic pipeline. The efficient-inference survey adds a second view: RAG is also a way to compress the input. In its words, instead of merging all information into an excessively long prompt, RAG adds only the relevant retrieved information, so the prompt becomes much shorter. [Certain, survey Section 4.1.4]
This is the third session on input compression. Session 88 deleted unimportant tokens from a long prompt, and session 89 rewrote the prompt into a short summary or a few learned vectors. RAG goes further: most of the input never enters the prompt at all. The survey names three methods that make retrieval smarter, and they are the core of this session:
- FLARE decides when to retrieve: only when the model is about to be unsure.
- REPLUG keeps the LLM frozen as a black box and uses it as a teacher to improve the retriever.
- Self-RAG trains the model to emit special reflection tokens that decide when to retrieve and grade what it retrieved.
Real systems use these ideas every day: search-backed chat assistants, coding assistants that pull the right files from a large repository, and video assistants that find a few clips in hours of footage. In all of them, the retriever decides what the expensive model must read, so it decides the cost.
RAG as input compression
Two ways to give a model knowledge
Let the user question have \(T_q\) tokens and the corpus (all documents) have \(T_c\) tokens. Split the corpus into chunks of \(c\) tokens each, and let the retriever pick the best \(k\) chunks. The two prompt lengths are:
- Read the symbols
- \(T_q\)
- tokens in the question and the fixed instructions (system prompt, template)
- \(T_c\)
- tokens in the whole corpus that might hold the answer
- \(k, c\)
- number of retrieved chunks, and tokens per chunk (often 128 to 1,024)
- \(\rho\)
- the compression ratio of the prompt: how many times shorter the RAG prompt is
With \(T_q = 100\), \(T_c = 100{,}000\), \(k = 5\), \(c = 256\): \(T_{\text{rag}} = 1{,}380\) and \(\rho = 100{,}100 / 1{,}380 = 72.5\). A prompt-pruning method such as LLMLingua aims for ratios of a few times up to about 20 times; RAG reaches 72 here because it never reads most of the corpus. [Likely, ratios as reported in the LLMLingua papers] The price is a new risk: if the retriever misses the right chunk, the model never sees it.
The cost math: prefill FLOPs
Recall from session 78 that the prompt is processed in one parallel prefill pass, then the answer is written token by token in decode. Prefill work has two parts. The weight matrices cost about \(2N\) FLOPs per token, where \(N\) is the number of non-embedding parameters (session 25). Attention adds a part that grows with the square of the length, because every token looks at all earlier tokens:
- Read the symbols
- \(T\)
- prompt length in tokens
- \(N\)
- non-embedding parameters (6.98 billion for Llama 3.1 8B; the input embedding is a lookup, and only the last position needs the output head in prefill)
- \(L, d\)
- number of layers, and model width \(d = n_h d_h\) (all query heads together)
- \(2LdT^2\)
- \(QK^\top\) and the weighted sum with \(V\): each is about \(2 d\) FLOPs per (query, key) pair, and a causal mask keeps about \(T^2/2\) pairs, so \(2 \times 2d \times T^2/2 = 2dT^2\) per layer
The attention term passes the weight term when \(2LdT^2 \gt 2NT\), that is when \(T \gt N/(Ld)\). For Llama 3.1 8B (\(L = 32\), \(d = 4{,}096\)) this is at \(T = 53{,}248\) tokens; for Llama 3.1 70B (\(L = 80\), \(d = 8{,}192\)) at \(104{,}448\). [Certain, from the configs and (90.2)] So below about 50k tokens, prefill cost grows almost linearly with the prompt; above it, the square term takes over and each extra token costs more than the one before.
The cost math: KV-cache memory
Session 80 gave the memory of the cache. Every prompt token stays in the cache for the whole answer:
- Read the symbols
- \(2\)
- one key and one value
- \(n_{kv}, d_h\)
- number of key/value heads (8 for Llama 3.1 8B and 70B, with GQA) and head size (128)
- \(b\)
- bytes per number: 2 for BF16
For Llama 3.1 8B this is 131,072 bytes (128 KiB) per token; for 70B, 327,680 bytes (320 KiB). The cache also makes each decode step slower, because each new token must read the whole cache from memory (session 82). A 100k-token prompt makes every one of the answer tokens slower, not only the first one.
Long context: \(T = 100{,}100\). Weights: \(2 \times 6.98\times10^9 \times 100{,}100 = 1.40\times10^{15}\) FLOPs. Attention: \(2 \times 32 \times 4{,}096 \times 100{,}100^2 = 2.63\times10^{15}\). Total \(4.02\times10^{15}\) FLOPs. KV cache: \(131{,}072 \times 100{,}100 = 13.1\) GB for one user.
RAG with \(k = 5\), \(c = 256\): \(T = 1{,}380\). Weights \(1.93\times10^{13}\), attention \(5.0\times10^{11}\), total \(1.98\times10^{13}\) FLOPs. KV cache: 0.18 GB.
Ratios: 204 times less prefill compute and 72.5 times less cache memory. The compute ratio is larger than the token ratio (72.5) because RAG also removes the square term. At an assumed 400 TFLOP/s of useful compute (a little under half of the dense BF16 peak of an H100, which is about 990 TFLOP/s [Likely]), prefill takes about 10.1 s against 0.049 s. With 16 users at once, the long-context caches need 210 GB, more than two 80 GB GPUs, while the RAG caches need 2.9 GB.
The same comparison for Llama 3.1 70B: about 67 s against 0.48 s of prefill, and 32.8 GB against 0.45 GB of cache per user.
How to read Figure 2
Start at the default (100k corpus, 5 chunks of 256). Move the corpus slider down to 8k: the ratio falls to about 6, and long context becomes a fair choice, especially if the same corpus is reused and its cache is kept (prefix caching). Move it up to 128k with the 70B model: prefill alone takes more than a minute. Now raise \(k\) to 20 and \(c\) to 1,024: the RAG prompt has 20,580 tokens and a lot of the saving is gone. Retrieval saves compute only if \(k c\) stays much smaller than \(T_c\).
What the cost math does not show
Retrieval is not free, but it is cheap. Encoding a 32-token query with a 110M-parameter encoder (the size of Contriever, a BERT-base model) costs about \(2 \times 110\times10^6 \times 32 = 7\times10^9\) FLOPs, which is 0.04% of the RAG prefill above. The search in a vector index takes milliseconds (session 60). The document embeddings are computed once, offline. [Likely]
Long context has two real advantages, and an honest answer in an interview names them:
- No retrieval misses. If the answer needs facts spread over many places, or the question is vague, a retriever can miss them. The whole document cannot be missed.
- Reuse. If many questions use the same document, its KV cache can be computed once and kept (prefix caching, session 134). Then prefill is paid once, but the memory is held for all users and every decode step still reads it.
Also, more context is not always better quality. Models use information at the start and end of a long prompt better than in the middle (“lost in the middle”). [Certain, Liu et al. 2023] A short prompt with the right chunks can give a better answer than a long prompt with the right chunk buried at position 50,000.
- offline: split the corpus into chunks of \(c\) tokens; encode each chunk \(d\) into a vector \(E_d(d)\); build a vector index
- input: question \(x\); number of chunks \(k\)
- \(\mathbf{e} \leftarrow E_q(x)\) // a small encoder, about \(10^{10}\) FLOPs
- \(D' \leftarrow\) the \(k\) chunks with the largest score \(s(d, x) = E_d(d)\cdot\mathbf{e}\) // formula (90.4), approximate nearest-neighbour search
- prompt \(\leftarrow\) instructions, then \(D'\), then \(x\) // \(T_q + kc\) tokens instead of \(T_q + T_c\)
- prefill the prompt; decode the answer with the KV cache
- return the answer and, if needed, the ids of the chunks used (for citations)
The score in line 4 is the dense-retrieval score from DPR and Contriever:
- Read the symbols
- \(E_d, E_q\)
- the document encoder and the query encoder (Contriever uses one shared encoder for both)
- \(\mathcal{D}\)
- all chunks in the corpus; \(D'\) is the small set we keep
Classic RAG has two weak points. It retrieves exactly once, with only the question as the query, even when a long answer needs different facts in different sentences. And the retriever was trained on its own, so it does not know which chunks actually help this LLM. FLARE attacks the first weakness, REPLUG the second, Self-RAG both.
FLARE: retrieve when the model is about to be unsure
The idea
Think of a model writing a short biography. Most sentences are easy: grammar, common facts. A few sentences hold a hard fact, such as a date. Retrieving before every sentence wastes work. Retrieving only once, before the answer, uses a query (the question) that may not mention the hard fact at all.
FLARE (Forward-Looking Active REtrieval, Jiang et al. 2023) asks the model itself. Before writing sentence \(t\), the model writes a draft of it without new documents. If every token of the draft has high probability, the model is confident: keep the draft. If some token has low probability, the model is unsure: use the draft to build a search query, retrieve, and write the sentence again with the documents. [Certain] The name says “forward-looking” because the query is built from what the model is about to say, not from what it has already said.
The math
Let \(\hat{s}_t = (\hat{s}_{t,1}, \dots, \hat{s}_{t,m})\) be the draft of sentence \(t\), generated from the input \(x\) and the earlier output \(y_{\lt t}\). FLARE-direct retrieves when any token is below a threshold \(\theta\):
The query is the draft with its unsure tokens masked. A wrong guess like “1816” would push the search toward wrong pages, so tokens with probability below a second threshold \(\beta\) are removed:
Then the sentence is written again, with the retrieved documents \(\mathcal{D}_{q_t}\) put in front:
- Read the symbols
- \(\theta\)
- trigger threshold. \(\theta = 0\): never retrieve. \(\theta = 1\): retrieve for every sentence. The paper used \(\theta = 0.8\) on 2WikiMultihopQA, ASQA and WikiAsp, and 0.4 on StrategyQA. [Certain]
- \(\beta\)
- masking threshold; the paper used \(\beta = 0.4\). [Certain] Larger \(\beta\) removes more tokens from the query.
- \(\mathcal{D}_{q_t}\)
- the documents found with query \(q_t\) (2 to 5 per retrieval in the paper, from BM25 over Wikipedia or from Bing) [Certain]
The paper also tried a second query type: send the unsure spans to another LLM that writes a question for each span (“When was Ada Lovelace born?”). And it tried FLARE-instruct, where the model is prompted to write [Search(query)] itself. FLARE-direct, with masked or generated queries, worked best. [Certain]
How to read Figure 3
At \(\theta = 0.8\), 4 of 6 sentences retrieve. Two of them (born in 1816, notes in 1842) hold real errors, and retrieval fixes them. Two others (Analytical Engine, died in 1852) were correct, so their retrieval was wasted but harmless. Set \(\theta = 0.5\): only the two wrong sentences retrieve, and the cost halves. Set \(\theta = 0\): no retrieval, and both errors stay in the answer. Now set \(\beta = 0.6\) and look at the queries: “Analytical” and “1852” are masked too, and the queries become vaguer. Thresholds trade cost against missed errors, and the best values depend on how well the model’s probabilities are calibrated.
The serving cost of FLARE
FLARE looks cheap because it retrieves less. But look at formula (90.7) from the point of view of the KV cache. The documents are put in front of \(x\) and \(y_{\lt t}\). A change at the front changes every later key and value (the causal mask makes position \(j\) depend on all earlier positions). So the cache of \(x\) and \(y_{\lt t}\) cannot be reused: each retrieval needs a new prefill of the whole prefix. The draft is also thrown away, so its decode steps are wasted:
- Read the symbols
- \(\mathcal{R}\)
- the set of sentences that triggered retrieval
- \(T_x, |y_{\lt t}|\)
- tokens of the input and of the answer written so far
- \(|\hat{s}_t|\)
- tokens of the thrown-away draft
At \(\theta = 0.8\), sentences 2, 4, 5 and 6 retrieve. The answer written before them has 7, 25, 35 and 47 tokens (sentence 5 is 12 tokens after its fix). With \(kc = 512\) and \(T_x = 8\): \(\Delta T_{\text{prefill}} = (520 + 7) + (520 + 25) + (520 + 35) + (520 + 47) = 2{,}194\) tokens, and the thrown-away drafts have \(8 + 10 + 11 + 5 = 34\) tokens.
At \(\theta = 1\) (retrieve for every sentence): 6 retrievals, 3,249 extra prefill tokens and 51 wasted draft tokens. At \(\theta = 0.5\): 2 retrievals, 1,082 tokens and 19 wasted tokens, and the same two errors are fixed. A single retrieval before the answer costs \(512 + 8 = 520\) tokens, but its query (“Write a short biography of Ada Lovelace”) is not about any specific fact.
A system can reduce the re-prefill cost by putting the documents after the current text instead of in front. Then the cache of the prefix stays valid and only the new documents are prefilled. This changes the prompt layout the model sees, so it is a quality trade-off. [Speculation: FLARE itself prepends the documents.]
The results show why the extra work can pay off. On 2WikiMultihopQA with text-davinci-003, exact match was 28.2 with no retrieval, 39.4 with one retrieval before the answer, 47.8 for the best baseline (question decomposition), and 51.0 for FLARE-direct. [Certain, FLARE Table]
- input: question \(x\); thresholds \(\theta, \beta\); chunks per retrieval \(k\)
- \(y \leftarrow\) empty
- while the answer is not finished:
- \(\hat{s} \leftarrow\) draft the next sentence from \((x, y)\), keeping each token’s probability
- if all probabilities \(\ge \theta\): \(y \leftarrow y + \hat{s}\); continue // confident: no retrieval
- \(q \leftarrow \hat{s}\) with tokens below \(\beta\) removed // formula (90.6)
- \(D \leftarrow\) retrieve \(k\) chunks with \(q\)
- throw away \(\hat{s}\); prefill \((D, x, y)\) again // the old cache is not valid, formula (90.8)
- \(s \leftarrow\) write the sentence again; \(y \leftarrow y + s\)
- return \(y\)
REPLUG: the frozen LLM teaches the retriever
A black-box LLM
Many teams use an LLM only through an API. They cannot change its weights or see its hidden states. REPLUG (Shi et al. 2023) is built for this case. It treats the LLM as a frozen black box and only adds things around it. [Certain] It has two parts: an inference trick (an ensemble over documents) and a training method for the retriever (LSR).
Inference: an ensemble over documents
Classic RAG puts all \(k\) chunks in one prompt. REPLUG runs the LLM \(k\) times instead, once per chunk, each time with only that chunk in front of the input. Then it mixes the \(k\) next-token distributions, giving more weight to chunks with a higher retrieval score:
- Read the symbols
- \(d \circ x\)
- chunk \(d\) followed by the input \(x\) (concatenation)
- \(p(y \mid d \circ x)\)
- the LLM’s probability of the next token (or answer) given one chunk
- \(\lambda(d, x)\)
- the weight of chunk \(d\): a softmax over the retrieval scores of the \(k\) chunks
Two reasons for the ensemble. Each pass sees a short prompt, so more chunks can be used than fit in one context window. And no chunk is “in the middle” of a long prompt. The paper used the top 10 documents. [Certain]
Training: LM-supervised retrieval (LSR)
Which chunks really help? The LLM can tell us: a chunk helps if it makes the correct continuation \(y\) more likely. REPLUG turns this into a target distribution for the retriever. On training text, split each sequence into an input \(x\) and its true continuation \(y\). Retrieve the top chunks, then compute two distributions over them:
The retriever is trained so that its distribution agrees with the LLM’s distribution:
- Read the symbols
- \(P_R\)
- the retriever’s view: how likely it is to pick each chunk. It depends on the retriever weights, so it gets the gradient.
- \(Q\)
- the LLM’s view: chunks that make \(y\) likely get high weight. The LLM is frozen; \(Q\) is a fixed target.
- \(\gamma, \beta\)
- softmax temperatures. Smaller values make the distributions sharper.
- \(\mathcal{B}\)
- a batch of training inputs
Only the retriever changes. The LLM gives only probabilities of \(y\), which many APIs return. Training changes the query encoder and the document encoder, so the stored document embeddings become old. REPLUG recomputes them and rebuilds the index every 3,000 steps. [Certain] The paper trained from Contriever on 800k sequences of 256 tokens from the Pile, with a corpus of 36M chunks of 128 tokens. [Certain]
How to read Figure 4
At the start, the retriever likes chunk 1 best (score 2.6), but the LLM finds chunk 1 almost useless (\(P_{LM} = 0.08\)) and chunk 2 very useful (0.52). The ensemble probability of the true continuation is only 0.218. Press “Train 1 step”: the KL falls from 1.887 to 0.624 and the ensemble probability rises to 0.361. After 10 steps the retriever puts about 90% of its weight on chunk 2 and the ensemble reaches 0.485. Now raise \(\beta\) to 1: the target \(Q\) becomes almost flat, because the four LM probabilities differ by less than 0.5, and the retriever learns little. The temperature decides how strongly the LLM’s preference is turned into a training signal. (In the real method the scores come from an encoder, and the gradient updates the encoder weights.)
Scores \(s = (2.6, 2.2, 1.8, 1.0)\) give ensemble weights \(\lambda = (0.431, 0.289, 0.194, 0.087)\). LM probabilities \(P_{LM}(y \mid d, x) = (0.08, 0.52, 0.04, 0.30)\). Ensemble (90.9): \(0.431 \times 0.08 + 0.289 \times 0.52 + 0.194 \times 0.04 + 0.087 \times 0.30 = 0.218\). A prompt with only chunk 2 would give 0.52.
Target with \(\beta = 0.1\): \(Q = \softmax(0.8, 5.2, 0.4, 3.0) = (0.011, 0.884, 0.007, 0.098)\). \(\KL(P_R \| Q) = 1.887\). The gradient of the KL with respect to the scores is \(P_{R,i}(\log P_{R,i} - \log Q_i - \KL)/\gamma\). One step with learning rate 1 gives \(s = (1.827, 3.068, 1.530, 1.174)\), KL 0.624, ensemble 0.361.
The serving cost of REPLUG
The ensemble is not free. With \(k\) chunks of \(c\) tokens and an input of \(T_x\) tokens, the two layouts cost:
The input \(x\) is processed \(k\) times, and during decoding there are \(k\) sequences, each with its own KV cache, and every generated token is computed \(k\) times. With \(k = 10\), \(c = 128\), \(T_x = 256\): one concatenated prompt has 1,536 tokens, while the ensemble has 10 sequences of 384 tokens, 3,840 tokens in total: 2.5 times more weight FLOPs. The attention term is smaller (\(10 \times 384^2 = 1.47\)M pairs against \(1{,}536^2 = 2.36\)M), but at this length the weight term dominates. The \(k\) sequences run as one batch, so latency grows less than cost. If the chunks come after a shared prefix (instructions), prefix caching can share it across the \(k\) sequences. [Likely]
Results: with the trained retriever, REPLUG improved GPT-3 (175B) language modeling by 6.3% (bits per byte from 0.80 to 0.75), and Codex on MMLU from 68.3 to 71.4 with the ensemble and 71.8 with LSR, which the paper reports as 4.5% and 5.1% gains. [Certain, REPLUG Tables 1 and 2]
- input: frozen LLM; retriever \(E\) (for example Contriever); corpus \(\mathcal{D}\); training texts
- encode all chunks of \(\mathcal{D}\) with \(E\); build the index
- for step \(= 1, 2, \dots\):
- sample a batch; split each text into input \(x\) and true continuation \(y\)
- retrieve the top chunks \(D'\) for each \(x\); compute \(P_R(d \mid x)\) // formula (90.10), with gradient
- for each \(d \in D'\): ask the LLM for \(P_{LM}(y \mid d, x)\); compute \(Q\) // no gradient through the LLM
- update \(E\) to lower \(\KL(P_R \| Q)\) // formula (90.11)
- every \(T\) steps: re-encode the corpus and rebuild the index // stored embeddings are stale; \(T = 3{,}000\) in the paper
- return the trained retriever
Self-RAG: the model decides and checks
Reflection tokens
FLARE decides with token probabilities, which are a rough signal. REPLUG never decides: it always retrieves. Self-RAG (Asai et al. 2023) teaches the model to make the decisions itself, by writing special reflection tokens into its output. [Certain] There are four types:
| Token | Looks at | Values | Question it answers |
|---|---|---|---|
Retrieve | input \(x\) (and output so far) | yes, no, continue | Do I need documents for the next segment? |
IsRel | \(x\) and one passage \(d\) | relevant, irrelevant | Is this passage about the question? |
IsSup | \(x\), \(d\) and the segment \(y_t\) | fully, partially, no support | Is what I wrote backed by the passage? |
IsUse | \(x\) and \(y_t\) | 5, 4, 3, 2, 1 | Is the answer useful overall? |
Training has two models. A critic (Llama 2 7B) learns to predict reflection tokens from labels that GPT-4 produced. Then the critic inserts reflection tokens into 150k instruction-output pairs, and the generator (Llama 2 7B or 13B) is trained with normal next-token loss on this text, so it learns to write the tokens itself. At inference only the generator is needed. [Certain]
Inference: retrieve on demand, then score the candidates
Before each segment, the model predicts the Retrieve token. Self-RAG retrieves when the normalized probability of “yes” is above a threshold \(\delta\):
The paper used \(\delta = 0.2\) for most tasks. [Certain] If it retrieves, the model processes \(K\) passages in parallel, one candidate segment per passage, and writes the critique tokens for each. Then it ranks the candidates by a score that mixes the segment’s probability with the critique:
- Read the symbols
- \(\hat{r}^G\)
- the best value of critique type \(G\) (for example “relevant” or “fully supported”)
- \(s_t^G\)
- the probability of that best value, normalized over the \(N^G\) values of the type
- \(w^G\)
- weights set at inference time, without retraining. Defaults in the paper: 1.0 for IsRel, 1.0 for IsSup, 0.5 for IsUse. [Certain] A user who cares most about citations can raise \(w^{\text{Sup}}\).
The paper keeps the best candidates in a segment-level beam search (beam width 2) and uses up to 5 passages from its retriever per step. [Certain]
How to read Figure 5
With the default weights (1, 1, 0.5), B wins with 2.35 against 1.90 for A. Set \(w^{\text{Sup}} = 0\): A wins (1.70 against 1.50), because without the support check, fluency and relevance decide. This is the case Self-RAG is built to avoid: a confident answer that the passage does not back up. Now raise \(\delta\) above 0.35: the model decides not to retrieve, no passages are read, and it answers from its own knowledge (score 0.45 + 0.5 × 0.55 = 0.725, only IsUse applies). The weights are a control knob at inference time, and the threshold is a cost knob.
The serving cost of Self-RAG
Self-RAG saves retrieval when it is not needed, like FLARE. But when it does retrieve, it runs \(K\) candidates in parallel (like the REPLUG ensemble), keeps a beam, and writes extra reflection tokens, which are extra decode steps. It also needs a fine-tuned model, so it does not work with a closed API. Its gain is quality: on PopQA, Self-RAG 13B scored 55.8 against 51.8 for Llama 2-chat 13B with the same retrieval and 29.3 for ChatGPT without retrieval. [Certain, Self-RAG Table 2] On ASQA exact match, ChatGPT with retrieval was higher (40.7 against 31.7). [Certain] So it is a strong method for small open models, not a win everywhere.
Putting the three side by side
| Classic RAG | FLARE | REPLUG | Self-RAG | |
|---|---|---|---|---|
| When to retrieve | once, before the answer | when a draft token is below \(\theta\) | once, before the answer | when \(p(\text{yes}) \gt \delta\) |
| Query | the question | masked draft of the next sentence | the input | input and output so far |
| LLM changes? | no | no (needs token probabilities) | no (needs probabilities of \(y\) for training) | yes, fine-tuned with reflection tokens |
| Retriever changes? | no | no | yes, LSR training | no |
| Extra serving cost | one prefill of \(T_q + kc\) | re-prefill and lost draft per retrieval (90.8) | \(k\) parallel sequences (90.12) | \(K\) parallel candidates, beam, reflection tokens |
The methods can be combined. A trained retriever (REPLUG) can serve FLARE’s queries. RECOMP from session 89 can compress the retrieved chunks before they enter the prompt, and LongLLMLingua can prune them. Each step reduces the tokens the expensive model must read.
Runnable code
The code below checks the cost formulas, runs the FLARE trigger and masked query, trains a tiny retriever with the REPLUG LSR loss through real autograd, and scores Self-RAG candidates. It runs on a CPU in about a second and prints the same numbers every time.
import torch, torch.nn.functional as F torch.manual_seed(0) # 1) Cost model, formulas (90.2) and (90.3): Llama 3.1 8B shapes L, d, n_kv, d_h, N_ne = 32, 4096, 8, 128, 6.98e9 def prefill_flops(T): return 2 * N_ne * T + 2 * L * d * T * T def kv_bytes(T): return 2 * L * n_kv * d_h * 2 * T # BF16 for name, T in [("whole corpus", 100_100), ("RAG, k=5, c=256", 5 * 256 + 100)]: print(f"{name:16s} T={T:7d} prefill {prefill_flops(T):.2e} FLOPs KV {kv_bytes(T)/1e9:.2f} GB") # 2) FLARE trigger and masked query, formulas (90.5) and (90.6) def flare(tokens, probs, theta=0.8, beta=0.4): if min(probs) >= theta: return None # confident: keep the draft return " ".join(t if p >= beta else "___" for t, p in zip(tokens, probs)) print(flare("She was born in 1816 in London .".split(), [0.97, 0.96, 0.95, 0.94, 0.34, 0.93, 0.62, 0.98])) # 3) REPLUG: ensemble (90.9) and LSR loss (90.10)-(90.11) with a tiny trainable retriever D, n_docs = 16, 4 enc = torch.nn.Linear(D, D, bias=False) # the retriever (shared query/doc encoder) x_raw, docs_raw = torch.randn(D), torch.randn(n_docs, D) p_lm = torch.tensor([0.08, 0.52, 0.04, 0.30]) # frozen LM: P(y | d, x) for each document gamma, beta = 1.0, 0.1 opt = torch.optim.SGD(enc.parameters(), lr=0.01) for step in range(101): s = enc(docs_raw) @ enc(x_raw) # s(d, x) = E(d) . E(x) log_P = F.log_softmax(s / gamma, dim=0) # retriever distribution P_R(d | x) Q = F.softmax(p_lm / beta, dim=0) # LM "teacher" distribution, no gradient loss = (log_P.exp() * (log_P - Q.log())).sum() # KL(P_R || Q) opt.zero_grad(); loss.backward(); opt.step() if step % 25 == 0: lam = F.softmax(s.detach(), dim=0) # ensemble weights (90.9) print(f"step {step:3d} KL {loss.item():.4f} ensemble p(y|x) {(lam * p_lm).sum().item():.3f}") # 4) Self-RAG segment score, formula (90.14): (p(y_t), s_Rel, s_Sup, s_Use) cands = {"A": (0.40, 0.95, 0.20, 0.70), "B": (0.30, 0.90, 0.85, 0.60), "C": (0.35, 0.30, 0.40, 0.50)} w = {"rel": 1.0, "sup": 1.0, "use": 0.5} score = {k: p + w["rel"] * r + w["sup"] * s + w["use"] * u for k, (p, r, s, u) in cands.items()} print({k: round(v, 2) for k, v in score.items()}, "best:", max(score, key=score.get))
Output (PyTorch 2.8, CPU):
whole corpus T= 100100 prefill 4.02e+15 FLOPs KV 13.12 GB
RAG, k=5, c=256 T= 1380 prefill 1.98e+13 FLOPs KV 0.18 GB
She was born in ___ in London .
step 0 KL 2.7279 ensemble p(y|x) 0.156
step 25 KL 0.0052 ensemble p(y|x) 0.484
step 50 KL 0.0007 ensemble p(y|x) 0.488
step 75 KL 0.0002 ensemble p(y|x) 0.489
step 100 KL 0.0001 ensemble p(y|x) 0.490
{'A': 1.9, 'B': 2.35, 'C': 1.3} best: B
Note in part 3 that the gradient flows only into enc, the retriever. p_lm is a plain tensor with no gradient: in a real system it comes from the frozen LLM, possibly over an API. The ensemble probability rises from 0.156 to 0.490 while the LLM did not change at all.
Where this is used
| Application | What is retrieved | Why retrieval instead of long context |
|---|---|---|
| Enterprise chat over manuals, tickets, wikis | a few chunks per question, with citations | corpus far larger than any context; access rules per document; formula (90.2) cost per query |
| Coding assistants over a large repository | files, functions, docs relevant to the edit | a repository can hold millions of tokens; only a few files matter for one change |
| Search-backed assistants (open web) | web pages from a search engine; FLARE used Bing for WikiAsp | the web changes daily; it cannot be in the weights or in one prompt |
| Long-form writing with facts (reports, biographies) | new documents per sentence when the model is unsure (FLARE, Self-RAG) | different sentences need different facts; one query is not enough |
| Video-RAG: questions over hours of video | a few clips or key frames, found by a text-to-video retriever | one hour at 1 frame per second with 576 tokens per frame (LLaVA-1.5) is 2,073,600 tokens; 16 retrieved frames are 9,216 (session 210) |
| Key-moment search and video fingerprinting | clip embeddings near the query clip or text | the search runs over vectors; the VLM only reads the few matches to describe or confirm them |
| Multimodal RAG over PDFs and slides | page images or regions plus text | each page image costs hundreds of visual tokens; retrieve pages first (session 219) |
Common mistakes
- Comparing only token counts. The compute ratio is larger than the token ratio when the long prompt is past \(N/(Ld)\) tokens, because of the square term in (90.2). And the cache cost (90.3) is paid again at every decode step.
- Ignoring retrieval misses. A 200 times cheaper answer is worthless if the right chunk is not in the top \(k\). Measure retrieval recall at \(k\) before tuning the LLM side (session 153).
- Making \(k c\) large “to be safe”. With 20 chunks of 1,024 tokens, the RAG prompt has over 20k tokens; most of the saving is gone, and the right chunk may be lost in the middle.
- Forgetting that a prepended document breaks the KV cache. In FLARE-style active retrieval, each retrieval at the front of the prompt forces a new prefill of everything after it (90.8).
- Counting the REPLUG ensemble as one prompt. It is \(k\) sequences: \(k\) times the input tokens, \(k\) caches, and \(k\) times the decode work (90.12).
- Not rebuilding the index after training the retriever. If the document encoder changes, old document embeddings no longer match new query embeddings. REPLUG re-encodes the corpus every 3,000 steps.
- Trusting raw token probabilities as confidence. FLARE’s thresholds assume reasonable calibration. Instruction tuning and some sampling settings can make probabilities over-confident; tune \(\theta\) on held-out data. [Likely]
- Mismatched encoders. Query and document vectors must come from the same retriever version (or the matching pair of encoders).
Interview questions
Why can RAG be seen as input compression?basic
Models now have 128k to 1M context. Why use RAG at all?core
Derive the prefill cost of a prompt of \(T\) tokens. When does attention dominate?core
Explain FLARE. What are \(\theta\) and \(\beta\)?core
What does active retrieval cost in a serving system?advanced
How does REPLUG train a retriever if the LLM is a black box?core
REPLUG ensemble or concatenating all chunks into one prompt: compare the costs.advanced
What are Self-RAG’s reflection tokens and how are they used at inference?core
FLARE, REPLUG, Self-RAG: which would you pick for a closed API model, and why?basic
Design question: answer questions over 500 hours of sports video. How do the ideas of this session apply?advanced
Your RAG answers got worse after you raised \(k\) from 5 to 30. Why might that happen?core
Exercises
- Llama 3.1 8B, corpus 30,000 tokens, \(T_q = 100\), \(k = 4\), \(c = 512\). Compute \(T_{\text{rag}}\), \(\rho\) and the KV memory of each prompt in BF16.
- For Llama 3.1 70B, at what prompt length does the attention term of (90.2) equal the weight term? Use \(N = 68.45\)B.
- In Figure 3, which sentences retrieve at \(\theta = 0.85\)? What is the masked query of sentence 5 at \(\beta = 0.7\)?
- In FLARE, a 500-token prefix has been written. One retrieval adds 3 chunks of 256 tokens in front, with \(T_x = 50\). How many tokens must be prefilled again?
- REPLUG with \(\gamma = 1\): scores \(s = (1, 0)\) for two chunks and \(P_{LM}(y \mid d, x) = (0.2, 0.6)\). Compute the ensemble probability (90.9), and \(Q\) with \(\beta = 0.1\).
- In Figure 5, with \(w^{\text{Rel}} = 1\), \(w^{\text{Use}} = 0.5\), find the value of \(w^{\text{Sup}}\) where A and B tie.
Answerscheck
Key takeaways
- RAG is input compression: the prompt holds \(T_q + kc\) tokens instead of \(T_q + T_c\) (90.1). It is the strongest form, because most of the corpus is never read.
- Prefill costs \(2NT + 2LdT^2\) FLOPs (90.2) and the cache costs \(2Ln_{kv}d_hbT\) bytes (90.3). For Llama 3.1 8B and a 100k corpus, RAG with 5 chunks of 256 saves about 204 times the compute and 72.5 times the memory.
- Long context still wins when facts are spread out, retrieval recall is poor, or a shared prefix can be cached and reused.
- FLARE retrieves only when a drafted token is below \(\theta\), with the unsure tokens masked out of the query (90.5, 90.6); each retrieval costs a new prefill and a lost draft (90.8).
- REPLUG ensembles over chunks for a black-box LLM (90.9) and trains the retriever to match the LLM’s preferences with \(\KL(P_R \| Q)\) (90.10, 90.11); the ensemble costs \(k\) sequences (90.12).
- Self-RAG fine-tunes the model to write reflection tokens: it retrieves when \(p(\text{yes}) \gt \delta\) and ranks candidates by probability plus weighted critique scores (90.13, 90.14).
Key formulas
References for this session
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Jiang et al. (2023). Active Retrieval Augmented Generation (FLARE)
- Shi et al. (2023). REPLUG: Retrieval-Augmented Black-Box Language Models
- Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Jiang et al. (2023). Active Retrieval Augmented Generation (FLARE)
- Shi et al. (2023). REPLUG: Retrieval-Augmented Black-Box Language Models
- Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning
- Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
- Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
- Xu et al. (2023). RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
- Jiang et al. (2023). LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models