LLM & VLM courseBlog
90Track 5: LLM inference and serving · session 13 of 65

RAG as input compression: FLARE, REPLUG, Self-RAG

Putting a whole document collection into the prompt is slow and costs a lot of GPU memory. Retrieval-augmented generation (RAG) puts only the few relevant pieces into the prompt, so it is also a way to compress the input. This session does the cost math for long context against retrieval, then studies three ways to make retrieval smarter: FLARE (retrieve only when the model is unsure), REPLUG (train the retriever with a frozen LLM as the teacher) and Self-RAG (the model decides and checks its own retrieval with special tokens).

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 60, 78, 80 and 81 (retrieval models, prefill and decode, the KV cache, inference metrics)

What you will learn

  • Why RAG is a form of input compression: the prompt holds k chunks instead of the whole corpus
  • The cost formulas for prefill FLOPs and KV-cache memory, and how to compare long context with retrieval for a real model
  • FLARE: draft the next sentence, retrieve only when a token is below a confidence threshold, and build the query by masking unsure tokens
  • REPLUG: an ensemble over retrieved documents for a black-box LLM, and LSR training that makes the retriever agree with the LLM
  • Self-RAG: reflection tokens that decide when to retrieve and score relevance, support and usefulness
  • The hidden serving costs of each method: re-prefill after each retrieval, k parallel sequences, extra decode tokens
  • When a long context is the better choice, and how to apply the same idea to video (retrieve clips, not all frames)

Why this matters

A company has 100,000 tokens of manuals. A user asks one question. You can put all 100,000 tokens in the prompt, or you can search the manuals first and put in only the five best paragraphs, about 1,300 tokens. For Llama 3.1 8B the first choice needs about 200 times more prefill compute and about 70 times more KV-cache memory. This session shows where those numbers come from.

Retrieval-augmented generation (RAG) means: before the model answers, a retriever (a search system) finds a few relevant pieces of text in a large collection, and we add only those pieces to the prompt. RAG was introduced to give models fresh and private knowledge and to reduce made-up facts; session 152 covers the basic pipeline. The efficient-inference survey adds a second view: RAG is also a way to compress the input. In its words, instead of merging all information into an excessively long prompt, RAG adds only the relevant retrieved information, so the prompt becomes much shorter. [Certain, survey Section 4.1.4]

This is the third session on input compression. Session 88 deleted unimportant tokens from a long prompt, and session 89 rewrote the prompt into a short summary or a few learned vectors. RAG goes further: most of the input never enters the prompt at all. The survey names three methods that make retrieval smarter, and they are the core of this session:

  • FLARE decides when to retrieve: only when the model is about to be unsure.
  • REPLUG keeps the LLM frozen as a black box and uses it as a teacher to improve the retriever.
  • Self-RAG trains the model to emit special reflection tokens that decide when to retrieve and grade what it retrieved.

Real systems use these ideas every day: search-backed chat assistants, coding assistants that pull the right files from a large repository, and video assistants that find a few clips in hours of footage. In all of them, the retriever decides what the expensive model must read, so it decides the cost.

RAG as input compression

Two ways to give a model knowledge

Let the user question have \(T_q\) tokens and the corpus (all documents) have \(T_c\) tokens. Split the corpus into chunks of \(c\) tokens each, and let the retriever pick the best \(k\) chunks. The two prompt lengths are:

\[ T_{\text{long}} = T_q + T_c, \qquad T_{\text{rag}} = T_q + k\,c, \qquad \rho = \frac{T_{\text{long}}}{T_{\text{rag}}} \tag{90.1} \]
Read the symbols
\(T_q\)
tokens in the question and the fixed instructions (system prompt, template)
\(T_c\)
tokens in the whole corpus that might hold the answer
\(k, c\)
number of retrieved chunks, and tokens per chunk (often 128 to 1,024)
\(\rho\)
the compression ratio of the prompt: how many times shorter the RAG prompt is

With \(T_q = 100\), \(T_c = 100{,}000\), \(k = 5\), \(c = 256\): \(T_{\text{rag}} = 1{,}380\) and \(\rho = 100{,}100 / 1{,}380 = 72.5\). A prompt-pruning method such as LLMLingua aims for ratios of a few times up to about 20 times; RAG reaches 72 here because it never reads most of the corpus. [Likely, ratios as reported in the LLMLingua papers] The price is a new risk: if the retriever misses the right chunk, the model never sees it.

question and generated textretrieved chunkscorpus that the LLM must read
Figure 1. Three ways to bring knowledge into the prompt. Top: put the whole corpus in the prompt (long context). Middle: retrieve once, before generation (classic RAG; REPLUG uses this shape). Bottom: retrieve during generation, only when it is needed (FLARE and Self-RAG). Bar widths are to scale for the worked example: 100,100 tokens against 1,380.

The cost math: prefill FLOPs

Recall from session 78 that the prompt is processed in one parallel prefill pass, then the answer is written token by token in decode. Prefill work has two parts. The weight matrices cost about \(2N\) FLOPs per token, where \(N\) is the number of non-embedding parameters (session 25). Attention adds a part that grows with the square of the length, because every token looks at all earlier tokens:

\[ F_{\text{prefill}}(T) \approx \underbrace{2\,N\,T}_{\text{weights}} \;+\; \underbrace{2\,L\,d\,T^2}_{\text{attention scores and values}} \tag{90.2} \]
Read the symbols
\(T\)
prompt length in tokens
\(N\)
non-embedding parameters (6.98 billion for Llama 3.1 8B; the input embedding is a lookup, and only the last position needs the output head in prefill)
\(L, d\)
number of layers, and model width \(d = n_h d_h\) (all query heads together)
\(2LdT^2\)
\(QK^\top\) and the weighted sum with \(V\): each is about \(2 d\) FLOPs per (query, key) pair, and a causal mask keeps about \(T^2/2\) pairs, so \(2 \times 2d \times T^2/2 = 2dT^2\) per layer

The attention term passes the weight term when \(2LdT^2 \gt 2NT\), that is when \(T \gt N/(Ld)\). For Llama 3.1 8B (\(L = 32\), \(d = 4{,}096\)) this is at \(T = 53{,}248\) tokens; for Llama 3.1 70B (\(L = 80\), \(d = 8{,}192\)) at \(104{,}448\). [Certain, from the configs and (90.2)] So below about 50k tokens, prefill cost grows almost linearly with the prompt; above it, the square term takes over and each extra token costs more than the one before.

The cost math: KV-cache memory

Session 80 gave the memory of the cache. Every prompt token stays in the cache for the whole answer:

\[ M_{\text{KV}}(T) = 2 \cdot L \cdot n_{kv} \cdot d_h \cdot b \cdot T \quad \text{bytes per sequence} \tag{90.3} \]
Read the symbols
\(2\)
one key and one value
\(n_{kv}, d_h\)
number of key/value heads (8 for Llama 3.1 8B and 70B, with GQA) and head size (128)
\(b\)
bytes per number: 2 for BF16

For Llama 3.1 8B this is 131,072 bytes (128 KiB) per token; for 70B, 327,680 bytes (320 KiB). The cache also makes each decode step slower, because each new token must read the whole cache from memory (session 82). A 100k-token prompt makes every one of the answer tokens slower, not only the first one.

Worked example: Llama 3.1 8B, 100k tokens of manuals

Long context: \(T = 100{,}100\). Weights: \(2 \times 6.98\times10^9 \times 100{,}100 = 1.40\times10^{15}\) FLOPs. Attention: \(2 \times 32 \times 4{,}096 \times 100{,}100^2 = 2.63\times10^{15}\). Total \(4.02\times10^{15}\) FLOPs. KV cache: \(131{,}072 \times 100{,}100 = 13.1\) GB for one user.

RAG with \(k = 5\), \(c = 256\): \(T = 1{,}380\). Weights \(1.93\times10^{13}\), attention \(5.0\times10^{11}\), total \(1.98\times10^{13}\) FLOPs. KV cache: 0.18 GB.

Ratios: 204 times less prefill compute and 72.5 times less cache memory. The compute ratio is larger than the token ratio (72.5) because RAG also removes the square term. At an assumed 400 TFLOP/s of useful compute (a little under half of the dense BF16 peak of an H100, which is about 990 TFLOP/s [Likely]), prefill takes about 10.1 s against 0.049 s. With 16 users at once, the long-context caches need 210 GB, more than two 80 GB GPUs, while the RAG caches need 2.9 GB.

The same comparison for Llama 3.1 70B: about 67 s against 0.48 s of prefill, and 32.8 GB against 0.45 GB of cache per user.

Corpus \(T_c\)100,000 Chunks \(k\)5 Chunk size \(c\)256 Users \(B\)16
Figure 2. Long context against retrieval, computed with formulas (90.1) to (90.3). The question and template are fixed at \(T_q = 100\) tokens. Bars use a log scale, because the two sides can differ by more than 100 times. Time assumes 400 TFLOP/s of useful compute.
How to read Figure 2

Start at the default (100k corpus, 5 chunks of 256). Move the corpus slider down to 8k: the ratio falls to about 6, and long context becomes a fair choice, especially if the same corpus is reused and its cache is kept (prefix caching). Move it up to 128k with the 70B model: prefill alone takes more than a minute. Now raise \(k\) to 20 and \(c\) to 1,024: the RAG prompt has 20,580 tokens and a lot of the saving is gone. Retrieval saves compute only if \(k c\) stays much smaller than \(T_c\).

What the cost math does not show

Retrieval is not free, but it is cheap. Encoding a 32-token query with a 110M-parameter encoder (the size of Contriever, a BERT-base model) costs about \(2 \times 110\times10^6 \times 32 = 7\times10^9\) FLOPs, which is 0.04% of the RAG prefill above. The search in a vector index takes milliseconds (session 60). The document embeddings are computed once, offline. [Likely]

Long context has two real advantages, and an honest answer in an interview names them:

  • No retrieval misses. If the answer needs facts spread over many places, or the question is vague, a retriever can miss them. The whole document cannot be missed.
  • Reuse. If many questions use the same document, its KV cache can be computed once and kept (prefix caching, session 134). Then prefill is paid once, but the memory is held for all users and every decode step still reads it.

Also, more context is not always better quality. Models use information at the start and end of a long prompt better than in the middle (“lost in the middle”). [Certain, Liu et al. 2023] A short prompt with the right chunks can give a better answer than a long prompt with the right chunk buried at position 50,000.

Algorithm 1: classic retrieve-then-generate (one retrieval)
  1. offline: split the corpus into chunks of \(c\) tokens; encode each chunk \(d\) into a vector \(E_d(d)\); build a vector index
  2. input: question \(x\); number of chunks \(k\)
  3. \(\mathbf{e} \leftarrow E_q(x)\) // a small encoder, about \(10^{10}\) FLOPs
  4. \(D' \leftarrow\) the \(k\) chunks with the largest score \(s(d, x) = E_d(d)\cdot\mathbf{e}\) // formula (90.4), approximate nearest-neighbour search
  5. prompt \(\leftarrow\) instructions, then \(D'\), then \(x\) // \(T_q + kc\) tokens instead of \(T_q + T_c\)
  6. prefill the prompt; decode the answer with the KV cache
  7. return the answer and, if needed, the ids of the chunks used (for citations)

The score in line 4 is the dense-retrieval score from DPR and Contriever:

\[ s(d, x) = E_d(d)^\top E_q(x), \qquad D' = \operatorname{top-}k_{\,d \in \mathcal{D}}\; s(d, x) \tag{90.4} \]
Read the symbols
\(E_d, E_q\)
the document encoder and the query encoder (Contriever uses one shared encoder for both)
\(\mathcal{D}\)
all chunks in the corpus; \(D'\) is the small set we keep

Classic RAG has two weak points. It retrieves exactly once, with only the question as the query, even when a long answer needs different facts in different sentences. And the retriever was trained on its own, so it does not know which chunks actually help this LLM. FLARE attacks the first weakness, REPLUG the second, Self-RAG both.

FLARE: retrieve when the model is about to be unsure

The idea

Think of a model writing a short biography. Most sentences are easy: grammar, common facts. A few sentences hold a hard fact, such as a date. Retrieving before every sentence wastes work. Retrieving only once, before the answer, uses a query (the question) that may not mention the hard fact at all.

FLARE (Forward-Looking Active REtrieval, Jiang et al. 2023) asks the model itself. Before writing sentence \(t\), the model writes a draft of it without new documents. If every token of the draft has high probability, the model is confident: keep the draft. If some token has low probability, the model is unsure: use the draft to build a search query, retrieve, and write the sentence again with the documents. [Certain] The name says “forward-looking” because the query is built from what the model is about to say, not from what it has already said.

The math

Let \(\hat{s}_t = (\hat{s}_{t,1}, \dots, \hat{s}_{t,m})\) be the draft of sentence \(t\), generated from the input \(x\) and the earlier output \(y_{\lt t}\). FLARE-direct retrieves when any token is below a threshold \(\theta\):

\[ \text{retrieve at step } t \iff \min_{i = 1..m}\; p\!\left(\hat{s}_{t,i} \mid x,\, y_{\lt t},\, \hat{s}_{t,\lt i}\right) \lt \theta \tag{90.5} \]

The query is the draft with its unsure tokens masked. A wrong guess like “1816” would push the search toward wrong pages, so tokens with probability below a second threshold \(\beta\) are removed:

\[ q_t = \operatorname{mask}_\beta(\hat{s}_t): \quad \hat{s}_{t,i} \mapsto \begin{cases} \text{(removed)} & p(\hat{s}_{t,i}) \lt \beta \\ \hat{s}_{t,i} & \text{otherwise} \end{cases} \tag{90.6} \]

Then the sentence is written again, with the retrieved documents \(\mathcal{D}_{q_t}\) put in front:

\[ y_t = \begin{cases} \hat{s}_t & \text{if no retrieval} \\ \operatorname{LM}\!\left(\mathcal{D}_{q_t},\, x,\, y_{\lt t}\right) & \text{otherwise} \end{cases} \tag{90.7} \]
Read the symbols
\(\theta\)
trigger threshold. \(\theta = 0\): never retrieve. \(\theta = 1\): retrieve for every sentence. The paper used \(\theta = 0.8\) on 2WikiMultihopQA, ASQA and WikiAsp, and 0.4 on StrategyQA. [Certain]
\(\beta\)
masking threshold; the paper used \(\beta = 0.4\). [Certain] Larger \(\beta\) removes more tokens from the query.
\(\mathcal{D}_{q_t}\)
the documents found with query \(q_t\) (2 to 5 per retrieval in the paper, from BM25 over Wikipedia or from Bing) [Certain]

The paper also tried a second query type: send the unsure spans to another LLM that writes a question for each span (“When was Ada Lovelace born?”). And it tried FLARE-instruct, where the model is prompted to write [Search(query)] itself. FLARE-direct, with masked or generated queries, worked best. [Certain]

Trigger \(\theta\)0.80 Mask \(\beta\)0.40
Figure 3. FLARE-direct on a short biography. Each line is a draft sentence; the small number after each word is its probability (made-up values for teaching, one word = one token). Words below \(\theta\) are outlined. A sentence with at least one outlined word triggers retrieval; its masked query (blue) drops the words below \(\beta\). Cost assumes \(k = 2\) chunks of 256 tokens per retrieval and an 8-token question.
How to read Figure 3

At \(\theta = 0.8\), 4 of 6 sentences retrieve. Two of them (born in 1816, notes in 1842) hold real errors, and retrieval fixes them. Two others (Analytical Engine, died in 1852) were correct, so their retrieval was wasted but harmless. Set \(\theta = 0.5\): only the two wrong sentences retrieve, and the cost halves. Set \(\theta = 0\): no retrieval, and both errors stay in the answer. Now set \(\beta = 0.6\) and look at the queries: “Analytical” and “1852” are masked too, and the queries become vaguer. Thresholds trade cost against missed errors, and the best values depend on how well the model’s probabilities are calibrated.

The serving cost of FLARE

FLARE looks cheap because it retrieves less. But look at formula (90.7) from the point of view of the KV cache. The documents are put in front of \(x\) and \(y_{\lt t}\). A change at the front changes every later key and value (the causal mask makes position \(j\) depend on all earlier positions). So the cache of \(x\) and \(y_{\lt t}\) cannot be reused: each retrieval needs a new prefill of the whole prefix. The draft is also thrown away, so its decode steps are wasted:

\[ \Delta T_{\text{prefill}} = \sum_{t \in \mathcal{R}} \big( k c + T_x + |y_{\lt t}| \big), \qquad \Delta T_{\text{decode}} = \sum_{t \in \mathcal{R}} |\hat{s}_t| \tag{90.8} \]
Read the symbols
\(\mathcal{R}\)
the set of sentences that triggered retrieval
\(T_x, |y_{\lt t}|\)
tokens of the input and of the answer written so far
\(|\hat{s}_t|\)
tokens of the thrown-away draft
FLARE cost on Figure 3

At \(\theta = 0.8\), sentences 2, 4, 5 and 6 retrieve. The answer written before them has 7, 25, 35 and 47 tokens (sentence 5 is 12 tokens after its fix). With \(kc = 512\) and \(T_x = 8\): \(\Delta T_{\text{prefill}} = (520 + 7) + (520 + 25) + (520 + 35) + (520 + 47) = 2{,}194\) tokens, and the thrown-away drafts have \(8 + 10 + 11 + 5 = 34\) tokens.

At \(\theta = 1\) (retrieve for every sentence): 6 retrievals, 3,249 extra prefill tokens and 51 wasted draft tokens. At \(\theta = 0.5\): 2 retrievals, 1,082 tokens and 19 wasted tokens, and the same two errors are fixed. A single retrieval before the answer costs \(512 + 8 = 520\) tokens, but its query (“Write a short biography of Ada Lovelace”) is not about any specific fact.

A system can reduce the re-prefill cost by putting the documents after the current text instead of in front. Then the cache of the prefix stays valid and only the new documents are prefilled. This changes the prompt layout the model sees, so it is a quality trade-off. [Speculation: FLARE itself prepends the documents.]

The results show why the extra work can pay off. On 2WikiMultihopQA with text-davinci-003, exact match was 28.2 with no retrieval, 39.4 with one retrieval before the answer, 47.8 for the best baseline (question decomposition), and 51.0 for FLARE-direct. [Certain, FLARE Table]

Algorithm 2: FLARE-direct
  1. input: question \(x\); thresholds \(\theta, \beta\); chunks per retrieval \(k\)
  2. \(y \leftarrow\) empty
  3. while the answer is not finished:
  4.   \(\hat{s} \leftarrow\) draft the next sentence from \((x, y)\), keeping each token’s probability
  5.   if all probabilities \(\ge \theta\): \(y \leftarrow y + \hat{s}\); continue // confident: no retrieval
  6.   \(q \leftarrow \hat{s}\) with tokens below \(\beta\) removed // formula (90.6)
  7.   \(D \leftarrow\) retrieve \(k\) chunks with \(q\)
  8.   throw away \(\hat{s}\); prefill \((D, x, y)\) again // the old cache is not valid, formula (90.8)
  9.   \(s \leftarrow\) write the sentence again; \(y \leftarrow y + s\)
  10. return \(y\)

REPLUG: the frozen LLM teaches the retriever

A black-box LLM

Many teams use an LLM only through an API. They cannot change its weights or see its hidden states. REPLUG (Shi et al. 2023) is built for this case. It treats the LLM as a frozen black box and only adds things around it. [Certain] It has two parts: an inference trick (an ensemble over documents) and a training method for the retriever (LSR).

Inference: an ensemble over documents

Classic RAG puts all \(k\) chunks in one prompt. REPLUG runs the LLM \(k\) times instead, once per chunk, each time with only that chunk in front of the input. Then it mixes the \(k\) next-token distributions, giving more weight to chunks with a higher retrieval score:

\[ p(y \mid x, D') = \sum_{d \in D'} \lambda(d, x)\; p(y \mid d \circ x), \qquad \lambda(d, x) = \frac{e^{s(d,x)}}{\sum_{d' \in D'} e^{s(d',x)}} \tag{90.9} \]
Read the symbols
\(d \circ x\)
chunk \(d\) followed by the input \(x\) (concatenation)
\(p(y \mid d \circ x)\)
the LLM’s probability of the next token (or answer) given one chunk
\(\lambda(d, x)\)
the weight of chunk \(d\): a softmax over the retrieval scores of the \(k\) chunks

Two reasons for the ensemble. Each pass sees a short prompt, so more chunks can be used than fit in one context window. And no chunk is “in the middle” of a long prompt. The paper used the top 10 documents. [Certain]

Training: LM-supervised retrieval (LSR)

Which chunks really help? The LLM can tell us: a chunk helps if it makes the correct continuation \(y\) more likely. REPLUG turns this into a target distribution for the retriever. On training text, split each sequence into an input \(x\) and its true continuation \(y\). Retrieve the top chunks, then compute two distributions over them:

\[ P_R(d \mid x) = \frac{e^{s(d,x)/\gamma}}{\sum_{d'} e^{s(d',x)/\gamma}}, \qquad Q(d \mid x, y) = \frac{e^{P_{LM}(y \mid d, x)/\beta}}{\sum_{d'} e^{P_{LM}(y \mid d', x)/\beta}} \tag{90.10} \]

The retriever is trained so that its distribution agrees with the LLM’s distribution:

\[ \mathcal{L}_{\text{LSR}} = \frac{1}{|\mathcal{B}|} \sum_{x \in \mathcal{B}} \KL\!\big( P_R(\cdot \mid x) \,\|\, Q(\cdot \mid x, y) \big) \tag{90.11} \]
Read the symbols
\(P_R\)
the retriever’s view: how likely it is to pick each chunk. It depends on the retriever weights, so it gets the gradient.
\(Q\)
the LLM’s view: chunks that make \(y\) likely get high weight. The LLM is frozen; \(Q\) is a fixed target.
\(\gamma, \beta\)
softmax temperatures. Smaller values make the distributions sharper.
\(\mathcal{B}\)
a batch of training inputs

Only the retriever changes. The LLM gives only probabilities of \(y\), which many APIs return. Training changes the query encoder and the document encoder, so the stored document embeddings become old. REPLUG recomputes them and rebuilds the index every 3,000 steps. [Certain] The paper trained from Contriever on 800k sequences of 256 tokens from the Pile, with a corpus of 36M chunks of 128 tokens. [Certain]

LM temperature \(\beta\)0.10
Figure 4. One REPLUG LSR example with four retrieved chunks. Blue bars: the retriever’s distribution \(P_R\) (with \(\gamma = 1\), so it equals the ensemble weights \(\lambda\)). Orange bars: the LM target \(Q\), from the LM probabilities of the true continuation (numbers under each chunk). Each training step is one gradient step (learning rate 1) on the four scores \(s(d,x)\) that lowers \(\KL(P_R \,\|\, Q)\).
How to read Figure 4

At the start, the retriever likes chunk 1 best (score 2.6), but the LLM finds chunk 1 almost useless (\(P_{LM} = 0.08\)) and chunk 2 very useful (0.52). The ensemble probability of the true continuation is only 0.218. Press “Train 1 step”: the KL falls from 1.887 to 0.624 and the ensemble probability rises to 0.361. After 10 steps the retriever puts about 90% of its weight on chunk 2 and the ensemble reaches 0.485. Now raise \(\beta\) to 1: the target \(Q\) becomes almost flat, because the four LM probabilities differ by less than 0.5, and the retriever learns little. The temperature decides how strongly the LLM’s preference is turned into a training signal. (In the real method the scores come from an encoder, and the gradient updates the encoder weights.)

The numbers in Figure 4

Scores \(s = (2.6, 2.2, 1.8, 1.0)\) give ensemble weights \(\lambda = (0.431, 0.289, 0.194, 0.087)\). LM probabilities \(P_{LM}(y \mid d, x) = (0.08, 0.52, 0.04, 0.30)\). Ensemble (90.9): \(0.431 \times 0.08 + 0.289 \times 0.52 + 0.194 \times 0.04 + 0.087 \times 0.30 = 0.218\). A prompt with only chunk 2 would give 0.52.

Target with \(\beta = 0.1\): \(Q = \softmax(0.8, 5.2, 0.4, 3.0) = (0.011, 0.884, 0.007, 0.098)\). \(\KL(P_R \| Q) = 1.887\). The gradient of the KL with respect to the scores is \(P_{R,i}(\log P_{R,i} - \log Q_i - \KL)/\gamma\). One step with learning rate 1 gives \(s = (1.827, 3.068, 1.530, 1.174)\), KL 0.624, ensemble 0.361.

The serving cost of REPLUG

The ensemble is not free. With \(k\) chunks of \(c\) tokens and an input of \(T_x\) tokens, the two layouts cost:

\[ \underbrace{T_{\text{concat}} = k c + T_x}_{\text{one sequence}} \qquad \text{vs} \qquad \underbrace{T_{\text{ens}} = k\,(c + T_x)}_{k \text{ sequences of } c + T_x} \tag{90.12} \]

The input \(x\) is processed \(k\) times, and during decoding there are \(k\) sequences, each with its own KV cache, and every generated token is computed \(k\) times. With \(k = 10\), \(c = 128\), \(T_x = 256\): one concatenated prompt has 1,536 tokens, while the ensemble has 10 sequences of 384 tokens, 3,840 tokens in total: 2.5 times more weight FLOPs. The attention term is smaller (\(10 \times 384^2 = 1.47\)M pairs against \(1{,}536^2 = 2.36\)M), but at this length the weight term dominates. The \(k\) sequences run as one batch, so latency grows less than cost. If the chunks come after a shared prefix (instructions), prefix caching can share it across the \(k\) sequences. [Likely]

Results: with the trained retriever, REPLUG improved GPT-3 (175B) language modeling by 6.3% (bits per byte from 0.80 to 0.75), and Codex on MMLU from 68.3 to 71.4 with the ensemble and 71.8 with LSR, which the paper reports as 4.5% and 5.1% gains. [Certain, REPLUG Tables 1 and 2]

Algorithm 3: REPLUG LSR training of the retriever
  1. input: frozen LLM; retriever \(E\) (for example Contriever); corpus \(\mathcal{D}\); training texts
  2. encode all chunks of \(\mathcal{D}\) with \(E\); build the index
  3. for step \(= 1, 2, \dots\):
  4.   sample a batch; split each text into input \(x\) and true continuation \(y\)
  5.   retrieve the top chunks \(D'\) for each \(x\); compute \(P_R(d \mid x)\) // formula (90.10), with gradient
  6.   for each \(d \in D'\): ask the LLM for \(P_{LM}(y \mid d, x)\); compute \(Q\) // no gradient through the LLM
  7.   update \(E\) to lower \(\KL(P_R \| Q)\) // formula (90.11)
  8.   every \(T\) steps: re-encode the corpus and rebuild the index // stored embeddings are stale; \(T = 3{,}000\) in the paper
  9. return the trained retriever

Self-RAG: the model decides and checks

Reflection tokens

FLARE decides with token probabilities, which are a rough signal. REPLUG never decides: it always retrieves. Self-RAG (Asai et al. 2023) teaches the model to make the decisions itself, by writing special reflection tokens into its output. [Certain] There are four types:

TokenLooks atValuesQuestion it answers
Retrieveinput \(x\) (and output so far)yes, no, continueDo I need documents for the next segment?
IsRel\(x\) and one passage \(d\)relevant, irrelevantIs this passage about the question?
IsSup\(x\), \(d\) and the segment \(y_t\)fully, partially, no supportIs what I wrote backed by the passage?
IsUse\(x\) and \(y_t\)5, 4, 3, 2, 1Is the answer useful overall?

Training has two models. A critic (Llama 2 7B) learns to predict reflection tokens from labels that GPT-4 produced. Then the critic inserts reflection tokens into 150k instruction-output pairs, and the generator (Llama 2 7B or 13B) is trained with normal next-token loss on this text, so it learns to write the tokens itself. At inference only the generator is needed. [Certain]

Inference: retrieve on demand, then score the candidates

Before each segment, the model predicts the Retrieve token. Self-RAG retrieves when the normalized probability of “yes” is above a threshold \(\delta\):

\[ \text{retrieve} \iff \frac{p(\texttt{Retrieve}=\text{yes})}{\sum_{r \in \{\text{yes, no, continue}\}} p(\texttt{Retrieve}=r)} \gt \delta \tag{90.13} \]

The paper used \(\delta = 0.2\) for most tasks. [Certain] If it retrieves, the model processes \(K\) passages in parallel, one candidate segment per passage, and writes the critique tokens for each. Then it ranks the candidates by a score that mixes the segment’s probability with the critique:

\[ f(y_t, d) = p(y_t \mid x, d, y_{\lt t}) + \sum_{G \in \{\text{Rel, Sup, Use}\}} w^G s_t^G, \qquad s_t^G = \frac{p_t(\hat{r}^G)}{\sum_{i=1}^{N^G} p_t(r^G_i)} \tag{90.14} \]
Read the symbols
\(\hat{r}^G\)
the best value of critique type \(G\) (for example “relevant” or “fully supported”)
\(s_t^G\)
the probability of that best value, normalized over the \(N^G\) values of the type
\(w^G\)
weights set at inference time, without retraining. Defaults in the paper: 1.0 for IsRel, 1.0 for IsSup, 0.5 for IsUse. [Certain] A user who cares most about citations can raise \(w^{\text{Sup}}\).

The paper keeps the best candidates in a segment-level beam search (beam width 2) and uses up to 5 passages from its retriever per step. [Certain]

\(w^{\text{Rel}}\)1.0 \(w^{\text{Sup}}\)1.0 \(w^{\text{Use}}\)0.5 Threshold \(\delta\)0.20
Figure 5. Self-RAG scoring of three candidate segments, one per retrieved passage, with formula (90.14). Made-up probabilities for teaching. The model’s normalized \(p(\texttt{Retrieve}=\text{yes})\) for this question is 0.35. Candidate A is fluent and about a relevant passage, but the passage does not support what it says. Candidate B is less fluent but well supported.
How to read Figure 5

With the default weights (1, 1, 0.5), B wins with 2.35 against 1.90 for A. Set \(w^{\text{Sup}} = 0\): A wins (1.70 against 1.50), because without the support check, fluency and relevance decide. This is the case Self-RAG is built to avoid: a confident answer that the passage does not back up. Now raise \(\delta\) above 0.35: the model decides not to retrieve, no passages are read, and it answers from its own knowledge (score 0.45 + 0.5 × 0.55 = 0.725, only IsUse applies). The weights are a control knob at inference time, and the threshold is a cost knob.

The serving cost of Self-RAG

Self-RAG saves retrieval when it is not needed, like FLARE. But when it does retrieve, it runs \(K\) candidates in parallel (like the REPLUG ensemble), keeps a beam, and writes extra reflection tokens, which are extra decode steps. It also needs a fine-tuned model, so it does not work with a closed API. Its gain is quality: on PopQA, Self-RAG 13B scored 55.8 against 51.8 for Llama 2-chat 13B with the same retrieval and 29.3 for ChatGPT without retrieval. [Certain, Self-RAG Table 2] On ASQA exact match, ChatGPT with retrieval was higher (40.7 against 31.7). [Certain] So it is a strong method for small open models, not a win everywhere.

Putting the three side by side

Classic RAGFLAREREPLUGSelf-RAG
When to retrieveonce, before the answerwhen a draft token is below \(\theta\)once, before the answerwhen \(p(\text{yes}) \gt \delta\)
Querythe questionmasked draft of the next sentencethe inputinput and output so far
LLM changes?nono (needs token probabilities)no (needs probabilities of \(y\) for training)yes, fine-tuned with reflection tokens
Retriever changes?nonoyes, LSR trainingno
Extra serving costone prefill of \(T_q + kc\)re-prefill and lost draft per retrieval (90.8)\(k\) parallel sequences (90.12)\(K\) parallel candidates, beam, reflection tokens

The methods can be combined. A trained retriever (REPLUG) can serve FLARE’s queries. RECOMP from session 89 can compress the retrieved chunks before they enter the prompt, and LongLLMLingua can prune them. Each step reduces the tokens the expensive model must read.

Runnable code

The code below checks the cost formulas, runs the FLARE trigger and masked query, trains a tiny retriever with the REPLUG LSR loss through real autograd, and scores Self-RAG candidates. It runs on a CPU in about a second and prints the same numbers every time.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# 1) Cost model, formulas (90.2) and (90.3): Llama 3.1 8B shapes
L, d, n_kv, d_h, N_ne = 32, 4096, 8, 128, 6.98e9
def prefill_flops(T):  return 2 * N_ne * T + 2 * L * d * T * T
def kv_bytes(T):       return 2 * L * n_kv * d_h * 2 * T          # BF16
for name, T in [("whole corpus", 100_100), ("RAG, k=5, c=256", 5 * 256 + 100)]:
    print(f"{name:16s} T={T:7d}  prefill {prefill_flops(T):.2e} FLOPs  KV {kv_bytes(T)/1e9:.2f} GB")

# 2) FLARE trigger and masked query, formulas (90.5) and (90.6)
def flare(tokens, probs, theta=0.8, beta=0.4):
    if min(probs) >= theta:
        return None                                    # confident: keep the draft
    return " ".join(t if p >= beta else "___" for t, p in zip(tokens, probs))
print(flare("She was born in 1816 in London .".split(),
            [0.97, 0.96, 0.95, 0.94, 0.34, 0.93, 0.62, 0.98]))

# 3) REPLUG: ensemble (90.9) and LSR loss (90.10)-(90.11) with a tiny trainable retriever
D, n_docs = 16, 4
enc = torch.nn.Linear(D, D, bias=False)               # the retriever (shared query/doc encoder)
x_raw, docs_raw = torch.randn(D), torch.randn(n_docs, D)
p_lm = torch.tensor([0.08, 0.52, 0.04, 0.30])          # frozen LM: P(y | d, x) for each document
gamma, beta = 1.0, 0.1
opt = torch.optim.SGD(enc.parameters(), lr=0.01)
for step in range(101):
    s = enc(docs_raw) @ enc(x_raw)                    # s(d, x) = E(d) . E(x)
    log_P = F.log_softmax(s / gamma, dim=0)            # retriever distribution P_R(d | x)
    Q = F.softmax(p_lm / beta, dim=0)                  # LM "teacher" distribution, no gradient
    loss = (log_P.exp() * (log_P - Q.log())).sum()     # KL(P_R || Q)
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 25 == 0:
        lam = F.softmax(s.detach(), dim=0)              # ensemble weights (90.9)
        print(f"step {step:3d}  KL {loss.item():.4f}  ensemble p(y|x) {(lam * p_lm).sum().item():.3f}")

# 4) Self-RAG segment score, formula (90.14): (p(y_t), s_Rel, s_Sup, s_Use)
cands = {"A": (0.40, 0.95, 0.20, 0.70), "B": (0.30, 0.90, 0.85, 0.60), "C": (0.35, 0.30, 0.40, 0.50)}
w = {"rel": 1.0, "sup": 1.0, "use": 0.5}
score = {k: p + w["rel"] * r + w["sup"] * s + w["use"] * u for k, (p, r, s, u) in cands.items()}
print({k: round(v, 2) for k, v in score.items()}, "best:", max(score, key=score.get))

Output (PyTorch 2.8, CPU):

whole corpus     T= 100100  prefill 4.02e+15 FLOPs  KV 13.12 GB
RAG, k=5, c=256  T=   1380  prefill 1.98e+13 FLOPs  KV 0.18 GB
She was born in ___ in London .
step   0  KL 2.7279  ensemble p(y|x) 0.156
step  25  KL 0.0052  ensemble p(y|x) 0.484
step  50  KL 0.0007  ensemble p(y|x) 0.488
step  75  KL 0.0002  ensemble p(y|x) 0.489
step 100  KL 0.0001  ensemble p(y|x) 0.490
{'A': 1.9, 'B': 2.35, 'C': 1.3} best: B

Note in part 3 that the gradient flows only into enc, the retriever. p_lm is a plain tensor with no gradient: in a real system it comes from the frozen LLM, possibly over an API. The ensemble probability rises from 0.156 to 0.490 while the LLM did not change at all.

Where this is used

ApplicationWhat is retrievedWhy retrieval instead of long context
Enterprise chat over manuals, tickets, wikisa few chunks per question, with citationscorpus far larger than any context; access rules per document; formula (90.2) cost per query
Coding assistants over a large repositoryfiles, functions, docs relevant to the edita repository can hold millions of tokens; only a few files matter for one change
Search-backed assistants (open web)web pages from a search engine; FLARE used Bing for WikiAspthe web changes daily; it cannot be in the weights or in one prompt
Long-form writing with facts (reports, biographies)new documents per sentence when the model is unsure (FLARE, Self-RAG)different sentences need different facts; one query is not enough
Video-RAG: questions over hours of videoa few clips or key frames, found by a text-to-video retrieverone hour at 1 frame per second with 576 tokens per frame (LLaVA-1.5) is 2,073,600 tokens; 16 retrieved frames are 9,216 (session 210)
Key-moment search and video fingerprintingclip embeddings near the query clip or textthe search runs over vectors; the VLM only reads the few matches to describe or confirm them
Multimodal RAG over PDFs and slidespage images or regions plus texteach page image costs hundreds of visual tokens; retrieve pages first (session 219)

Common mistakes

  • Comparing only token counts. The compute ratio is larger than the token ratio when the long prompt is past \(N/(Ld)\) tokens, because of the square term in (90.2). And the cache cost (90.3) is paid again at every decode step.
  • Ignoring retrieval misses. A 200 times cheaper answer is worthless if the right chunk is not in the top \(k\). Measure retrieval recall at \(k\) before tuning the LLM side (session 153).
  • Making \(k c\) large “to be safe”. With 20 chunks of 1,024 tokens, the RAG prompt has over 20k tokens; most of the saving is gone, and the right chunk may be lost in the middle.
  • Forgetting that a prepended document breaks the KV cache. In FLARE-style active retrieval, each retrieval at the front of the prompt forces a new prefill of everything after it (90.8).
  • Counting the REPLUG ensemble as one prompt. It is \(k\) sequences: \(k\) times the input tokens, \(k\) caches, and \(k\) times the decode work (90.12).
  • Not rebuilding the index after training the retriever. If the document encoder changes, old document embeddings no longer match new query embeddings. REPLUG re-encodes the corpus every 3,000 steps.
  • Trusting raw token probabilities as confidence. FLARE’s thresholds assume reasonable calibration. Instruction tuning and some sampling settings can make probabilities over-confident; tune \(\theta\) on held-out data. [Likely]
  • Mismatched encoders. Query and document vectors must come from the same retriever version (or the matching pair of encoders).

Interview questions

Why can RAG be seen as input compression?basic
The knowledge a model may need is a large corpus of \(T_c\) tokens. Putting it all in the prompt costs prefill compute (90.2) and KV memory (90.3) for every token. RAG keeps only \(k\) chunks of \(c\) tokens, so the prompt shrinks from \(T_q + T_c\) to \(T_q + kc\). For 100k corpus tokens and 5 chunks of 256, that is a 72.5 times shorter prompt. The survey on efficient inference lists RAG with prompt pruning and summarization for this reason.
Models now have 128k to 1M context. Why use RAG at all?core
Cost and quality. Cost: for Llama 3.1 8B, a 100k prompt needs about \(4.0\times10^{15}\) prefill FLOPs and 13.1 GB of cache per user, against \(2.0\times10^{13}\) and 0.18 GB for a 1,380-token RAG prompt; every decode step also reads the whole cache. Quality: models use the middle of long prompts worse (“lost in the middle”). Scale: many corpora are far larger than any context. Long context still wins when facts are spread everywhere, when retrieval recall is poor, or when one document serves many questions and its cache can be reused with prefix caching.
Derive the prefill cost of a prompt of \(T\) tokens. When does attention dominate?core
Weights: about \(2N\) FLOPs per token, so \(2NT\). Attention: \(QK^\top\) and the weighted sum of values cost about \(2d\) FLOPs each per (query, key) pair; with a causal mask there are about \(T^2/2\) pairs per layer, so \(2LdT^2\) in total. Attention dominates when \(T \gt N/(Ld)\): 53,248 tokens for Llama 3.1 8B and 104,448 for 70B. FlashAttention changes the memory traffic, not this FLOP count.
Explain FLARE. What are \(\theta\) and \(\beta\)?core
FLARE drafts the next sentence without retrieval. If any token has probability below \(\theta\), the model is unsure, so it retrieves; the query is the draft with tokens below \(\beta\) masked, so a wrong guess (a wrong date) does not steer the search. Then it writes the sentence again with the documents. \(\theta\) controls how often it retrieves (0 never, 1 always; the paper used 0.8 on several tasks); \(\beta\) controls how much of the draft is removed from the query (0.4 in the paper).
What does active retrieval cost in a serving system?advanced
Each retrieval prepends new documents. Because of the causal mask, every key and value after the insertion point changes, so the cache of the input and the text so far must be recomputed: \(kc + T_x + |y_{\lt t}|\) prefill tokens per retrieval (90.8). The draft sentence is thrown away, which wastes decode steps. The retrieval call also adds a network round trip in the middle of decoding, which hurts time per output token and makes batching harder. Mitigations: retrieve less (higher confidence bar), append documents after the prefix, or batch the retrieval calls of many requests.
How does REPLUG train a retriever if the LLM is a black box?core
For a training text split into \(x\) and continuation \(y\), retrieve top chunks and ask the frozen LLM for \(P_{LM}(y \mid d, x)\) for each chunk. Turn these into a target \(Q = \softmax(P_{LM}/\beta)\) over chunks, and the retriever scores into \(P_R = \softmax(s/\gamma)\). Minimize \(\KL(P_R \| Q)\); gradients go only into the retriever. Only output probabilities of the LLM are needed. Because the document encoder changes, re-encode the corpus and rebuild the index periodically (every 3,000 steps in the paper).
REPLUG ensemble or concatenating all chunks into one prompt: compare the costs.advanced
Concatenation: one sequence of \(kc + T_x\) tokens, one cache, attention over all chunks at once. Ensemble: \(k\) sequences of \(c + T_x\) tokens: the input is processed \(k\) times, there are \(k\) caches, and each output token is decoded \(k\) times. For \(k = 10\), \(c = 128\), \(T_x = 256\): 1,536 against 3,840 tokens, so 2.5 times more weight FLOPs, though fewer attention pairs (1.47M against 2.36M). The ensemble runs as a batch, so latency rises less than cost. It avoids context-length limits and the lost-in-the-middle effect.
What are Self-RAG’s reflection tokens and how are they used at inference?core
Retrieve (yes/no/continue), IsRel (relevant/irrelevant), IsSup (fully/partially/no support) and IsUse (1 to 5). The generator writes them as normal tokens. It retrieves when the normalized probability of \(\texttt{Retrieve}=\text{yes}\) is above \(\delta\) (0.2 in the paper). Then it writes one candidate per passage in parallel and ranks them by segment probability plus a weighted sum of the normalized probabilities of the best critique values (90.14), with weights 1, 1, 0.5 by default. The weights can be changed at inference without retraining, for example to favor supported answers.
FLARE, REPLUG, Self-RAG: which would you pick for a closed API model, and why?basic
REPLUG fits a closed model best: it changes only the retriever and needs only output probabilities (for LSR training) or nothing at all (for retrieval plus ensemble). FLARE also works with a closed model if the API returns token log-probabilities, and costs extra calls per retrieval. Self-RAG needs fine-tuning the generator to emit reflection tokens, so it needs an open model.
Design question: answer questions over 500 hours of sports video. How do the ideas of this session apply?advanced
The full video cannot enter a VLM: one hour at 1 frame per second with 576 tokens per frame is about 2.07M tokens. Index clips offline with a video-text encoder; at query time retrieve a few clips (16 frames is 9,216 tokens). Use active retrieval for multi-part answers (a second search when the VLM is unsure which game or player), and train the clip retriever with VLM feedback in the REPLUG style if answers are logged. Measure recall at \(k\) on key moments, cost per query with (90.2) and (90.3), and latency of the search plus prefill (session 210).
Your RAG answers got worse after you raised \(k\) from 5 to 30. Why might that happen?core
More chunks add more noise: weak chunks can distract the model, and the best chunk may now sit in the middle of a long prompt where models use information less well. The prompt is also 6 times longer, so prefill and cache cost grow. Better: improve ranking (a re-ranker or an LSR-trained retriever), keep \(k\) small, compress the chunks (RECOMP, LongLLMLingua), or use an ensemble so each chunk is read on its own.

Exercises

  1. Llama 3.1 8B, corpus 30,000 tokens, \(T_q = 100\), \(k = 4\), \(c = 512\). Compute \(T_{\text{rag}}\), \(\rho\) and the KV memory of each prompt in BF16.
  2. For Llama 3.1 70B, at what prompt length does the attention term of (90.2) equal the weight term? Use \(N = 68.45\)B.
  3. In Figure 3, which sentences retrieve at \(\theta = 0.85\)? What is the masked query of sentence 5 at \(\beta = 0.7\)?
  4. In FLARE, a 500-token prefix has been written. One retrieval adds 3 chunks of 256 tokens in front, with \(T_x = 50\). How many tokens must be prefilled again?
  5. REPLUG with \(\gamma = 1\): scores \(s = (1, 0)\) for two chunks and \(P_{LM}(y \mid d, x) = (0.2, 0.6)\). Compute the ensemble probability (90.9), and \(Q\) with \(\beta = 0.1\).
  6. In Figure 5, with \(w^{\text{Rel}} = 1\), \(w^{\text{Use}} = 0.5\), find the value of \(w^{\text{Sup}}\) where A and B tie.
Answerscheck
1. \(T_{\text{rag}} = 100 + 4 \times 512 = 2{,}148\); \(T_{\text{long}} = 30{,}100\); \(\rho = 14.0\). KV: \(131{,}072 \times 30{,}100 = 3.95\) GB against \(131{,}072 \times 2{,}148 = 0.28\) GB. 2. \(T = N/(Ld) = 68.45\times10^9/(80 \times 8{,}192) \approx 104{,}400\) tokens (104,448 with the exact \(N\)). 3. The minimum probabilities of the six sentences are 0.88, 0.34, 0.83, 0.55, 0.29, 0.58, so sentences 2, 3, 4, 5 and 6 retrieve (all but sentence 1). At \(\beta = 0.7\), sentence 5 drops 1842 (0.29), with (0.64) and first (0.52); “computer” is 0.70, not below 0.7, so it stays: “In ___ she published notes ___ the ___ computer program .”. 4. \(3 \times 256 + 50 + 500 = 1{,}318\) tokens. 5. \(\lambda = \softmax(1, 0) = (0.731, 0.269)\); ensemble \(= 0.731 \times 0.2 + 0.269 \times 0.6 = 0.308\). \(Q = \softmax(2, 6) = (0.018, 0.982)\). 6. A: \(0.40 + 0.95 + 0.20w + 0.35 = 1.70 + 0.20w\); B: \(0.30 + 0.90 + 0.85w + 0.30 = 1.50 + 0.85w\). Equal at \(0.65 w = 0.20\), \(w^{\text{Sup}} = 0.308\). Above that, B wins.

Key takeaways

  • RAG is input compression: the prompt holds \(T_q + kc\) tokens instead of \(T_q + T_c\) (90.1). It is the strongest form, because most of the corpus is never read.
  • Prefill costs \(2NT + 2LdT^2\) FLOPs (90.2) and the cache costs \(2Ln_{kv}d_hbT\) bytes (90.3). For Llama 3.1 8B and a 100k corpus, RAG with 5 chunks of 256 saves about 204 times the compute and 72.5 times the memory.
  • Long context still wins when facts are spread out, retrieval recall is poor, or a shared prefix can be cached and reused.
  • FLARE retrieves only when a drafted token is below \(\theta\), with the unsure tokens masked out of the query (90.5, 90.6); each retrieval costs a new prefill and a lost draft (90.8).
  • REPLUG ensembles over chunks for a black-box LLM (90.9) and trains the retriever to match the LLM’s preferences with \(\KL(P_R \| Q)\) (90.10, 90.11); the ensemble costs \(k\) sequences (90.12).
  • Self-RAG fine-tunes the model to write reflection tokens: it retrieves when \(p(\text{yes}) \gt \delta\) and ranks candidates by probability plus weighted critique scores (90.13, 90.14).

Key formulas

\[ F_{\text{prefill}} \approx 2NT + 2LdT^2, \qquad M_{\text{KV}} = 2Ln_{kv}d_h b T, \qquad T_{\text{rag}} = T_q + kc \]
\[ p(y \mid x) = \sum_{d} \lambda(d,x)\, p(y \mid d \circ x), \qquad \mathcal{L}_{\text{LSR}} = \KL\!\big(P_R \,\|\, Q\big), \qquad \text{retrieve} \iff \min_i p(\hat{s}_{t,i}) \lt \theta \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  3. Jiang et al. (2023). Active Retrieval Augmented Generation (FLARE)
  4. Shi et al. (2023). REPLUG: Retrieval-Augmented Black-Box Language Models
  5. Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
  6. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  7. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  8. Jiang et al. (2023). Active Retrieval Augmented Generation (FLARE)
  9. Shi et al. (2023). REPLUG: Retrieval-Augmented Black-Box Language Models
  10. Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
  11. Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning
  12. Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering
  13. Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
  14. Xu et al. (2023). RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
  15. Jiang et al. (2023). LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
  16. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  17. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  18. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models