LLM & VLM courseBlog
89Track 5: LLM inference and serving · session 12 of 65

Prompt summary and soft-prompt compression: RECOMP, Gisting, AutoCompressors, ICAE

Long prompts cost prefill compute and KV-cache memory. This session shows two ways to make them much shorter without changing what the model can do: rewrite the text into a short summary with a small trained compressor (RECOMP), or replace it with a few learned vectors that the model reads like a long prompt (soft prompts, Gisting, AutoCompressors and ICAE). We also work out when compression really saves compute.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: IntermediateBefore this: Sessions 18, 56, 80 and 88

What you will learn

  • The difference between hard prompts (text) and soft prompts (vectors), and between offline and online compression
  • How RECOMP trains extractive and abstractive compressors with the target LLM as the judge, and why it may return an empty summary
  • How a soft prompt can imitate a long prompt by minimizing a KL divergence, and why that needs one optimization per prompt
  • How the gist attention mask forces an instruction into a few gist tokens, and why the real saving is cache storage
  • How AutoCompressors carry summary vectors across segments, and how ICAE compresses a context into memory slots with a LoRA encoder and a frozen decoder
  • A break-even rule that tells when online compression saves compute, and what it always saves in KV memory

Why this matters

A RAG system puts five retrieved passages in front of a question: about 660 tokens of text for a question of maybe 15 tokens. A chat product puts the same 500-token system prompt in front of every user message. Long prompts are paid for twice: once in prefill compute, and again in KV-cache memory for every step of decoding.

Session 88 made prompts shorter by deleting tokens that carry little information. The kept tokens stayed exactly as they were. This session looks at two stronger ideas from the efficient-inference survey (Zhou et al., sections 4.1.2 and 4.1.3):

  • Prompt summary. A small model rewrites the long text into a short text. The output is still normal words, so any LLM can read it, even one behind an API. The main example is RECOMP.
  • Soft-prompt compression. The long text is replaced by a few vectors that are not words at all. The LLM reads them in place of token embeddings or as extra keys and values. Examples: Gisting, AutoCompressors and the In-context Autoencoder (ICAE).

The numbers are large. On Natural Questions, RECOMP shrinks five passages from 660 tokens to 37 tokens (5.6%) and the exact-match score falls only from 39.39 to 36.57. [Certain, Table 2 of the RECOMP paper] Gisting compresses a 26-token instruction into a single token. [Certain] AutoCompressors turn a 2,048-token segment into 50 vectors, about 41 tokens per vector. [Certain] The price is always the same question: what is lost, and who pays the cost of compressing?

You will need prompt tuning and prefix tuning from session 56, the KV-cache size formula from session 80, and attention masks from session 18. The next session, session 90, goes one step back: instead of compressing retrieved text, retrieve less of it.

Two ways to make a prompt shorter

Call the original context \(C\) with \(n\) tokens, and the compressed form \(\tilde C\) with \(c\) positions. The compression ratio is \(\rho = n / c\). The LLM then does prefill over \(c\) positions instead of \(n\), and keeps \(c\) positions in its KV cache. For a dense model with \(N\) parameters:

\[ \text{prefill FLOPs} \approx 2 N c, \qquad \text{KV bytes} = 2 \, L \, n_{kv} \, d_h \, b \cdot c, \qquad \rho = \frac{n}{c} \tag{89.1} \]
Read the symbols
\(N\)
number of model parameters (6.74 billion for Llama 2 7B)
\(c\)
number of positions the LLM actually reads: summary tokens, or soft vectors
\(L, n_{kv}, d_h\)
layers, key-value heads and head size; for Llama 2 7B: 32, 32 and 128
\(b\)
bytes per number: 2 for bf16, 4 for fp32
the factor 2 in KV bytes
one key and one value per position

(The \(2Nc\) rule ignores the attention-score FLOPs, which grow with \(c^2\). They only make long prompts even more expensive, so compression helps more than the rule shows.) The two families differ in what \(\tilde C\) is:

Hard prompt (text)Soft prompt (vectors)
What the LLM readsReal tokens from the vocabularyVectors in embedding (or key/value) space that need not be any real token
ExamplesPruning (LLMLingua), summary (RECOMP, SemanticCompression)PromptCompression, Gisting, AutoCompressors, ICAE
Works with a black-box API?YesNo: you must feed vectors into the model
Readable by a human?Yes, you can debug itNo
Typical ratio \(\rho\)2× to 20×4× (ICAE) to 26× (Gisting) and more

The survey also splits soft-prompt methods by when compression happens. Offline methods compress a fixed prefix (a system prompt, a task instruction) once and reuse it for many requests: PromptCompression and Gisting. Online methods compress each new context when it arrives: AutoCompressors and ICAE. This split decides who pays for compression, and Figure 1 makes it concrete.

LLM prefill FLOPscompressor FLOPs (divided by \(R\))KV cache of the context
Figure 1. Cost of one request for Llama 2 7B (\(N = 6.74\)B, bf16 KV cache of 0.5 MiB per position), using formula (89.1). Choosing a method sets \(\rho\) to the ratio reported in its paper; you can then move the slider. The compressor costs are rough estimates: 110M parameters for the RECOMP extractive encoder, 775M for the abstractive T5-large, and the full 7B LLM for Gisting, AutoCompressor and ICAE, whose encoders are the LLM itself.
How to read Figure 1

Each pair of bars compares “no compression” (left) and the chosen method (right) for compute, and the same for KV memory. Start with RECOMP extractive: the compressor is tiny, so compute falls almost by \(\rho\) at once. Now click ICAE with reuse \(R = 1\): the total compute is higher than no compression, because the encoder is the same 7B model reading all \(n\) tokens. The KV cache still shrinks by 4×. Move \(R\) up to 2 and the compute bar drops below the baseline. Gisting has the same structure, but its prompt is a fixed instruction shared by many users, so a large \(R\) is natural. The extra sequential steps in the readout matter for latency: an abstractive summary must be generated token by token before the LLM can start.

Prompt summary with RECOMP

RECOMP (Xu, Shi and Choi, 2023) stands for “Retrieve, Compress, Prepend”. A retriever returns the top 5 passages (each a 100-word piece of Wikipedia). Before they go into the LLM, a small compressor turns them into a short text that keeps only what the question needs. [Certain] The LLM itself does not change. It is used only as a judge while the compressor is trained.

The key design choice is the training signal. A generic summarizer keeps what a human would call important. RECOMP keeps what helps this LLM answer. The paper reports that its compressors beat off-the-shelf summarization models for this reason. [Certain]

The extractive compressor

The extractive version selects whole sentences. It is a dual encoder: one encoder maps the input \(x\) (the question) to a vector, the same kind of encoder maps each candidate sentence \(s_i\) to a vector, and the score is their dot product. It starts from Contriever, a retrieval model with about 110M parameters. [Certain]

\[ \text{score}(x, s_i) = E(x)^\top E(s_i), \qquad \tilde C = \text{top-}k \text{ sentences by score} \tag{89.2} \]

To train it we need to know which sentence is “good”. RECOMP asks the frozen LLM \(M\). For language modeling, the positive sentence is the one that gives the target text the highest likelihood when it is put in front of \(x\). For question answering, it is a sentence that makes \(M\) produce the correct answer. [Certain]

\[ p^\ast = \argmax_{s_i \in S} \; \log P_M(y \mid s_i, x) \tag{89.3} \]

Negatives are up to five sentences that Contriever scores high, but whose LLM score is clearly lower than the positive's. [Certain] These are “hard negatives”: they look relevant but do not help. The loss is the usual contrastive loss of session 60:

\[ \mathcal{L}_{\text{ext}} = -\log \frac{\exp\big(\text{score}(x, p^\ast)\big)}{\exp\big(\text{score}(x, p^\ast)\big) + \sum_{j} \exp\big(\text{score}(x, n_j)\big)} \tag{89.4} \]
Read the symbols
\(E(\cdot)\)
the encoder that maps a text to one vector
\(S\)
the candidate sentences (the top 20 sentences from the top 5 passages in the paper)
\(P_M(y \mid s_i, x)\)
the probability the frozen LLM gives to the right output \(y\) when sentence \(s_i\) comes before \(x\)
\(p^\ast, n_j\)
the positive sentence and the hard negatives
Figure 2. A toy extractive compressor. Eight candidate sentences from three retrieved passages, each with a score from formula (89.2) (toy numbers, chosen by hand) and its token count. Green sentences are kept and put in front of the question. If no sentence reaches the minimum score \(\tau\), the compressor returns nothing.
How to read Figure 2

For question A, the sentence with the answer has the top score, so \(k = 1\) already keeps it with a small fraction of the tokens. Raising \(k\) adds tokens but little new information. Switch to question B: the passages are about a different topic, all scores are low, and with \(\tau = 0.5\) the compressor returns an empty string. The LLM then answers from its own knowledge, which is better than reading misleading text. This is RECOMP’s selective augmentation. Set \(\tau\) to 0 to see what happens without it.

The abstractive compressor

Selecting sentences cannot merge facts that are spread over several passages. The abstractive version writes a new summary. It is an encoder-decoder initialized from T5-large (775M parameters). [Certain] There is no human-written training data for “a summary that helps Flan-UL2 answer this question”, so RECOMP builds it by distillation (session 59):

  1. A large teacher (GPT-3.5-turbo) writes query-focused summaries of the top 5 passages. For language modeling it tries four prompts and keeps the best one.
  2. Each candidate summary is tested with the target LLM. It is kept only if putting it in front of the input improves the end task.
  3. If no candidate helps, the training target is the empty string.
\[ \tilde C^{\ast} = \begin{cases} \argmax_{\tilde C \in \mathcal{S}} \ \text{task}(M, \tilde C, x) & \text{if this beats } \text{task}(M, \varnothing, x) \\ \varnothing \ (\text{empty}) & \text{otherwise} \end{cases} \tag{89.5} \]
Read the symbols
\(\mathcal{S}\)
the candidate summaries written by the teacher
\(\text{task}(M, \tilde C, x)\)
how well the frozen LLM does when \(\tilde C\) is put before \(x\): the likelihood of the gold continuation, or whether the answer is correct
\(\varnothing\)
no context at all: the LLM answers from its own knowledge

The student T5 is then trained with ordinary cross-entropy to produce \(\tilde C^\ast\) from \((x, \text{passages})\). Empty targets are common: 25% of examples for NQ, 16% for TriviaQA, 4% for HotpotQA and 24% for WikiText. [Certain, Table 5 of the paper] So the compressor learns when to stay silent.

Worked example: RECOMP on Natural Questions, then the serving bill

The paper’s numbers for Flan-UL2 (20B) on NQ (Table 2): no retrieval gives 21.99 exact match (EM). The full top 5 passages give 39.39 EM with 660 tokens. The extractive compressor gives 36.57 EM with 37 tokens, and the abstractive one 37.04 EM with 36 tokens. [Certain]

So the extractive compressor keeps \(37/660 = 5.6\%\) of the tokens. Retrieval added \(39.39 - 21.99 = 17.40\) EM points. The compressed context still gives \(36.57 - 21.99 = 14.58\) of them, which is 84% of the gain (86% for abstractive), at about 1/18 of the tokens.

Now apply formula (89.1) to a Llama 2 7B reader. Prefill: \(2 \times 6.74\text{B} \times 660 = 8.90\) TFLOPs per request without compression, and \(2 \times 6.74\text{B} \times 37 = 0.50\) TFLOPs with it. The compressor adds roughly \(2 \times 110\text{M} \times 660 = 0.15\) TFLOPs. KV cache in bf16 (0.5 MiB per token): 330 MiB versus 18.5 MiB per request. For a batch of 64 requests that is 20.6 GiB versus 1.2 GiB of cache.

Two more results from the paper. The compressors trained for GPT-2 also helped the larger GPT2-XL and GPT-J on language modeling, so a compressor can transfer to other readers. [Certain] And the target LLM is used only through its outputs, so the same recipe works for an API model. A related method, SemanticCompression (Fei et al., 2023), splits a long text into sentences, groups them by topic, and summarizes each group with a pretrained model; the authors report handling texts 6 to 8 times longer than the context window, without fine-tuning the LLM. [Certain, as reported]

Soft prompts: a few vectors that act like a long prompt

A prompt reaches the model as a matrix of embeddings, one row per token. Nothing forces those rows to be embeddings of real words. Session 56 used this fact for prompt tuning: learn \(k\) free vectors \(Z \in \R^{k \times d}\) that make a frozen model do a task well. The same idea can be used for compression. Instead of a task loss, we ask the model with \(Z\) to behave like the model with the long prompt \(P\). Wingate, Shoeybi and Sorensen (2022) called this prompt compression; it is the “PromptCompression” of the survey:

\[ Z^\ast = \argmin_{Z \in \R^{k \times d}} \ \E_{x} \Big[ \KL\big( P_M(\cdot \mid P, x) \,\big\|\, P_M(\cdot \mid Z, x) \big) \Big] \tag{89.6} \]
Read the symbols
\(P\)
the long, fixed prompt we want to replace (for example a system prompt)
\(Z\)
\(k\) learned vectors, each of the model width \(d\); \(k\) is much smaller than the length of \(P\)
\(x\)
inputs that could follow the prompt; the expectation averages over them
\(\KL(\cdot \| \cdot)\)
how far the student's next-token distributions are from the teacher's (the model with the full prompt)

The model weights stay frozen; only \(Z\) is trained, by backpropagation through the model. The authors found that heavily compressed prompts keep the broad concepts and sentiment of the original but lose fine details, and that some complex prompts can be squeezed into a single token. [Certain, as reported] The weakness: you must run a new optimization for every prompt. That is fine for one system prompt used a million times. It is useless for a RAG context that changes every request.

The rest of this session is about one question: how do we get \(Z\) for a new prompt without running an optimization? The answer of all three methods below is the same: train the LLM itself (or a copy of it) to produce \(Z\) in one forward pass. This is called amortized compression: we pay once in training, then compressing a new prompt costs one forward pass.

Gisting: compress instructions into gist tokens

Gisting (Mu, Li and Goodman, 2023) targets task instructions such as “Translate this to French:” or “Write a haiku about the input.” It adds \(k\) new gist tokens \(G = (g_1, \dots, g_k)\) to the vocabulary and places them right after the instruction \(t\). Then it changes only the attention mask during instruction fine-tuning: tokens after the gist tokens may not attend to the instruction. [Certain]

\[ M_{ij} = \begin{cases} 0 & \text{if } i \text{ is after } G \text{ and } j \in t \quad \text{(the blocked block)} \\ 1 & \text{if } j \le i \ \text{otherwise (causal)} \\ 0 & \text{if } j \gt i \end{cases} \tag{89.7} \]
Read the symbols
\(M_{ij}\)
1 if position \(i\) (a query) may attend to position \(j\) (a key), 0 if not
\(t\)
the instruction tokens, placed first
\(G\)
the gist tokens, placed right after \(t\); they can see \(t\)
“after \(G\)”
the input \(x\) and the output \(y\); they see \(G\), \(x\) and earlier \(y\), but not \(t\)

Think about what this forces. The only path from the instruction to the answer goes through the gist tokens. To predict \(y\) well, the model must learn to pack everything useful from \(t\) into the keys and values of \(G\), in every layer. The training loss is the usual instruction-tuning loss, only with the new mask:

\[ \mathcal{L}_{\text{gist}} = - \sum_{\tau} \log P_\theta\big(y_\tau \mid t, G, x, y_{\lt \tau}; \ M\big) \tag{89.8} \]

So gisting costs nothing extra in training beyond normal instruction tuning. [Certain] At inference, run the model over \((t, G)\) once, keep only the keys and values at the \(k\) gist positions, and throw away those of \(t\). Every later request with the same instruction starts from that tiny cache. This is the offline case of Figure 1 with a large reuse \(R\). Gist tokens are prefix tuning (session 56) whose prefix is predicted by the model from the instruction instead of learned per task; the survey calls this a meta-learning step. [Certain]

allowedblocked by the gist maskfuture (causal)
Figure 3. The attention mask of formula (89.7) for a decoder-only model. Rows are queries, columns are keys. The instruction “Translate to French :” is followed by \(k\) gist tokens, then the input and the output. Click a row label to see what that token can read.
How to read Figure 3

With the causal mask, every token sees everything before it, including the instruction. Switch to the gist mask: a red block appears in the lower left. All tokens after the gist tokens lose sight of the instruction; only the gist rows themselves still read it. Click the row of “chat”: it reads the gist token(s), the input and earlier output, nothing else. The number of blocked pairs equals \((\text{tokens after } G) \times (\text{instruction length})\). More gist tokens give the instruction a wider “pipe”, but the paper found results were generally insensitive to \(k\) between 1 and 10. [Certain]

Algorithm 1: training and using gist tokens
  1. training input: instruction data \((t, x, y)\), a pretrained LM, \(k\) new gist embeddings
  2. for each batch: build the sequence \([t;\ G;\ x;\ y]\)
  3.   build mask \(M\) by formula (89.7) // causal, plus: rows after G cannot see t
  4.   loss \(\leftarrow\) cross-entropy on \(y\) only, formula (89.8); update all weights
  5. inference, once per instruction: run the model on \([t;\ G]\); keep the K and V of the \(k\) gist positions in every layer
  6. inference, per request: prefill \(x\) with the gist KV as a cached prefix; decode \(y\) as usual
  7. the instruction’s own KV is never stored // k positions instead of |t|

The results. Gisting was trained on “Alpaca+”, about 130K instruction examples, with LLaMA-7B and FLAN-T5-XXL. [Certain] With one gist token on LLaMA-7B, ROUGE-L was 57.8 versus 58.0 for the uncompressed model on seen tasks, 46.6 versus 48.1 on unseen tasks, and 23.9 versus 27.0 on human-written instructions; When ChatGPT compared answers, the gist model's win rates against the full model were 48.6%, 49.7% and 45.8% on the three splits (50% would mean a tie). [Certain, Table 1] Compute: on LLaMA-7B, FLOPs per request fell from 915 to 552 GFLOPs (40%) versus no caching. [Certain, Table 3]

Worked example: why the real gain of Gisting is memory, not FLOPs

The fair baseline is not “no caching” but instruction caching: keep the full KV cache of the instruction. Against that, gist caching saves only 0.11% of FLOPs and 1.0% of time on LLaMA-7B, because most FLOPs go to the new input tokens. [Certain]

The gain is storage. LLaMA-7B has 32 layers of width 4,096, so one position of KV cache in fp32 is \(2 \times 32 \times 4096 \times 4 = 1{,}048{,}576\) bytes, the 1.05 MB the paper reports. A 26-token instruction costs 27.3 MB; one gist token costs 1.05 MB. So the same memory holds 26 times more cached prompts. [Certain] In bf16 the numbers halve (13 MiB versus 0.5 MiB), the ratio does not.

AutoCompressors: summary vectors for any long context

AutoCompressors (Chevalier, Wettig, Ajith and Chen, 2023) apply the gist idea to long documents, online. Split a document into segments \(S_1, S_2, \dots\) of 1,024 to 2,048 tokens. Append \(\kappa = 50\) special summary tokens to each segment. The model's outputs at those positions are the summary vectors \(\sigma_i\). They are passed on as a soft prompt to every later segment. [Certain]

\[ \sigma_i = \text{LM}_\theta\big(\, [\,\sigma_1; \dots; \sigma_{i-1};\ S_i;\ \langle\text{Sum}\rangle_{1..\kappa}\,] \,\big)_{\text{last } \kappa \text{ positions}} \tag{89.9} \]
Read the symbols
\(S_i\)
segment \(i\) of the document, as normal tokens
\(\langle\text{Sum}\rangle_{1..\kappa}\)
\(\kappa\) learned summary-token embeddings (\(\kappa = 50\) in the paper)
\(\sigma_i \in \R^{\kappa \times d}\)
the outputs at the summary positions: the summary vectors of segment \(i\)
\([\,\sigma_1; \dots; \sigma_{i-1}\,]\)
summary accumulation: all earlier summaries, concatenated, as a soft prompt of length \((i-1)\kappa\)

Training is unsupervised: the loss is plain next-token prediction on the tokens of every segment. A segment can only learn about earlier segments through the summary vectors, so the model is pushed to put useful information into them. Gradients flow back through the summary vectors into earlier segments, like backpropagation through time in an RNN, but they are stopped after 2 compression steps to save memory. [Certain] The paper fine-tunes OPT-1.3B and OPT-2.7B on sequences of up to 30,720 tokens, and Llama-2-7B on 15B tokens with LoRA of rank 16. [Certain]

segment tokens (2,048)summary vectors from earlier segmentsnew summary tokens (\(\kappa = 50\))
Figure 4. An AutoCompressor reads a document of up to 15 segments of 2,048 tokens (30,720 tokens). At each step it sees the summary vectors of earlier segments, the new segment, and 50 summary tokens whose outputs become the next summary. Widths are not to scale.
How to read Figure 4

Step through the segments. The row of each step is what the model reads in one forward pass. With accumulation on, the green soft prompt grows by 50 vectors per segment: after 14 segments it is 700 vectors, still much less than the 28,672 tokens it stands for. The readout compares the attention span with plain full attention over the whole document. Turn accumulation off: the model only sees the last summary (like the Recurrent Memory Transformer), so the span stays flat, but older information must survive many rounds of re-compression.

The results. With Llama-2-7B, perplexity on a segment was 5.40 with no context and 5.07 when 6,144 tokens of earlier context were compressed into 150 summary vectors. Full attention over the plain 6,144 tokens was better still, at 4.76. [Certain, Table 3] So summary vectors keep a real part of the information, but not all of it. For in-context learning, 50 summary vectors of demonstrations often matched or beat 150 tokens of plain-text demonstrations: for example 79.6 versus 74.5 accuracy on AG News and 94.2 versus 92.4 on SST-2. [Certain, Table 4] Summary vectors of retrieved passages can also be precomputed and stored next to the passages, which makes them useful for retrieval-augmented LMs. [Certain]

ICAE: an in-context autoencoder with memory slots

The In-context Autoencoder (Ge et al., 2023) treats compression as an autoencoder: an encoder squeezes a context into \(k\) vectors called memory slots, and a decoder must rebuild the context from them. Both parts are the same LLM:

  • Encoder: the LLM plus a LoRA adapter (rank 128 on the query and value projections). It reads the context \(C\) followed by \(k\) learned memory tokens \(e_1, \dots, e_k\). The outputs at those \(k\) positions are the memory slots \(\mathbf{m}\). [Certain]
  • Decoder: the original, frozen LLM. It reads the memory slots as a soft prompt, followed by a task prompt. [Certain]
\[ \mathbf{m} = \text{LLM}_{\theta + \Delta_{\text{LoRA}}}\big([\,C;\ e_1, \dots, e_k\,]\big)_{\text{last } k}, \qquad \mathcal{L}_{\text{pre}} = \lambda\, \mathcal{L}_{\text{AE}} + (1 - \lambda)\, \mathcal{L}_{\text{LM}} \tag{89.10} \]
Read the symbols
\(\Delta_{\text{LoRA}}\)
the only new weights of the encoder (plus the memory-token embeddings), about 1% of the model
\(\mathbf{m} \in \R^{k \times d}\)
the memory slots; \(k = 128\) for a 512-token context gives \(\rho = 4\)
\(\mathcal{L}_{\text{AE}}\)
autoencoding loss: given \(\mathbf{m}\) and a special [AE] token, the frozen decoder must reproduce \(C\) word by word
\(\mathcal{L}_{\text{LM}}\)
text-continuation loss: given \(\mathbf{m}\), predict the text that comes after \(C\)
\(\lambda\)
the mix; values between 0.4 and 0.6 worked best

Pretraining uses the Pile (200K updates). Autoencoding alone would teach the slots to store surface text; the continuation loss pushes them to also store meaning that is useful later. Then ICAE is fine-tuned on PwC (“prompt with context”), 240K examples where GPT-4 wrote questions and instructions about Pile texts, so that the slots work for real tasks and not only for copying. [Certain]

Figure 5. ICAE. Left: the encoder (LLM + LoRA) reads the context and \(k\) memory tokens. Right: the frozen LLM reads only the \(k\) memory slots plus a task prompt, and is trained to produce the target. The three buttons show the three training tasks.
How to read Figure 5

Compare the three targets. In autoencoding, the target is the context itself, so the slots must keep almost every word. In continuation, the target is the next text, so the slots must keep what predicts the future. In fine-tuning, the target is an answer to a question about the context. Move \(k\): fewer slots give a higher ratio and a smaller KV cache for the decoder, at a loss of detail. The paper's main setting is \(k = 128\), \(\rho = 4\).

The results. On Llama-7B with 512 tokens in 128 slots, the frozen decoder rebuilds the context with BLEU 99.1. [Certain] Text continuation perplexity rose from 9.01 with the real context to 9.50 with the slots. [Certain] Latency on one A100 (generation length 128): for \(8 \times 2{,}048\)-token inputs, total time fell from 24.0 to 7.3 (3.3×), with 3.4 of that spent on compression; the paper says the speedup can exceed 7× when the slots are cached in advance. [Certain, Table 7]

Two things are worth noticing. First, the decoder is frozen: ICAE does not change the LLM, it only learns to write in a “language” the LLM already reads. Second, nearly perfect reconstruction at 4× says that a 4,096-wide vector can hold about four tokens of text exactly. The paper links this to memorization ability: what the model knows well (normal text) compresses better than random text. [Certain, as discussed in the paper]

Algorithm 2: ICAE online compression at inference
  1. input: context \(C\) (\(n\) tokens), task prompt \(q\), encoder (LLM + LoRA), frozen decoder LLM, \(k\) memory embeddings
  2. if the slots of \(C\) are in the cache then \(\mathbf{m} \leftarrow\) cache[\(C\)] // offline / reuse case: free
  3. else \(\mathbf{m} \leftarrow\) encoder\(([C;\ e_{1..k}])\) at the last \(k\) positions; cache[\(C\)] \(\leftarrow \mathbf{m}\) // one parallel pass, cost \(\approx 2N(n + k)\)
  4. prefill the frozen LLM on \([\mathbf{m};\ q]\) // k + |q| positions instead of n + |q|
  5. decode the answer with a KV cache of \(k + |q|\) + generated positions

When does compression pay? The break-even rule

Every method moves cost from the LLM to a compressor. Say the compressor costs \(F_{\text{comp}}\) FLOPs for one context, and the result is used by \(R\) requests. Compression saves compute when

\[ \frac{F_{\text{comp}}}{R} + 2 N c \ \lt\ 2 N n \quad \Longleftrightarrow \quad R \gt \frac{F_{\text{comp}}}{2N(n - c)} \tag{89.11} \]

For RECOMP, \(F_{\text{comp}}\) comes from a model 60 times smaller than a 7B reader, so the rule holds even at \(R = 1\). For ICAE and AutoCompressors, the encoder is the LLM itself, \(F_{\text{comp}} \approx 2N(n + k)\), so you need \(R \gt (n + k)/(n - c)\). For ICAE with \(n = 512\), \(k = c = 128\): \(R \gt 640/384 = 1.67\). Used once, the context costs 1.5 times the compute of no compression. Used twice, it costs 0.875 times. The KV cache, however, shrinks by \(\rho\) from the first request, and decode steps, which are limited by memory bandwidth (session 82), read a cache that is \(\rho\) times smaller. [Likely: this is why ICAE's single-use latency still improves]

Code: the gist mask, and a soft prompt that imitates a long prompt

The code builds a tiny frozen 2-layer attention model with random weights. Part 1 checks the key property of formula (89.7): with the gist mask, the outputs for the input are exactly the same whether the model sees the whole prompt or only the cached gist keys and values. With a normal causal mask they are not. Part 2 trains 2 soft vectors to imitate a 12-token prompt with the KL objective of formula (89.6). It runs on a CPU in a few seconds and is deterministic (tested with PyTorch 2.x).

import torch, torch.nn.functional as F
torch.manual_seed(0)
V, D, H, NL = 50, 32, 4, 2          # vocab, width, heads, layers (a tiny frozen "LLM")
emb = torch.randn(V, D) * 0.5
Wqkv = [torch.randn(D, 3 * D) / D**0.5 for _ in range(NL)]
Wo   = [torch.randn(D, D) / D**0.5 for _ in range(NL)]
Wout = torch.randn(D, V) / D**0.5

def layer(h, l, mask, past=None):
    """One attention layer. past = (K, V) of extra positions placed before h."""
    q, k, v = (h @ Wqkv[l]).split(D, dim=-1)
    if past is not None:
        k, v = torch.cat([past[0], k]), torch.cat([past[1], v])
    sh = lambda t: t.view(-1, H, D // H).transpose(0, 1)          # (H, T, D/H)
    a = (sh(q) @ sh(k).transpose(1, 2)) / (D // H) ** 0.5
    a = a.masked_fill(~mask, float("-inf")).softmax(-1)
    o = (a @ sh(v)).transpose(0, 1).reshape(-1, D)
    return h + o @ Wo[l], (k, v)

def forward(h, mask):
    kvs = []
    for l in range(NL):
        h, kv = layer(h, l, mask); kvs.append(kv)
    return h, kvs

# 1) the gist mask, formula (89.7): after the gist, nobody sees the prompt
def gist_mask(p, g, x):
    T = p + g + x
    m = torch.tril(torch.ones(T, T, dtype=torch.bool))            # causal
    m[p + g:, :p] = False                                         # input rows cannot see prompt columns
    return m

p, g, x = 6, 1, 4
prompt, gist_tok, inp = torch.randint(0, V, (p,)), torch.tensor([V - 1]), torch.randint(0, V, (x,))
h0 = emb[torch.cat([prompt, gist_tok, inp])]
out_full, kvs = forward(h0, gist_mask(p, g, x))

# 2) cache only the gist keys/values, throw the prompt away, run the input alone
h = emb[inp]
m_in = torch.cat([torch.ones(x, g, dtype=torch.bool), torch.tril(torch.ones(x, x, dtype=torch.bool))], 1)
for l in range(NL):
    K, Vv = kvs[l]
    h, _ = layer(h, l, m_in, past=(K[p:p + g], Vv[p:p + g]))
print("gist mask, cached gist only :", (h - out_full[p + g:]).abs().max().item())   # ~3e-7: identical

out_causal, kvs_c = forward(h0, torch.tril(torch.ones(p + g + x, p + g + x, dtype=torch.bool)))
h = emb[inp]
for l in range(NL):
    K, Vv = kvs_c[l]
    h, _ = layer(h, l, m_in, past=(K[p:p + g], Vv[p:p + g]))
print("causal mask, cached gist only:", (h - out_causal[p + g:]).abs().max().item())  # ~1.19: info lost

# 3) soft-prompt distillation, formula (89.6): k vectors imitate a long prompt
k = 2
long_prompt = torch.randint(0, V, (12,))
soft = torch.nn.Parameter(torch.randn(k, D) * 0.5)               # the only trainable numbers
opt = torch.optim.Adam([soft], lr=0.05)
causal = lambda T: torch.tril(torch.ones(T, T, dtype=torch.bool))
def kl_on(xs):
    T = emb[torch.cat([long_prompt, xs])]
    with torch.no_grad():
        t_logits = forward(T, causal(len(T)))[0][-len(xs):] @ Wout                 # teacher: full prompt
    S = torch.cat([soft, emb[xs]])
    s_logits = forward(S, causal(len(S)))[0][-len(xs):] @ Wout                     # student: k vectors
    return F.kl_div(s_logits.log_softmax(-1), t_logits.log_softmax(-1), log_target=True, reduction="batchmean")
gen = torch.Generator().manual_seed(1)
test = [torch.randint(0, V, (5,), generator=gen) for _ in range(20)]
ev = lambda: sum(kl_on(t).item() for t in test) / len(test)
print("KL before:", round(ev(), 4))                                                # 0.1747
for step in range(300):
    xs = torch.randint(0, V, (5,), generator=gen)
    loss = kl_on(xs); opt.zero_grad(); loss.backward(); opt.step()
print("KL after :", round(ev(), 4), "| prompt", len(long_prompt), "tokens ->", k, "soft tokens")   # 0.0221

Part 1 prints a difference of about \(3 \times 10^{-7}\) (float rounding) with the gist mask and about 1.19 with the causal mask. Part 2 lowers the average KL from 0.1747 to 0.0221: two vectors recover most of the behavior of a 12-token prompt. In this toy the prompt is random tokens; with real models the same loop is what PromptCompression runs per prompt, and what Gisting, AutoCompressors and ICAE learn to replace by one forward pass.

The four methods side by side

MethodCompressesOutputWhat is trainedReported ratioOffline / online
RECOMP extractiveRetrieved passagesSelected sentences (text)110M dual encoder; LLM frozen\(660 \to 37\) tokens (\(\approx 18\times\)) on NQOnline, cheap
RECOMP abstractiveRetrieved passagesA new summary, or empty775M T5-large, distilled from GPT-3.5\(660 \to 36\) tokens on NQOnline, adds decode steps
PromptCompressionOne fixed promptSoft vectorsThe vectors only, per promptDown to 1 token for some promptsOffline
GistingTask instructionsKV of \(k\) gist tokensThe whole LM, with the gist maskUp to 26×Offline (once per instruction)
AutoCompressorAny long text, per segment50 summary vectors per segmentThe LM (OPT full; Llama-2 with LoRA)\(2{,}048 \to 50\) (\(\approx 41\times\))Online, or precomputed
ICAEAny context (512 tokens in the main setting)\(k\) memory slotsLoRA encoder (\(\approx 1\%\)); decoder frozen4×Online, or cached

Where this is used

ApplicationGood choiceWhy
RAG over documents with an API LLMExtractive or abstractive summary (RECOMP style)Only text can be sent to an API; fewer input tokens means a lower bill and faster prefill
Chat product with a long, fixed system promptPrefix caching first; gist-style compression if KV memory is the limitThe prompt is shared by all users (large \(R\)); gists store far fewer positions than the full prompt cache
Many-shot in-context classificationAutoCompressor summary vectors of the demonstrations50 vectors of demonstrations matched or beat 150 plain tokens on several tasks
Long-document QA on your own modelICAE or AutoCompressor, slots cached per documentCompress once at indexing time, answer many questions with a 4× to 40× smaller cache
Video-RAG and video QAText captions of retrieved clips, compressed by a RECOMP-style query-focused summarizer [Speculation]A video clip can turn into thousands of visual tokens or long captions; a query-focused summary keeps only what the question needs
Key-moment detection over long videosPer-segment summary vectors carried forward, AutoCompressor style [Speculation]Segments of frames play the role of text segments; a running soft memory avoids attending to every earlier frame
Agents with long tool outputsAbstractive summary of each tool result before it enters the history [Likely]Tool outputs (web pages, logs) are long and mostly irrelevant; the empty-summary option drops useless results

Common mistakes

  • Comparing to “no caching” instead of prefix caching. For a fixed instruction, the strong baseline is caching its KV. Gisting saves 40% of FLOPs against no caching but only 0.11% against instruction caching; its real win is 26× less storage.
  • Counting only LLM tokens. An online compressor that runs the LLM itself (ICAE, AutoCompressor) costs about as much as the prefill it replaces. Use formula (89.11) with your real reuse \(R\).
  • Forgetting time to first token. An abstractive summarizer generates its summary token by token. That latency is on the critical path before the LLM starts.
  • Using a generic summarizer. A summary that reads well for a human can drop the one number the question needs. RECOMP trains with the end-task signal of the target LLM, and reports beating off-the-shelf summarizers.
  • Never returning an empty context. If retrieval failed, a compressed version of useless text is still misleading. Let the compressor output nothing (selective augmentation).
  • Reusing soft prompts across model versions. Summary vectors and memory slots are tied to the exact weights that produced them and that read them. Recompute them after any update.
  • Expecting soft prompts to be portable or inspectable. You cannot send them to an API, and you cannot read them to debug a wrong answer. Keep the original text for audit.
  • Pushing the ratio too far. AutoCompressor perplexity at 150 vectors (5.07) was still worse than full context (4.76). Measure task quality at your target ratio; do not assume the paper’s ratio transfers to your data.

Interview questions

What is the difference between prompt pruning, prompt summary and soft-prompt compression?basic
Pruning deletes tokens and keeps the rest unchanged (Selective Context, LLMLingua). Summary rewrites the context into a new, shorter text (RECOMP, SemanticCompression); the output is still readable text, so any LLM, including an API model, can use it. Soft-prompt compression replaces the context by learned vectors that are not real tokens (Gisting, AutoCompressors, ICAE); these give higher ratios but need access to the model's embeddings or KV cache and are tied to one model.
How is RECOMP's extractive compressor trained, and why not just use a similarity model?core
It is a dual encoder (initialized from Contriever, about 110M parameters) that scores each candidate sentence against the input. The labels come from the frozen target LLM: the positive is the sentence that most increases the likelihood of the correct output (or makes the LM answer correctly), and hard negatives are sentences the retriever scores high but that help the LM less. It is trained with a contrastive loss. A plain similarity model selects what looks related; RECOMP selects what actually helps the reader LLM.
What is selective augmentation and why does it matter?core
The compressor may output an empty string when no retrieved content helps. For the abstractive compressor, training targets are empty whenever no teacher summary improved the target LM's result (25% of NQ examples, 24% of WikiText). This avoids feeding misleading text, which can make the LLM worse than no retrieval at all, and it saves all the context tokens for those requests.
Explain the gist attention mask. Why does it allow caching only the gist tokens?core
Gist tokens are inserted after the instruction. The mask is causal, plus: every position after the gist tokens is forbidden from attending to the instruction tokens. The gist tokens themselves can see the instruction. So the only path from the instruction to later tokens is through the gist positions' keys and values in every layer. At inference, the outputs after the gists depend on the instruction only through those KV entries, so you can compute them once, drop the instruction's KV, and get exactly the same results. The code above checks this numerically.
Gisting reports a 40% FLOP reduction. Is that the right comparison for a production system?advanced
No. 40% (915 to 552 GFLOPs on LLaMA-7B) is versus re-encoding the instruction every time. A production server would use prefix caching of the instruction's KV. Against that, gisting saves only 0.11% of FLOPs and 1% of time, since compute is dominated by the new input tokens. The real benefit is storage and context space: one gist position instead of about 26, so about 26 times more cached prompts in the same memory (1.05 MB per position in fp32 on LLaMA-7B).
How do AutoCompressors train without labels, and what is summary accumulation?core
A document is cut into segments. After each segment the model reads κ = 50 summary tokens; their outputs become summary vectors that are prepended as a soft prompt to later segments. The loss is ordinary next-token prediction on every segment, so the only way to use information from earlier segments is through the summaries, which forces them to be useful. Summary accumulation means concatenating the summaries of all previous segments, not just the last one, so the soft prompt grows by κ per segment. Gradients flow through summaries for 2 compression steps, then are stopped.
Describe ICAE's architecture and its two pretraining objectives.core
The encoder is the LLM with a LoRA adapter (rank 128 on Q and V, about 1% extra parameters) reading the context plus k memory tokens; the outputs at those positions are the memory slots. The decoder is the frozen original LLM. Pretraining mixes an autoencoding loss (rebuild the context from the slots after an [AE] token) and a continuation loss (predict the text after the context), with λ around 0.5. Then it is fine-tuned on prompt-with-context data so slots support question answering. On Llama-7B, 512 tokens in 128 slots reconstruct with BLEU 99.1.
An online compressor that runs the 7B model itself over the context: can it ever save prefill compute?advanced
Not for a single use. The encoder pass costs about \(2N(n + k)\), already more than the \(2Nn\) prefill it replaces, and the LLM still prefills the \(c\) slots. It saves compute only if the compressed result is reused: \(R \gt (n + k)/(n - c)\), which is \(R \ge 2\) for ICAE at \(512 \to 128\). Even at \(R = 1\) it still shrinks the KV cache by \(\rho\), which allows larger batches and makes memory-bound decode steps faster, and the encoder's prefill is a parallel, compute-bound pass.
How do soft prompts relate to prompt tuning and prefix tuning?core
Prompt tuning learns input vectors per task; prefix tuning learns keys and values in every layer per task. PromptCompression is prompt tuning with a distillation objective: match the model's outputs with the full prompt (KL, formula 89.6), optimized separately for each prompt. Gist tokens are like a prefix in every layer, but predicted by the model from the instruction in one forward pass (amortized), so new instructions need no optimization. AutoCompressor summary vectors and ICAE memory slots are amortized input-level soft prompts for arbitrary contexts.
You serve RAG with a closed API model. Which of these methods can you use, and how would you evaluate it?basic
Only text methods: pruning or a RECOMP-style summarizer, since you cannot feed vectors to an API. Train or prompt a small compressor that is query-focused and allowed to return nothing. Evaluate end-task accuracy (EM, F1, or judged quality) against full passages and against no retrieval, the fraction of tokens kept, the extra latency of the compressor (especially for abstractive), and failure cases where the answer sentence was dropped.
What limits how far soft-prompt compression can go?advanced
Each soft vector has \(d\) numbers (4,096 for a 7B model), so in principle it can carry a lot, but the encoder must learn to write and the frozen decoder must be able to read that information. Results show the trade-off: ICAE rebuilds text almost exactly at 4×; AutoCompressors at about 41× keep part of the information (perplexity 5.07 versus 4.76 with full context); Gisting at 26× works for short instructions, not long documents. Text that the model knows well compresses better than random text. Higher ratios lose fine details first, such as exact numbers and names. [Likely]

Exercises

  1. RECOMP's abstractive compressor gives 37.04 EM with 36 tokens on NQ (full passages: 39.39 with 660 tokens; no retrieval: 21.99). What fraction of tokens is kept, and what fraction of the retrieval gain is kept?
  2. In Figure 3 with an instruction of 4 tokens, \(k = 1\) gist token and 6 tokens after it, how many query–key pairs does the gist mask block? Give the general formula.
  3. A service caches 100 different system prompts of 500 tokens each for Llama 2 7B in bf16 (0.5 MiB per position). How much memory do the caches need? How much with one gist token per prompt?
  4. An AutoCompressor reads a 30,720-token document in segments of 2,048 tokens with κ = 50 and accumulation. How many positions does the model read in the last forward pass? How many would full attention over the document need at that point?
  5. For ICAE with \(n = 2048\) and \(k = 512\), using formula (89.11) with \(F_{\text{comp}} = 2N(n + k)\), what reuse \(R\) is needed before compute is saved?
Answerscheck
1. \(36/660 = 5.5\%\) of the tokens. Gain kept: \((37.04 - 21.99)/(39.39 - 21.99) = 15.05/17.40 = 86\%\). 2. Every row after the gist (6 rows) loses every instruction column (4 columns): \(6 \times 4 = 24\) pairs. In general: \((\text{number of tokens after } G) \times |t|\). 3. \(100 \times 500 \times 0.5\) MiB \(= 25{,}000\) MiB \(\approx 24.4\) GiB. With one gist position each: \(100 \times 0.5 = 50\) MiB. 4. 15 segments; the last pass reads \(14 \times 50 = 700\) summary vectors \(+\ 2{,}048\) tokens \(+\ 50\) summary tokens \(= 2{,}798\) positions. Full attention would cover all 30,720 tokens. 5. \(R \gt (2048 + 512)/(2048 - 512) = 2560/1536 = 1.67\), so \(R \ge 2\).

Key takeaways

  • Compression cuts prefill compute and KV memory by the ratio \(\rho = n/c\) (89.1). Text outputs (summaries) work with any LLM; soft vectors give higher ratios but only for the model they were trained with.
  • RECOMP trains its compressors with the target LLM as the judge (89.3–89.5): \(660 \to 37\) tokens on NQ while keeping about 84% of the retrieval gain, and it may return nothing when retrieval does not help.
  • Soft prompts can imitate a long prompt (89.6). Gisting learns to produce them in one pass through a modified attention mask (89.7, 89.8); its main win is 26× less cache storage, not FLOPs.
  • AutoCompressors (89.9) carry 50 summary vectors per 2,048-token segment; ICAE (89.10) uses a LoRA encoder and a frozen decoder, 4× with near-perfect reconstruction.
  • Online compressors that run the LLM itself save compute only with reuse (89.11); they still shrink the KV cache from the first request.

Key formulas

\[ \rho = \frac{n}{c}, \qquad p^\ast = \argmax_{s_i} \log P_M(y \mid s_i, x), \qquad Z^\ast = \argmin_Z \E_x \KL\big(P_M(\cdot \mid P, x) \,\|\, P_M(\cdot \mid Z, x)\big) \]
\[ \mathcal{L}_{\text{pre}} = \lambda \mathcal{L}_{\text{AE}} + (1 - \lambda) \mathcal{L}_{\text{LM}}, \qquad R \gt \frac{F_{\text{comp}}}{2N(n - c)} \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Xu et al. (2023). RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
  3. Mu et al. (2023). Learning to Compress Prompts with Gist Tokens
  4. Chevalier et al. (2023). Adapting Language Models to Compress Contexts (AutoCompressors)
  5. Ge et al. (2023). In-context Autoencoder for Context Compression in a Large Language Model
  6. Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
  7. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  8. Xu et al. (2023). RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
  9. Mu et al. (2023). Learning to Compress Prompts with Gist Tokens
  10. Chevalier et al. (2023). Adapting Language Models to Compress Contexts (AutoCompressors)
  11. Ge et al. (2023). In-context Autoencoder for Context Compression in a Large Language Model
  12. Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
  13. Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning
  14. Wingate, Shoeybi & Sorensen (2022). Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models
  15. Fei et al. (2023). Extending Context Window of Large Language Models via Semantic Compression
  16. Jiang et al. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models