Prompt summary and soft-prompt compression: RECOMP, Gisting, AutoCompressors, ICAE
Long prompts cost prefill compute and KV-cache memory. This session shows two ways to make them much shorter without changing what the model can do: rewrite the text into a short summary with a small trained compressor (RECOMP), or replace it with a few learned vectors that the model reads like a long prompt (soft prompts, Gisting, AutoCompressors and ICAE). We also work out when compression really saves compute.
What you will learn
- The difference between hard prompts (text) and soft prompts (vectors), and between offline and online compression
- How RECOMP trains extractive and abstractive compressors with the target LLM as the judge, and why it may return an empty summary
- How a soft prompt can imitate a long prompt by minimizing a KL divergence, and why that needs one optimization per prompt
- How the gist attention mask forces an instruction into a few gist tokens, and why the real saving is cache storage
- How AutoCompressors carry summary vectors across segments, and how ICAE compresses a context into memory slots with a LoRA encoder and a frozen decoder
- A break-even rule that tells when online compression saves compute, and what it always saves in KV memory
Why this matters
A RAG system puts five retrieved passages in front of a question: about 660 tokens of text for a question of maybe 15 tokens. A chat product puts the same 500-token system prompt in front of every user message. Long prompts are paid for twice: once in prefill compute, and again in KV-cache memory for every step of decoding.
Session 88 made prompts shorter by deleting tokens that carry little information. The kept tokens stayed exactly as they were. This session looks at two stronger ideas from the efficient-inference survey (Zhou et al., sections 4.1.2 and 4.1.3):
- Prompt summary. A small model rewrites the long text into a short text. The output is still normal words, so any LLM can read it, even one behind an API. The main example is RECOMP.
- Soft-prompt compression. The long text is replaced by a few vectors that are not words at all. The LLM reads them in place of token embeddings or as extra keys and values. Examples: Gisting, AutoCompressors and the In-context Autoencoder (ICAE).
The numbers are large. On Natural Questions, RECOMP shrinks five passages from 660 tokens to 37 tokens (5.6%) and the exact-match score falls only from 39.39 to 36.57. [Certain, Table 2 of the RECOMP paper] Gisting compresses a 26-token instruction into a single token. [Certain] AutoCompressors turn a 2,048-token segment into 50 vectors, about 41 tokens per vector. [Certain] The price is always the same question: what is lost, and who pays the cost of compressing?
You will need prompt tuning and prefix tuning from session 56, the KV-cache size formula from session 80, and attention masks from session 18. The next session, session 90, goes one step back: instead of compressing retrieved text, retrieve less of it.
Two ways to make a prompt shorter
Call the original context \(C\) with \(n\) tokens, and the compressed form \(\tilde C\) with \(c\) positions. The compression ratio is \(\rho = n / c\). The LLM then does prefill over \(c\) positions instead of \(n\), and keeps \(c\) positions in its KV cache. For a dense model with \(N\) parameters:
- Read the symbols
- \(N\)
- number of model parameters (6.74 billion for Llama 2 7B)
- \(c\)
- number of positions the LLM actually reads: summary tokens, or soft vectors
- \(L, n_{kv}, d_h\)
- layers, key-value heads and head size; for Llama 2 7B: 32, 32 and 128
- \(b\)
- bytes per number: 2 for bf16, 4 for fp32
- the factor 2 in KV bytes
- one key and one value per position
(The \(2Nc\) rule ignores the attention-score FLOPs, which grow with \(c^2\). They only make long prompts even more expensive, so compression helps more than the rule shows.) The two families differ in what \(\tilde C\) is:
| Hard prompt (text) | Soft prompt (vectors) | |
|---|---|---|
| What the LLM reads | Real tokens from the vocabulary | Vectors in embedding (or key/value) space that need not be any real token |
| Examples | Pruning (LLMLingua), summary (RECOMP, SemanticCompression) | PromptCompression, Gisting, AutoCompressors, ICAE |
| Works with a black-box API? | Yes | No: you must feed vectors into the model |
| Readable by a human? | Yes, you can debug it | No |
| Typical ratio \(\rho\) | 2× to 20× | 4× (ICAE) to 26× (Gisting) and more |
The survey also splits soft-prompt methods by when compression happens. Offline methods compress a fixed prefix (a system prompt, a task instruction) once and reuse it for many requests: PromptCompression and Gisting. Online methods compress each new context when it arrives: AutoCompressors and ICAE. This split decides who pays for compression, and Figure 1 makes it concrete.
How to read Figure 1
Each pair of bars compares “no compression” (left) and the chosen method (right) for compute, and the same for KV memory. Start with RECOMP extractive: the compressor is tiny, so compute falls almost by \(\rho\) at once. Now click ICAE with reuse \(R = 1\): the total compute is higher than no compression, because the encoder is the same 7B model reading all \(n\) tokens. The KV cache still shrinks by 4×. Move \(R\) up to 2 and the compute bar drops below the baseline. Gisting has the same structure, but its prompt is a fixed instruction shared by many users, so a large \(R\) is natural. The extra sequential steps in the readout matter for latency: an abstractive summary must be generated token by token before the LLM can start.
Prompt summary with RECOMP
RECOMP (Xu, Shi and Choi, 2023) stands for “Retrieve, Compress, Prepend”. A retriever returns the top 5 passages (each a 100-word piece of Wikipedia). Before they go into the LLM, a small compressor turns them into a short text that keeps only what the question needs. [Certain] The LLM itself does not change. It is used only as a judge while the compressor is trained.
The key design choice is the training signal. A generic summarizer keeps what a human would call important. RECOMP keeps what helps this LLM answer. The paper reports that its compressors beat off-the-shelf summarization models for this reason. [Certain]
The extractive compressor
The extractive version selects whole sentences. It is a dual encoder: one encoder maps the input \(x\) (the question) to a vector, the same kind of encoder maps each candidate sentence \(s_i\) to a vector, and the score is their dot product. It starts from Contriever, a retrieval model with about 110M parameters. [Certain]
To train it we need to know which sentence is “good”. RECOMP asks the frozen LLM \(M\). For language modeling, the positive sentence is the one that gives the target text the highest likelihood when it is put in front of \(x\). For question answering, it is a sentence that makes \(M\) produce the correct answer. [Certain]
Negatives are up to five sentences that Contriever scores high, but whose LLM score is clearly lower than the positive's. [Certain] These are “hard negatives”: they look relevant but do not help. The loss is the usual contrastive loss of session 60:
- Read the symbols
- \(E(\cdot)\)
- the encoder that maps a text to one vector
- \(S\)
- the candidate sentences (the top 20 sentences from the top 5 passages in the paper)
- \(P_M(y \mid s_i, x)\)
- the probability the frozen LLM gives to the right output \(y\) when sentence \(s_i\) comes before \(x\)
- \(p^\ast, n_j\)
- the positive sentence and the hard negatives
How to read Figure 2
For question A, the sentence with the answer has the top score, so \(k = 1\) already keeps it with a small fraction of the tokens. Raising \(k\) adds tokens but little new information. Switch to question B: the passages are about a different topic, all scores are low, and with \(\tau = 0.5\) the compressor returns an empty string. The LLM then answers from its own knowledge, which is better than reading misleading text. This is RECOMP’s selective augmentation. Set \(\tau\) to 0 to see what happens without it.
The abstractive compressor
Selecting sentences cannot merge facts that are spread over several passages. The abstractive version writes a new summary. It is an encoder-decoder initialized from T5-large (775M parameters). [Certain] There is no human-written training data for “a summary that helps Flan-UL2 answer this question”, so RECOMP builds it by distillation (session 59):
- A large teacher (GPT-3.5-turbo) writes query-focused summaries of the top 5 passages. For language modeling it tries four prompts and keeps the best one.
- Each candidate summary is tested with the target LLM. It is kept only if putting it in front of the input improves the end task.
- If no candidate helps, the training target is the empty string.
- Read the symbols
- \(\mathcal{S}\)
- the candidate summaries written by the teacher
- \(\text{task}(M, \tilde C, x)\)
- how well the frozen LLM does when \(\tilde C\) is put before \(x\): the likelihood of the gold continuation, or whether the answer is correct
- \(\varnothing\)
- no context at all: the LLM answers from its own knowledge
The student T5 is then trained with ordinary cross-entropy to produce \(\tilde C^\ast\) from \((x, \text{passages})\). Empty targets are common: 25% of examples for NQ, 16% for TriviaQA, 4% for HotpotQA and 24% for WikiText. [Certain, Table 5 of the paper] So the compressor learns when to stay silent.
The paper’s numbers for Flan-UL2 (20B) on NQ (Table 2): no retrieval gives 21.99 exact match (EM). The full top 5 passages give 39.39 EM with 660 tokens. The extractive compressor gives 36.57 EM with 37 tokens, and the abstractive one 37.04 EM with 36 tokens. [Certain]
So the extractive compressor keeps \(37/660 = 5.6\%\) of the tokens. Retrieval added \(39.39 - 21.99 = 17.40\) EM points. The compressed context still gives \(36.57 - 21.99 = 14.58\) of them, which is 84% of the gain (86% for abstractive), at about 1/18 of the tokens.
Now apply formula (89.1) to a Llama 2 7B reader. Prefill: \(2 \times 6.74\text{B} \times 660 = 8.90\) TFLOPs per request without compression, and \(2 \times 6.74\text{B} \times 37 = 0.50\) TFLOPs with it. The compressor adds roughly \(2 \times 110\text{M} \times 660 = 0.15\) TFLOPs. KV cache in bf16 (0.5 MiB per token): 330 MiB versus 18.5 MiB per request. For a batch of 64 requests that is 20.6 GiB versus 1.2 GiB of cache.
Two more results from the paper. The compressors trained for GPT-2 also helped the larger GPT2-XL and GPT-J on language modeling, so a compressor can transfer to other readers. [Certain] And the target LLM is used only through its outputs, so the same recipe works for an API model. A related method, SemanticCompression (Fei et al., 2023), splits a long text into sentences, groups them by topic, and summarizes each group with a pretrained model; the authors report handling texts 6 to 8 times longer than the context window, without fine-tuning the LLM. [Certain, as reported]
Soft prompts: a few vectors that act like a long prompt
A prompt reaches the model as a matrix of embeddings, one row per token. Nothing forces those rows to be embeddings of real words. Session 56 used this fact for prompt tuning: learn \(k\) free vectors \(Z \in \R^{k \times d}\) that make a frozen model do a task well. The same idea can be used for compression. Instead of a task loss, we ask the model with \(Z\) to behave like the model with the long prompt \(P\). Wingate, Shoeybi and Sorensen (2022) called this prompt compression; it is the “PromptCompression” of the survey:
- Read the symbols
- \(P\)
- the long, fixed prompt we want to replace (for example a system prompt)
- \(Z\)
- \(k\) learned vectors, each of the model width \(d\); \(k\) is much smaller than the length of \(P\)
- \(x\)
- inputs that could follow the prompt; the expectation averages over them
- \(\KL(\cdot \| \cdot)\)
- how far the student's next-token distributions are from the teacher's (the model with the full prompt)
The model weights stay frozen; only \(Z\) is trained, by backpropagation through the model. The authors found that heavily compressed prompts keep the broad concepts and sentiment of the original but lose fine details, and that some complex prompts can be squeezed into a single token. [Certain, as reported] The weakness: you must run a new optimization for every prompt. That is fine for one system prompt used a million times. It is useless for a RAG context that changes every request.
The rest of this session is about one question: how do we get \(Z\) for a new prompt without running an optimization? The answer of all three methods below is the same: train the LLM itself (or a copy of it) to produce \(Z\) in one forward pass. This is called amortized compression: we pay once in training, then compressing a new prompt costs one forward pass.
Gisting: compress instructions into gist tokens
Gisting (Mu, Li and Goodman, 2023) targets task instructions such as “Translate this to French:” or “Write a haiku about the input.” It adds \(k\) new gist tokens \(G = (g_1, \dots, g_k)\) to the vocabulary and places them right after the instruction \(t\). Then it changes only the attention mask during instruction fine-tuning: tokens after the gist tokens may not attend to the instruction. [Certain]
- Read the symbols
- \(M_{ij}\)
- 1 if position \(i\) (a query) may attend to position \(j\) (a key), 0 if not
- \(t\)
- the instruction tokens, placed first
- \(G\)
- the gist tokens, placed right after \(t\); they can see \(t\)
- “after \(G\)”
- the input \(x\) and the output \(y\); they see \(G\), \(x\) and earlier \(y\), but not \(t\)
Think about what this forces. The only path from the instruction to the answer goes through the gist tokens. To predict \(y\) well, the model must learn to pack everything useful from \(t\) into the keys and values of \(G\), in every layer. The training loss is the usual instruction-tuning loss, only with the new mask:
So gisting costs nothing extra in training beyond normal instruction tuning. [Certain] At inference, run the model over \((t, G)\) once, keep only the keys and values at the \(k\) gist positions, and throw away those of \(t\). Every later request with the same instruction starts from that tiny cache. This is the offline case of Figure 1 with a large reuse \(R\). Gist tokens are prefix tuning (session 56) whose prefix is predicted by the model from the instruction instead of learned per task; the survey calls this a meta-learning step. [Certain]
How to read Figure 3
With the causal mask, every token sees everything before it, including the instruction. Switch to the gist mask: a red block appears in the lower left. All tokens after the gist tokens lose sight of the instruction; only the gist rows themselves still read it. Click the row of “chat”: it reads the gist token(s), the input and earlier output, nothing else. The number of blocked pairs equals \((\text{tokens after } G) \times (\text{instruction length})\). More gist tokens give the instruction a wider “pipe”, but the paper found results were generally insensitive to \(k\) between 1 and 10. [Certain]
- training input: instruction data \((t, x, y)\), a pretrained LM, \(k\) new gist embeddings
- for each batch: build the sequence \([t;\ G;\ x;\ y]\)
- build mask \(M\) by formula (89.7) // causal, plus: rows after G cannot see t
- loss \(\leftarrow\) cross-entropy on \(y\) only, formula (89.8); update all weights
- inference, once per instruction: run the model on \([t;\ G]\); keep the K and V of the \(k\) gist positions in every layer
- inference, per request: prefill \(x\) with the gist KV as a cached prefix; decode \(y\) as usual
- the instruction’s own KV is never stored // k positions instead of |t|
The results. Gisting was trained on “Alpaca+”, about 130K instruction examples, with LLaMA-7B and FLAN-T5-XXL. [Certain] With one gist token on LLaMA-7B, ROUGE-L was 57.8 versus 58.0 for the uncompressed model on seen tasks, 46.6 versus 48.1 on unseen tasks, and 23.9 versus 27.0 on human-written instructions; When ChatGPT compared answers, the gist model's win rates against the full model were 48.6%, 49.7% and 45.8% on the three splits (50% would mean a tie). [Certain, Table 1] Compute: on LLaMA-7B, FLOPs per request fell from 915 to 552 GFLOPs (40%) versus no caching. [Certain, Table 3]
The fair baseline is not “no caching” but instruction caching: keep the full KV cache of the instruction. Against that, gist caching saves only 0.11% of FLOPs and 1.0% of time on LLaMA-7B, because most FLOPs go to the new input tokens. [Certain]
The gain is storage. LLaMA-7B has 32 layers of width 4,096, so one position of KV cache in fp32 is \(2 \times 32 \times 4096 \times 4 = 1{,}048{,}576\) bytes, the 1.05 MB the paper reports. A 26-token instruction costs 27.3 MB; one gist token costs 1.05 MB. So the same memory holds 26 times more cached prompts. [Certain] In bf16 the numbers halve (13 MiB versus 0.5 MiB), the ratio does not.
AutoCompressors: summary vectors for any long context
AutoCompressors (Chevalier, Wettig, Ajith and Chen, 2023) apply the gist idea to long documents, online. Split a document into segments \(S_1, S_2, \dots\) of 1,024 to 2,048 tokens. Append \(\kappa = 50\) special summary tokens to each segment. The model's outputs at those positions are the summary vectors \(\sigma_i\). They are passed on as a soft prompt to every later segment. [Certain]
- Read the symbols
- \(S_i\)
- segment \(i\) of the document, as normal tokens
- \(\langle\text{Sum}\rangle_{1..\kappa}\)
- \(\kappa\) learned summary-token embeddings (\(\kappa = 50\) in the paper)
- \(\sigma_i \in \R^{\kappa \times d}\)
- the outputs at the summary positions: the summary vectors of segment \(i\)
- \([\,\sigma_1; \dots; \sigma_{i-1}\,]\)
- summary accumulation: all earlier summaries, concatenated, as a soft prompt of length \((i-1)\kappa\)
Training is unsupervised: the loss is plain next-token prediction on the tokens of every segment. A segment can only learn about earlier segments through the summary vectors, so the model is pushed to put useful information into them. Gradients flow back through the summary vectors into earlier segments, like backpropagation through time in an RNN, but they are stopped after 2 compression steps to save memory. [Certain] The paper fine-tunes OPT-1.3B and OPT-2.7B on sequences of up to 30,720 tokens, and Llama-2-7B on 15B tokens with LoRA of rank 16. [Certain]
How to read Figure 4
Step through the segments. The row of each step is what the model reads in one forward pass. With accumulation on, the green soft prompt grows by 50 vectors per segment: after 14 segments it is 700 vectors, still much less than the 28,672 tokens it stands for. The readout compares the attention span with plain full attention over the whole document. Turn accumulation off: the model only sees the last summary (like the Recurrent Memory Transformer), so the span stays flat, but older information must survive many rounds of re-compression.
The results. With Llama-2-7B, perplexity on a segment was 5.40 with no context and 5.07 when 6,144 tokens of earlier context were compressed into 150 summary vectors. Full attention over the plain 6,144 tokens was better still, at 4.76. [Certain, Table 3] So summary vectors keep a real part of the information, but not all of it. For in-context learning, 50 summary vectors of demonstrations often matched or beat 150 tokens of plain-text demonstrations: for example 79.6 versus 74.5 accuracy on AG News and 94.2 versus 92.4 on SST-2. [Certain, Table 4] Summary vectors of retrieved passages can also be precomputed and stored next to the passages, which makes them useful for retrieval-augmented LMs. [Certain]
ICAE: an in-context autoencoder with memory slots
The In-context Autoencoder (Ge et al., 2023) treats compression as an autoencoder: an encoder squeezes a context into \(k\) vectors called memory slots, and a decoder must rebuild the context from them. Both parts are the same LLM:
- Encoder: the LLM plus a LoRA adapter (rank 128 on the query and value projections). It reads the context \(C\) followed by \(k\) learned memory tokens \(e_1, \dots, e_k\). The outputs at those \(k\) positions are the memory slots \(\mathbf{m}\). [Certain]
- Decoder: the original, frozen LLM. It reads the memory slots as a soft prompt, followed by a task prompt. [Certain]
- Read the symbols
- \(\Delta_{\text{LoRA}}\)
- the only new weights of the encoder (plus the memory-token embeddings), about 1% of the model
- \(\mathbf{m} \in \R^{k \times d}\)
- the memory slots; \(k = 128\) for a 512-token context gives \(\rho = 4\)
- \(\mathcal{L}_{\text{AE}}\)
- autoencoding loss: given \(\mathbf{m}\) and a special [AE] token, the frozen decoder must reproduce \(C\) word by word
- \(\mathcal{L}_{\text{LM}}\)
- text-continuation loss: given \(\mathbf{m}\), predict the text that comes after \(C\)
- \(\lambda\)
- the mix; values between 0.4 and 0.6 worked best
Pretraining uses the Pile (200K updates). Autoencoding alone would teach the slots to store surface text; the continuation loss pushes them to also store meaning that is useful later. Then ICAE is fine-tuned on PwC (“prompt with context”), 240K examples where GPT-4 wrote questions and instructions about Pile texts, so that the slots work for real tasks and not only for copying. [Certain]
How to read Figure 5
Compare the three targets. In autoencoding, the target is the context itself, so the slots must keep almost every word. In continuation, the target is the next text, so the slots must keep what predicts the future. In fine-tuning, the target is an answer to a question about the context. Move \(k\): fewer slots give a higher ratio and a smaller KV cache for the decoder, at a loss of detail. The paper's main setting is \(k = 128\), \(\rho = 4\).
The results. On Llama-7B with 512 tokens in 128 slots, the frozen decoder rebuilds the context with BLEU 99.1. [Certain] Text continuation perplexity rose from 9.01 with the real context to 9.50 with the slots. [Certain] Latency on one A100 (generation length 128): for \(8 \times 2{,}048\)-token inputs, total time fell from 24.0 to 7.3 (3.3×), with 3.4 of that spent on compression; the paper says the speedup can exceed 7× when the slots are cached in advance. [Certain, Table 7]
Two things are worth noticing. First, the decoder is frozen: ICAE does not change the LLM, it only learns to write in a “language” the LLM already reads. Second, nearly perfect reconstruction at 4× says that a 4,096-wide vector can hold about four tokens of text exactly. The paper links this to memorization ability: what the model knows well (normal text) compresses better than random text. [Certain, as discussed in the paper]
- input: context \(C\) (\(n\) tokens), task prompt \(q\), encoder (LLM + LoRA), frozen decoder LLM, \(k\) memory embeddings
- if the slots of \(C\) are in the cache then \(\mathbf{m} \leftarrow\) cache[\(C\)] // offline / reuse case: free
- else \(\mathbf{m} \leftarrow\) encoder\(([C;\ e_{1..k}])\) at the last \(k\) positions; cache[\(C\)] \(\leftarrow \mathbf{m}\) // one parallel pass, cost \(\approx 2N(n + k)\)
- prefill the frozen LLM on \([\mathbf{m};\ q]\) // k + |q| positions instead of n + |q|
- decode the answer with a KV cache of \(k + |q|\) + generated positions
When does compression pay? The break-even rule
Every method moves cost from the LLM to a compressor. Say the compressor costs \(F_{\text{comp}}\) FLOPs for one context, and the result is used by \(R\) requests. Compression saves compute when
For RECOMP, \(F_{\text{comp}}\) comes from a model 60 times smaller than a 7B reader, so the rule holds even at \(R = 1\). For ICAE and AutoCompressors, the encoder is the LLM itself, \(F_{\text{comp}} \approx 2N(n + k)\), so you need \(R \gt (n + k)/(n - c)\). For ICAE with \(n = 512\), \(k = c = 128\): \(R \gt 640/384 = 1.67\). Used once, the context costs 1.5 times the compute of no compression. Used twice, it costs 0.875 times. The KV cache, however, shrinks by \(\rho\) from the first request, and decode steps, which are limited by memory bandwidth (session 82), read a cache that is \(\rho\) times smaller. [Likely: this is why ICAE's single-use latency still improves]
Code: the gist mask, and a soft prompt that imitates a long prompt
The code builds a tiny frozen 2-layer attention model with random weights. Part 1 checks the key property of formula (89.7): with the gist mask, the outputs for the input are exactly the same whether the model sees the whole prompt or only the cached gist keys and values. With a normal causal mask they are not. Part 2 trains 2 soft vectors to imitate a 12-token prompt with the KL objective of formula (89.6). It runs on a CPU in a few seconds and is deterministic (tested with PyTorch 2.x).
import torch, torch.nn.functional as F torch.manual_seed(0) V, D, H, NL = 50, 32, 4, 2 # vocab, width, heads, layers (a tiny frozen "LLM") emb = torch.randn(V, D) * 0.5 Wqkv = [torch.randn(D, 3 * D) / D**0.5 for _ in range(NL)] Wo = [torch.randn(D, D) / D**0.5 for _ in range(NL)] Wout = torch.randn(D, V) / D**0.5 def layer(h, l, mask, past=None): """One attention layer. past = (K, V) of extra positions placed before h.""" q, k, v = (h @ Wqkv[l]).split(D, dim=-1) if past is not None: k, v = torch.cat([past[0], k]), torch.cat([past[1], v]) sh = lambda t: t.view(-1, H, D // H).transpose(0, 1) # (H, T, D/H) a = (sh(q) @ sh(k).transpose(1, 2)) / (D // H) ** 0.5 a = a.masked_fill(~mask, float("-inf")).softmax(-1) o = (a @ sh(v)).transpose(0, 1).reshape(-1, D) return h + o @ Wo[l], (k, v) def forward(h, mask): kvs = [] for l in range(NL): h, kv = layer(h, l, mask); kvs.append(kv) return h, kvs # 1) the gist mask, formula (89.7): after the gist, nobody sees the prompt def gist_mask(p, g, x): T = p + g + x m = torch.tril(torch.ones(T, T, dtype=torch.bool)) # causal m[p + g:, :p] = False # input rows cannot see prompt columns return m p, g, x = 6, 1, 4 prompt, gist_tok, inp = torch.randint(0, V, (p,)), torch.tensor([V - 1]), torch.randint(0, V, (x,)) h0 = emb[torch.cat([prompt, gist_tok, inp])] out_full, kvs = forward(h0, gist_mask(p, g, x)) # 2) cache only the gist keys/values, throw the prompt away, run the input alone h = emb[inp] m_in = torch.cat([torch.ones(x, g, dtype=torch.bool), torch.tril(torch.ones(x, x, dtype=torch.bool))], 1) for l in range(NL): K, Vv = kvs[l] h, _ = layer(h, l, m_in, past=(K[p:p + g], Vv[p:p + g])) print("gist mask, cached gist only :", (h - out_full[p + g:]).abs().max().item()) # ~3e-7: identical out_causal, kvs_c = forward(h0, torch.tril(torch.ones(p + g + x, p + g + x, dtype=torch.bool))) h = emb[inp] for l in range(NL): K, Vv = kvs_c[l] h, _ = layer(h, l, m_in, past=(K[p:p + g], Vv[p:p + g])) print("causal mask, cached gist only:", (h - out_causal[p + g:]).abs().max().item()) # ~1.19: info lost # 3) soft-prompt distillation, formula (89.6): k vectors imitate a long prompt k = 2 long_prompt = torch.randint(0, V, (12,)) soft = torch.nn.Parameter(torch.randn(k, D) * 0.5) # the only trainable numbers opt = torch.optim.Adam([soft], lr=0.05) causal = lambda T: torch.tril(torch.ones(T, T, dtype=torch.bool)) def kl_on(xs): T = emb[torch.cat([long_prompt, xs])] with torch.no_grad(): t_logits = forward(T, causal(len(T)))[0][-len(xs):] @ Wout # teacher: full prompt S = torch.cat([soft, emb[xs]]) s_logits = forward(S, causal(len(S)))[0][-len(xs):] @ Wout # student: k vectors return F.kl_div(s_logits.log_softmax(-1), t_logits.log_softmax(-1), log_target=True, reduction="batchmean") gen = torch.Generator().manual_seed(1) test = [torch.randint(0, V, (5,), generator=gen) for _ in range(20)] ev = lambda: sum(kl_on(t).item() for t in test) / len(test) print("KL before:", round(ev(), 4)) # 0.1747 for step in range(300): xs = torch.randint(0, V, (5,), generator=gen) loss = kl_on(xs); opt.zero_grad(); loss.backward(); opt.step() print("KL after :", round(ev(), 4), "| prompt", len(long_prompt), "tokens ->", k, "soft tokens") # 0.0221
Part 1 prints a difference of about \(3 \times 10^{-7}\) (float rounding) with the gist mask and about 1.19 with the causal mask. Part 2 lowers the average KL from 0.1747 to 0.0221: two vectors recover most of the behavior of a 12-token prompt. In this toy the prompt is random tokens; with real models the same loop is what PromptCompression runs per prompt, and what Gisting, AutoCompressors and ICAE learn to replace by one forward pass.
The four methods side by side
| Method | Compresses | Output | What is trained | Reported ratio | Offline / online |
|---|---|---|---|---|---|
| RECOMP extractive | Retrieved passages | Selected sentences (text) | 110M dual encoder; LLM frozen | \(660 \to 37\) tokens (\(\approx 18\times\)) on NQ | Online, cheap |
| RECOMP abstractive | Retrieved passages | A new summary, or empty | 775M T5-large, distilled from GPT-3.5 | \(660 \to 36\) tokens on NQ | Online, adds decode steps |
| PromptCompression | One fixed prompt | Soft vectors | The vectors only, per prompt | Down to 1 token for some prompts | Offline |
| Gisting | Task instructions | KV of \(k\) gist tokens | The whole LM, with the gist mask | Up to 26× | Offline (once per instruction) |
| AutoCompressor | Any long text, per segment | 50 summary vectors per segment | The LM (OPT full; Llama-2 with LoRA) | \(2{,}048 \to 50\) (\(\approx 41\times\)) | Online, or precomputed |
| ICAE | Any context (512 tokens in the main setting) | \(k\) memory slots | LoRA encoder (\(\approx 1\%\)); decoder frozen | 4× | Online, or cached |
Where this is used
| Application | Good choice | Why |
|---|---|---|
| RAG over documents with an API LLM | Extractive or abstractive summary (RECOMP style) | Only text can be sent to an API; fewer input tokens means a lower bill and faster prefill |
| Chat product with a long, fixed system prompt | Prefix caching first; gist-style compression if KV memory is the limit | The prompt is shared by all users (large \(R\)); gists store far fewer positions than the full prompt cache |
| Many-shot in-context classification | AutoCompressor summary vectors of the demonstrations | 50 vectors of demonstrations matched or beat 150 plain tokens on several tasks |
| Long-document QA on your own model | ICAE or AutoCompressor, slots cached per document | Compress once at indexing time, answer many questions with a 4× to 40× smaller cache |
| Video-RAG and video QA | Text captions of retrieved clips, compressed by a RECOMP-style query-focused summarizer [Speculation] | A video clip can turn into thousands of visual tokens or long captions; a query-focused summary keeps only what the question needs |
| Key-moment detection over long videos | Per-segment summary vectors carried forward, AutoCompressor style [Speculation] | Segments of frames play the role of text segments; a running soft memory avoids attending to every earlier frame |
| Agents with long tool outputs | Abstractive summary of each tool result before it enters the history [Likely] | Tool outputs (web pages, logs) are long and mostly irrelevant; the empty-summary option drops useless results |
Common mistakes
- Comparing to “no caching” instead of prefix caching. For a fixed instruction, the strong baseline is caching its KV. Gisting saves 40% of FLOPs against no caching but only 0.11% against instruction caching; its real win is 26× less storage.
- Counting only LLM tokens. An online compressor that runs the LLM itself (ICAE, AutoCompressor) costs about as much as the prefill it replaces. Use formula (89.11) with your real reuse \(R\).
- Forgetting time to first token. An abstractive summarizer generates its summary token by token. That latency is on the critical path before the LLM starts.
- Using a generic summarizer. A summary that reads well for a human can drop the one number the question needs. RECOMP trains with the end-task signal of the target LLM, and reports beating off-the-shelf summarizers.
- Never returning an empty context. If retrieval failed, a compressed version of useless text is still misleading. Let the compressor output nothing (selective augmentation).
- Reusing soft prompts across model versions. Summary vectors and memory slots are tied to the exact weights that produced them and that read them. Recompute them after any update.
- Expecting soft prompts to be portable or inspectable. You cannot send them to an API, and you cannot read them to debug a wrong answer. Keep the original text for audit.
- Pushing the ratio too far. AutoCompressor perplexity at 150 vectors (5.07) was still worse than full context (4.76). Measure task quality at your target ratio; do not assume the paper’s ratio transfers to your data.
Interview questions
What is the difference between prompt pruning, prompt summary and soft-prompt compression?basic
How is RECOMP's extractive compressor trained, and why not just use a similarity model?core
What is selective augmentation and why does it matter?core
Explain the gist attention mask. Why does it allow caching only the gist tokens?core
Gisting reports a 40% FLOP reduction. Is that the right comparison for a production system?advanced
How do AutoCompressors train without labels, and what is summary accumulation?core
Describe ICAE's architecture and its two pretraining objectives.core
An online compressor that runs the 7B model itself over the context: can it ever save prefill compute?advanced
How do soft prompts relate to prompt tuning and prefix tuning?core
You serve RAG with a closed API model. Which of these methods can you use, and how would you evaluate it?basic
What limits how far soft-prompt compression can go?advanced
Exercises
- RECOMP's abstractive compressor gives 37.04 EM with 36 tokens on NQ (full passages: 39.39 with 660 tokens; no retrieval: 21.99). What fraction of tokens is kept, and what fraction of the retrieval gain is kept?
- In Figure 3 with an instruction of 4 tokens, \(k = 1\) gist token and 6 tokens after it, how many query–key pairs does the gist mask block? Give the general formula.
- A service caches 100 different system prompts of 500 tokens each for Llama 2 7B in bf16 (0.5 MiB per position). How much memory do the caches need? How much with one gist token per prompt?
- An AutoCompressor reads a 30,720-token document in segments of 2,048 tokens with κ = 50 and accumulation. How many positions does the model read in the last forward pass? How many would full attention over the document need at that point?
- For ICAE with \(n = 2048\) and \(k = 512\), using formula (89.11) with \(F_{\text{comp}} = 2N(n + k)\), what reuse \(R\) is needed before compute is saved?
Answerscheck
Key takeaways
- Compression cuts prefill compute and KV memory by the ratio \(\rho = n/c\) (89.1). Text outputs (summaries) work with any LLM; soft vectors give higher ratios but only for the model they were trained with.
- RECOMP trains its compressors with the target LLM as the judge (89.3–89.5): \(660 \to 37\) tokens on NQ while keeping about 84% of the retrieval gain, and it may return nothing when retrieval does not help.
- Soft prompts can imitate a long prompt (89.6). Gisting learns to produce them in one pass through a modified attention mask (89.7, 89.8); its main win is 26× less cache storage, not FLOPs.
- AutoCompressors (89.9) carry 50 summary vectors per 2,048-token segment; ICAE (89.10) uses a LoRA encoder and a frozen decoder, 4× with near-perfect reconstruction.
- Online compressors that run the LLM itself save compute only with reuse (89.11); they still shrink the KV cache from the first request.
Key formulas
References for this session
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Xu et al. (2023). RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
- Mu et al. (2023). Learning to Compress Prompts with Gist Tokens
- Chevalier et al. (2023). Adapting Language Models to Compress Contexts (AutoCompressors)
- Ge et al. (2023). In-context Autoencoder for Context Compression in a Large Language Model
- Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Xu et al. (2023). RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
- Mu et al. (2023). Learning to Compress Prompts with Gist Tokens
- Chevalier et al. (2023). Adapting Language Models to Compress Contexts (AutoCompressors)
- Ge et al. (2023). In-context Autoencoder for Context Compression in a Large Language Model
- Li & Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Lester et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning
- Wingate, Shoeybi & Sorensen (2022). Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models
- Fei et al. (2023). Extending Context Window of Large Language Models via Semantic Compression
- Jiang et al. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models