compress a long RAG prompt
In this lab you build a long-context question-answering test with 20 retrieved chunks per question, prune the prompt at 2×, 5× and 10× with LLMLingua-style and LongLLMLingua-style token scores from GPT-2, and let a small chat model answer. You measure prompt tokens, time to first token and accuracy at each ratio, and compare with the simple alternative: send fewer, better chunks.
What you will learn
- How to build a fair test for prompt compression: answers only in the context, automatic checking, distractor chunks and a controlled answer position
- How to compute self-information, the question-aware contrastive score and the document score r_k with one small causal LM
- How to prune a RAG prompt to a target ratio, and why the effective ratio in reader tokens is lower than the target
- How to measure accuracy, answer survival and time to first token, including the cost of the compressor
- Why question-agnostic token pruning fails on multi-document QA while document selection and reordering keep the answer
- When compression pays for latency: the break-even test m·N_s < N_r(1 − 1/ρ)
- Why a reranker with a smaller top-k is the first thing to try
Why this matters
A RAG system (retrieval-augmented generation) puts retrieved text into the prompt, and the prompt gets long fast. Twenty chunks of a few hundred tokens each is already thousands of tokens. Every token costs prefill time, KV-cache memory and, with an API, money. In this lab you take a long RAG prompt, cut it to one half, one fifth and one tenth of its size, and measure what you gain and what you lose.
Session 88 (session 88) explained the methods: Selective Context, LLMLingua and LongLLMLingua delete the tokens that a small language model finds easy to guess. Papers report large ratios with little loss. LLMLingua reports up to 20× on reasoning prompts, and LongLLMLingua reports better accuracy with about 4× fewer tokens on multi-document question answering [Certain]. A lab is where you check such claims on your data. You will:
- build a long-context QA test: 50 questions, each with 20 retrieved chunks, and the answer hidden in one of them;
- apply LLMLingua-style pruning at 2×, 5× and 10×, in four versions: one that ignores the question and three that use it;
- measure prompt tokens, time to first token (TTFT) and answer accuracy at each ratio;
- compare with the simplest alternative: better retrieval, which sends fewer, better chunks and does not touch the text.
The result on our test is clear, and it teaches more than a table in a paper. Question-agnostic token pruning destroys the answers above 2×. The document step of LongLLMLingua (score whole chunks, drop the weak ones, put the best first) saves most of them. The token score used inside the kept chunks decides the rest, and one natural question-aware score deletes exactly the answer. A good reranker with fewer chunks is as accurate as the full prompt and costs almost nothing. You will see why, token by token.
Setup
You need Python 3.9 or newer, PyTorch and Hugging Face transformers. Two models play two roles:
- The compressor is GPT-2 small (124M parameters). It only scores tokens. LongLLMLingua uses LLaMA-2-7B-Chat as its main compressor and tests a GPT-2-small-sized model (named GPT2-dolly in its appendix) in an ablation [Certain]. We use the small one so that everything runs on a laptop.
- The reader is Qwen2.5-0.5B-Instruct, a small chat model that reads the prompt and answers. In production the reader is your big LLM. A small reader makes the test harder for compression: it has less world knowledge to repair broken text.
pip install torch transformers python data.py # prints one test question and one chunk python compress_lab.py 50 # the full experiment (Step 4), about 20 to 60 minutes on a laptop
Everything is deterministic: the test set comes from a seeded random generator, the scores come from one forward pass, and the reader decodes greedily. The only thing that changes between runs is the time measurements. All accuracy and token numbers on this page come from one run of this code. Our timings come from an Apple laptop that was running other heavy jobs at the same time. Treat them as rough, and measure on your own hardware.
Step 1: build a long-context QA test
A good test for prompt compression needs three things. The answer must be in the context and nowhere else, so the model cannot answer from memory. The answer must be checkable automatically. And there must be distractors: chunks that look like the answer chunk but are about something else, as real retrieval returns.
We generate fictional research stations. Each chunk describes one station with seven sentences: where it is, who opened it and when, the staff count, the supply ship, the access code of the main lab, and two filler sentences. The order of the sentences after the first one is shuffled. A question asks for one fact of one station, for example “What is the access code for the main lab at Cinder Station?”. The other 19 chunks describe other stations with the same kinds of facts, so every chunk contains an access code. Only the station name tells them apart.
We also control where the answer chunk sits among the 20: at position 1, 5, 10, 15 or 20, ten questions each. This lets us see the “lost in the middle” effect (Liu et al., 2023): models use information at the start and the end of a long context better than in the middle [Certain]. Figure 4 shows whether our small reader has this problem at our prompt length.
"""Synthetic long-context RAG QA test. Deterministic (seed 0).""" import random NAMES = ["Kestrel","Alder","Birch","Cobalt","Dunmore","Ember","Falcon","Granite","Harbor","Iris","Juniper","Kodiak", "Lumen","Maple","Nimbus","Onyx","Pioneer","Quartz","Raven","Sable","Tundra","Umber","Vesper","Willow", "Yarrow","Zephyr","Aurora","Basalt","Cedar","Delta","Echo","Fjord","Glacier","Heron","Ivory","Jasper", "Krona","Larch","Marlin","Nova","Oriole","Pumice","Quill","Ridge","Solace","Talon","Upland","Valor", "Wren","Xenon","Yukon","Zenith","Arbor","Bramble","Cinder","Dorado","Elm","Flint","Garnet","Hollow"] FIRST = ["Amara","Jonas","Leila","Tomas","Nadia","Felix","Ines","Omar","Mina","Lukas","Sara","Ravi","Elena","Hugo","Yara","Pavel"] LAST = ["Lind","Okafor","Moreau","Tanaka","Sorensen","Haddad","Novak","Reyes","Bauer","Costa","Ivanova","Park","Ahmadi","Silva","Weber","Kowal"] COAST = ["north coast","south valley","eastern plateau","western islands","high desert","river delta"] FILL = ["Visitors must register at the main gate before entering.","The canteen serves hot meals three times a day.", "Most buildings are heated by a small geothermal plant.","Staff rotate between day and night shifts every week.", "A weekly newsletter reports on the research results.","The library holds old field notebooks and maps.", "Heavy storms sometimes close the road in winter.","A small clinic treats minor injuries on site."] ATTRS = ["code","staff","founder","year","ship"] QUESTION = {"code":"What is the access code for the main lab at {n} Station?", "staff":"How many staff members work at {n} Station?", "founder":"Who founded {n} Station?", "year":"In which year was {n} Station opened?", "ship":"Every how many days does the supply ship arrive at {n} Station?"} def make_entity(rng, name): return {"name":name,"code":str(rng.randint(1000,9999)),"staff":str(rng.randint(12,95)), "founder":f"{rng.choice(FIRST)} {rng.choice(LAST)}","year":str(rng.randint(1961,2019)), "ship":str(rng.randint(5,30)),"where":rng.choice(COAST)} def chunk_text(rng, e): s = [f"{e['name']} Station is a research base on the {e['where']}.", f"It was opened in {e['year']} by Dr. {e['founder']}.", f"Today {e['staff']} staff members live and work there.", rng.choice(FILL), f"The supply ship arrives every {e['ship']} days.", f"The access code for the main lab at {e['name']} Station is {e['code']}.", rng.choice(FILL)] head, tail = s[:1], s[1:] rng.shuffle(tail) return " ".join(head + tail) def build(n_q=50, k=20, seed=0): rng = random.Random(seed) positions = [1, 5, 10, 15, 20] items = [] for qi in range(n_q): names = rng.sample(NAMES, k) ents = [make_entity(rng, n) for n in names] chunks = [chunk_text(rng, e) for e in ents] pos = positions[qi % 5] gold = pos - 1 # gold entity is the one at this position attr = ATTRS[(qi // 5) % 5] e = ents[gold] items.append({"chunks":chunks,"gold":gold,"pos":pos,"attr":attr, "question":QUESTION[attr].format(n=e["name"]),"answer":e[attr]}) return items if __name__ == "__main__": it = build() print(it[0]["question"], it[0]["answer"], it[0]["pos"]) # ... Cinder Station? 5104 1 print(it[0]["chunks"][0])
One question has 1,487 context tokens for GPT-2 (the compressor) and 1,860 prompt tokens for Qwen (the reader), counting the chat template, the instruction and the question. Over all 50 questions, the full prompt has 1,871 reader tokens on average. This is shorter than most production RAG prompts, but the structure is the same; Step 6 scales the numbers to 20,000 tokens.
Step 2: score every token with a small model
Prompt pruning rests on one idea from information theory. If a small model can easily guess a token from the tokens before it, the token carries little information, and the reader can probably do without it. The self-information (surprisal) of token \(x_i\) is
- Read the symbols
- \(x_i\)
- the \(i\)-th token of a retrieved chunk, in the compressor’s tokenizer (GPT-2 here)
- \(x_{\lt i}\)
- all tokens before it in the same chunk
- \(p_{\text{small}}\)
- the probability the small model gives to the true next token
- \(s_i\)
- surprisal in nats (natural log). Divide by \(\ln 2\) to get bits, as in formula (88.3). High \(s_i\): surprising, keep it. Low \(s_i\): predictable, can go.
This score ignores the question. Session 88 showed the fix from LongLLMLingua: compare the token’s probability with and without the question in front. A token that becomes much more likely once the model has read the question is related to the question. We test two forms of this contrastive score: the difference of log-probabilities \(c_i\), and the difference of token perplexities \(c_i^{\text{ppl}}\), which is the form written in the paper, formula (88.11):
- Read the symbols
- \(x^{\text{que}}\)
- the question tokens, placed before the chunk
- \(c_i\)
- how much the question raises the log-probability of token \(x_i\). It is a pointwise mutual information between the token and the question. It does not care whether the token was surprising to begin with.
- \(c_i^{\text{ppl}}\)
- the same comparison on the perplexity scale \(1/p\). Because \(e^{s_i}\) multiplies everything, it is large only for tokens that are both surprising and helped by the question. A token the question does not touch (\(c_i = 0\)) scores 0; one the question makes less likely scores below 0.
LongLLMLingua also scores whole documents, before any token is touched (the coarse step). The document score is the average loss of the question, plus a fixed sentence, when the small model reads the document first:
- Read the symbols
- \(x^{\text{doc}}_k\)
- the \(k\)-th retrieved chunk, \(k = 1, \dots, 20\)
- \(x^{\text{que,restrict}}\)
- the question followed by the sentence “We can get the answer to this question in the given documents.” (the restrictive statement from the paper) [Certain]
- \(N_c\)
- number of tokens in that question-plus-sentence
- \(r_k\)
- a loss: lower means the document makes the question easier to predict, so the document is more relevant
Look at real GPT-2 scores for the answer chunk of our first question. The station name “ C” + “inder” costs 10.37 + 7.09 nats the first time, and only 0.55 + 0.01 nats the second time, because GPT-2 has just seen it. The access code “ 5” + “104” costs 4.36 + 6.82 nats. But the filler words “ rotate” (8.97) and “ Costa” (9.71) cost more. A pure surprisal ranking keeps these before the code. The contrastive scores tell another story: “ code” gets +5.75 and “ lab” +5.54 because the question mentions them, while the digits get only +0.12 and +0.27 (the question does not make the number guessable). The document score separates the gold chunk well: \(r_k = 2.55\) for the answer chunk, against 3.08 to 3.32 for the other 19.
- input: chunks \(d_1, \dots, d_K\); question \(q\); small causal LM
- for \(k = 1, \dots, K\):
- tokenize \(d_k\) with the small model’s tokenizer: \(x_1, \dots, x_n\)
- one forward pass on [BOS, \(x_1..x_n\)]: \(s_i = -\ln p(x_i \mid x_{\lt i})\) // (92.1)
- one forward pass on [BOS, \(q\), \(x_1..x_n\)]: \(c_i = \ln p(x_i \mid q, x_{\lt i}) + s_i\) // (92.2)
- one forward pass on [BOS, \(x_1..x_n\), \(q\) + restrict]: \(r_k\) = mean loss on the question part // (92.3)
- return all \(s_i\), \(c_i\), \(r_k\) // cost: 3 passes over the context (1 pass for plain LLMLingua)
Step 3: prune at 2×, 5× and 10×
Now delete tokens. Session 88 used \(\tau\) for the fraction of tokens kept, formula (88.1). Here we name the compression ratio \(\rho = 1/\tau\): 2×, 5× or 10×. With \(L\) context tokens, the budget is
- Read the symbols
- \(L\)
- context tokens in the compressor’s tokenizer (all chunks together, without instruction and question)
- \(\rho\)
- the target compression ratio; \(\tau = 1/\rho\) is the fraction kept
- \(B\)
- the token budget; ties in the score are broken by position (earlier first), so the result is deterministic
- \(\mathcal{K}\)
- the kept tokens; they stay in their original order, and each chunk keeps its “Document [k]:” label
We keep the instruction and the question whole, as LLMLingua’s budget controller does: they are short and every word matters. We compare four ways to fill the budget:
- Random tokens: keep \(B\) random tokens. A sanity check: any method must beat it.
- Token pruning (LLMLingua style, question-agnostic): keep the \(B\) tokens with the highest surprisal \(s_i\).
- Coarse + fine (LongLLMLingua style, question-aware): first sort documents by \(r_k\) and keep the best ones until they hold \(2B\) tokens (the coarse step); then keep the \(B\) tokens with the highest score inside them (the fine step). The kept documents stay in \(r_k\) order, most relevant first: this is LongLLMLingua’s document reordering. We try three fine scores: \(s_i\) (“coarse docs + self-information”), \(c_i\) and \(c_i^{\text{ppl}}\) from (92.2).
- Fewer chunks (better retrieval, Step 5): keep the top \(20/\rho\) chunks of a reranker, uncompressed.
- input: chunk scores from Algorithm 1; ratio \(\rho\)
- \(L \leftarrow\) total context tokens; \(B \leftarrow \operatorname{round}(L/\rho)\) // (92.4)
- sort chunks by \(r_k\), lowest first; take chunks in this order while the kept total is below \(2L/\rho\) // coarse step
- list all tokens of the kept chunks (in the new chunk order) with their fine scores
- keep the \(B\) tokens with the highest fine score (\(s_i\), \(c_i\) or \(c_i^{\text{ppl}}\)); break ties by position // fine step
- for each kept chunk: join its kept tokens in their original order; drop the chunk if no token is left
- return the instruction, the compressed chunks with their labels, and the full question
Figure 2 runs exactly this algorithm in your browser on the first question of the test, with the real GPT-2 scores. Try the four scores at 5×, and hover over a token to see its score.
How to read Figure 2
Start with token pruning at 2×: the answer chunk keeps most of its words and “5104” survives. Move to 3× or 5×: the digits disappear, while names such as “Nadia Costa” and words such as “rotate” stay. Self-information keeps what is surprising, and every chunk is full of surprising names and numbers, so the budget is spread over all 20 chunks. Now switch to coarse + si. The coarse step puts the answer chunk at rank 1 (it has by far the lowest \(r_k\)) and drops most other chunks: at 5× only 8 of 20 are left. With fewer chunks, the fine step can keep about half of each, and the answer survives even at 20×. Then try coarse + ci: on this question the code also survives, but look at what the score likes. It keeps “access code for the main lab at” in every chunk, because the question predicts those words, while the digits get only +0.12 and +0.27. On many other questions of the test this is not enough (Step 4). Coarse + cippl multiplies by \(e^{s_i}\), so surprising tokens come back. In the hidden-chunk mode, the figure shows the answer chunk and the first three other chunks in the order the reader will see them.
The context has \(L = 1,487\) GPT-2 tokens, so \(B = \operatorname{round}(1,487/5) = \operatorname{round}(297.4) = 297\) tokens.
Token pruning keeps every token with surprisal above about 6.85 nats (the score of the 297-th best token). The digits “ 5” (4.36) and “104” (6.82) are both below it, so the code is lost. The reader then answered “The access code for the main lab at Cinder Station is 409.”.
Coarse + fine: the coarse step keeps documents in \(r_k\) order until they hold at least \(2 \cdot 297.4 = 594.8\) tokens. That is 8 documents with 603 tokens. The fine step keeps 297 of those 603 tokens, about 49% of each kept chunk on average. With the fine score \(c_i\) the code “5104” survives, and the reader answered “The access code for the main lab at Cinder Station is 5104.”.
Step 4: measure tokens, TTFT and accuracy
For each of the 50 questions, we build 22 prompts (full; and 7 methods × 3 ratios), let the reader answer each one greedily (up to 24 new tokens), and record three numbers.
Tokens. Count the prompt in the reader’s tokenizer, with the chat template, instruction and question. This is what you pay for and what prefill processes. The effective ratio is
- Read the symbols
- \(N_{\text{full}}, N_{\text{comp}}\)
- reader prompt tokens of the full and the compressed prompt
It is smaller than the target \(\rho\) for two reasons. The fixed parts (template, instruction, question, labels) are not compressed. And text made of isolated kept tokens tokenizes worse: the reader’s tokenizer needs more tokens for “ Station86 meals three.” than the words suggest. On our test, token pruning at 5× gives 541 reader tokens, so \(\rho_{\text{eff}} = 3.46\), not 5.
Accuracy and answer survival. A question counts as correct if the gold answer string appears in the reader’s reply. We also check whether the answer string is still in the compressed context at all. With \(Q\) questions, answers \(a_q\), replies \(y_q\) and compressed contexts \(\tilde C_q\):
- Read the symbols
- \(\mathbb{1}[\cdot]\)
- 1 if the condition is true, else 0
- \(a \subseteq y\)
- the string \(a\) appears inside the string \(y\) (case-insensitive)
- Surv
- answer survival: on this test, the reader cannot be right if the answer is gone (except by luck), so Surv is close to an upper bound for Acc. It is cheap to compute: no reader call needed. Use it to reject bad settings early.
TTFT. Time to first token, formula (81.1): time for the prefill of the prompt plus the first decode step, here with an empty queue. For a fair comparison we also must add the compressor time, see Step 6.
How to read Figure 3
In accuracy mode, the full prompt scores 100%. Token pruning (red) holds 70% at 2× but falls to 16% at 5×, close to random tokens (dashed grey, 10%). Coarse docs + \(s_i\) (orange) stays at 80%, 72% and 72%. Coarse docs + \(c_i\) (dashed) uses the same documents, but its fine score sinks it to 32% at 5×. Fewer chunks with BM25 (green) gives 100%, 100% and 100%. Switch to answer survives: each line drops at the same place as in accuracy mode, so the errors come from the compressor deleting the answer, not from the reader misreading it. Switch to reader tokens: the methods do not reach the same size for the same target, because of formula (92.5). Turn on “coarse docs + \(c_i^{\text{ppl}}\)” to see the paper’s form of the score: 32% at 5×, no better than \(c_i\). The reason is in (92.2): \(c_i^{\text{ppl}} = e^{s_i}(1 - e^{-c_i})\). For the digit “104” of question 1 (\(s_i = 6.82\), \(c_i = 0.27\)) this is about 217, while “ code” (\(s_i = 5.97\), \(c_i = 5.75\)) gets about 391. A token the question does not help stays low, however surprising it is. Turn on the \(r_k\) reranker too: used alone as a reranker, the same document score is as good as BM25.
What we measured for TTFT on the laptop (reader only, median of 5 runs, CPU, 8 threads):
| prompt tokens | 125 | 250 | 500 | 1,000 | 1,860 |
|---|---|---|---|---|---|
| TTFT (ms) | 999 | 1,923 | 4,504 | 10,541 | 17,778 |
The laptop was running other heavy jobs during this measurement, so the absolute times are slow; read the trend. Between 250 and 2,000 tokens, prefill time grows about linearly with the prompt length, as formula (81.5) says. So a 5× shorter prompt gives roughly a 4× to 6× shorter TTFT for the reader. The compressor costs extra: GPT-2 small took 2,453 ms for one pass over the 1,487 context tokens (chunk by chunk) and 9,287 ms for the three passes of Algorithm 1. On this laptop, with this tiny reader, the three GPT-2 passes cost 52% of the full prefill they try to save. Step 6 explains this with formula (92.9): three passes of GPT-2 (\(3 \times 124\)M = 373M parameters) against a reader of 494M parameters at 5× (\(494\text{M} \times 0.8 = 395\)M) pay off only barely, even in theory. Compression pays when the reader is much larger than the compressor.
Step 5: compare with better retrieval
Prompt compression fixes a prompt that already contains too much. The other way to get a short prompt is to retrieve fewer, better chunks. Instead of 20 chunks, send only the top \(k\) after a reranker (a second, more precise scoring step) has sorted them:
- Read the symbols
- \(K\)
- number of retrieved chunks (20)
- \(\pi\)
- the reranker’s order: \(\pi(1)\) is the chunk it thinks most relevant
- \(k\)
- chunks kept: 10, 4 and 2 for \(\rho\) = 2, 5 and 10. The chunks are sent whole, so the text stays readable.
We try two rerankers. BM25 is the classic keyword score (session 60); it costs microseconds on a CPU. The second uses \(r_k\) from formula (92.3), the LongLLMLingua coarse score, alone. On our test, BM25 puts the gold chunk first in 50 of 50 questions, and \(r_k\) in 50 of 50. The station name in the question is a rare word that appears in only one chunk, so keyword matching is very strong here. Real data is harder (synonyms, paraphrases), and a cross-encoder reranker or a dense retriever is the usual choice [Likely].
How to read Figure 4
Look at the black bars first: the full prompt is right at every position, 100% from position 1 to 20. On our prompts of about 1,900 tokens the small reader shows no “lost in the middle” effect; the effect in Liu et al. was measured on longer contexts and other models, so do not expect it at every length. Our errors come from compression, not from position. Green (BM25) also stays at 100% everywhere: it sends the answer chunk first and whole. Orange (coarse docs + \(s_i\)) moves the answer chunk to the front as well (the coarse step reorders by \(r_k\)), so its errors do not follow the original position; they come from tokens deleted inside the answer chunk. Red (token pruning) keeps the original order and deletes the answer at most positions, so it is low everywhere above 2×. With only 10 questions per bar, one question changes a bar by 10 points: read the pattern, not small differences.
The comparison tells you where the gain comes from. On this test almost all of it is document selection and ordering. That is the coarse step of LongLLMLingua, and a reranker does the same job. The token-level deletion inside the kept chunks adds compression, but it also adds risk: it is the step that can break numbers and names. A practical order of work is: (1) improve retrieval and reranking, (2) lower top-\(k\), (3) only then add token-level compression, measured on your own questions.
Step 6: when does compression pay?
Compression adds a step before the reader. With the lower-bound model of session 81 (formula 81.5: prefill of \(P\) tokens on a model of \(N\) parameters costs about \(2NP\) FLOPs), the time to first token with compression is
- Read the symbols
- \(P\)
- prompt length in tokens (we ignore the small difference between the two tokenizers)
- \(N_s, N_r\)
- parameters of the small compressor and of the reader
- \(m\)
- number of compressor passes over the prompt (1 for LLMLingua-style surprisal, 3 in our Algorithm 1)
- \(F\)
- GPU compute speed in FLOPs per second (H100 SXM, BF16, dense: about \(989.5 \times 10^{12}\))
Compression lowers TTFT only if this is less than the full prefill \(2 N_r P / F\). Divide both sides by \(2P/F\):
The prompt length cancels out. Whether compression pays for latency depends only on model sizes, passes and ratio. (For money, the test is different: you pay the API per token, and the compressor may run on your own cheap hardware.) Two more effects help compression and are not in (92.9): the reader’s attention cost grows with \(P^2\) for long prompts, and the KV cache is \(\rho\) times smaller (Llama 3.1 8B stores 128 KiB per token, session 80).
Reader Llama 3.1 8B (\(N_r = 8.03 \times 10^9\)). Full prefill \(\ge 2 \cdot 8.03\times10^9 \cdot 20{,}000 / 989.5\times10^{12} = 324.6\) ms. At 5×: 64.9 ms, a saving of 259.7 ms.
Compressor GPT-2 small, 3 passes: \(2 \cdot 124.4\times10^6 \cdot 60{,}000 / 989.5\times10^{12} = 15.1\) ms. Total \(15.1 + 64.9 = 80.0\) ms instead of 324.6 ms: a 4.06× faster TTFT (lower bound). Test (92.9): \(3 \times 0.124 = 0.37\)B \(\lt 8.03 \times 0.8 = 6.42\)B. It pays.
Compressor Llama-2-7B, 3 passes (LongLLMLingua’s main setting): \(2 \cdot 6.738\times10^9 \cdot 60{,}000 / 989.5\times10^{12} = 817.1\) ms. That is more than the whole full prefill. Test: \(3 \times 6.74 = 20.2\)B \(\gt 6.42\)B. For an 8B reader it does not pay for latency; the reader must have more than \(20.2 / 0.8 = 25.3\)B parameters at 5×.
KV cache of the reader: \(20{,}000 \times 128\) KiB \(= 2.44\) GiB for the full prompt, 0.49 GiB at 5×.
How to read Figure 5
With GPT-2 as compressor, the orange part is tiny and the speed-up is close to \(\rho\). Pick Llama-2-7B ×3 with the 8B reader: the orange bar is longer than the whole top bar, and compression makes TTFT worse at every prompt length, because \(P\) cancels in (92.9). Switch the reader to 70B: now the same compressor pays at 5×. XLM-R-large ×1 is close to the LLMLingua-2 setup, a token classifier with one pass; we count its 560M parameters at 2 FLOPs each per token, which overestimates it because much of an encoder’s size is the embedding table, a cheap lookup [Likely]. Real compressors run far below peak speed on short chunks, so measure them.
The full program
This is the whole experiment in one file. It imports build() from data.py (Step 1). Run python compress_lab.py 1 first: on question 1 it must print 1,860 reader tokens for the full prompt, and accuracy 0 for token pruning at 5× and 10×. Timings will differ on your machine; token counts and accuracies must not.
# compress_lab.py: prompt compression for long RAG prompts, end to end import math, random, re, time, sys, torch from transformers import AutoTokenizer, AutoModelForCausalLM from data import build # the test set generator (Step 1) torch.manual_seed(0) dev = "cuda" if torch.cuda.is_available() else "cpu" # small model that scores tokens (the "compressor") and the model that answers (the "reader") ctok = AutoTokenizer.from_pretrained("gpt2") cmod = AutoModelForCausalLM.from_pretrained("gpt2", dtype=torch.float32).eval().to(dev) rtok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct") rmod = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype=torch.float32).eval().to(dev) BOS = ctok.bos_token_id RESTRICT = " We can get the answer to this question in the given documents." @torch.no_grad() def logp(prefix, ids): """log p(ids[i] | prefix, ids[:i]) for every token i: one forward pass of the small LM.""" x = torch.tensor([[BOS] + prefix + ids], device=dev) lp = torch.log_softmax(cmod(x).logits[0, :-1].float(), -1) tgt = x[0, 1:] return lp[torch.arange(len(tgt), device=dev), tgt][len(prefix):].tolist() def score(chunks, question): q = ctok.encode(" " + question) qr = ctok.encode(" " + question + RESTRICT) docs = [] for c in chunks: ids = ctok.encode(" " + c) lp, lpq = logp([], ids), logp(q, ids) r_k = -sum(logp(ids, qr)) / len(qr) # (92.3) lower = more relevant docs.append(dict(ids=ids, surp=[-a for a in lp], # (92.1) contrast=[b - a for a, b in zip(lp, lpq)], # (92.2) rk=r_k)) return docs def prune(docs, ratio, field, order=None): """Keep the budget B = N / ratio highest-scoring tokens; keep their order (Algorithm 2).""" order = order if order is not None else list(range(len(docs))) N = sum(len(d["ids"]) for d in docs) B = max(1, round(N / ratio)) # (92.4) flat = [(k, i) for k in order for i in range(len(docs[k]["ids"]))] best = sorted(range(len(flat)), key=lambda j: (-docs[flat[j][0]][field][flat[j][1]], j))[:B] keep = set(flat[j] for j in best) out = [] for k in order: ids = [t for i, t in enumerate(docs[k]["ids"]) if (k, i) in keep] if ids: out.append(ctok.decode(ids).strip()) return out def coarse(docs, ratio): """LongLLMLingua coarse step: sort documents by r_k, keep them until 2x the token budget.""" rank = sorted(range(len(docs)), key=lambda k: (docs[k]["rk"], k)) B, keep, n = sum(len(d["ids"]) for d in docs) / ratio, [], 0 for k in rank: if n >= 2 * B: break keep.append(k); n += len(docs[k]["ids"]) return keep # most relevant first def bm25(chunks, q, k1=1.2, b=0.75): W = lambda s: re.findall(r"[a-z0-9]+", s.lower()) D = [W(c) for c in chunks]; N = len(D); avg = sum(map(len, D)) / N df = {w: sum(w in d for d in D) for w in set(W(q))} def s(d): return sum(math.log(1 + (N - df[w] + .5) / (df[w] + .5)) * d.count(w) * (k1 + 1) / (d.count(w) + k1 * (1 - b + b * len(d) / avg)) for w in df) return sorted(range(N), key=lambda i: (-s(D[i]), i)) def prompt(docs, q): body = "\n".join(f"Document [{i+1}]: {d}" for i, d in enumerate(docs)) msg = "Answer the question using only the documents below. Reply with the answer only." return rtok.apply_chat_template([{"role": "user", "content": f"{msg}\n\n{body}\n\nQuestion: {q}"}], tokenize=False, add_generation_prompt=True) @torch.no_grad() def answer(p, max_new=24): ids = rtok(p, return_tensors="pt").input_ids.to(dev) if dev == "cuda": torch.cuda.synchronize() t0 = time.perf_counter() out = rmod(ids, use_cache=True) # prefill nxt = out.logits[0, -1].argmax().view(1, 1) if dev == "cuda": torch.cuda.synchronize() ttft = time.perf_counter() - t0 # time to first token gen, past = [nxt.item()], out.past_key_values for _ in range(max_new - 1): # greedy decode o = rmod(nxt, past_key_values=past, use_cache=True); past = o.past_key_values nxt = o.logits[0, -1].argmax().view(1, 1) if nxt.item() in (rtok.eos_token_id, rtok.convert_tokens_to_ids("<|im_end|>")): break gen.append(nxt.item()) return ids.shape[1], ttft, rtok.decode(gen) items = build()[: int(sys.argv[1]) if len(sys.argv) > 1 else 50] rows = [] answer(prompt(["warm-up text"], "warm-up?")) # first call is slow: do not time it for it in items: docs = score(it["chunks"], it["question"]) runs = [("full", 1, it["chunks"])] for r in (2, 5, 10): k = round(20 / r) runs += [("token", r, prune(docs, r, "surp")), # LLMLingua-style ("coarse_s", r, prune(docs, r, "surp", coarse(docs, r))), # coarse docs + s_i ("qaware", r, prune(docs, r, "contrast", coarse(docs, r))), # coarse docs + c_i ("bm25", r, [it["chunks"][i] for i in bm25(it["chunks"], it["question"])[:k]])] for name, r, kept in runs: n, ttft, out = answer(prompt(kept, it["question"])) rows.append((name, r, n, ttft, it["answer"].lower() in out.lower())) for name, r in sorted({(a, b) for a, b, *_ in rows}, key=lambda x: (x[1], x[0])): sel = [x for x in rows if x[0] == name and x[1] == r] print(f"{name:8s} {r:2d}x tokens {sum(x[2] for x in sel)/len(sel):6.0f} " f"TTFT {1000*sum(x[3] for x in sel)/len(sel):6.0f} ms acc {sum(x[4] for x in sel)/len(sel):.2f}")
Our full run used the same functions, plus the random baseline, the \(c_i^{\text{ppl}}\) variant and the \(r_k\) reranker, on the 50 questions. It ran the models on the Apple GPU (MPS) to save time; the TTFT table above comes from a separate CPU measurement.
Where this is used
| Setting | What prompt compression or better retrieval does there |
|---|---|
| Enterprise RAG over documents | Rerank and cut top-\(k\) first; add question-aware compression when the chunks are long and the context budget is tight |
| Paid LLM APIs | Input tokens are billed; a small local compressor (LLMLingua package) reduces the bill even when it does not reduce latency |
| Long few-shot and chain-of-thought prompts | LLMLingua’s original use: compress demonstrations (GSM8K, BBH); it reports up to 20× with little loss [Certain] |
| Meeting and call transcripts | LLMLingua-2 was trained with data distilled from GPT-4 on meeting transcripts (MeetingBank) [Likely]; transcripts are long and redundant, a good case for token deletion |
| Video-RAG and video QA | ASR transcripts and frame captions of a long video become thousands of tokens; select the relevant segments first (retrieval over time), then compress the text; key-moment detection works the same way: find the segment, not every word |
| Agents with long tool outputs | Web pages and logs returned by tools are pruned or summarized before the next model call (see session 89) |
| Long-context serving | Shorter prompts mean shorter prefill and a smaller KV cache, so more requests fit in one batch (session 80) |
Common mistakes
- Counting the ratio in the wrong tokenizer. The compressor counts GPT-2 tokens of the context; you pay for reader tokens of the whole prompt. Our 5× target gave only 3.46× in reader tokens for token pruning. Always report formula (92.5).
- Measuring only token counts. A 10× shorter prompt that loses the answer is worthless. Measure accuracy (or at least answer survival, 92.6) on questions where you know the answer.
- Using question-agnostic pruning for QA. Self-information keeps the surprising tokens of every chunk; in RAG every chunk is full of surprising names and numbers. On our test, token pruning at 2× already dropped accuracy from 100% to 70%.
- Forgetting the compressor time. TTFT includes it. A 7B compressor in front of an 8B reader makes TTFT worse (92.9), whatever the prompt length.
- Compressing the instruction and the question. They are short, and one lost word (“not”, a name) changes the task. Keep them whole and compress only the retrieved context.
- Skipping the retrieval baseline. If fewer, better chunks give the same accuracy at the same size, you do not need a compressor model. On our test BM25 with 4 chunks was as accurate as the full prompt.
- Testing on a benchmark the reader has memorized. If the model knows the answers, compression looks harmless. Use private or synthetic facts, as in Step 1.
- Breaking structure. Deleting tokens inside JSON, code, tables or numbers produces invalid text. Protect such spans (the LLMLingua package has a
force_tokensoption for this), or do not compress them.
Interview questions
What does LLMLingua-style pruning delete, and why does that work at all?basic
Why did question-agnostic pruning fail on our RAG test, and why did one question-aware variant fail too?core
Write the end-to-end latency with compression. When does it pay?core
How would you build an evaluation set for prompt compression?core
Why is the effective compression ratio lower than the target?basic
What is the “lost in the middle” effect, and how do compression methods interact with it?core
Compression versus a smaller top-k: how do you decide?advanced
Why does LongLLMLingua need three forward passes, and how would you make the compressor cheaper?advanced
Does prompt compression change decode speed?advanced
A pruned prompt gives a correct answer but the user complains about wrong names in the answer. What happened?core
Exercises
- For question 1, \(L = 1,487\). Compute the budget \(B\) at 2×, 5× and 10× with formula (92.4). Check your values in Figure 2.
- Our reader prompt has 1,860 tokens for question 1, and BM25 with 4 chunks gives 415 tokens. What is \(\rho_{\text{eff}}\)? Why is it below 5?
- Use (92.9): with the 70B reader (\(70.55 \times 10^9\) parameters) and Llama-2-7B ×3 as compressor, what is the smallest ratio \(\rho\) at which compression lowers TTFT?
- Change the test so that the answer needs two chunks (for example “Which station has more staff, Cinder or Maple?”). Predict what happens to BM25 with 2 chunks and to question-aware pruning, then run it.
- Install
llmlinguaand runcompress_prompt(..., rank_method="longllmlingua", rate=0.2)with the default model on 10 test questions. Compare tokens and accuracy with our Algorithm 2. - Protect digits: give every token that contains a digit a score of \(+\infty\) in token pruning. How much does accuracy at 5× improve? What is the price?
Answerscheck
round sends .5 to the even number); \(B = \operatorname{round}(297.4) = 297\); \(B = \operatorname{round}(148.7) = 149\). 2. \(1,860/415 = 4.48\). The template, instruction, question and labels (about 150 tokens) are kept in both prompts. 3. Need \(3 \times 6.738 = 20.21 \lt 70.55\,(1 - 1/\rho)\), so \(1/\rho \lt 0.7135\), \(\rho \gt 1.40\). 4. BM25 with 2 chunks can hold both stations only if both names rank in the top 2, which is likely here because the question names them; question-aware pruning keeps chunks with the lowest \(r_k\), and the second chunk may rank lower, so check survival of both numbers. 5. Depends on your run: report both \(\rho_{\text{eff}}\) and accuracy. 6. The digits of every chunk now take budget first, about 5 digit tokens per chunk, 99 for question 1, which is a third of the 297-token budget at 5×, so less room is left for words; you trade readable sentences for numbers that the reader may no longer link to the right station.Key takeaways
- A good compression test has answers only in the context, automatic checking, realistic distractors and a controlled answer position (Step 1).
- LLMLingua-style pruning keeps high self-information tokens (92.1); LongLLMLingua adds a document score \(r_k\) (92.3) and a question-aware token score (92.2), then reorders documents.
- Measure reader tokens (92.5), accuracy and answer survival (92.6) and TTFT including the compressor (92.8), at several ratios, against full-prompt and random baselines.
- On our test: full prompt 100%; token pruning 16% at 5×; coarse docs + \(s_i\) 72% at 5×, but coarse docs + \(c_i\) only 32%; BM25 with 4 chunks 100%. Most of the gain comes from choosing and ordering documents.
- Compression lowers TTFT only if \(mN_s \lt N_r(1 - 1/\rho)\) (92.9): small compressors in front of big readers. For API cost, fewer tokens always save money.
Key formulas
References for this session
- Jiang et al. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- Jiang et al. (2023). LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
- Jiang et al. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- Jiang et al. (2023). LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
- Pan, Wu, Jiang et al. (2024). LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
- Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
- Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners
- Qwen Team (2024). Qwen2.5 Technical Report
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models