benchmark a serving system
In this lab you serve a 7B model with vLLM on one GPU, drive it with an open-loop load generator, and plot TTFT, TPOT and throughput against the request rate and the prompt length. Then you turn on prefix caching, chunked prefill, FP8 or INT4 weights, and speculative decoding, one at a time, and explain each change with the survey’s taxonomy. A small simulator tells you what to expect, so you can tell a real effect from a broken benchmark.
What you will learn
- How to write down a serving setup that someone else can reproduce: GPU, model, engine version and every scheduler flag
- How to compute the KV-cache capacity of your GPU and the largest batch it allows, before you run anything
- How to generate load correctly: open loop (Poisson arrivals) against closed loop (fixed concurrency), and why the send time must be the scheduled time
- How TTFT, TPOT, ITL, throughput and goodput change as the load rises, and where the knee comes from (queueing once the batch is full)
- How prompt length moves TTFT, TPOT and the maximum throughput, through compute and through KV-cache capacity
- What prefix caching, chunked prefill, FP8, INT4 and speculative decoding each change, at low load and at overload
- How to place each change in the survey’s taxonomy (memory management, batching, quantization, speculative decoding) and predict which metric it moves
Why this matters
Every serving decision ends with a benchmark: which GPU to rent, how many replicas to run, whether FP8 is worth it, whether to turn on speculative decoding. A benchmark that is set up wrong gives numbers that look precise and mean nothing. In this lab you build a benchmark you can trust. You serve a real 7B model with vLLM on one GPU, send it load the way users send it, and measure how the latency and the throughput change as the load and the prompt length grow. Then you change one thing at a time and explain each change with the map of the efficient-inference survey.
This lab closes Track 5. It uses almost every session before it. Session 81 defined TTFT, TPOT and throughput. Session 84 showed the latency–throughput trade-off of one batch. Session 133 followed one request through a server. Session 134 built paged KV blocks and prefix caching. Session 135 added continuous batching and chunked prefill, session 136 the queue, and Session 138 the engine you will now run. Session 141 did capacity math on paper; here you check it against a running server.
The lab has four parts, one per key point of the session:
- Serve Qwen2.5-7B-Instruct with vLLM on an NVIDIA L4 (and, optionally, with TensorRT-LLM on the same GPU).
- Plot TTFT, TPOT and throughput against the request rate, and against the prompt length.
- Turn on prefix caching, change chunked prefill, switch to FP8 or INT4 weights, and turn on speculative decoding, one at a time.
- Explain each change with the survey’s taxonomy: which part of the system it touches and which bottleneck it removes.
The setup: one GPU, one model, every flag written down
A benchmark result is only useful if someone else can repeat it. So you write down the whole setup before the first run: the GPU, the model and its precision, the engine and its version, and every scheduler setting. Do not rely on defaults. Defaults change between versions, and a different default is the most common reason two teams get different numbers for “the same” benchmark.
The GPU. An NVIDIA L4: 24 GB of memory, 300 GB/s of memory bandwidth, 121 TFLOP/s of dense BF16 matrix math and 242 TFLOP/s of dense FP8 [Certain: NVIDIA lists 242 and 485 “with sparsity” and says dense is one half]. It is a cheap 72 W inference card, available in most clouds. Any GPU with at least 24 GB works; your numbers will scale with its bandwidth and FLOP/s.
The model. Qwen2.5-7B-Instruct, in BF16. It has an Apache-2.0 license, so no access request is needed. From its config.json [Certain]: \(L = 28\) layers, width \(d = 3{,}584\), 28 query heads and \(n_{kv} = 4\) key-value heads of size \(d_h = 128\) (grouped-query attention), feed-forward width 18,944, a vocabulary of 152,064, and an output layer that is not tied to the input embedding. Hugging Face counts 7,615,616,512 parameters.
- Read the symbols
- \(C_{\text{KV}}\)
- how many tokens of KV cache fit on the GPU, summed over all running requests
- \(u\)
- the fraction of GPU memory the engine may use (vLLM’s
--gpu-memory-utilization) - \(M\)
- GPU memory in bytes
- \(W\)
- bytes of the stored weights
- \(R\)
- memory kept for everything else: activations of the largest step, CUDA graphs, the sampler; here an assumed 1.5 GB
- \(\text{kv}_{\text{tok}}\)
- bytes of cache per token: a key and a value (the 2) for every layer and every KV head
- \(s\)
- bytes per number in the cache: 2 for BF16
Weights in BF16: \(7{,}615{,}616{,}512 \times 2 = 15.23\) GB. Cache per token: \(2 \cdot 28 \cdot 4 \cdot 128 \cdot 2 = 57{,}344\) bytes (56 KiB). With \(u = 0.9\) and \(R = 1.5\) GB: \(0.9 \cdot 24 - 15.23 - 1.5 = 4.87\) GB for the cache, so \(C_{\text{KV}} = 84{,}904\) tokens.
One benchmark request has 1,024 prompt tokens and 256 output tokens, 1,280 in total. So at most \(\lfloor 84{,}904 / 1{,}280 \rfloor = 66\) requests can be in the batch at once. This one number explains a lot of what follows: no scheduler setting can run more than about 66 of these requests together on this GPU in BF16. With FP8 weights (8.71 GB) the cache grows to 198,702 tokens, room for 155 requests.
vLLM prints its own version of this number at start-up, as the KV-cache size in tokens and the maximum concurrency for the configured model length [Likely: the exact wording of the log line changes between versions]. Compare it with your estimate. If they differ a lot, find out why before you benchmark.
The engine. Install vLLM in a fresh environment and record its version (pip show vllm). Start the server with every setting written out:
# the baseline server: BF16, chunked prefill with a 2,048-token budget, NO prefix caching
vllm serve Qwen/Qwen2.5-7B-Instruct \
--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 \
--max-num-seqs 128 --max-num-batched-tokens 2048 \
--enable-chunked-prefill --disable-prefix-caching \
--port 8000
--max-num-batched-tokens is the token budget of one engine iteration (the Sarathi-Serve “chunk size”); --max-num-seqs caps the number of requests in the batch. The baseline turns prefix caching off, because the random benchmark prompts share nothing and we want to add each feature ourselves. Recent vLLM versions spell the switch --disable-prefix-caching; older ones use --no-enable-prefix-caching [Certain for the current docs]. Check with vllm serve --help. When the log says the server is ready, send one request by hand to check that it answers.
What to measure, and how to generate the load
A streaming server returns tokens one chunk at a time. For each request \(i\) the client records three kinds of times: when the request was scheduled to be sent, when the first token arrived, and when every later chunk arrived. From these come the per-request metrics of session 81:
- Read the symbols
- \(t^{\text{sched}}_i\)
- the time the load generator planned to send request \(i\) (not the time it actually managed to send it)
- \(t^{\text{first}}_i, t^{\text{last}}_i\)
- arrival of the first and of the last output token
- \(O_i\)
- output tokens of request \(i\); TPOT averages over the \(O_i - 1\) gaps after the first token
- \(\text{ITL}_{i,j}\)
- inter-token latency: the gap between two chunks; its p99 shows the stalls that the TPOT average hides
- \(\tau_1, \tau_2\)
- the service-level objectives (SLOs) for TTFT and TPOT; goodput counts only the requests that meet both, per second
Report percentiles, not means. The median (p50) tells you what a typical user sees; p99 tells you what the unlucky one in a hundred sees, and it is usually what an SLO is written on. Goodput is the most honest single number: requests per second that were useful to their users. DistServe made it the main target for LLM serving [Likely].
Open loop, closed loop, and the scheduled send time
There are two ways to generate load. In a closed loop, \(N\) virtual users each send a request, wait for the answer, and send the next. In an open loop, requests arrive on a schedule that does not depend on the server, for example a Poisson process with rate \(\lambda\): the gaps between arrivals are random with mean \(1/\lambda\), like real users who do not know each other.
The two answer different questions. A closed loop can never overload the server: when it slows down, the users slow down with it, so the queue stays short and the latency looks fine. Real traffic does not slow down because your server is slow. So for capacity questions use an open loop and sweep \(\lambda\). The two are linked by Little’s law [Certain]:
- Read the symbols
- \(\bar N\)
- the average number of requests inside the server (waiting or running)
- \(\lambda\)
- the arrival rate in requests per second (in a steady state it equals the completion rate)
- \(\bar W\)
- the average time a request spends inside: queueing + TTFT + the decode time, so about \(\text{TTFT} + (O-1)\,\text{TPOT}\)
For example, if the server answers 1 request per second and each request takes about 20 s from arrival to last token, then about 20 requests are inside at any time. A closed loop with 20 users would produce about the same state. But a closed loop with 200 users at a slow server only shows you a long queue at a lower rate; it never shows you the rate at which the queue starts to grow without bound.
The last trap is subtle. If the load generator measures TTFT from the moment it actually sent the request, and it sends late because it was busy or blocked, the waiting time disappears from the numbers. This is called coordinated omission: the slow moments of the server delay the measurements that would have shown them. Always measure from the scheduled send time.
- input: rate \(\lambda\), number of requests \(n\), prompt and output lengths, seed
- \(t \leftarrow\) now; for \(i = 1, \dots, n\):
- \(t \leftarrow t + \text{Exp}(\lambda)\) // Poisson arrivals; \(t\) is the scheduled time
- wait until \(t\); start request \(i\) in its own thread or task, streaming, with
ignore_eos// never wait for the answer here - the task records \(t^{\text{sched}}_i = t\) and the arrival time of every chunk
- wait for all tasks; compute (142.2) and (142.3); report p50 and p99
- repeat for each \(\lambda\) in the sweep; discard nothing, but check that the run was long enough (several hundred requests)
ignore_eos makes the model generate exactly the requested number of tokens, even if it would stop earlier. Without it, the output lengths depend on what the model writes, and two configurations are no longer doing the same work.
vLLM ships a benchmark client that does all of this [Certain]. One run at 2 requests per second:
vllm bench serve --backend vllm --model Qwen/Qwen2.5-7B-Instruct --port 8000 \ --dataset-name random --random-input-len 1024 --random-output-len 256 \ --num-prompts 300 --request-rate 2 --ignore-eos --seed 1 \ --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,99 \ --goodput ttft:1000 tpot:100 --save-result
Its documented behavior [Certain]: --request-rate gives Poisson arrivals (inf sends everything at time 0); --burstiness below 1 makes the arrivals burstier; --max-concurrency caps the requests in flight, which turns the run into a partly closed loop; --goodput takes SLOs in milliseconds. We also give a 60-line client of our own below, so you can see every step and run it against any OpenAI-compatible server.
A model to compare against
Before you rent the GPU, predict the result. A prediction tells you what “normal” looks like, so you notice when a run is broken. Our simulator is the continuous-batching loop of session 135, with a cost model from session 82 for each iteration.
- Read the symbols
- \(j\)
- one request in this iteration: a decode (\(n_j = 1\)) or a piece of a prefill (\(n_j\) up to the token budget)
- \(p_j\)
- tokens of request \(j\) already in the cache
- \(N_{\text{body}}, N_{\text{out}}\)
- weights of the 28 blocks (6.53 billion) and of the output layer (0.545 billion); \(b\) is the number of positions that produce logits
- \(W_{\text{read}}\)
- weight bytes read once per iteration: 14.14 GB in BF16 (the embedding lookup reads only a few rows)
- \(\eta_c, \eta_m\)
- assumed fractions of peak FLOP/s and bandwidth that real kernels reach: 0.6 and 0.8
- \(t_{\text{oh}}\)
- assumed fixed cost of one iteration (scheduling, launches, sampling): 2 ms
The scheduler copies vLLM’s main rule Session 138: in every iteration, first one token for each running decode, then the rest of the token budget goes to unfinished prefills, then new requests are admitted first-come-first-served while the budget and the KV cache allow. Three simplifications make it smaller than vLLM: it reserves the whole KV cache of a request at admission (vLLM allocates blocks as it goes and preempts when memory runs out, so it packs a few more requests); it ignores sampling, detokenization and network time; and the efficiencies are constants, while real kernels are faster at some shapes than others. Treat its numbers as a sketch with the right shape, good to maybe 20 to 50% [Speculation].
One decode step, batch 1. It reads 14.14 GB of weights: at the spec bandwidth that is 47.1 ms, and with \(\eta_m = 0.8\) and the 2 ms overhead, 60.9 ms. So no setting can make TPOT much below about 50 to 60 ms for this model in BF16 on an L4.
One prefill of 1,024 tokens. \(F = 13.58\) TFLOP; at \(0.6 \times 121\) TFLOP/s that is 187 ms, plus 2 ms: 189 ms. Its bytes would take only 59 ms, so prefill is compute-bound. The lowest possible TTFT is about 0.19 s.
The full batch. At most \(B = 66\) requests fit (worked example above). With an average context of about 1,152 tokens, one decode step of 66 requests reads \(14.14 + 66 \cdot 1{,}153 \cdot 57{,}344 / 10^9 = 18.50\) GB and takes 79.1 ms. Each request needs one prefill and 256 output tokens, so a batch of 66 is replaced about every \(66\,t_{\text{pre}} + 256\, t_{\text{dec}}(66)\) seconds:
That is about \(2.02 \times 256 = 516\) output tokens per second. The full simulation gives 2.05 requests per second (525 tokens per second) when every request is sent at once. Above about 2 requests per second the queue must grow without limit, whatever the scheduler does.
Part 1: TTFT, TPOT and throughput against the load
Sweep the request rate: 0.25, 0.5, 1, 1.5, 2, 2.5, 3 and 4 requests per second, and inf, 300 requests each, 1,024 prompt tokens and 256 output tokens. A shell loop with the command above and --request-rate $r does it; the slow rates take longest (300 requests at 0.25 per second need 20 minutes), so the whole sweep takes about an hour. Restart nothing between runs, but send a few warm-up requests first. Then plot each metric against the rate. Figure 1 shows what the simulator predicts.
How to read Figure 1
Start at the default, 2 requests per second, with SLOs of 1 s and 100 ms. TTFT p50 is still 271 ms, close to the 189 ms of a lone prefill, but TPOT p50 has grown from 64 ms (at 0.25 requests per second) to 102 ms, because the batch is bigger and every iteration reads more cache and carries prefill chunks. Throughput is 464 tokens per second, yet goodput is only 0.75 requests per second: most requests miss the 100 ms TPOT SLO. Move the rate to 2.25 and then 2.5: the TTFT curves jump from below one second to 4 to 14 seconds. That is the knee: the batch is full (about 66 requests), new requests wait in a queue, and the queue grows for as long as the run lasts. Throughput stops growing at about 525 tokens per second. Now loosen the TPOT SLO to 120 ms: the highest rate that meets both SLOs at p99 moves from 1.5 to 2 requests per second.
What the curves say
- Below the knee, TTFT is set by the prefill, TPOT by the batch. TTFT stays near the cost of one prefill. TPOT grows slowly and steadily with the rate, because more requests share each iteration, which then reads more KV cache and carries more prefill work.
- At the knee, TTFT explodes and TPOT flattens. Once the KV cache is full, new requests wait outside. Their waiting time is part of TTFT, and it grows with the length of the run, so p99 TTFT above the knee is not a property of the server but of how long you ran the test. The running requests are not affected, so TPOT stops growing.
- Throughput saturates; goodput falls. Above the knee the server does the same amount of work per second. Goodput falls to zero, because almost every request now misses the TTFT SLO.
- The useful capacity is the rate where the SLO breaks, not where throughput peaks. With the default SLOs that is 1.5 requests per second in the simulator, 73% of the maximum.
Part 2: the effect of prompt length
Now fix a moderate rate and change the prompt: 256, 512, 1,024, 2,048, 4,096 and 8,192 tokens, still with 256 output tokens. Also run each length with --request-rate inf to find the maximum throughput. Longer prompts act through two channels: more prefill compute per request (TTFT), and fewer requests in the KV cache (capacity and throughput).
How to read Figure 2
At 0.5 requests per second, TTFT p50 grows from 92 ms at 256 tokens to 502 ms at 2,048 and 1.47 s at 4,096: prefill is compute-bound, so its time follows the prompt tokens (and the attention term, which grows with the square). At 8,192 tokens the server cannot keep up any more: it serves only about 0.29 requests per second, so the queue grows and TTFT is minutes. The maximum throughput falls from 1,163 tokens per second at 256 tokens to 525 at 1,024 and 74 at 8,192. Two causes add up: each request brings more prefill work, and only 10 requests of 8,448 tokens fit in the cache, against 165 requests of 512 tokens (256 in, 256 out). Switch to 1 request per second: 4,096 tokens now break the server too.
This is why long-context serving is a capacity problem first. The cache per token is fixed by the model; a longer context means fewer requests per GPU. Grouped-query attention, KV-cache quantization (FP8 cache: --kv-cache-dtype fp8), and prefix caching all attack this, and so does moving to a GPU with more memory.
Part 3: one change at a time
Now change the server, one setting per experiment, and re-run the same sweep. Keep everything else fixed: same seed, same number of requests, same lengths, same vLLM version. If two settings change at once, you cannot say which one caused the effect.
- run the baseline sweep (Part 1) and save the JSON results with the full command line
- for each change \(c\) (prefix caching, chunked prefill, FP8, INT4, speculation):
- write down the prediction: which bottleneck \(c\) attacks, which metric should move, at low load and at overload
- restart the server with only \(c\) changed; check the start-up log (KV-cache size, quantization, speculation settings)
- re-run the same sweep, with the same seed, on the workload that \(c\) needs (shared prefixes for prefix caching, real text for speculation)
- plot baseline and \(c\) together; compare with the prediction; if they disagree, find out why before the next change
Prefix caching
Turn it on with --enable-prefix-caching. Random prompts share nothing, so you need a workload with shared prefixes, as in a chat product with one long system prompt or a RAG system that sends the same documents many times. vLLM’s client has one [Certain]: --dataset-name prefix_repetition --prefix-repetition-prefix-len 768 --prefix-repetition-suffix-len 256 --prefix-repetition-num-prefixes 10 --prefix-repetition-output-len 256. Run the baseline (cache off) on the same workload, so that the comparison is fair. The engine hashes each full 16-token block together with all the blocks before it session 134; a new request whose first blocks match skips their prefill.
Prediction. A cached request computes only 256 of its 1,024 prompt tokens, so its prefill drops from 189 ms to about 49 ms. TTFT should fall about 2 to 4 times at low load. At overload the gain is much bigger: less prefill work per request means more requests per second, and the shared prefix is stored once, so more requests fit in the cache.
Chunked prefill and the token budget
The baseline already uses chunked prefill with a 2,048-token budget. Try budgets of 512 and 8,192, and try --disable-chunked-prefill [Certain: the flag is in the current docs; some vLLM versions may not support every combination]. Without chunking, a new prompt runs as a whole in its own iteration and every running decode waits for it. With a small budget, a prompt is split into small pieces that ride along with the decodes.
Prediction. The cost of a prefill does not go away; chunking only spreads it. So TTFT rises a little with a small budget, while the long gaps between tokens (ITL p99) shrink. Look at ITL p99, not at TPOT p50: the average hides the stalls.
FP8 and INT4 weights
For FP8, add --quantization fp8: vLLM quantizes the BF16 checkpoint to FP8 when it loads it, and on GPUs with FP8 tensor cores (Ada, such as the L4, and Hopper) it also runs the matrix multiplies in FP8 [Likely: support depends on the GPU and the vLLM version; check the log]. For INT4, serve the official AWQ checkpoint Qwen/Qwen2.5-7B-Instruct-AWQ; vLLM reads its quantization method from the config. Measure quality too: run a small accuracy check (a few hundred questions from a benchmark you trust) next to the speed test, because a faster wrong answer is not a win (session 112).
Prediction. At low load, decode is memory-bound, so TPOT follows the weight bytes: 14.14 GB in BF16, 7.62 GB in FP8, about 4.56 GB in INT4 (the embedding and the output layer stay in BF16). FP8 also doubles the math speed, so prefill gets faster too. INT4 AWQ keeps the math in BF16 and has to unpack the weights, so at large batches and in prefill it can be slower than BF16 [Likely]; our simulator models this with a low compute efficiency (0.35), which is an assumption. Both free memory for the KV cache, so the batch can grow: 155 requests in FP8 and 196 in INT4 instead of 66.
Speculative decoding
The simplest drafter needs no second model: n-gram (prompt-lookup) decoding looks for the last few generated tokens in the prompt and proposes the tokens that followed them there. The vLLM docs give this example [Certain]:
vllm serve Qwen/Qwen2.5-7B-Instruct ... \
--speculative-config '{"method": "ngram", "num_speculative_tokens": 4,
"prompt_lookup_min": 2, "prompt_lookup_max": 5}'
Do not benchmark it on random token prompts: the outputs will not repeat the prompt, so almost no draft is accepted. Use real text where the answer copies from the prompt, such as summarization, code editing or RAG, through one of the client’s file-based datasets. Read the acceptance rate from the server’s metrics; vLLM reports accepted and drafted tokens [Likely: the metric names change between versions].
With \(k\) draft tokens and a probability \(\alpha\) that each one is accepted (independently, for simplicity), one verification pass yields on average
- Read the symbols
- \(k\)
- draft tokens checked in one pass (here 4); the pass costs the compute of \(k + 1\) tokens per request
- \(\alpha\)
- acceptance probability of each draft token; 0 for random prompts, higher when the output copies the input
Prediction. With \(k = 4\) and \(\alpha = 0.7\), each pass gives \(2.77\) tokens for about the cost of one memory-bound step, so TPOT at low load should fall about 2.5 to 3 times. The cost is compute: \(k + 1\) tokens per request per step, and they also use the token budget that prefills need. With \(\alpha = 0\) you pay the compute and get nothing. The vLLM docs say the same in words: speculation aims to reduce inter-token latency “under medium-to-low QPS”, memory-bound workloads [Certain].
How to read Figure 3
Prefix caching at 1 request per second: TTFT p50 233 → 103 ms, and at 4 requests per second the server still keeps up, at 876 instead of 528 tokens per second. Chunk budget: at 1 request per second, a budget of 512 gives ITL p99 97 ms against 194 ms for 2,048 and 444 ms with chunking off, while TTFT p50 rises from 237 to 306 ms. In the simulator a small budget also raises the overload throughput (642 against 529 tokens per second), because more prefill work hides under the memory time of decode steps; a budget of 256 goes too far: TTFT p50 at 2 requests per second is 645 ms, and the overload throughput falls back to 598. FP8: TPOT p50 at 1 request per second falls from 76.9 to 39.2 ms, and the knee moves beyond 4 requests per second (TTFT p50 is still 150 ms there). INT4: TPOT is lowest at low load (26 ms at 0.5 requests per second), but at 2 requests per second TTFT p50 is 830 ms against 271 ms, and from 3 requests per second on TPOT is about twice that of BF16 (about 205 against 113 ms), because the slow BF16 math on unpacked weights now dominates. Speculation: at \(\alpha = 0.7\), TPOT p50 at 1 request per second falls from 76.9 to 26.5 ms; set \(\alpha = 0\) and every curve gets a bit worse. In our simulator speculation keeps helping at overload when \(\alpha \ge 0.3\), because the KV cache keeps the batch at about 66 requests, well below the point where the steps become compute-bound; a real server pays extra costs we do not model (drafting, the verification kernels, rejected tokens), so expect smaller gains [Likely].
Part 4: explain each change with the survey’s taxonomy
The efficient-inference survey (Zhou et al.) sorts every technique into three levels [Certain]: data-level (change the input or the output organization), model-level (change the model: structure, quantization, sparsity, distillation, dynamic inference) and system-level (change how the same model runs: the inference engine, the serving system, hardware). The system level splits again into the inference engine (graph and operator optimization, speculative decoding, offloading) and the serving system (memory management, continuous batching, scheduling, distributed systems). Placing a change on this map tells you what it can and cannot do. A serving-system change can never make one forward pass faster; a model-level change can, but it may change the outputs.
How to read Figure 4
Prefix caching and chunked prefill sit in the same column and the same branch (serving system), but on different leaves: one changes what is stored (memory management), the other what goes into one iteration (batching). FP8 and INT4 sit on the same model-level leaf, yet they behave differently in Figure 3, because FP8 also speeds up the math while INT4 only cuts bytes. Speculation is a system-level change in the inference engine: it keeps the model and the outputs (for exact verification) and changes how many tokens one pass produces.
| Change | Survey place | Bottleneck it attacks | Should move | Simulator, baseline → changed |
|---|---|---|---|---|
| Prefix caching | 6.2.1 memory management | repeated prefill of the same tokens; duplicate KV blocks | TTFT; capacity | TTFT p50 233 → 103 ms at 1 req/s; 528 → 876 tok/s at 4 req/s |
| Chunk budget 512 | 6.2.2 continuous batching | prefills that stall decodes | ITL p99 down, TTFT up a little | ITL p99 194 → 97 ms, TTFT p50 237 → 306 ms at 1 req/s |
| FP8 W8A8 | 5.2.1 quantization | weight bytes (decode) and FLOP/s (prefill) | TPOT, TTFT, capacity | TPOT p50 76.9 → 39.2 ms at 1 req/s; 529 → 900 tok/s at 4 req/s |
| INT4 AWQ | 5.2.1 quantization | weight bytes only | TPOT at low load | TPOT p50 68.1 → 26.4 ms at 0.5 req/s; 102 → 91 ms at 2 req/s, TTFT p50 271 → 830 ms |
| n-gram speculation, \(\alpha = 0.7\) | 6.1.2 speculative decoding | one token per weight read | TPOT at low load | TPOT p50 76.9 → 26.5 ms at 1 req/s |
The pattern to remember: serving-system changes (memory management, batching, scheduling) move the queue and the capacity, and show up mostly near and above the knee. Model-level and engine-level changes move the cost of one iteration, and show up at every load. A change that saves bytes helps most where decode is memory-bound (low load, small batch); a change that adds compute to save bytes (speculation, INT4 unpacking) loses some of its gain as the batch grows.
Optional: TensorRT-LLM on the same L4
TensorRT-LLM has its own OpenAI-compatible server, trtllm-serve (session 131). Its documented flags include --max_batch_size, --max_num_tokens (the token budget of one iteration) and --kv_cache_free_gpu_memory_fraction [Certain]:
trtllm-serve Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8001 \
--max_batch_size 128 --max_num_tokens 2048 --kv_cache_free_gpu_memory_fraction 0.9
# the same client, pointed at the OpenAI-compatible endpoint
vllm bench serve --backend openai --base-url http://localhost:8001 --endpoint /v1/completions \
--model Qwen/Qwen2.5-7B-Instruct --dataset-name random --random-input-len 1024 \
--random-output-len 256 --num-prompts 300 --request-rate 2 --ignore-eos --seed 1
openai is one of the client’s documented back-ends, and /v1/completions its default endpoint [Certain]. To compare engines fairly, match three things. The same work: identical prompts and output lengths, with ignore_eos on both servers (check that the other server honors it). The same memory: TensorRT-LLM’s fraction is of the memory left after the weights, vLLM’s fraction is of the whole GPU, so “0.9” is not the same setting; compare the KV-cache sizes that both servers print. The same scheduler limits: batch size and token budget. Then report the sweep, not one point: one engine may win at low load and the other at overload.
The code
Two short programs. The first is an open-loop load generator that speaks the OpenAI streaming protocol, so it works with vLLM, trtllm-serve, SGLang and others. It uses only the Python standard library. It sends token ids as the prompt (the completions API accepts a list of integers, so the prompt length is exact), uses one thread per request so that a slow answer never delays the next send, and measures from the scheduled time.
# bench.py: an open-loop load generator for any OpenAI-compatible server (vLLM, trtllm-serve, SGLang) # Standard library only. Usage: python3 bench.py URL MODEL RATE [N] [INPUT] [OUTPUT] [PREFIX] import json, math, random, sys, threading, time, urllib.request def one_request(url, model, prompt, out_len, t_sched, rec): body = json.dumps({"model": model, "prompt": prompt, "max_tokens": out_len, "temperature": 0, "ignore_eos": True, "stream": True}).encode() req = urllib.request.Request(url + "/v1/completions", body, {"Content-Type": "application/json"}) chunks = [] # arrival time of every streamed chunk with text with urllib.request.urlopen(req, timeout=600) as resp: for line in resp: # server-sent events: b"data: {...}\n" line = line.strip() if not line.startswith(b"data: ") or line == b"data: [DONE]": continue if json.loads(line[6:])["choices"][0].get("text"): chunks.append(time.perf_counter()) rec.append({"sched": t_sched, "chunks": chunks, "out": out_len}) def run(url, model, rate, n=200, inp=1024, out=256, prefix=0, nprefix=10, seed=0): rnd = random.Random(seed) prefixes = [[rnd.randrange(1000, 30000) for _ in range(prefix)] for _ in range(nprefix)] rec, threads, t0 = [], [], time.perf_counter() t_next = t0 for i in range(n): if rate != math.inf: t_next += rnd.expovariate(rate) # Poisson arrivals: open loop prompt = (prefixes[rnd.randrange(nprefix)] if prefix else []) + \ [rnd.randrange(1000, 30000) for _ in range(inp - prefix)] # token ids time.sleep(max(0.0, t_next - time.perf_counter())) th = threading.Thread(target=one_request, args=(url, model, prompt, out, t_next, rec)) th.start(); threads.append(th) for th in threads: th.join() return summarize(rec, t0) def pct(a, p): a = sorted(a); k = (len(a) - 1) * p / 100; lo, hi = math.floor(k), math.ceil(k) return a[lo] + (a[hi] - a[lo]) * (k - lo) def summarize(rec, t0): ttft = [r["chunks"][0] - r["sched"] for r in rec] # from the SCHEDULED send time tpot = [(r["chunks"][-1] - r["chunks"][0]) / (r["out"] - 1) for r in rec] itl = [b - a for r in rec for a, b in zip(r["chunks"], r["chunks"][1:])] dur = max(r["chunks"][-1] for r in rec) - t0 return {"ttft_p50_ms": 1e3 * pct(ttft, 50), "ttft_p99_ms": 1e3 * pct(ttft, 99), "tpot_p50_ms": 1e3 * pct(tpot, 50), "tpot_p99_ms": 1e3 * pct(tpot, 99), "itl_p99_ms": 1e3 * pct(itl, 99), "req_per_s": len(rec) / dur, "out_tok_per_s": sum(r["out"] for r in rec) / dur} if __name__ == "__main__": a = sys.argv[1:] url, model, rate = a[0], a[1], float(a[2]) args = [int(x) for x in a[3:]] print(json.dumps({k: round(v, 2) for k, v in run(url, model, rate, *args).items()}))
Test the client before you trust it. This fake server waits 50 ms plus 0.1 ms per prompt token, then streams one token every 20 ms:
# mock_server.py: a fake OpenAI-compatible streaming server to dry-run bench.py without a GPU import json, socket, sys, time from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer class H(BaseHTTPRequestHandler): def log_message(self, *a): pass def do_POST(self): self.connection.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) # no Nagle delay q = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) self.send_response(200); self.send_header("Content-Type", "text/event-stream"); self.end_headers() time.sleep(0.05 + 1e-4 * len(q["prompt"])) # fake prefill: 50 ms + 0.1 ms per token for _ in range(q["max_tokens"]): self.wfile.write(b'data: {"choices": [{"text": " x"}]}\n\n'); self.wfile.flush() time.sleep(0.02) # fake decode: 20 ms per token self.wfile.write(b"data: [DONE]\n\n") ThreadingHTTPServer(("127.0.0.1", int(sys.argv[1]) if len(sys.argv) > 1 else 8011), H).serve_forever()
Our dry run (python3 mock_server.py 8011, then python3 bench.py http://127.0.0.1:8011 mock 5 20 1024 32) printed TTFT p50 171 ms, TPOT p50 28 ms and 3.5 requests per second [Certain, our run]. Two lessons. TPOT was 28 ms, not 20: a direct check showed that time.sleep(0.02) itself took 28.6 ms on our busy laptop, so the client was right and the fake server was slow. Always check the measuring tool against a known load. And 20 requests sent at 5 per second came out as 3.5 per second: in a short run the time to finish the last requests is a large part of the total. Use several hundred requests per point.
The second program is the simulator behind Figures 1 to 3, the “expected result” of this lab. It is deterministic: the same seed gives the same numbers, here and in the figures.
# sim.py: a deterministic model of a continuous-batching server (Qwen2.5-7B, BF16 KV cache, one L4) import math def rng(seed): # mulberry32, the same generator as the page figures st = [seed & 0xFFFFFFFF] mul = lambda a, b: (a * b) & 0xFFFFFFFF def r(): st[0] = (st[0] + 0x6D2B79F5) & 0xFFFFFFFF; s = st[0] t = mul(s ^ (s >> 15), 1 | s) t = ((t + mul(t ^ (t >> 7), 61 | t)) & 0xFFFFFFFF) ^ t return (t ^ (t >> 14)) / 4294967296 return r V, D, L, H, KVH, DH = 152064, 3584, 28, 28, 4, 128 # Qwen2.5-7B config.json N_EMB = V * D; N_BODY = 7_615_616_512 - 2 * N_EMB # untied embedding and output layer KV_TOK = 2 * L * KVH * DH * 2 # 57,344 bytes per token BW, MEM, ETA_M, T_OH, RESERVE, UTIL = 300e9, 24e9, 0.8, 0.002, 1.5e9, 0.9 # L4 + assumptions FMT = {"bf16": (2.0, 121e12, 0.6), "fp8": (1.0, 242e12, 0.6), "int4": (0.53125, 121e12, 0.35)} def weights(f): # (bytes stored, bytes read per step) sb = FMT[f][0]; return sb * N_BODY + 4 * N_EMB, sb * N_BODY + 2 * N_EMB def kv_capacity(f): return int((UTIL * MEM - weights(f)[0] - RESERVE) // KV_TOK) def step_time(pieces, f): # pieces: (new tokens, past tokens, logits) per sequence _, peak, eta = FMT[f]; F = 0.0; Q = weights(f)[1] for n, p, nl in pieces: F += 2 * N_BODY * n + 2 * N_EMB * nl + 4 * L * H * DH * (n * p + n * (n + 1) / 2) Q += (p + n) * KV_TOK return max(F / (peak * eta), Q / (BW * ETA_M)) + T_OH def workload(n, rate, inp, out, seed=1, prefix=0, nprefix=10): r = rng(seed); t = 0.0; reqs = [] for _ in range(n): if rate != math.inf: t += -math.log(1 - r()) / rate # Poisson arrivals pid = int(r() * nprefix) if prefix else -1 reqs.append(dict(arr=t, P=inp, O=out, pid=pid, pre=prefix)) return reqs def simulate(reqs, fmt="bf16", chunked=True, budget=2048, max_seqs=128, prefix_cache=False, spec_k=0, alpha=0.0, seed=7, max_len=8192): r = rng(seed); cap = kv_capacity(fmt); waiting = list(range(len(reqs))); running = [] used = 0; resident = set(); clock = 0.0; finished = 0 st = [dict(done=0, gen=0, first=None, last=None, times=[], need=0) for _ in reqs] def admit(i): # reserve the whole KV of a request, FCFS nonlocal used q = reqs[i]; hit = prefix_cache and q["pid"] in resident need = q["P"] + q["O"] - (q["pre"] if hit else 0) if used + need > cap or len(running) >= max_seqs: return False used += need; st[i]["need"] = need; st[i]["done"] = (q["pre"] // 16) * 16 if hit else 0 if prefix_cache and q["pid"] >= 0: resident.add(q["pid"]) running.append(i); waiting.remove(i); return True while finished < len(reqs): if not running and reqs[waiting[0]]["arr"] > clock: clock = reqs[waiting[0]]["arr"] arrived = [i for i in waiting if reqs[i]["arr"] <= clock] pieces, plan = [], []; left = budget if chunked else max(budget, max_len) if chunked: for i in running: # 1. every decoding sequence: 1 token (k + 1 if speculating) if st[i]["done"] >= reqs[i]["P"]: pieces.append((1 + spec_k, reqs[i]["P"] + st[i]["gen"], 1 + spec_k)) plan.append((i, "d")); left -= 1 + spec_k for i in running: # 2. continue unfinished prefills q = reqs[i] if st[i]["done"] < q["P"] and left > 0: c = min(left, q["P"] - st[i]["done"]); pieces.append((c, st[i]["done"], 1)) plan.append((i, c)); left -= c for i in arrived: # 3. admit new requests while budget and memory allow if left <= 0 or not admit(i): break c = min(left, reqs[i]["P"] - st[i]["done"]); pieces.append((c, st[i]["done"], 1)) plan.append((i, c)); left -= c else: # prefill-first: whole prompts, decodes wait for i in arrived: q = reqs[i]; hit = prefix_cache and q["pid"] in resident if q["P"] - ((q["pre"] // 16) * 16 if hit else 0) > left or not admit(i): break c = q["P"] - st[i]["done"]; pieces.append((c, st[i]["done"], 1)); plan.append((i, c)); left -= c if not pieces: for i in running: pieces.append((1 + spec_k, reqs[i]["P"] + st[i]["gen"], 1 + spec_k)); plan.append((i, "d")) if not pieces: # idle: jump to the next arrival clock = min(reqs[i]["arr"] for i in waiting if reqs[i]["arr"] > clock); continue clock += step_time(pieces, fmt) for i, w in plan: s, q = st[i], reqs[i] if w == "d": a = 0 while a < spec_k and r() < alpha: a += 1 # accepted draft tokens k = min(a + 1, q["O"] - s["gen"]); s["gen"] += k; s["last"] = clock; s["times"].append(clock) else: s["done"] += w if s["done"] >= q["P"]: s["gen"] = 1; s["first"] = s["last"] = clock; s["times"].append(clock) if s["first"] is not None and s["gen"] >= q["O"]: running.remove(i); used -= s["need"]; finished += 1 return metrics(reqs, st) def pct(a, p): a = sorted(a); k = (len(a) - 1) * p / 100; lo, hi = math.floor(k), math.ceil(k) return a[lo] + (a[hi] - a[lo]) * (k - lo) def metrics(reqs, st): ttft = [s["first"] - q["arr"] for q, s in zip(reqs, st)] tpot = [(s["last"] - s["first"]) / (q["O"] - 1) for q, s in zip(reqs, st)] itl = [b - a for s in st for a, b in zip(s["times"], s["times"][1:])] dur = max(s["last"] for s in st) - min(q["arr"] for q in reqs) return dict(ttft50=pct(ttft, 50), ttft99=pct(ttft, 99), tpot50=pct(tpot, 50), tpot99=pct(tpot, 99), itl99=pct(itl, 99), thr=sum(q["O"] for q in reqs) / dur, rps=len(reqs) / dur) if __name__ == "__main__": print("KV capacity:", kv_capacity("bf16"), "tokens") for rate in [0.5, 1, 2, 3, 4]: m = simulate(workload(300, rate, 1024, 256)) print(f"{rate:4} req/s TTFT p50 {m['ttft50']:.3f} s p99 {m['ttft99']:.3f} s " f"TPOT p50 {1e3 * m['tpot50']:.1f} ms ITL p99 {1e3 * m['itl99']:.0f} ms {m['thr']:.0f} tok/s")
Its output:
KV capacity: 84904 tokens 0.5 req/s TTFT p50 0.224 s p99 0.596 s TPOT p50 68.1 ms ITL p99 191 ms 126 tok/s 1 req/s TTFT p50 0.237 s p99 0.639 s TPOT p50 76.9 ms ITL p99 194 ms 246 tok/s 2 req/s TTFT p50 0.271 s p99 0.947 s TPOT p50 102.4 ms ITL p99 377 ms 464 tok/s 3 req/s TTFT p50 13.139 s p99 31.828 s TPOT p50 112.0 ms ITL p99 377 ms 523 tok/s 4 req/s TTFT p50 26.180 s p99 54.831 s TPOT p50 113.3 ms ITL p99 377 ms 529 tok/s
When you have your measurements, fit the simulator to them: change \(\eta_m\) until TPOT at 0.25 requests per second matches, \(\eta_c\) until TTFT at low load matches, and \(R\) until the KV capacity matches vLLM’s log. If the fitted simulator then also predicts the knee, you understand your server. If it does not, the difference is the interesting part: preemption, scheduling details, or costs outside the GPU.
Where this is used
| Where | What you benchmark | What it decides |
|---|---|---|
| Capacity planning (session 141) | Goodput against rate under the product’s SLOs | Requests per GPU, number of replicas, cost per million tokens |
| Choosing a GPU or an engine | The same sweep on each candidate, same workload | Price-performance at the operating point, not at one peak number |
| Chat products with a long system prompt | Prefix caching on a shared-prefix workload | TTFT and capacity; how to order the prompt so the shared part comes first |
| Quantization roll-out | Speed sweep plus an accuracy check on the same model | Whether FP8 or INT4 is worth the quality risk |
| Video question answering and Video-RAG | Long prompts (thousands of visual tokens per clip), Part 2 sweep | Frames per request, visual-token compression, chunk budget; TTFT dominates |
| Video retrieval and key-moment detection with VLM captioning | Offline throughput at --request-rate inf | Cost of captioning a video archive; batch jobs care about throughput, not TTFT |
| Agents with many short calls | Short prompts with shared prefixes, high rate | Prefix caching, speculation on tool-output copying |
| Performance regression tests | A fixed small sweep after every engine upgrade | Catches a changed default or a slower kernel before users do |
Common mistakes
- Closed-loop load for a capacity question. A fixed number of users can never overload the server, so the knee never appears. Use Poisson arrivals and sweep the rate.
- Measuring from the actual send time. When the client is late, the waiting time vanishes from TTFT (coordinated omission). Measure from the scheduled time, and give each request its own thread or task.
- Relying on defaults. Prefix caching, chunked prefill, the token budget and CUDA graphs have different defaults in different versions. Write every flag, and record the version.
- Benchmarking prefix caching or speculation on random prompts. Random prompts share no prefix and give the n-gram drafter nothing to copy, so both features look useless, or the cache looks magic if the client reuses the same prompt by mistake.
- Letting output lengths vary between configurations. Without
ignore_eos, a quantized model that stops earlier looks faster. Fix the output length for speed tests; test quality separately. - Reporting the mean, or one rate. The mean hides the tail, and one rate hides the knee. Report p50 and p99 over a sweep, and goodput under a stated SLO.
- Runs that are too short or not warmed up. The first requests pay for CUDA graph capture and caches; a short run is dominated by its start and its end. Warm up, then use several hundred requests per point.
- Changing two things at once. FP8 also frees memory, which changes the batch; switching engines also changes defaults. One change per experiment, and check the start-up log each time.
Interview questions
What is the difference between open-loop and closed-loop load, and which one should a capacity benchmark use?basic
Why does p99 TTFT jump by orders of magnitude at a certain request rate, while TPOT barely changes?core
What is goodput, and why can it fall while throughput rises?basic
How do you estimate the maximum request rate of one GPU before running any benchmark?core
Chunked prefill: what does the token budget trade, and how would you choose it?core
FP8 and INT4 both cut weight bytes. Why can INT4 win at low load and lose at high load?advanced
Why does speculative decoding help at low load and help less, or hurt, at high load?advanced
You turned on prefix caching and TTFT dropped 10 times. What do you check before you believe it?core
How would you compare vLLM and TensorRT-LLM fairly on the same GPU?advanced
ignore_eos honored by both, the same client against both OpenAI-compatible endpoints, and matched scheduler limits (maximum batch, token budget). Match memory by the KV-cache size each engine reports, not by the memory flag, because vLLM’s fraction is of the whole GPU and TensorRT-LLM’s of the memory left after the weights. Warm up both. Run the full rate sweep and report p50 and p99 TTFT and TPOT, throughput and goodput under one SLO. Record the versions. Expect the winner to depend on the operating point.Your benchmark shows TTFT p99 of 0.4 s at 4 requests per second, but the server only completes 2 requests per second. What is wrong?core
What do you record so that someone else can reproduce your serving benchmark?basic
ignore_eos; warm-up; and the raw per-request results, not only the summary, so that other percentiles can be computed later.Exercises
- Use (142.1) to compute the KV-cache capacity of the L4 for the FP8 model (weights 8.71 GB) and the number of 1,280-token requests that fit. How many more is that than in BF16?
- In the simulator at 1.5 requests per second, TTFT p50 is 252 ms and TPOT p50 is 88.4 ms for 256 output tokens. Use Little’s law to estimate how many requests are inside the server. Does it fit in the KV cache?
- With \(k = 4\), compute the expected tokens per verification pass for \(\alpha = 0.5\). How much compute does a pass cost compared with a plain decode step, in tokens per request?
- Compute the lowest possible TPOT at batch 1 for the INT4 model (4.557 GB read per step), first at the spec bandwidth and then with the simulator’s assumptions (\(\eta_m = 0.8\), 2 ms overhead).
- A colleague benchmarks with
--max-concurrency 16and reports that the server handles 8 requests per second with TTFT p99 of 0.5 s. What did they really measure? How do you get the capacity instead? - Run the lab: the baseline sweep and at least two changes on a real GPU. Fit \(\eta_m\), \(\eta_c\) and \(R\) of the simulator to your baseline, and check whether the fitted simulator predicts your knee within 20%.
Answerscheck
Key takeaways
- Write down the whole setup and every flag; defaults change between versions. Compute the KV capacity first: for Qwen2.5-7B in BF16 on an L4 it is about 85,000 tokens, 66 requests of 1,280 tokens.
- Use open-loop load (Poisson arrivals), measure from the scheduled send time, fix output lengths with
ignore_eos, and run several hundred requests per point after a warm-up. - Sweep the rate. Below the knee, TTFT is one prefill and TPOT grows slowly; at the knee the batch is full, TTFT explodes and throughput saturates. Capacity is the rate where the SLO breaks, read from goodput.
- Longer prompts cost compute (TTFT) and cache capacity (fewer requests at once); long context is a capacity problem first.
- Change one thing at a time, on the workload it needs: shared prefixes for prefix caching, real text for speculation, an accuracy check for quantization.
- Serving-system changes (memory management, batching) move the queue and the capacity; model-level and engine-level changes (quantization, speculation) move the cost of each iteration. Byte-saving changes help most where decode is memory-bound.
Key formulas
References for this session
- vLLM project (docs). vLLM documentation
- NVIDIA (code). TensorRT-LLM
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
- Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
- Qwen Team (2024). Qwen2.5-7B-Instruct model card and config.json (Hugging Face)
- NVIDIA (2023). NVIDIA L4 Tensor Core GPU (product page and specifications)
- vLLM project (2026). vllm bench serve: command-line reference
- Saxena (2023). Prompt Lookup Decoding (code repository)
- Little (1961). A Proof for the Queuing Formula: L = λW