vLLM internals
vLLM is the open-source engine behind many LLM services. Inside it, one small loop does everything: a scheduler gives each request some tokens from a fixed budget, a block manager hands out 16-token KV blocks and keeps a prefix cache for free, and GPU workers run the batch with CUDA graphs and tensor parallelism. This session reads that loop from the source code of v0.32.0, with real defaults and the memory math that decides whether your server starts at all.
What you will learn
- The processes of a vLLM V1 server (API server, EngineCore, GPU workers) and what happens in one engine step
- How the V1 scheduler works with one token budget and no prefill or decode phase, and how chunked prefill, prefix caching and speculative decoding fit into it
- How the block manager allocates 16-token KV blocks, hashes full blocks, and turns its free queue into an LRU prefix cache
- Why V1 preempts by recompute and not by swap, with a cost model and the measurements of the PagedAttention paper
- How CUDA graphs are captured for a ladder of batch sizes, how batches are padded, and what piecewise mode does
- How tensor and pipeline parallelism split weights and the KV cache, and what the all-reduces cost
- What max_num_seqs, max_num_batched_tokens, max_model_len and gpu_memory_utilization really control, with their defaults and the start-up memory math
Why this matters
vLLM is the open-source LLM engine you are most likely to meet at work. Many companies run it in production, cloud providers offer it as a managed service, and other systems copy its ideas. When an interviewer asks “how would you serve this model?”, the expected answer often starts with vLLM, and the follow-up questions go inside it: what does the scheduler do, why did a request get preempted, why does the server refuse to start, what does gpu_memory_utilization really mean?
The last five sessions gave you the ideas one by one: the request path in session 133, paged KV memory and prefix caching in session 134, continuous batching and chunked prefill in session 135, preemption policies in session 136, and multi-GPU serving in session 137. This session shows how one real engine puts all of them together. You will see that the pieces fit into one small loop, and that most tuning knobs are limits on that loop.
We describe the V1 engine, which is the only engine in vLLM since version 0.11.0 (October 2025); the release notes say: “V1 is the only engine in the codebase now.” [Certain] Defaults and file names below are read from the source code of v0.32.0 (October 2026). vLLM releases every two weeks or so, and defaults do change (you will see two examples), so always check the version you run. [Certain]
The big picture: processes and one engine step
When you run vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4, vLLM starts several operating-system processes. The architecture docs list them: [Certain]
- API server (1 by default): speaks HTTP (the OpenAI-compatible API), applies the chat template, tokenizes, loads images, and streams the text back.
- EngineCore (1 per data-parallel rank): runs the scheduler and the KV cache manager in a busy loop, and sends work to the GPU workers. API servers and engine cores talk over ZMQ sockets (a fast message library).
- GPU workers (1 per GPU, so TP × PP per engine core): each loads its part of the weights, owns its part of the KV cache, and runs the forward pass through a model runner, which also captures the CUDA graphs.
So TP = 4 on one node gives 1 + 1 + 4 = 6 processes. The split is on purpose: tokenizing and HTTP work in Python are slow, and in a separate process they do not delay the GPU loop. [Certain for the layout; Likely for this reason as the main one]
One turn of the EngineCore loop is called a step (or iteration). Each step runs one forward pass of the model over a batch of tokens from many requests.
- input: waiting queue, running list, KV block pool, limits
max_num_seqsandmax_num_batched_tokens - Move newly arrived requests (sent by the API server over ZMQ) into the waiting queue
scheduler.schedule(): decide how many tokens each request runs now; allocate KV blocks // Algorithm 2- Send the scheduler output (request ids, token counts, new block ids) to all workers
- Each worker builds input ids, positions and block tables, picks a CUDA graph if the batch shape has one, and runs the forward pass // TP all-reduce inside
- Sample one token per request whose prompt is complete (or verify draft tokens if speculative decoding is on)
scheduler.update_from_output(): append tokens, check stop conditions, free blocks of finished requests- Send new tokens to the API server, which detokenizes and streams them
- output: one new token for each running request that finished its prompt // with async scheduling, step k+1 is scheduled while step k runs
Async scheduling is on by default in v0.32.0 unless an option is incompatible with it (for example pipeline parallelism with the older model runner, or some speculative methods). It lets the CPU prepare the next step while the GPU runs the current one, so the GPU does not wait for Python. [Certain]
vllm serve with tensor parallelism 4 on one node, and the path of one engine step through them. Names follow the vLLM source (v0.32.0).How to read Figure 1
Each box is a process or a part of one. Step through: the highlighted box is where the work happens, and the arrow shows what is sent. Note that only small messages (token ids, block ids) travel between processes; the weights and the KV cache never leave the GPUs. Step 6 shows the two all-reduces per layer that tensor parallelism needs.
The scheduler: one token budget, no phases
Older engines had a “prefill phase” and a “decode phase”. The V1 scheduler does not. A comment at the top of Scheduler.schedule() says: “There’s no ‘decoding phase’ nor ‘prefill phase’ in the scheduler. Each request just has the num_computed_tokens and num_tokens_with_spec.” [Certain] Each request \(i\) has a number of known tokens \(T_i\) (prompt plus generated tokens, plus draft tokens if speculation is on) and a number of tokens whose KV is already computed, \(C_i\). The scheduler’s only job is to give each request some tokens \(n_i\) so that \(C_i\) catches up with \(T_i\):
- Read the symbols
- \(T_i\)
- tokens request \(i\) knows now: prompt + generated (+ drafts)
- \(C_i\)
- tokens whose KV is already in the cache (computed before, or found in the prefix cache)
- \(n_i\)
- tokens request \(i\) runs in this step; for a decoding request it is 1
- \(B, B_{\text{left}}\)
- the token budget of one step, and what is left of it when request \(i\) is visited
This one rule covers many features. A new prompt of 10,000 tokens with a budget of 8,192 gets 8,192 now and the rest next step: that is chunked prefill. A request whose first 6,000 tokens are in the prefix cache starts with \(C_i = 6000\): that is prefix caching. A request with 3 draft tokens has \(T_i - C_i = 4\): that is speculative decoding. A decoding request has \(T_i - C_i = 1\). The forward pass does not care which is which; the attention kernel gets, for each request, its new tokens and its block table.
- input: running list (in admission order), waiting queue, token budget \(B\)
- for each request \(r\) in running, while budget is left: // decodes and unfinished prefills first
- \(n \leftarrow \min(T_r - C_r, B_{\text{left}})\)
- while
allocate_slots(r, n)fails: preempt the last request in running (free its blocks, set \(C = 0\), put it at the front of waiting); stop if \(r\) itself was preempted - schedule \(n\) tokens for \(r\); \(B_{\text{left}} \leftarrow B_{\text{left}} - n\)
- if nobody was preempted in this step: while waiting is not empty, budget is left and \(|\text{running}| \lt\)
max_num_seqs: - \(r \leftarrow\) head of waiting; \(C_r \leftarrow\) longest prefix-cache hit (full blocks, at most \(T_r - 1\) tokens)
- \(n \leftarrow \min(T_r - C_r, B_{\text{left}})\); if the blocks for \(n\) tokens are not free, stop
- move \(r\) to running; schedule \(n\) tokens
- output: \(\{r: n_r\}\), new block ids, ids of preempted requests
Three details matter in practice. First, running requests are served before waiting ones, so decodes are not starved by new prompts; the docs say the policy “batches all pending decode requests before scheduling any prefill operations”. [Certain] Second, when memory runs out, the victim is the most recently admitted request (self.running[-1]); with the priority policy it is the one with the worst priority. [Certain] Third, a prefix-cache hit can cover at most \(T - 1\) tokens, because the model must run at least the last prompt token to produce the logits of the next one. [Certain] The setting long_prefill_token_threshold (0, off, by default) caps the chunk of a single long prompt, so that a few long prompts cannot take the whole budget. [Certain]
The block manager: blocks, reference counts and the prefix cache
The KV cache manager (KVCacheManager, with a BlockPool inside) owns all KV blocks of the GPU. A block holds the keys and values of block_size tokens, 16 by default, for every layer. [Certain] The PagedAttention paper chose 16 because it is “large enough to efficiently utilize the GPU and small enough to avoid significant internal fragmentation in most workloads”. [Certain] The memory of one token is
- Read the symbols
- \(2\)
- one key and one value vector
- \(n_{\text{layers}}, n_{\text{kv}}, d_{\text{head}}\)
- layers, KV heads (fewer than query heads with GQA) and head size
- \(\text{TP}, \text{PP}\)
- tensor- and pipeline-parallel sizes: each GPU stores only its KV heads and its layers; if TP is larger than \(n_{\text{kv}}\), heads are copied
- \(b\)
block_size; a request needs enough blocks for all its computed tokens
For Llama 3.1 8B (32 layers, 8 KV heads of size 128) in BF16, \(s_{\text{tok}} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB), so one 16-token block is 2 MiB. For Llama 3.1 70B (80 layers, 8 KV heads) it is 327,680 bytes per token, and with TP = 4 each GPU stores 2 heads: 81,920 bytes per token per GPU.
Each block has a reference count (how many requests use it). When a block becomes full, it gets a hash that depends on the whole prefix before it:
Here \(t\) are token ids and “extra” holds anything else that changes the KV values: the LoRA adapter id, hashes of images in the block, and an optional cache_salt per user or tenant. The default hash is SHA-256 since v0.11. [Certain] Because \(h_k\) contains \(h_{k-1}\), two blocks with the same tokens but different histories get different hashes, so a hit on block \(k\) means the whole prefix up to block \(k\) is the same. Only full blocks are cached. [Certain] (The hash table and the matching rules are explained in session 134.)
The free block queue is a doubly linked list of blocks with reference count 0. Freed blocks that have a hash keep it and go to the tail; blocks without a hash go to the head. A new allocation takes the head; if that block still has a hash, its cache entry is evicted at that moment. A finished request frees its blocks in reverse order, so its last block (the least likely to be shared) is evicted first. This gives LRU eviction with no extra work. [Certain] So in vLLM, prefix caching costs almost nothing: the cache is simply the free memory that has not been reused yet. It is on by default. [Certain]
Preemption: recompute, and why not swap
A decoding request needs a new block every \(b\) tokens. If many requests grow at the same time, the pool can run out in the middle of a step. vLLM then preempts: it takes the victim’s blocks away and puts the victim back into the waiting queue. Later the request must get its KV back. There are two classic ways, both described in the PagedAttention paper: [Certain]
- Swap: copy the victim’s blocks to CPU memory, and copy them back later. The old V0 engine had this (
preemption_mode="swap"with a CPUswap_space). - Recompute: drop the blocks. When the request runs again, its prompt plus all tokens it generated so far are processed as one new prompt. The paper notes this is faster than the original work, because the generated tokens are now handled in one prefill pass, not one by one.
V1 does only recompute: _preempt_request frees the blocks, sets num_computed_tokens = 0 and puts the request at the front of the waiting queue. [Certain] The docs say “recomputation has lower overhead in the V1 architecture”. [Certain] A simple cost model shows why the choice is close, and what it depends on:
- Read the symbols
- \(n, P, d\)
- tokens of the victim, model parameters, hidden size; \(2Pn\) is the matmul work and \(2 n_{\text{layers}} d n^2\) the causal attention work of one prefill
- \(u F\)
- achieved fraction \(u\) (MFU) of the GPU’s peak BF16 rate \(F\) (989 TFLOP/s for an H100 SXM, dense)
- \(\eta_{\text{hit}}\)
- fraction of the victim’s tokens still found in the prefix cache when it comes back
- \(t_c, e\)
- fixed cost of one CPU–GPU copy, and bytes of K (or V) per token per layer on one GPU; there is one copy per block, per layer, for K and for V, each way
Recompute: \(2 \cdot 8.03\text{e}9 \cdot 2000 = 32.1\) TFLOP for the matmuls plus 1.05 TFLOP for attention (3% of the total). At \(u = 0.5\): \(33.2\text{e}12 / 494.5\text{e}12 = 67.1\) ms of GPU time.
Swap: the KV is \(2000 \cdot 128\) KiB \(= 262\) MB. With 16-token blocks there are \(\lceil 2000/16 \rceil = 125\) blocks, so \(2 \cdot 32 \cdot 125 = 8{,}000\) copies of 32 KiB each way. With our assumptions \(t_c = 5\ \mu s\) and 50 GB/s over PCIe 5.0: \(8000 \cdot (5 + 0.66)\ \mu s = 45.2\) ms out and the same back, 90.5 ms in total. With 128-token blocks: 21.0 ms. The values of \(t_c\) and the bandwidth are our assumptions. [Likely]
This matches the paper’s measurement: “swapping incurs excessive overhead with small block sizes”, because many small copies cannot use the PCIe bandwidth, while recompute does not depend on the block size; for blocks of 16 to 64 tokens “the two methods exhibit comparable end-to-end performance.” [Certain]
How to read Figure 2
The two curves are the cost of each method for every victim length; the vertical line is the length you chose. Recompute grows a bit faster than linearly (the attention term \(n^2\)); swap grows linearly, and its height depends strongly on the block size, because every block is one copy per layer for K and for V. Move the block size from 16 to 128: swap becomes about 4× cheaper, recompute does not change. Raise the prefix-cache hit: recompute falls in proportion. This is a model with assumed constants, not a measurement.
In V1 there are two more reasons for recompute. The victim’s full blocks keep their hashes and go to the tail of the free queue, so when the victim returns soon, much of its prefix may still be in the prefix cache (\(\eta_{\text{hit}} \gt 0\)), and recompute becomes nearly free. And recompute needs no CPU buffer, no copy streams and no second allocator. [Certain for the mechanism; Likely as the design reason] Figure 3 shows both effects in a small simulation of Algorithm 2.
Five requests: R0, R2 and R4 start with the same 8-token system prompt (two blocks); max_num_seqs = 4. Step 0: R0 runs its 10-token prompt and R1 gets only 2 of its 6 prompt tokens (chunked prefill: the budget is 12). Step 2: R2 arrives and finds its first 8 tokens in the prefix cache, so it computes only 4. Step 7 ends with 0 free blocks. Step 8: R1 needs a new block, so R3, the last admitted, is preempted and loses 9 computed tokens. Step 10: R3 comes back and finds 8 of its 10 tokens still cached; it computes only 2.
In total: 19 steps, 3 preemptions, 31 computed tokens thrown away, but only 82 tokens computed against 79 with no preemption at all, because the prefix cache returned most of the lost work. With prefix caching off: 2 preemptions, 9 tokens thrown away, and 104 tokens computed (95 is the minimum without caching). Our Python code below gives the same numbers.
How to read Figure 3
The grid is the block pool. A filled cell is a block in use, colored by its owner; a block shared by several requests has a thick border and shows how many use it (×2). A cell with only a colored outline is free but still holds a cached prefix block (it can be hit, or evicted when reused). The rows below show each request: state, tokens computed out of tokens known, and the tokens it runs in this step. Use Next and Back to step. Lower the pool to 10 blocks to see repeated preemptions; turn prefix caching off and compare “tokens computed” in the readout. A larger budget finishes prompts in fewer steps; max_num_seqs = 1 turns the engine into one-at-a-time serving.
Memory at start-up: what gpu_memory_utilization means
vLLM does not grow the KV cache on demand. At start-up each worker measures what it needs and gives the rest of its budget to the block pool. The code in gpu_worker.py does this: [Certain]
- Requested memory =
gpu_memory_utilization× total GPU memory. Note: total, not free. Another process on the same GPU is not subtracted for you. - Load the weights, then run a profile run: a dummy forward pass with
max_num_batched_tokenstokens, to measure the peak activation memory. - Estimate the memory of the CUDA graphs (on by default since v0.21.0, per the log message in the code).
- KV memory: take the requested memory and subtract the weights, the peak activations, other non-KV memory and the CUDA graph estimate. Divide by the bytes of one block.
- Read the symbols
- \(u_{\text{gpu}}\)
gpu_memory_utilization: 0.9 for a long time, 0.92 in v0.20.0 and later [Certain from the code of v0.15.0 and v0.20.0]- \(M_W\)
- weights on this GPU (about the total divided by TP × PP)
- \(M_{\text{act}}, M_{\text{graph}}\)
- peak activations of the profile run, and CUDA graph memory; we use 3 GB together, an assumption [Likely]
- concurrency
- how many requests of the full
max_model_lenfit at once; vLLM logs this number as “Maximum concurrency”
Start-up also checks that at least one request of max_model_len tokens fits. If not, it stops with “To serve at least one request with the model’s max seq len (…) … KV cache is needed, which is larger than the available KV cache memory”, and it prints an estimated maximum length that would fit. [Certain] max_model_len defaults to the model’s context length from its config (131,072 for Llama 3.1); -1 or auto asks vLLM to pick the largest length that fits. [Certain]
Llama 3.1 8B, one GPU, BF16. Requested \(0.92 \cdot 80 = 73.6\) GB; weights 16.06 GB (8,030,261,248 parameters × 2 bytes). \(M_{\text{KV}} = 73.6 - 16.06 - 3 = 54.54\) GB, so \(N_{\text{blocks}} = \lfloor 54.54\text{e}9 / (16 \cdot 131072) \rfloor = 26{,}006\) blocks, 416,096 tokens. Concurrency at 131,072 tokens: 3.17. Requests of 2,000 tokens (125 blocks each): 208 at once. With gpu_memory_utilization 0.90: 25,243 blocks.
Llama 3.1 70B, TP = 4. 70,553,706,496 parameters, 141.11 GB in BF16, 35.28 GB per GPU. \(M_{\text{KV}} = 73.6 - 35.28 - 3 = 35.32\) GB per GPU at 81,920 bytes per token: about 26,950 blocks, 431,000 tokens, concurrency 3.29 at 131,072.
Llama 3.1 70B, TP = 2. Weights 70.55 GB per GPU leave \(0.05\) GB. One 131,072-token request needs 21.5 GB per GPU, so vLLM refuses to start. Lowering max_model_len does not help here: there is almost no memory at all. You need TP = 4, FP8 weights, or bigger GPUs.
How to read Figure 4
The bar is the memory of one GPU. The readout gives the KV pool in GB, blocks and tokens, the concurrency vLLM would log, and whether start-up succeeds. Pick 70B on an H100 with TP 1 or 2 to see the error; TP 4 works. Switch the KV cache to FP8: twice as many tokens fit in the same memory. Move the utilization: every 0.01 on an 80 GB GPU is 0.8 GB, about 6,100 tokens of 8B KV in BF16.
CUDA graphs for decode
A decode step of a small model is made of hundreds of short kernels, and the CPU can be too slow to launch them; the GPU then waits. A CUDA graph records the kernel launches of one step once and replays them with one call (session 127). A graph is recorded for fixed tensor shapes, so vLLM captures one graph per batch size, from a list of capture sizes, at start-up. A batch of \(n\) tokens is padded up to the next captured size:
This is the default rule in v0.32.0; the cap is 1,024 instead of 512 on data-center Blackwell GPUs. [Certain] With max_num_seqs = 256 this gives 51 graphs up to 512 tokens. A decode batch of 37 requests runs the graph for 40 (7.5% padding), 257 runs as 272 (5.5%). Averaged over batch sizes 1 to 512, padding wastes 3.3% of the rows. A batch larger than \(c_{\max}\) runs without a graph.
Not every batch can be one full graph. Attention for a batch that mixes prefills of different lengths with decodes has shapes that change every step. So the default mode is FULL_AND_PIECEWISE: the code says it captures “full cudagraph for decode batches and piecewise cudagraph for prefill and mixed prefill-decode batches”. A piecewise graph covers everything between the attention calls (the matmuls, norms, activations) while attention runs outside the graph. [Certain] The optimization level controls this: -O1 uses piecewise graphs only, -O2 (the default) uses FULL_AND_PIECEWISE, and --enforce-eager skips both compilation and graphs for the fastest start-up. [Certain]
max_num_batched_tokens = 8,192, and the padding of one decode batch. The lower panel shows the padding waste for every batch size up to the largest graph.How to read Figure 5
Each tick on the top line is one captured graph. The marker is your batch, the arrow goes to the graph that runs it, and the readout gives the waste. The lower curve shows waste for all batch sizes: it jumps right after each captured size and falls to zero at the next one. Below 8 the steps are 1, 2, 4; from 8 to 248 the step is 8; above 256 it is 16. Set max_num_seqs to 1,024 (the H100 server default): the cap stays at 512, so decode batches of more than 512 requests run without CUDA graphs.
Tensor and pipeline parallelism inside vLLM
With --tensor-parallel-size N each layer is split over N GPUs in the Megatron style (session 48): the QKV and first MLP matmuls are split by columns, the output and second MLP matmuls by rows, so each layer needs two all-reduces of the activations. KV heads are split too, so each GPU keeps only its own heads’ KV blocks, and all GPUs use the same block ids: one scheduler and one block table serve all of them. [Certain] The cost per step of the all-reduces, with a ring algorithm, is roughly
One all-reduce moves \(256 \cdot 8192 \cdot 2 = 4.19\) MB. At an assumed 450 GB/s per direction over NVLink: \(2 \cdot \tfrac34 \cdot 4.19\text{e}6 / 450\text{e}9 = 14.0\ \mu s\), times 160 all-reduces = 2.24 ms per step, plus a latency of a few microseconds each. [Likely for the bandwidth] Reading the 35.3 GB of weights on each GPU at 3.35 TB/s takes at least 10.5 ms. So communication adds roughly 20% here; across nodes, over slower links, it would dominate, and that is why TP is used inside a node.
Pipeline parallelism (--pipeline-parallel-size) puts groups of layers on different GPUs or nodes. One batch would leave all but one stage idle, so the EngineCore keeps a batch queue of PP batches in flight: it schedules a new batch while earlier ones are still moving through the pipeline. The code comment says “PP requires PP-size concurrent batches to fill the pipeline.” [Certain] The docs suggest TP within a node first, then PP across nodes. Data parallelism (--data-parallel-size) runs separate engine cores, each with its own scheduler and KV cache; for MoE models, expert parallelism shares the experts across them. [Certain] Prefill–decode disaggregation, from session 137, is done in vLLM with KV connectors that move blocks between instances. [Likely for the details]
Prefix caching, chunked prefill and speculative decoding inside one loop
These three features were whole sessions earlier. In vLLM V1 each is a small change to \(C_i\), \(T_i\) or the budget of formula (138.1), plus some code in the worker:
| Feature | What changes in the scheduler | What changes in the worker | Default in v0.32.0 |
|---|---|---|---|
| Prefix caching | at admission, \(C_i\) jumps to the length of the longest cached chain of full blocks (at most \(T_i - 1\)) | nothing: the block table simply points at shared blocks | on, SHA-256 hashes [Certain] |
| Chunked prefill | \(n_i\) is cut by the budget left; decodes are scheduled first | attention handles “some new tokens + cached prefix” for every request | on whenever the model supports it [Certain] |
| Speculative decoding | \(T_i\) includes \(k\) draft tokens, so a decoding request asks for \(k + 1\) tokens of budget | a drafter proposes tokens; the rejection sampler accepts a prefix and rejected slots are rolled back | off; methods include ngram, eagle, eagle3, medusa, mtp, draft_model, suffix [Certain] |
| Multimodal inputs | image or video tokens count in \(T_i\); the encoder output is cached in an encoder cache whose size defaults to max_num_batched_tokens | the vision encoder runs before the language model for new images | on for VLMs [Certain] |
Speculation uses budget and KV blocks for tokens that may be rejected. With acceptance rate \(\alpha\) and \(k\) drafts, a step gives on average \((1 - \alpha^{k+1})/(1 - \alpha)\) tokens (session 128): 2.53 for \(\alpha = 0.7\), \(k = 3\), while it uses 4 tokens of budget. At low load the budget is free and speculation wins; at high load those 4 slots could have been 4 other requests, so the gain shrinks. [Likely]
The key settings and what they trade
Most of the time you tune vLLM with a few limits on the loop. The defaults for max_num_seqs and max_num_batched_tokens depend on the GPU and on how you start vLLM (the LLM class for offline jobs, or the OpenAI server). From arg_utils.py in v0.32.0: [Certain]
| GPU memory | max_num_seqs | max_num_batched_tokens (server) | max_num_batched_tokens (LLM class) |
|---|---|---|---|
| under 70 GB, or any A100 | 256 | 2,048 | 8,192 |
| 70 GB or more (H100, H200) | 1,024 | 8,192 | 16,384 |
| 160 GB or more (B200, B300) | 1,024 | 16,384 | 16,384 |
The code notes that a large token budget on an A100 reduced throughput, hence the extra check. --performance-mode throughput doubles both defaults. [Certain]
| Setting | What it limits | Raise it when | Lower it when |
|---|---|---|---|
max_num_seqs | requests in the running list; also sizes buffers and CUDA graphs | the KV pool is half empty and the GPU is not saturated | preemptions are frequent, or per-token latency is too high |
max_num_batched_tokens | tokens in one step (prefill chunk + decodes) | TTFT is too high and throughput matters; docs suggest more than 8,192 for small models on big GPUs | long prompts cause latency spikes for decoding users (docs: 2,048 gives better ITL) |
max_model_len | prompt + output length per request | users need longer context | start-up fails, or you never need 128k tokens |
gpu_memory_utilization | share of total GPU memory this instance may use | the GPU is dedicated and you want more KV | another process shares the GPU, or you hit OOM |
block_size | tokens per KV block (16) | rarely; some attention backends need other sizes | rarely |
kv_cache_dtype | bytes per KV value | — | use fp8 to double capacity, after an accuracy check |
enforce_eager, -O | compilation and CUDA graphs | -O2 (default) in production | --enforce-eager for debugging and fast start-up |
tensor_parallel_size, pipeline_parallel_size | GPUs per replica | the model or its KV does not fit | the model fits: more replicas (data parallel) give more throughput per GPU |
- input: model, GPU, traffic (prompt and output lengths, arrival rate), SLOs for TTFT and time per output token
- Set
max_model_lento the longest request you accept, not the model maximum - Start the server; read “KV cache size: … tokens” and “Maximum concurrency” in the log // formula (138.5)
- Set
max_num_seqsnear KV tokens ÷ (average prompt + output length) and the GPU’s decode sweet spot - Replay real traffic; watch preemptions, KV usage, TTFT and inter-token latency
- if preemptions are frequent: lower
max_num_seqs, use an FP8 KV cache, or add GPUs - if ITL spikes when long prompts arrive: lower
max_num_batched_tokensor setlong_prefill_token_threshold; if TTFT is too high: raise it - output: settings that meet the SLOs at the highest request rate (goodput)
PyTorch code
The program has four parts. (1) The toy V1 scheduler of Figure 3: token budget, running first, recompute preemption of the last request, a free queue with LRU eviction, and prefix-cache hashes of full blocks. (2) A paged KV cache in PyTorch: the worker writes K and V through a block table and reads them back by gathering blocks; attention is unchanged. (3) The CUDA graph capture sizes of formula (138.6). (4) The start-up memory plan of formula (138.5). It runs on CPU and prints the same every time.
import math, torch BS = 4 # tokens per block (vLLM's default is 16) SYS = list(range(100, 108)) # an 8-token system prompt shared by R0, R2, R4 REQS = [(0, SYS + [1, 2], 14), (0, [3, 4, 5, 6, 7, 8], 18), (2, SYS + [9, 10, 11, 12], 8), (3, [13, 14, 15, 16, 17], 12), (4, SYS + [18], 6)] # (arrival step, prompt ids, output length) def simulate(num_blocks, budget, max_seqs=4, caching=True, log=None): free = list(range(num_blocks)) # free queue: the head is reused (and evicted) first bhash, cached, ref = {}, {}, [0] * num_blocks # block -> hash, hash -> block, reference counts R = [dict(id=i, arr=a, toks=list(p), out=o, gen=0, comp=0, blocks=[], nh=0, h=0) for i, (a, p, o) in enumerate(REQS)] waiting, running, done = [], [], 0 st = dict(steps=0, preemptions=0, discarded=0, hits=0, computed=0) def bh(prev, r, k): return hash((prev, tuple(r['toks'][k*BS:(k+1)*BS]))) def take(): b = free.pop(0) if b in bhash: del cached[bhash.pop(b)] # reusing a cached free block evicts its prefix entry ref[b] = 1; return b def grow(r, n): # allocate_slots: blocks for comp + n tokens, or fail need = math.ceil((r['comp'] + n) / BS) - len(r['blocks']) if need > len(free): return False r['blocks'] += [take() for _ in range(need)]; return True def release(r): # free in reverse order: tail blocks are evicted first for b in reversed(r['blocks']): ref[b] -= 1 if ref[b] == 0: if b in bhash: free.append(b) # cached: keep it, put it at the tail (LRU) else: free.insert(0, b) # not cached: reuse it first r['blocks'], r['nh'], r['h'] = [], 0, 0 def register(r): # every newly full block gets a hash while caching and r['nh'] < r['comp'] // BS: h = bh(r['h'], r, r['nh']); b = r['blocks'][r['nh']] if h not in cached and b not in bhash: cached[h] = b; bhash[b] = h r['h'] = h; r['nh'] += 1 step = 0 while done < len(R): waiting += [r for r in R if r['arr'] == step] tb, sched, pre = budget, {}, [] i = 0 while i < len(running) and tb > 0: # 1) RUNNING first: decodes and unfinished prefills r = running[i]; n = min(len(r['toks']) - r['comp'], tb); ok = True while not grow(r, n): # out of blocks: preempt the LAST running request v = running.pop(); st['discarded'] += v['comp']; release(v); v['comp'] = 0 waiting.insert(0, v); pre.append(v['id']); st['preemptions'] += 1; sched.pop(v['id'], None) if v is r: ok = False; break if not ok: break sched[r['id']] = n; tb -= n; i += 1 while not pre and waiting and tb > 0 and len(running) < max_seqs: # 2) WAITING, if nobody was preempted r = waiting[0]; hitb, h = [], 0 if caching: # longest cached chain of full blocks, at most len - 1 tokens for k in range((len(r['toks']) - 1) // BS): hk = bh(h, r, k) if hk not in cached: break h = hk; hitb.append(cached[hk]) hit = len(hitb) * BS; n = min(len(r['toks']) - hit, tb) if math.ceil((hit + n) / BS) - len(hitb) > len(free) - sum(ref[b] == 0 for b in hitb): break for b in hitb: # "touch": a cached block leaves the free queue if ref[b] == 0: free.remove(b) ref[b] += 1 r['blocks'], r['comp'], r['nh'], r['h'] = list(hitb), hit, len(hitb), h st['hits'] += hit; grow(r, n); waiting.pop(0); running.append(r); sched[r['id']] = n; tb -= n for r in list(running): # 3) execute, then sample one token where the prompt is done if r['id'] not in sched: continue r['comp'] += sched[r['id']]; st['computed'] += sched[r['id']]; register(r) if r['comp'] == len(r['toks']): r['gen'] += 1; r['toks'].append(1000 * (r['id'] + 1) + r['gen']) if r['gen'] == r['out']: release(r); running.remove(r); done += 1 if log is not None: log.append((step, dict(sched), list(pre), len(free))) step += 1 st['steps'] = step return st for nb, tb, c in [(16, 12, True), (14, 12, True), (14, 12, False), (10, 12, True), (40, 32, True)]: print(f"blocks={nb:2d} budget={tb:2d} prefix_cache={c!s:5}:", simulate(nb, tb, caching=c)) # 2. What a worker does with a block table: gather paged K/V and check attention is unchanged torch.manual_seed(0) L_tok, H, D, NB = 37, 2, 8, 32 # tokens of one request, heads, head dim, blocks in the pool k_pool = torch.zeros(NB, BS, H, D); v_pool = torch.zeros(NB, BS, H, D) K, V, q = torch.randn(L_tok, H, D), torch.randn(L_tok, H, D), torch.randn(H, D) table = torch.randperm(NB)[:math.ceil(L_tok / BS)] # the request's blocks: anywhere in the pool for t in range(L_tok): # write: slot = table[t // BS] * BS + t % BS k_pool[table[t // BS], t % BS] = K[t]; v_pool[table[t // BS], t % BS] = V[t] Kp = k_pool[table].reshape(-1, H, D)[:L_tok]; Vp = v_pool[table].reshape(-1, H, D)[:L_tok] att = lambda K, V: torch.einsum('ht,thd->hd', torch.softmax(torch.einsum('hd,thd->ht', q, K) / D**0.5, -1), V) print("block table:", table.tolist(), " max |paged - contiguous|:", (att(Kp, Vp) - att(K, V)).abs().max().item()) # 3. CUDA graph capture sizes (vLLM's default rule) and the padding of a decode batch def capture_sizes(max_num_seqs, max_num_batched_tokens=8192, blackwell=False): top = min(max_num_seqs * 2, 1024 if blackwell else 512, max_num_batched_tokens) s = [1, 2, 4] + list(range(8, 256, 8)) + list(range(256, top + 1, 16)) return sorted(set([x for x in s if x <= top] + ([max_num_seqs] if max_num_seqs <= top else []))) S = capture_sizes(256) pad = lambda n: next((x for x in S if x >= n), None) print("graphs:", len(S), "largest:", S[-1], " batch 37 ->", pad(37), " 257 ->", pad(257), " 600 ->", pad(600)) # 4. The KV cache budget at start-up: util * total - weights - other, in blocks of 16 tokens def kv_plan(total_gb, util, weights_gb, other_gb, kv_bytes_per_token, max_model_len, block=16): kv = util * total_gb - weights_gb - other_gb blocks = int(kv * 1e9 // (kv_bytes_per_token * block)) need = max_model_len * kv_bytes_per_token / 1e9 return round(kv, 2), blocks, round(blocks * block / max_model_len, 2), kv >= need print("8B, 1 x H100 :", kv_plan(80, 0.92, 16.06, 3.0, 131072, 131072)) print("70B, TP=4 :", kv_plan(80, 0.92, 141.11 / 4, 3.0, 81920, 131072)) print("70B, TP=2 :", kv_plan(80, 0.92, 141.11 / 2, 3.0, 163840, 131072))
Output with PyTorch 2.8 on CPU:
blocks=16 budget=12 prefix_cache=True : {'steps': 19, 'preemptions': 0, 'discarded': 0, 'hits': 16, 'computed': 79}
blocks=14 budget=12 prefix_cache=True : {'steps': 19, 'preemptions': 3, 'discarded': 31, 'hits': 44, 'computed': 82}
blocks=14 budget=12 prefix_cache=False: {'steps': 20, 'preemptions': 2, 'discarded': 9, 'hits': 0, 'computed': 104}
blocks=10 budget=12 prefix_cache=True : {'steps': 28, 'preemptions': 4, 'discarded': 46, 'hits': 44, 'computed': 97}
blocks=40 budget=32 prefix_cache=True : {'steps': 18, 'preemptions': 0, 'discarded': 0, 'hits': 16, 'computed': 79}
block table: [28, 13, 31, 18, 29, 16, 14, 11, 8, 4] max |paged - contiguous|: 0.0
graphs: 51 largest: 512 batch 37 -> 40 257 -> 272 600 -> None
8B, 1 x H100 : (54.54, 26006, 3.17, True)
70B, TP=4 : (35.32, 26948, 3.29, True)
70B, TP=2 : (0.05, 17, 0.0, False)
Read the first block of lines. With 16 blocks no request is preempted. With 14 blocks there are 3 preemptions and 31 tokens are thrown away, but only 3 extra tokens are computed (82 against 79), because the prefix cache still holds most of the victims’ blocks. With caching off, 9 lost tokens cost 9 extra tokens of compute (104 against 95). With 10 blocks the queue thrashes: 28 steps instead of 19.
The real thing, on a GPU (not run here; check flag names against your version):
# server: 8B on one H100, 32k context, FP8 KV cache, more prefill per step vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768 \ --gpu-memory-utilization 0.92 --max-num-seqs 512 --max-num-batched-tokens 8192 \ --kv-cache-dtype fp8 --enable-prefix-caching # offline: 70B over 4 GPUs with tensor parallelism from vllm import LLM, SamplingParams llm = LLM(model="meta-llama/Llama-3.1-70B-Instruct", tensor_parallel_size=4, max_model_len=16384, gpu_memory_utilization=0.92) outs = llm.generate(["The capital of France is"], SamplingParams(temperature=0.0, max_tokens=16)) print(outs[0].outputs[0].text)
Where this is used
| Application | What matters inside vLLM | Settings to look at |
|---|---|---|
| Chat assistants and APIs | decodes first, chunked prefill so long prompts do not stall streams | max_num_batched_tokens, max_num_seqs, CUDA graphs on |
| RAG and agents with long shared system prompts | prefix-cache hits turn repeated prefill into a lookup | prefix caching on, stable prompt order, cache_salt per tenant |
| Offline batch jobs (labeling, synthetic data, evaluation) | throughput: large budget, many sequences, preemption is acceptable | LLM class defaults, --performance-mode throughput |
| VLMs: image question answering, documents | image tokens count in the budget; encoder cache; image hashes in block hashes | max_num_batched_tokens \(\ge\) tokens of one image, limit_mm_per_prompt |
| Video-RAG, video QA and key-moment detection | thousands of frame tokens per request make prefill heavy and KV large | max_model_len, frames per request, FP8 KV cache, TP |
| Video retrieval and classification with an LLM re-ranker | many short requests sharing an instruction prefix | prefix caching, high max_num_seqs |
| RL training (rollouts for GRPO or PPO) | huge batches of samples from the same prompt; weights are updated between rounds | prefix caching (shared prompt), throughput settings; see session 75 |
Common mistakes
- Leaving
max_model_lenat 131,072 “to be safe”. Start-up then needs room for one full request, concurrency logs look tiny, and on small GPUs vLLM refuses to start. Set the length you really accept. - Two vLLM instances on one GPU with the default utilization. Each asks for 92% of total memory, not of free memory; the second fails. Give each a share, for example 0.45.
- Confusing
max_num_seqswithmax_num_batched_tokens. The first limits requests in the batch; the second limits tokens in one step. 256 decoding requests use only 256 tokens of budget. - Ignoring preemption warnings. Frequent preemptions mean the pool is too small for the admission limit; latency becomes unstable. Lower
max_num_seqsor add KV memory. - Benchmarking with
--enforce-eager. It is meant for debugging; decode of small models can be much slower without CUDA graphs. - Using TP across nodes. Two all-reduces per layer over a slow network dominate the step; use PP across nodes and TP inside a node.
- Copying settings from an old blog post. V0 options such as
swap_spaceandpreemption_mode, and the old defaults (utilization 0.9, one fixedmax_num_seqs), are gone or changed. - Turning on speculative decoding for a high-load batch job. Draft tokens take budget and KV blocks; at high load the gain is small or negative. Measure at your real load.
Interview questions
Walk me through one step of the vLLM V1 engine.basic
What is the difference between max_num_seqs and max_num_batched_tokens?basic
max_num_seqs caps the number of requests in the running list at once; it bounds KV demand and sizes buffers and CUDA graphs. max_num_batched_tokens caps the tokens processed in one forward pass: all decode tokens plus prefill chunks. 300 decoding requests need 300 tokens of budget; a single 10,000-token prompt needs 10,000 tokens and is chunked if the budget is smaller. Defaults in v0.32.0 depend on the GPU: on an H100 server, 1,024 sequences and 8,192 tokens; on smaller GPUs or A100, 256 and 2,048.Why does the V1 scheduler have no prefill and decode phases?core
What happens when the KV cache is full in the middle of a step?core
allocate_slots fails for the request being scheduled. The scheduler preempts the last request in the running list (the most recently admitted under FCFS, the lowest priority under the priority policy): it frees its blocks, sets its computed tokens to 0 and puts it at the front of the waiting queue. It repeats until the allocation succeeds or the request itself was the victim. No new requests are admitted in a step with preemptions. When the victim returns, its prompt and generated tokens are prefilled again; full blocks that were not yet reused are found in the prefix cache, so part of the work is often free.Recompute or swap for preemption: what decides it, and why did vLLM V1 keep only recompute?advanced
How does vLLM decide how many KV blocks to allocate, and what does gpu_memory_utilization mean exactly?core
Why does vLLM capture CUDA graphs only for some batch sizes, and what is piecewise mode?core
Llama 3.1 70B on two 80 GB GPUs fails to start. Explain and fix.core
How do tensor parallelism and the KV cache interact in vLLM?advanced
How do prefix caching and preemption interact?advanced
Users complain that streams freeze for a moment when someone sends a very long document. What do you change?core
Exercises
- Compute the KV bytes per token per GPU for Llama 3.1 70B with TP = 8 in BF16, and with TP = 16.
- Llama 3.1 8B on an L40S (48 GB nominal) at utilization 0.92 with 3 GB for activations: how many 16-token blocks, and what is the concurrency at
max_model_len32,768? - With
max_num_seqs= 64, list the largest CUDA graph and the number of graphs. Which graph runs a decode batch of 50? - A server has a budget of 2,048 tokens and 300 decoding requests. A 5,000-token prompt arrives. How many steps until its prefill is done, if nothing else changes?
- Use (138.4) to find the swap time for the 2,000-token 8B request with 64-token blocks, keeping \(t_c = 5\ \mu s\) and 50 GB/s.
- Two vLLM servers must share one 80 GB GPU. Which utilization would you give each, and what goes wrong with the defaults?
Answerscheck
Key takeaways
- vLLM V1 runs an API server, one EngineCore (scheduler + KV cache manager) per data-parallel rank, and one worker per GPU; only small messages cross processes.
- The scheduler has no phases: each request gets \(\min(T - C, \text{budget left})\) tokens (138.1). Chunked prefill, prefix caching and speculation are changes to \(T\), \(C\) and the budget.
- KV blocks are 16 tokens; full blocks get chained hashes (138.3), and the free queue doubles as an LRU prefix cache, on by default.
- When blocks run out, the last admitted request is preempted and later recomputed; V1 has no swap. Prefix caching often makes recompute cheap.
- At start-up, \(M_{\text{KV}} = u_{\text{gpu}} M_{\text{total}} - M_W - M_{\text{act}} - M_{\text{graph}}\) (138.5); start-up fails if one
max_model_lenrequest does not fit. - CUDA graphs are captured for a ladder of sizes up to min(2 ×
max_num_seqs, 512) and batches are padded (138.6); mixed batches use piecewise graphs. - Tune with four limits:
max_model_len,gpu_memory_utilization,max_num_seqsandmax_num_batched_tokens, and read defaults from the version you run.
Key formulas
References for this session
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- vLLM project (docs). vLLM documentation
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- vLLM project (docs). vLLM documentation
- vLLM project (code). vLLM v0.32.0 source: vllm/v1/core/sched/scheduler.py, vllm/v1/core/block_pool.py, vllm/v1/worker/gpu_worker.py, vllm/config/ and vllm/engine/arg_utils.py
- vLLM project (docs). vLLM documentation: Optimization and Tuning (preemption, chunked prefill, optimization levels, parallelism)
- vLLM project (docs). vLLM documentation: Architecture Overview (V1 process architecture)
- vLLM project (docs). vLLM documentation: Automatic Prefix Caching (design)
- vLLM project (2025). vLLM v0.11.0 release notes (removal of the V0 engine)
- Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
- Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models