LLM & VLM courseBlog
138Track 5: LLM inference and serving · session 61 of 65

vLLM internals

vLLM is the open-source engine behind many LLM services. Inside it, one small loop does everything: a scheduler gives each request some tokens from a fixed budget, a block manager hands out 16-token KV blocks and keeps a prefix cache for free, and GPU workers run the batch with CUDA graphs and tensor parallelism. This session reads that loop from the source code of v0.32.0, with real defaults and the memory math that decides whether your server starts at all.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 to 1.5 hours)Level: IntermediateBefore this: Sessions 133 (the serving stack), 134 (memory management), 135 (batching), 136 (scheduling) and 127 (CUDA graphs)

What you will learn

  • The processes of a vLLM V1 server (API server, EngineCore, GPU workers) and what happens in one engine step
  • How the V1 scheduler works with one token budget and no prefill or decode phase, and how chunked prefill, prefix caching and speculative decoding fit into it
  • How the block manager allocates 16-token KV blocks, hashes full blocks, and turns its free queue into an LRU prefix cache
  • Why V1 preempts by recompute and not by swap, with a cost model and the measurements of the PagedAttention paper
  • How CUDA graphs are captured for a ladder of batch sizes, how batches are padded, and what piecewise mode does
  • How tensor and pipeline parallelism split weights and the KV cache, and what the all-reduces cost
  • What max_num_seqs, max_num_batched_tokens, max_model_len and gpu_memory_utilization really control, with their defaults and the start-up memory math

Why this matters

vLLM is the open-source LLM engine you are most likely to meet at work. Many companies run it in production, cloud providers offer it as a managed service, and other systems copy its ideas. When an interviewer asks “how would you serve this model?”, the expected answer often starts with vLLM, and the follow-up questions go inside it: what does the scheduler do, why did a request get preempted, why does the server refuse to start, what does gpu_memory_utilization really mean?

The last five sessions gave you the ideas one by one: the request path in session 133, paged KV memory and prefix caching in session 134, continuous batching and chunked prefill in session 135, preemption policies in session 136, and multi-GPU serving in session 137. This session shows how one real engine puts all of them together. You will see that the pieces fit into one small loop, and that most tuning knobs are limits on that loop.

We describe the V1 engine, which is the only engine in vLLM since version 0.11.0 (October 2025); the release notes say: “V1 is the only engine in the codebase now.” [Certain] Defaults and file names below are read from the source code of v0.32.0 (October 2026). vLLM releases every two weeks or so, and defaults do change (you will see two examples), so always check the version you run. [Certain]

The big picture: processes and one engine step

When you run vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4, vLLM starts several operating-system processes. The architecture docs list them: [Certain]

  • API server (1 by default): speaks HTTP (the OpenAI-compatible API), applies the chat template, tokenizes, loads images, and streams the text back.
  • EngineCore (1 per data-parallel rank): runs the scheduler and the KV cache manager in a busy loop, and sends work to the GPU workers. API servers and engine cores talk over ZMQ sockets (a fast message library).
  • GPU workers (1 per GPU, so TP × PP per engine core): each loads its part of the weights, owns its part of the KV cache, and runs the forward pass through a model runner, which also captures the CUDA graphs.

So TP = 4 on one node gives 1 + 1 + 4 = 6 processes. The split is on purpose: tokenizing and HTTP work in Python are slow, and in a separate process they do not delay the GPU loop. [Certain for the layout; Likely for this reason as the main one]

One turn of the EngineCore loop is called a step (or iteration). Each step runs one forward pass of the model over a batch of tokens from many requests.

Algorithm 1: one EngineCore step (V1, simplified)
  1. input: waiting queue, running list, KV block pool, limits max_num_seqs and max_num_batched_tokens
  2. Move newly arrived requests (sent by the API server over ZMQ) into the waiting queue
  3. scheduler.schedule(): decide how many tokens each request runs now; allocate KV blocks // Algorithm 2
  4. Send the scheduler output (request ids, token counts, new block ids) to all workers
  5. Each worker builds input ids, positions and block tables, picks a CUDA graph if the batch shape has one, and runs the forward pass // TP all-reduce inside
  6. Sample one token per request whose prompt is complete (or verify draft tokens if speculative decoding is on)
  7. scheduler.update_from_output(): append tokens, check stop conditions, free blocks of finished requests
  8. Send new tokens to the API server, which detokenizes and streams them
  9. output: one new token for each running request that finished its prompt // with async scheduling, step k+1 is scheduled while step k runs

Async scheduling is on by default in v0.32.0 unless an option is incompatible with it (for example pipeline parallelism with the older model runner, or some speculative methods). It lets the CPU prepare the next step while the GPU runs the current one, so the GPU does not wait for Python. [Certain]

Figure 1. The processes of vllm serve with tensor parallelism 4 on one node, and the path of one engine step through them. Names follow the vLLM source (v0.32.0).
How to read Figure 1

Each box is a process or a part of one. Step through: the highlighted box is where the work happens, and the arrow shows what is sent. Note that only small messages (token ids, block ids) travel between processes; the weights and the KV cache never leave the GPUs. Step 6 shows the two all-reduces per layer that tensor parallelism needs.

The scheduler: one token budget, no phases

Older engines had a “prefill phase” and a “decode phase”. The V1 scheduler does not. A comment at the top of Scheduler.schedule() says: “There’s no ‘decoding phase’ nor ‘prefill phase’ in the scheduler. Each request just has the num_computed_tokens and num_tokens_with_spec.” [Certain] Each request \(i\) has a number of known tokens \(T_i\) (prompt plus generated tokens, plus draft tokens if speculation is on) and a number of tokens whose KV is already computed, \(C_i\). The scheduler’s only job is to give each request some tokens \(n_i\) so that \(C_i\) catches up with \(T_i\):

\[ n_i = \min\!\big(T_i - C_i,\ B_{\text{left}}\big), \qquad \sum_{i} n_i \le B = \texttt{max\_num\_batched\_tokens}, \qquad |\text{running}| \le \texttt{max\_num\_seqs} \tag{138.1} \]
Read the symbols
\(T_i\)
tokens request \(i\) knows now: prompt + generated (+ drafts)
\(C_i\)
tokens whose KV is already in the cache (computed before, or found in the prefix cache)
\(n_i\)
tokens request \(i\) runs in this step; for a decoding request it is 1
\(B, B_{\text{left}}\)
the token budget of one step, and what is left of it when request \(i\) is visited

This one rule covers many features. A new prompt of 10,000 tokens with a budget of 8,192 gets 8,192 now and the rest next step: that is chunked prefill. A request whose first 6,000 tokens are in the prefix cache starts with \(C_i = 6000\): that is prefix caching. A request with 3 draft tokens has \(T_i - C_i = 4\): that is speculative decoding. A decoding request has \(T_i - C_i = 1\). The forward pass does not care which is which; the attention kernel gets, for each request, its new tokens and its block table.

Algorithm 2: Scheduler.schedule() in V1 (FCFS policy, simplified)
  1. input: running list (in admission order), waiting queue, token budget \(B\)
  2. for each request \(r\) in running, while budget is left: // decodes and unfinished prefills first
  3.   \(n \leftarrow \min(T_r - C_r, B_{\text{left}})\)
  4.   while allocate_slots(r, n) fails: preempt the last request in running (free its blocks, set \(C = 0\), put it at the front of waiting); stop if \(r\) itself was preempted
  5.   schedule \(n\) tokens for \(r\); \(B_{\text{left}} \leftarrow B_{\text{left}} - n\)
  6. if nobody was preempted in this step: while waiting is not empty, budget is left and \(|\text{running}| \lt\) max_num_seqs:
  7.   \(r \leftarrow\) head of waiting; \(C_r \leftarrow\) longest prefix-cache hit (full blocks, at most \(T_r - 1\) tokens)
  8.   \(n \leftarrow \min(T_r - C_r, B_{\text{left}})\); if the blocks for \(n\) tokens are not free, stop
  9.   move \(r\) to running; schedule \(n\) tokens
  10. output: \(\{r: n_r\}\), new block ids, ids of preempted requests

Three details matter in practice. First, running requests are served before waiting ones, so decodes are not starved by new prompts; the docs say the policy “batches all pending decode requests before scheduling any prefill operations”. [Certain] Second, when memory runs out, the victim is the most recently admitted request (self.running[-1]); with the priority policy it is the one with the worst priority. [Certain] Third, a prefix-cache hit can cover at most \(T - 1\) tokens, because the model must run at least the last prompt token to produce the logits of the next one. [Certain] The setting long_prefill_token_threshold (0, off, by default) caps the chunk of a single long prompt, so that a few long prompts cannot take the whole budget. [Certain]

The block manager: blocks, reference counts and the prefix cache

The KV cache manager (KVCacheManager, with a BlockPool inside) owns all KV blocks of the GPU. A block holds the keys and values of block_size tokens, 16 by default, for every layer. [Certain] The PagedAttention paper chose 16 because it is “large enough to efficiently utilize the GPU and small enough to avoid significant internal fragmentation in most workloads”. [Certain] The memory of one token is

\[ s_{\text{tok}} = 2 \cdot \frac{n_{\text{layers}}}{\text{PP}} \cdot \max\!\Big(1, \frac{n_{\text{kv}}}{\text{TP}}\Big) \cdot d_{\text{head}} \cdot \text{bytes}, \qquad \text{blocks}_i = \left\lceil \frac{C_i + n_i}{b} \right\rceil \tag{138.2} \]
Read the symbols
\(2\)
one key and one value vector
\(n_{\text{layers}}, n_{\text{kv}}, d_{\text{head}}\)
layers, KV heads (fewer than query heads with GQA) and head size
\(\text{TP}, \text{PP}\)
tensor- and pipeline-parallel sizes: each GPU stores only its KV heads and its layers; if TP is larger than \(n_{\text{kv}}\), heads are copied
\(b\)
block_size; a request needs enough blocks for all its computed tokens

For Llama 3.1 8B (32 layers, 8 KV heads of size 128) in BF16, \(s_{\text{tok}} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB), so one 16-token block is 2 MiB. For Llama 3.1 70B (80 layers, 8 KV heads) it is 327,680 bytes per token, and with TP = 4 each GPU stores 2 heads: 81,920 bytes per token per GPU.

Each block has a reference count (how many requests use it). When a block becomes full, it gets a hash that depends on the whole prefix before it:

\[ h_k = H\big(h_{k-1},\ (t_{kb}, \dots, t_{kb+b-1}),\ \text{extra}\big), \qquad h_{-1} = \text{seed} \tag{138.3} \]

Here \(t\) are token ids and “extra” holds anything else that changes the KV values: the LoRA adapter id, hashes of images in the block, and an optional cache_salt per user or tenant. The default hash is SHA-256 since v0.11. [Certain] Because \(h_k\) contains \(h_{k-1}\), two blocks with the same tokens but different histories get different hashes, so a hit on block \(k\) means the whole prefix up to block \(k\) is the same. Only full blocks are cached. [Certain] (The hash table and the matching rules are explained in session 134.)

The free block queue is a doubly linked list of blocks with reference count 0. Freed blocks that have a hash keep it and go to the tail; blocks without a hash go to the head. A new allocation takes the head; if that block still has a hash, its cache entry is evicted at that moment. A finished request frees its blocks in reverse order, so its last block (the least likely to be shared) is evicted first. This gives LRU eviction with no extra work. [Certain] So in vLLM, prefix caching costs almost nothing: the cache is simply the free memory that has not been reused yet. It is on by default. [Certain]

Preemption: recompute, and why not swap

A decoding request needs a new block every \(b\) tokens. If many requests grow at the same time, the pool can run out in the middle of a step. vLLM then preempts: it takes the victim’s blocks away and puts the victim back into the waiting queue. Later the request must get its KV back. There are two classic ways, both described in the PagedAttention paper: [Certain]

  • Swap: copy the victim’s blocks to CPU memory, and copy them back later. The old V0 engine had this (preemption_mode="swap" with a CPU swap_space).
  • Recompute: drop the blocks. When the request runs again, its prompt plus all tokens it generated so far are processed as one new prompt. The paper notes this is faster than the original work, because the generated tokens are now handled in one prefill pass, not one by one.

V1 does only recompute: _preempt_request frees the blocks, sets num_computed_tokens = 0 and puts the request at the front of the waiting queue. [Certain] The docs say “recomputation has lower overhead in the V1 architecture”. [Certain] A simple cost model shows why the choice is close, and what it depends on:

\[ t_{\text{rec}} = (1 - \eta_{\text{hit}})\,\frac{2 P n + 2 n_{\text{layers}} d\, n^2}{\text{TP}\cdot u \cdot F}, \qquad t_{\text{swap}} = 2 \cdot 2 n_{\text{layers}} \left\lceil \frac{n}{b} \right\rceil \left( t_c + \frac{b\, e}{\mathrm{BW}_{\text{PCIe}}} \right) \tag{138.4} \]
Read the symbols
\(n, P, d\)
tokens of the victim, model parameters, hidden size; \(2Pn\) is the matmul work and \(2 n_{\text{layers}} d n^2\) the causal attention work of one prefill
\(u F\)
achieved fraction \(u\) (MFU) of the GPU’s peak BF16 rate \(F\) (989 TFLOP/s for an H100 SXM, dense)
\(\eta_{\text{hit}}\)
fraction of the victim’s tokens still found in the prefix cache when it comes back
\(t_c, e\)
fixed cost of one CPU–GPU copy, and bytes of K (or V) per token per layer on one GPU; there is one copy per block, per layer, for K and for V, each way
Worked example: preempt a 2,000-token Llama 3.1 8B request on one H100

Recompute: \(2 \cdot 8.03\text{e}9 \cdot 2000 = 32.1\) TFLOP for the matmuls plus 1.05 TFLOP for attention (3% of the total). At \(u = 0.5\): \(33.2\text{e}12 / 494.5\text{e}12 = 67.1\) ms of GPU time.

Swap: the KV is \(2000 \cdot 128\) KiB \(= 262\) MB. With 16-token blocks there are \(\lceil 2000/16 \rceil = 125\) blocks, so \(2 \cdot 32 \cdot 125 = 8{,}000\) copies of 32 KiB each way. With our assumptions \(t_c = 5\ \mu s\) and 50 GB/s over PCIe 5.0: \(8000 \cdot (5 + 0.66)\ \mu s = 45.2\) ms out and the same back, 90.5 ms in total. With 128-token blocks: 21.0 ms. The values of \(t_c\) and the bandwidth are our assumptions. [Likely]

This matches the paper’s measurement: “swapping incurs excessive overhead with small block sizes”, because many small copies cannot use the PCIe bandwidth, while recompute does not depend on the block size; for blocks of 16 to 64 tokens “the two methods exhibit comparable end-to-end performance.” [Certain]

recompute (GPU compute)swap out + swap in (PCIe)
Figure 2. The cost model (138.4) of getting a preempted request back: recompute against swap, on an H100 (989 TFLOP/s BF16), with our assumptions \(t_c = 5\ \mu s\) per copy and 50 GB/s over PCIe. Log scales on both axes.
How to read Figure 2

The two curves are the cost of each method for every victim length; the vertical line is the length you chose. Recompute grows a bit faster than linearly (the attention term \(n^2\)); swap grows linearly, and its height depends strongly on the block size, because every block is one copy per layer for K and for V. Move the block size from 16 to 128: swap becomes about 4× cheaper, recompute does not change. Raise the prefix-cache hit: recompute falls in proportion. This is a model with assumed constants, not a measurement.

In V1 there are two more reasons for recompute. The victim’s full blocks keep their hashes and go to the tail of the free queue, so when the victim returns soon, much of its prefix may still be in the prefix cache (\(\eta_{\text{hit}} \gt 0\)), and recompute becomes nearly free. And recompute needs no CPU buffer, no copy streams and no second allocator. [Certain for the mechanism; Likely as the design reason] Figure 3 shows both effects in a small simulation of Algorithm 2.

Worked example: Algorithm 2 with 14 blocks of 4 tokens and a budget of 12 tokens

Five requests: R0, R2 and R4 start with the same 8-token system prompt (two blocks); max_num_seqs = 4. Step 0: R0 runs its 10-token prompt and R1 gets only 2 of its 6 prompt tokens (chunked prefill: the budget is 12). Step 2: R2 arrives and finds its first 8 tokens in the prefix cache, so it computes only 4. Step 7 ends with 0 free blocks. Step 8: R1 needs a new block, so R3, the last admitted, is preempted and loses 9 computed tokens. Step 10: R3 comes back and finds 8 of its 10 tokens still cached; it computes only 2.

In total: 19 steps, 3 preemptions, 31 computed tokens thrown away, but only 82 tokens computed against 79 with no preemption at all, because the prefix cache returned most of the lost work. With prefix caching off: 2 preemptions, 9 tokens thrown away, and 104 tokens computed (95 is the minimum without caching). Our Python code below gives the same numbers.

R0R1R2R3R4free, no hash
Figure 3. A step-by-step simulation of the V1 scheduler (Algorithm 2) with recompute preemption and the block pool’s free queue. Toy sizes: blocks of 4 tokens, five requests; R0, R2 and R4 share an 8-token system prompt. The code is the same as in the Python program below.
How to read Figure 3

The grid is the block pool. A filled cell is a block in use, colored by its owner; a block shared by several requests has a thick border and shows how many use it (×2). A cell with only a colored outline is free but still holds a cached prefix block (it can be hit, or evicted when reused). The rows below show each request: state, tokens computed out of tokens known, and the tokens it runs in this step. Use Next and Back to step. Lower the pool to 10 blocks to see repeated preemptions; turn prefix caching off and compare “tokens computed” in the readout. A larger budget finishes prompts in fewer steps; max_num_seqs = 1 turns the engine into one-at-a-time serving.

Memory at start-up: what gpu_memory_utilization means

vLLM does not grow the KV cache on demand. At start-up each worker measures what it needs and gives the rest of its budget to the block pool. The code in gpu_worker.py does this: [Certain]

  1. Requested memory = gpu_memory_utilization × total GPU memory. Note: total, not free. Another process on the same GPU is not subtracted for you.
  2. Load the weights, then run a profile run: a dummy forward pass with max_num_batched_tokens tokens, to measure the peak activation memory.
  3. Estimate the memory of the CUDA graphs (on by default since v0.21.0, per the log message in the code).
  4. KV memory: take the requested memory and subtract the weights, the peak activations, other non-KV memory and the CUDA graph estimate. Divide by the bytes of one block.
\[ M_{\text{KV}} = u_{\text{gpu}} M_{\text{total}} - M_W - M_{\text{act}} - M_{\text{graph}}, \qquad N_{\text{blocks}} = \left\lfloor \frac{M_{\text{KV}}}{b \cdot s_{\text{tok}}} \right\rfloor, \qquad \text{concurrency} = \frac{N_{\text{blocks}}\, b}{\texttt{max\_model\_len}} \tag{138.5} \]
Read the symbols
\(u_{\text{gpu}}\)
gpu_memory_utilization: 0.9 for a long time, 0.92 in v0.20.0 and later [Certain from the code of v0.15.0 and v0.20.0]
\(M_W\)
weights on this GPU (about the total divided by TP × PP)
\(M_{\text{act}}, M_{\text{graph}}\)
peak activations of the profile run, and CUDA graph memory; we use 3 GB together, an assumption [Likely]
concurrency
how many requests of the full max_model_len fit at once; vLLM logs this number as “Maximum concurrency”

Start-up also checks that at least one request of max_model_len tokens fits. If not, it stops with “To serve at least one request with the model’s max seq len (…) … KV cache is needed, which is larger than the available KV cache memory”, and it prints an estimated maximum length that would fit. [Certain] max_model_len defaults to the model’s context length from its config (131,072 for Llama 3.1); -1 or auto asks vLLM to pick the largest length that fits. [Certain]

Worked example: three deployments on 80 GB H100s (80 GB nominal, 3 GB for activations and graphs)

Llama 3.1 8B, one GPU, BF16. Requested \(0.92 \cdot 80 = 73.6\) GB; weights 16.06 GB (8,030,261,248 parameters × 2 bytes). \(M_{\text{KV}} = 73.6 - 16.06 - 3 = 54.54\) GB, so \(N_{\text{blocks}} = \lfloor 54.54\text{e}9 / (16 \cdot 131072) \rfloor = 26{,}006\) blocks, 416,096 tokens. Concurrency at 131,072 tokens: 3.17. Requests of 2,000 tokens (125 blocks each): 208 at once. With gpu_memory_utilization 0.90: 25,243 blocks.

Llama 3.1 70B, TP = 4. 70,553,706,496 parameters, 141.11 GB in BF16, 35.28 GB per GPU. \(M_{\text{KV}} = 73.6 - 35.28 - 3 = 35.32\) GB per GPU at 81,920 bytes per token: about 26,950 blocks, 431,000 tokens, concurrency 3.29 at 131,072.

Llama 3.1 70B, TP = 2. Weights 70.55 GB per GPU leave \(0.05\) GB. One 131,072-token request needs 21.5 GB per GPU, so vLLM refuses to start. Lowering max_model_len does not help here: there is almost no memory at all. You need TP = 4, FP8 weights, or bigger GPUs.

weights (per GPU)activations + CUDA graphs (3 GB)KV block poolnot requested (\(1 - u_{\text{gpu}}\))
Figure 4. The start-up memory plan of one GPU from formula (138.5), with 16-token blocks. GPU sizes are nominal; the 3 GB for activations and graphs is our assumption, the real value comes from the profile run.
How to read Figure 4

The bar is the memory of one GPU. The readout gives the KV pool in GB, blocks and tokens, the concurrency vLLM would log, and whether start-up succeeds. Pick 70B on an H100 with TP 1 or 2 to see the error; TP 4 works. Switch the KV cache to FP8: twice as many tokens fit in the same memory. Move the utilization: every 0.01 on an 80 GB GPU is 0.8 GB, about 6,100 tokens of 8B KV in BF16.

CUDA graphs for decode

A decode step of a small model is made of hundreds of short kernels, and the CPU can be too slow to launch them; the GPU then waits. A CUDA graph records the kernel launches of one step once and replays them with one call (session 127). A graph is recorded for fixed tensor shapes, so vLLM captures one graph per batch size, from a list of capture sizes, at start-up. A batch of \(n\) tokens is padded up to the next captured size:

\[ \mathcal{C} = \{1, 2, 4\} \cup \{8, 16, \dots, 248\} \cup \{256, 272, \dots, c_{\max}\}, \qquad c_{\max} = \min(2\,\texttt{max\_num\_seqs},\ 512,\ B), \qquad \text{pad}(n) = \min\{c \in \mathcal{C} : c \ge n\} \tag{138.6} \]

This is the default rule in v0.32.0; the cap is 1,024 instead of 512 on data-center Blackwell GPUs. [Certain] With max_num_seqs = 256 this gives 51 graphs up to 512 tokens. A decode batch of 37 requests runs the graph for 40 (7.5% padding), 257 runs as 272 (5.5%). Averaged over batch sizes 1 to 512, padding wastes 3.3% of the rows. A batch larger than \(c_{\max}\) runs without a graph.

Not every batch can be one full graph. Attention for a batch that mixes prefills of different lengths with decodes has shapes that change every step. So the default mode is FULL_AND_PIECEWISE: the code says it captures “full cudagraph for decode batches and piecewise cudagraph for prefill and mixed prefill-decode batches”. A piecewise graph covers everything between the attention calls (the matmuls, norms, activations) while attention runs outside the graph. [Certain] The optimization level controls this: -O1 uses piecewise graphs only, -O2 (the default) uses FULL_AND_PIECEWISE, and --enforce-eager skips both compilation and graphs for the fastest start-up. [Certain]

Figure 5. CUDA graph capture sizes from rule (138.6) with max_num_batched_tokens = 8,192, and the padding of one decode batch. The lower panel shows the padding waste for every batch size up to the largest graph.
How to read Figure 5

Each tick on the top line is one captured graph. The marker is your batch, the arrow goes to the graph that runs it, and the readout gives the waste. The lower curve shows waste for all batch sizes: it jumps right after each captured size and falls to zero at the next one. Below 8 the steps are 1, 2, 4; from 8 to 248 the step is 8; above 256 it is 16. Set max_num_seqs to 1,024 (the H100 server default): the cap stays at 512, so decode batches of more than 512 requests run without CUDA graphs.

Tensor and pipeline parallelism inside vLLM

With --tensor-parallel-size N each layer is split over N GPUs in the Megatron style (session 48): the QKV and first MLP matmuls are split by columns, the output and second MLP matmuls by rows, so each layer needs two all-reduces of the activations. KV heads are split too, so each GPU keeps only its own heads’ KV blocks, and all GPUs use the same block ids: one scheduler and one block table serve all of them. [Certain] The cost per step of the all-reduces, with a ring algorithm, is roughly

\[ t_{\text{AR}} \approx 2\, n_{\text{layers}} \cdot 2\,\frac{\text{TP} - 1}{\text{TP}} \cdot \frac{n\, d \cdot \text{bytes}}{\mathrm{BW}_{\text{link}}} \tag{138.7} \]
Worked example: Llama 3.1 70B, TP = 4, a decode batch of 256

One all-reduce moves \(256 \cdot 8192 \cdot 2 = 4.19\) MB. At an assumed 450 GB/s per direction over NVLink: \(2 \cdot \tfrac34 \cdot 4.19\text{e}6 / 450\text{e}9 = 14.0\ \mu s\), times 160 all-reduces = 2.24 ms per step, plus a latency of a few microseconds each. [Likely for the bandwidth] Reading the 35.3 GB of weights on each GPU at 3.35 TB/s takes at least 10.5 ms. So communication adds roughly 20% here; across nodes, over slower links, it would dominate, and that is why TP is used inside a node.

Pipeline parallelism (--pipeline-parallel-size) puts groups of layers on different GPUs or nodes. One batch would leave all but one stage idle, so the EngineCore keeps a batch queue of PP batches in flight: it schedules a new batch while earlier ones are still moving through the pipeline. The code comment says “PP requires PP-size concurrent batches to fill the pipeline.” [Certain] The docs suggest TP within a node first, then PP across nodes. Data parallelism (--data-parallel-size) runs separate engine cores, each with its own scheduler and KV cache; for MoE models, expert parallelism shares the experts across them. [Certain] Prefill–decode disaggregation, from session 137, is done in vLLM with KV connectors that move blocks between instances. [Likely for the details]

Prefix caching, chunked prefill and speculative decoding inside one loop

These three features were whole sessions earlier. In vLLM V1 each is a small change to \(C_i\), \(T_i\) or the budget of formula (138.1), plus some code in the worker:

FeatureWhat changes in the schedulerWhat changes in the workerDefault in v0.32.0
Prefix cachingat admission, \(C_i\) jumps to the length of the longest cached chain of full blocks (at most \(T_i - 1\))nothing: the block table simply points at shared blockson, SHA-256 hashes [Certain]
Chunked prefill\(n_i\) is cut by the budget left; decodes are scheduled firstattention handles “some new tokens + cached prefix” for every requeston whenever the model supports it [Certain]
Speculative decoding\(T_i\) includes \(k\) draft tokens, so a decoding request asks for \(k + 1\) tokens of budgeta drafter proposes tokens; the rejection sampler accepts a prefix and rejected slots are rolled backoff; methods include ngram, eagle, eagle3, medusa, mtp, draft_model, suffix [Certain]
Multimodal inputsimage or video tokens count in \(T_i\); the encoder output is cached in an encoder cache whose size defaults to max_num_batched_tokensthe vision encoder runs before the language model for new imageson for VLMs [Certain]

Speculation uses budget and KV blocks for tokens that may be rejected. With acceptance rate \(\alpha\) and \(k\) drafts, a step gives on average \((1 - \alpha^{k+1})/(1 - \alpha)\) tokens (session 128): 2.53 for \(\alpha = 0.7\), \(k = 3\), while it uses 4 tokens of budget. At low load the budget is free and speculation wins; at high load those 4 slots could have been 4 other requests, so the gain shrinks. [Likely]

The key settings and what they trade

Most of the time you tune vLLM with a few limits on the loop. The defaults for max_num_seqs and max_num_batched_tokens depend on the GPU and on how you start vLLM (the LLM class for offline jobs, or the OpenAI server). From arg_utils.py in v0.32.0: [Certain]

GPU memorymax_num_seqsmax_num_batched_tokens (server)max_num_batched_tokens (LLM class)
under 70 GB, or any A1002562,0488,192
70 GB or more (H100, H200)1,0248,19216,384
160 GB or more (B200, B300)1,02416,38416,384

The code notes that a large token budget on an A100 reduced throughput, hence the extra check. --performance-mode throughput doubles both defaults. [Certain]

SettingWhat it limitsRaise it whenLower it when
max_num_seqsrequests in the running list; also sizes buffers and CUDA graphsthe KV pool is half empty and the GPU is not saturatedpreemptions are frequent, or per-token latency is too high
max_num_batched_tokenstokens in one step (prefill chunk + decodes)TTFT is too high and throughput matters; docs suggest more than 8,192 for small models on big GPUslong prompts cause latency spikes for decoding users (docs: 2,048 gives better ITL)
max_model_lenprompt + output length per requestusers need longer contextstart-up fails, or you never need 128k tokens
gpu_memory_utilizationshare of total GPU memory this instance may usethe GPU is dedicated and you want more KVanother process shares the GPU, or you hit OOM
block_sizetokens per KV block (16)rarely; some attention backends need other sizesrarely
kv_cache_dtypebytes per KV value—use fp8 to double capacity, after an accuracy check
enforce_eager, -Ocompilation and CUDA graphs-O2 (default) in production--enforce-eager for debugging and fast start-up
tensor_parallel_size, pipeline_parallel_sizeGPUs per replicathe model or its KV does not fitthe model fits: more replicas (data parallel) give more throughput per GPU
Algorithm 3: tuning a vLLM deployment
  1. input: model, GPU, traffic (prompt and output lengths, arrival rate), SLOs for TTFT and time per output token
  2. Set max_model_len to the longest request you accept, not the model maximum
  3. Start the server; read “KV cache size: … tokens” and “Maximum concurrency” in the log // formula (138.5)
  4. Set max_num_seqs near KV tokens ÷ (average prompt + output length) and the GPU’s decode sweet spot
  5. Replay real traffic; watch preemptions, KV usage, TTFT and inter-token latency
  6. if preemptions are frequent: lower max_num_seqs, use an FP8 KV cache, or add GPUs
  7. if ITL spikes when long prompts arrive: lower max_num_batched_tokens or set long_prefill_token_threshold; if TTFT is too high: raise it
  8. output: settings that meet the SLOs at the highest request rate (goodput)

PyTorch code

The program has four parts. (1) The toy V1 scheduler of Figure 3: token budget, running first, recompute preemption of the last request, a free queue with LRU eviction, and prefix-cache hashes of full blocks. (2) A paged KV cache in PyTorch: the worker writes K and V through a block table and reads them back by gathering blocks; attention is unchanged. (3) The CUDA graph capture sizes of formula (138.6). (4) The start-up memory plan of formula (138.5). It runs on CPU and prints the same every time.

import math, torch
BS = 4                                              # tokens per block (vLLM's default is 16)
SYS = list(range(100, 108))                         # an 8-token system prompt shared by R0, R2, R4
REQS = [(0, SYS + [1, 2], 14), (0, [3, 4, 5, 6, 7, 8], 18), (2, SYS + [9, 10, 11, 12], 8),
        (3, [13, 14, 15, 16, 17], 12), (4, SYS + [18], 6)]   # (arrival step, prompt ids, output length)

def simulate(num_blocks, budget, max_seqs=4, caching=True, log=None):
    free = list(range(num_blocks))                  # free queue: the head is reused (and evicted) first
    bhash, cached, ref = {}, {}, [0] * num_blocks   # block -> hash, hash -> block, reference counts
    R = [dict(id=i, arr=a, toks=list(p), out=o, gen=0, comp=0, blocks=[], nh=0, h=0)
         for i, (a, p, o) in enumerate(REQS)]
    waiting, running, done = [], [], 0
    st = dict(steps=0, preemptions=0, discarded=0, hits=0, computed=0)
    def bh(prev, r, k): return hash((prev, tuple(r['toks'][k*BS:(k+1)*BS])))
    def take():
        b = free.pop(0)
        if b in bhash: del cached[bhash.pop(b)]     # reusing a cached free block evicts its prefix entry
        ref[b] = 1; return b
    def grow(r, n):                                 # allocate_slots: blocks for comp + n tokens, or fail
        need = math.ceil((r['comp'] + n) / BS) - len(r['blocks'])
        if need > len(free): return False
        r['blocks'] += [take() for _ in range(need)]; return True
    def release(r):                                 # free in reverse order: tail blocks are evicted first
        for b in reversed(r['blocks']):
            ref[b] -= 1
            if ref[b] == 0:
                if b in bhash: free.append(b)       # cached: keep it, put it at the tail (LRU)
                else: free.insert(0, b)             # not cached: reuse it first
        r['blocks'], r['nh'], r['h'] = [], 0, 0
    def register(r):                                # every newly full block gets a hash
        while caching and r['nh'] < r['comp'] // BS:
            h = bh(r['h'], r, r['nh']); b = r['blocks'][r['nh']]
            if h not in cached and b not in bhash: cached[h] = b; bhash[b] = h
            r['h'] = h; r['nh'] += 1
    step = 0
    while done < len(R):
        waiting += [r for r in R if r['arr'] == step]
        tb, sched, pre = budget, {}, []
        i = 0
        while i < len(running) and tb > 0:          # 1) RUNNING first: decodes and unfinished prefills
            r = running[i]; n = min(len(r['toks']) - r['comp'], tb); ok = True
            while not grow(r, n):                   # out of blocks: preempt the LAST running request
                v = running.pop(); st['discarded'] += v['comp']; release(v); v['comp'] = 0
                waiting.insert(0, v); pre.append(v['id']); st['preemptions'] += 1; sched.pop(v['id'], None)
                if v is r: ok = False; break
            if not ok: break
            sched[r['id']] = n; tb -= n; i += 1
        while not pre and waiting and tb > 0 and len(running) < max_seqs:   # 2) WAITING, if nobody was preempted
            r = waiting[0]; hitb, h = [], 0
            if caching:                             # longest cached chain of full blocks, at most len - 1 tokens
                for k in range((len(r['toks']) - 1) // BS):
                    hk = bh(h, r, k)
                    if hk not in cached: break
                    h = hk; hitb.append(cached[hk])
            hit = len(hitb) * BS; n = min(len(r['toks']) - hit, tb)
            if math.ceil((hit + n) / BS) - len(hitb) > len(free) - sum(ref[b] == 0 for b in hitb): break
            for b in hitb:                          # "touch": a cached block leaves the free queue
                if ref[b] == 0: free.remove(b)
                ref[b] += 1
            r['blocks'], r['comp'], r['nh'], r['h'] = list(hitb), hit, len(hitb), h
            st['hits'] += hit; grow(r, n); waiting.pop(0); running.append(r); sched[r['id']] = n; tb -= n
        for r in list(running):                     # 3) execute, then sample one token where the prompt is done
            if r['id'] not in sched: continue
            r['comp'] += sched[r['id']]; st['computed'] += sched[r['id']]; register(r)
            if r['comp'] == len(r['toks']):
                r['gen'] += 1; r['toks'].append(1000 * (r['id'] + 1) + r['gen'])
                if r['gen'] == r['out']: release(r); running.remove(r); done += 1
        if log is not None: log.append((step, dict(sched), list(pre), len(free)))
        step += 1
    st['steps'] = step
    return st

for nb, tb, c in [(16, 12, True), (14, 12, True), (14, 12, False), (10, 12, True), (40, 32, True)]:
    print(f"blocks={nb:2d} budget={tb:2d} prefix_cache={c!s:5}:", simulate(nb, tb, caching=c))

# 2. What a worker does with a block table: gather paged K/V and check attention is unchanged
torch.manual_seed(0)
L_tok, H, D, NB = 37, 2, 8, 32                     # tokens of one request, heads, head dim, blocks in the pool
k_pool = torch.zeros(NB, BS, H, D); v_pool = torch.zeros(NB, BS, H, D)
K, V, q = torch.randn(L_tok, H, D), torch.randn(L_tok, H, D), torch.randn(H, D)
table = torch.randperm(NB)[:math.ceil(L_tok / BS)]  # the request's blocks: anywhere in the pool
for t in range(L_tok):                              # write: slot = table[t // BS] * BS + t % BS
    k_pool[table[t // BS], t % BS] = K[t]; v_pool[table[t // BS], t % BS] = V[t]
Kp = k_pool[table].reshape(-1, H, D)[:L_tok]; Vp = v_pool[table].reshape(-1, H, D)[:L_tok]
att = lambda K, V: torch.einsum('ht,thd->hd', torch.softmax(torch.einsum('hd,thd->ht', q, K) / D**0.5, -1), V)
print("block table:", table.tolist(), " max |paged - contiguous|:", (att(Kp, Vp) - att(K, V)).abs().max().item())

# 3. CUDA graph capture sizes (vLLM's default rule) and the padding of a decode batch
def capture_sizes(max_num_seqs, max_num_batched_tokens=8192, blackwell=False):
    top = min(max_num_seqs * 2, 1024 if blackwell else 512, max_num_batched_tokens)
    s = [1, 2, 4] + list(range(8, 256, 8)) + list(range(256, top + 1, 16))
    return sorted(set([x for x in s if x <= top] + ([max_num_seqs] if max_num_seqs <= top else [])))
S = capture_sizes(256)
pad = lambda n: next((x for x in S if x >= n), None)
print("graphs:", len(S), "largest:", S[-1], " batch 37 ->", pad(37), " 257 ->", pad(257), " 600 ->", pad(600))

# 4. The KV cache budget at start-up: util * total - weights - other, in blocks of 16 tokens
def kv_plan(total_gb, util, weights_gb, other_gb, kv_bytes_per_token, max_model_len, block=16):
    kv = util * total_gb - weights_gb - other_gb
    blocks = int(kv * 1e9 // (kv_bytes_per_token * block))
    need = max_model_len * kv_bytes_per_token / 1e9
    return round(kv, 2), blocks, round(blocks * block / max_model_len, 2), kv >= need
print("8B, 1 x H100 :", kv_plan(80, 0.92, 16.06, 3.0, 131072, 131072))
print("70B, TP=4    :", kv_plan(80, 0.92, 141.11 / 4, 3.0, 81920, 131072))
print("70B, TP=2    :", kv_plan(80, 0.92, 141.11 / 2, 3.0, 163840, 131072))

Output with PyTorch 2.8 on CPU:

blocks=16 budget=12 prefix_cache=True : {'steps': 19, 'preemptions': 0, 'discarded': 0, 'hits': 16, 'computed': 79}
blocks=14 budget=12 prefix_cache=True : {'steps': 19, 'preemptions': 3, 'discarded': 31, 'hits': 44, 'computed': 82}
blocks=14 budget=12 prefix_cache=False: {'steps': 20, 'preemptions': 2, 'discarded': 9, 'hits': 0, 'computed': 104}
blocks=10 budget=12 prefix_cache=True : {'steps': 28, 'preemptions': 4, 'discarded': 46, 'hits': 44, 'computed': 97}
blocks=40 budget=32 prefix_cache=True : {'steps': 18, 'preemptions': 0, 'discarded': 0, 'hits': 16, 'computed': 79}
block table: [28, 13, 31, 18, 29, 16, 14, 11, 8, 4]  max |paged - contiguous|: 0.0
graphs: 51 largest: 512  batch 37 -> 40  257 -> 272  600 -> None
8B, 1 x H100 : (54.54, 26006, 3.17, True)
70B, TP=4    : (35.32, 26948, 3.29, True)
70B, TP=2    : (0.05, 17, 0.0, False)

Read the first block of lines. With 16 blocks no request is preempted. With 14 blocks there are 3 preemptions and 31 tokens are thrown away, but only 3 extra tokens are computed (82 against 79), because the prefix cache still holds most of the victims’ blocks. With caching off, 9 lost tokens cost 9 extra tokens of compute (104 against 95). With 10 blocks the queue thrashes: 28 steps instead of 19.

The real thing, on a GPU (not run here; check flag names against your version):

# server: 8B on one H100, 32k context, FP8 KV cache, more prefill per step
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768 \
    --gpu-memory-utilization 0.92 --max-num-seqs 512 --max-num-batched-tokens 8192 \
    --kv-cache-dtype fp8 --enable-prefix-caching

# offline: 70B over 4 GPUs with tensor parallelism
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3.1-70B-Instruct", tensor_parallel_size=4,
          max_model_len=16384, gpu_memory_utilization=0.92)
outs = llm.generate(["The capital of France is"], SamplingParams(temperature=0.0, max_tokens=16))
print(outs[0].outputs[0].text)

Where this is used

ApplicationWhat matters inside vLLMSettings to look at
Chat assistants and APIsdecodes first, chunked prefill so long prompts do not stall streamsmax_num_batched_tokens, max_num_seqs, CUDA graphs on
RAG and agents with long shared system promptsprefix-cache hits turn repeated prefill into a lookupprefix caching on, stable prompt order, cache_salt per tenant
Offline batch jobs (labeling, synthetic data, evaluation)throughput: large budget, many sequences, preemption is acceptableLLM class defaults, --performance-mode throughput
VLMs: image question answering, documentsimage tokens count in the budget; encoder cache; image hashes in block hashesmax_num_batched_tokens \(\ge\) tokens of one image, limit_mm_per_prompt
Video-RAG, video QA and key-moment detectionthousands of frame tokens per request make prefill heavy and KV largemax_model_len, frames per request, FP8 KV cache, TP
Video retrieval and classification with an LLM re-rankermany short requests sharing an instruction prefixprefix caching, high max_num_seqs
RL training (rollouts for GRPO or PPO)huge batches of samples from the same prompt; weights are updated between roundsprefix caching (shared prompt), throughput settings; see session 75

Common mistakes

  • Leaving max_model_len at 131,072 “to be safe”. Start-up then needs room for one full request, concurrency logs look tiny, and on small GPUs vLLM refuses to start. Set the length you really accept.
  • Two vLLM instances on one GPU with the default utilization. Each asks for 92% of total memory, not of free memory; the second fails. Give each a share, for example 0.45.
  • Confusing max_num_seqs with max_num_batched_tokens. The first limits requests in the batch; the second limits tokens in one step. 256 decoding requests use only 256 tokens of budget.
  • Ignoring preemption warnings. Frequent preemptions mean the pool is too small for the admission limit; latency becomes unstable. Lower max_num_seqs or add KV memory.
  • Benchmarking with --enforce-eager. It is meant for debugging; decode of small models can be much slower without CUDA graphs.
  • Using TP across nodes. Two all-reduces per layer over a slow network dominate the step; use PP across nodes and TP inside a node.
  • Copying settings from an old blog post. V0 options such as swap_space and preemption_mode, and the old defaults (utilization 0.9, one fixed max_num_seqs), are gone or changed.
  • Turning on speculative decoding for a high-load batch job. Draft tokens take budget and KV blocks; at high load the gain is small or negative. Measure at your real load.

Interview questions

Walk me through one step of the vLLM V1 engine.basic
The API server tokenizes requests and sends them over ZMQ to the EngineCore process. In each step the scheduler gives every running request its next tokens (1 for a decode, a chunk for an unfinished prompt) within the token budget, allocating KV blocks; if blocks run out it preempts the last admitted request. If nothing was preempted, it admits waiting requests, using prefix-cache hits. The scheduler output (request ids, token counts, block ids) goes to the GPU workers, one per GPU, which build inputs and block tables, run the forward pass (a CUDA graph for decode-only batches) with tensor-parallel all-reduces, and sample. The scheduler appends the tokens, frees finished requests’ blocks, and the API server detokenizes and streams. With async scheduling the next step is scheduled while this one runs.
What is the difference between max_num_seqs and max_num_batched_tokens?basic
max_num_seqs caps the number of requests in the running list at once; it bounds KV demand and sizes buffers and CUDA graphs. max_num_batched_tokens caps the tokens processed in one forward pass: all decode tokens plus prefill chunks. 300 decoding requests need 300 tokens of budget; a single 10,000-token prompt needs 10,000 tokens and is chunked if the budget is smaller. Defaults in v0.32.0 depend on the GPU: on an H100 server, 1,024 sequences and 8,192 tokens; on smaller GPUs or A100, 256 and 2,048.
Why does the V1 scheduler have no prefill and decode phases?core
Every request is described by two numbers: tokens known \(T\) and tokens computed \(C\). Each step gives each request \(n = \min(T - C, \text{budget left})\) tokens. A new prompt, a chunk of a long prompt, a decode, a request resuming after preemption, a request with a prefix-cache hit, and a request with draft tokens are all the same case with different \(T\) and \(C\). This makes chunked prefill, prefix caching and speculative decoding work together without special code paths in the scheduler, and the attention kernel handles mixed batches.
What happens when the KV cache is full in the middle of a step?core
allocate_slots fails for the request being scheduled. The scheduler preempts the last request in the running list (the most recently admitted under FCFS, the lowest priority under the priority policy): it frees its blocks, sets its computed tokens to 0 and puts it at the front of the waiting queue. It repeats until the allocation succeeds or the request itself was the victim. No new requests are admitted in a step with preemptions. When the victim returns, its prompt and generated tokens are prefilled again; full blocks that were not yet reused are found in the prefix cache, so part of the work is often free.
Recompute or swap for preemption: what decides it, and why did vLLM V1 keep only recompute?advanced
Recompute costs GPU compute: one prefill of the victim’s tokens, about \(2Pn\) FLOPs, independent of block size. Swap costs PCIe transfers out and back: bytes \(n \cdot s_{\text{tok}}\) but also one copy per block, per layer, for K and V, so small blocks give many small copies that cannot use the bandwidth. The PagedAttention paper measured that swap is bad with small blocks and that for blocks of 16 to 64 the two are comparable. In V1, the victim’s full blocks stay in the prefix cache until reused, so recompute is often partly free, and it needs no CPU buffer or extra allocator. For a 2,000-token Llama 3.1 8B request on an H100, our model gives about 67 ms of recompute at 50% MFU versus about 90 ms of swap with 16-token blocks.
How does vLLM decide how many KV blocks to allocate, and what does gpu_memory_utilization mean exactly?core
At start-up each worker requests utilization × total GPU memory (total, not free). It loads the weights, runs a profile forward pass at the maximum token budget to find peak activation memory, estimates CUDA graph memory, and gives the rest to the KV pool, divided into blocks of block_size × bytes per token. For Llama 3.1 8B on an 80 GB H100 at 0.92 with about 3 GB of activations: 54.5 GB, 26,006 blocks, about 416,000 tokens. Start-up fails if one request of max_model_len does not fit. The default was 0.9, it is 0.92 in recent versions.
Why does vLLM capture CUDA graphs only for some batch sizes, and what is piecewise mode?core
A CUDA graph replays fixed shapes, so one graph is needed per batch size. vLLM captures sizes 1, 2, 4, then steps of 8 to 248, then steps of 16 up to min(2 × max_num_seqs, 512), and pads a batch to the next size (for example 37 to 40). Capturing every size would cost start-up time and memory; the ladder keeps padding at about 3% on average. Mixed prefill and decode batches have attention shapes that change every step, so in the default FULL_AND_PIECEWISE mode decode-only batches replay one full graph, while mixed batches use piecewise graphs: everything between attention calls is graphed, attention runs eagerly. Batches above the largest size run without graphs.
Llama 3.1 70B on two 80 GB GPUs fails to start. Explain and fix.core
With TP = 2 each GPU holds 70.55 GB of BF16 weights. At utilization 0.92 the budget is 73.6 GB, leaving about 0.05 GB after weights and roughly 3 GB of activations, while one request of the default 131,072 tokens needs 21.5 GB of KV per GPU. vLLM reports that the KV cache needed for max_model_len is larger than the available memory. Lowering max_model_len cannot fix it because there is almost no KV memory at all. Options: TP = 4 (35 GB KV per GPU), FP8 weights (about 35 GB per GPU on 2 GPUs), or GPUs with more memory such as H200.
How do tensor parallelism and the KV cache interact in vLLM?advanced
Attention heads, including KV heads, are split across the TP GPUs. Each GPU stores the KV of its own heads, in the same block ids, so one scheduler and one block table drive all workers, and a block costs \(1/\text{TP}\) of the bytes on each GPU. For Llama 3.1 70B with 8 KV heads and TP = 4, each GPU stores 2 heads: 81,920 bytes per token. If TP is larger than the number of KV heads (TP = 16 with 8 heads), heads are replicated and KV per GPU stops shrinking. TP also frees weight memory per GPU, which goes to the KV pool, but costs two all-reduces per layer per step.
How do prefix caching and preemption interact?advanced
When a request is preempted, its blocks are freed, but full blocks keep their hashes and go to the tail of the free queue, so they are the last to be reused. If the victim is rescheduled before those blocks are taken by others, its prefix-cache lookup finds them, and only the uncached tail is recomputed. In our simulation with 14 blocks, 31 tokens were thrown away by preemptions but only 3 extra tokens were computed. Under heavy memory pressure the blocks are reused quickly, and the benefit disappears; then recompute costs a full prefill.
Users complain that streams freeze for a moment when someone sends a very long document. What do you change?core
Each step runs all decodes plus prefill chunks up to max_num_batched_tokens. A long prompt fills the budget with a large chunk, so the step takes longer and every decoding user waits for it: inter-token latency spikes. Lower max_num_batched_tokens (the docs say 2,048 gives better ITL), or set long_prefill_token_threshold so one long prompt cannot take the whole budget. The cost is higher time to first token for the long prompt. For a strict separation, use prefill–decode disaggregation.

Exercises

  1. Compute the KV bytes per token per GPU for Llama 3.1 70B with TP = 8 in BF16, and with TP = 16.
  2. Llama 3.1 8B on an L40S (48 GB nominal) at utilization 0.92 with 3 GB for activations: how many 16-token blocks, and what is the concurrency at max_model_len 32,768?
  3. With max_num_seqs = 64, list the largest CUDA graph and the number of graphs. Which graph runs a decode batch of 50?
  4. A server has a budget of 2,048 tokens and 300 decoding requests. A 5,000-token prompt arrives. How many steps until its prefill is done, if nothing else changes?
  5. Use (138.4) to find the swap time for the 2,000-token 8B request with 64-token blocks, keeping \(t_c = 5\ \mu s\) and 50 GB/s.
  6. Two vLLM servers must share one 80 GB GPU. Which utilization would you give each, and what goes wrong with the defaults?
Answerscheck
1. TP = 8: one KV head per GPU, \(2 \cdot 80 \cdot 1 \cdot 128 \cdot 2 = 40{,}960\) bytes. TP = 16: still 40,960, because 8 heads cannot be split over 16 GPUs and each head is stored twice. 2. \(0.92 \cdot 48 = 44.16\) GB; \(44.16 - 16.06 - 3 = 25.10\) GB; \(25.10\text{e}9 / 2{,}097{,}152 = 11{,}968\) blocks (191,488 tokens); concurrency \(191488 / 32768 = 5.84\). 3. \(c_{\max} = \min(128, 512, 8192) = 128\): sizes 1, 2, 4, 8, 16, …, 128, which is 19 graphs; a batch of 50 runs the graph for 56. 4. Each step the 300 decodes take 300 tokens, leaving 1,748 for the prompt: \(\lceil 5000/1748 \rceil = 3\) steps (1,748 + 1,748 + 1,504). 5. \(\lceil 2000/64 \rceil = 32\) blocks, \(2 \cdot 32 \cdot 32 = 2{,}048\) copies of 128 KiB each way; one copy \(5 + 2.62 = 7.62\ \mu s\); one way 15.6 ms, both ways 31.2 ms. 6. Each instance asks for utilization × total memory, so two at 0.92 would ask for 184% and the second fails at start-up. Give each about 0.45 (36 GB), and check that weights + activations + a useful KV pool fit in that share.

Key takeaways

  • vLLM V1 runs an API server, one EngineCore (scheduler + KV cache manager) per data-parallel rank, and one worker per GPU; only small messages cross processes.
  • The scheduler has no phases: each request gets \(\min(T - C, \text{budget left})\) tokens (138.1). Chunked prefill, prefix caching and speculation are changes to \(T\), \(C\) and the budget.
  • KV blocks are 16 tokens; full blocks get chained hashes (138.3), and the free queue doubles as an LRU prefix cache, on by default.
  • When blocks run out, the last admitted request is preempted and later recomputed; V1 has no swap. Prefix caching often makes recompute cheap.
  • At start-up, \(M_{\text{KV}} = u_{\text{gpu}} M_{\text{total}} - M_W - M_{\text{act}} - M_{\text{graph}}\) (138.5); start-up fails if one max_model_len request does not fit.
  • CUDA graphs are captured for a ladder of sizes up to min(2 × max_num_seqs, 512) and batches are padded (138.6); mixed batches use piecewise graphs.
  • Tune with four limits: max_model_len, gpu_memory_utilization, max_num_seqs and max_num_batched_tokens, and read defaults from the version you run.

Key formulas

\[ n_i = \min(T_i - C_i, B_{\text{left}}), \qquad s_{\text{tok}} = 2 \frac{n_{\text{layers}}}{\text{PP}} \max\!\Big(1, \frac{n_{\text{kv}}}{\text{TP}}\Big) d_{\text{head}} \cdot \text{bytes}, \qquad h_k = H(h_{k-1}, \text{tokens}_k, \text{extra}) \]
\[ M_{\text{KV}} = u_{\text{gpu}} M_{\text{total}} - M_W - M_{\text{act}} - M_{\text{graph}}, \qquad N_{\text{blocks}} = \left\lfloor \frac{M_{\text{KV}}}{b\, s_{\text{tok}}} \right\rfloor, \qquad \text{pad}(n) = \min\{c \in \mathcal{C} : c \ge n\} \]

References for this session

  1. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  2. vLLM project (docs). vLLM documentation
  3. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  4. vLLM project (docs). vLLM documentation
  5. vLLM project (code). vLLM v0.32.0 source: vllm/v1/core/sched/scheduler.py, vllm/v1/core/block_pool.py, vllm/v1/worker/gpu_worker.py, vllm/config/ and vllm/engine/arg_utils.py
  6. vLLM project (docs). vLLM documentation: Optimization and Tuning (preemption, chunked prefill, optimization levels, parallelism)
  7. vLLM project (docs). vLLM documentation: Architecture Overview (V1 process architecture)
  8. vLLM project (docs). vLLM documentation: Automatic Prefix Caching (design)
  9. vLLM project (2025). vLLM v0.11.0 release notes (removal of the V0 engine)
  10. Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
  11. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  12. Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  13. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
  14. Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
  15. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models