Distributed serving and prefill–decode disaggregation
A large model is served by many GPUs at once. This session shows how tensor and pipeline parallelism split one model for serving, why the two phases of generation (prefill and decode) fight when they share a GPU, and how Splitwise, DistServe and TetriInfer give each phase its own GPUs and move the KV cache between them. It ends with SpotServe, which serves on cheap cloud GPUs that can be taken away at any time.
What you will learn
- Tensor-parallel and pipeline-parallel serving: what each costs per token, and why tensor parallelism inside a node is the default for latency
- Why prefill and decode interfere on one GPU, measured as stalls in time per output token
- Goodput, the queueing view of TTFT from DistServe, and how to choose the ratio of prefill GPUs to decode GPUs
- Splitwise, DistServe and TetriInfer: what each one adds (cheaper hardware for decode, placement search, chunked prefill and length prediction)
- How large the KV cache is, how long it takes to move, and why layer-wise transfer hides most of the cost
- SpotServe: re-planning the parallel layout when a spot instance is taken away, device mapping as bipartite matching, and token-level recovery
- When disaggregation helps and when it does not
Why this matters
Llama 3.1 70B in BF16 is 141 GB of weights. One H100 has 80 GB. So the smallest way to serve it at full precision is already a small distributed system: at least two GPUs that must talk to each other for every token. Big chat services use thousands of GPUs, and the way those GPUs are grouped decides the cost per token and the latency users feel.
This session answers three questions that come up in every serving design review and in many interviews:
- How do we split one model over several GPUs for serving? Tensor parallelism and pipeline parallelism, which you met for training in session 48 and session 49, behave differently at inference time. Tensor parallelism makes each token faster. Pipeline parallelism does not; it only adds capacity.
- Should the two phases of generation share GPUs? Prefill (reading the prompt) is compute-bound; decode (writing one token at a time) is memory-bound (session 78). When they share a GPU, a long prefill stalls every running decode. Splitwise (Patel et al., 2023), DistServe (Zhong et al., 2024) and TetriInfer (Hu et al., 2024) put them on different GPUs. This is called prefill–decode disaggregation, or PD disaggregation. [Certain]
- What does it cost to move the KV cache, and can we serve on cheap GPUs that may disappear? After prefill, the KV cache must travel to the decode GPU. And cloud providers sell spare GPUs (spot instances) at large discounts but can take them back with little warning. SpotServe (Miao et al., 2023) serves LLMs on such GPUs. [Certain]
The efficient-inference survey of Zhou et al. lists these systems under serving-system optimization, as “distributed systems”. [Certain] The ideas are now in production: Moonshot AI’s Kimi service runs on Mooncake, a disaggregated design, and DeepSeek-V3 is deployed with separate prefill and decode units (32 GPUs per prefill unit, 320 GPUs per decode unit). [Certain] vLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo all offer disaggregated serving. [Likely]
Two phases that want different machines
Recall the two phases (session 78, session 81):
- Prefill runs the whole prompt (\(s\) tokens) through the model at once. It does about \(2N\) FLOPs per token for \(N\) weights, so it is a large matrix multiply. It is compute-bound: the GPU’s arithmetic units are the limit. It produces the first output token and the KV cache of the prompt. Its latency is the time to first token (TTFT).
- Decode makes one token per step for every running request. Each step reads all weights and the KV cache of every request in the batch, but does little arithmetic per byte. It is memory-bound. Its speed is the time per output token (TPOT), also called inter-token latency (ITL).
The two phases also like different settings. Prefill wants many GPUs per request to cut TTFT, and gains little from batching (one long prompt already fills the GPU). Decode wants big batches, because the weights are read once per step for the whole batch, and a lot of memory for the KV cache. Splitwise measured both on real GPUs: prompt throughput stops growing above about 2,048 batched tokens, while token throughput keeps growing with batch size up to 64, where memory runs out; and capping a DGX-H100’s power from 700 W to 350 W hurts prompt latency a lot but token latency almost not at all. [Certain]
Interference
Modern servers use continuous batching (session 135): new requests join the running batch at any step. With the simplest policy, when a new prompt arrives the next step runs its prefill, and all decoding requests wait for it. A 2,048-token prefill of Llama 3.1 8B takes about 69 ms on an H100 (at an assumed 50% of peak FLOPs), while a decode step takes about 5 to 6 ms. So every new prompt makes every running user see a pause of more than 10 decode steps. TetriInfer measured this kind of interference: one heavy prefill in the batch made a light decode step 5× slower, and decode slowed prefill up to about 2.5×. [Certain]
Chunked prefill (session 135, Sarathi-Serve) softens this by cutting the prompt into chunks and mixing a chunk into each decode step. Disaggregation removes it: prefill GPUs only prefill, decode GPUs only decode. Figure 1 compares two GPUs used both ways.
How to read Figure 1
Each bar is one step of a GPU. In the colocated rows, the wide orange-red bars are prefills; every decode on that GPU waits for them. Look at the readout: the average TPOT is almost the same in both designs, but the worst gap between two tokens of one user is about 74 ms colocated and about 14 ms disaggregated. In the disaggregated design the worst gap is between the first and the second token: the KV cache must travel (5.4 ms here) and the request then waits for the next decode step to start. Now raise the prompt length to 8,192 at 4 requests/s: the single prefill GPU needs 0.30 s per prompt, which is more than the 0.25 s between arrivals, so its queue grows and TTFT explodes. The colocated design shares the prefill work over both GPUs and survives. Disaggregation needs the right ratio of prefill to decode GPUs; that is the subject of Figure 4.
Spreading one model over many GPUs
Before we separate the phases, we need the tools to put one copy of the model on several GPUs. Inside a model copy there are two main ways; across copies there is the trivial third one.
- Data parallelism (replicas). Copy the whole model to several GPU groups and send each request to one copy. No communication between copies. It adds throughput, not speed per request.
- Tensor parallelism (TP). Split every weight matrix over \(t\) GPUs, as in Megatron-LM (session 48). Each GPU holds \(1/t\) of every layer and computes \(1/t\) of every matrix multiply. After the attention block and after the MLP block, the partial results are summed with an all-reduce (session 43): two all-reduces per layer. [Certain]
- Pipeline parallelism (PP). Give each of \(p\) GPUs a consecutive group of layers (a stage). A token’s hidden state flows from stage to stage with point-to-point sends. Communication is small, so PP works even over slow links between machines.
In decode, a step is memory-bound, so its time is about “bytes read divided by memory bandwidth”. TP divides the bytes per GPU by \(t\) and adds communication:
- Read the symbols
- \(W\)
- bytes of all weights (141 GB for Llama 3.1 70B in BF16)
- \(B, c\)
- number of sequences in the batch, and their average context length (tokens)
- \(\kappa\)
- KV-cache bytes per token for the whole model (equation 137.6 below; 320 KiB for Llama 3.1 70B)
- \(t\)
- tensor-parallel degree (number of GPUs that share each layer)
- \(B_{\text{hbm}}\)
- memory bandwidth of one GPU (3.35 TB/s for an H100 SXM)
- \(L, h, b\)
- number of layers, hidden size, bytes per number (2 for BF16)
- \(\alpha\)
- fixed start-up time of one all-reduce (a few microseconds; we assume 10 µs inside a node)
- \(B_{\text{link}}\)
- bandwidth of the GPU-to-GPU link per direction (about 450 GB/s for H100 NVLink, about 50 GB/s for a 400 Gb/s InfiniBand port)
The first term is the useful work. The second term is the price of TP: \(2L = 160\) all-reduces per token for a 70B model. Each one moves only \(B \cdot h \cdot 2\) bytes (16 KB per sequence), so at small batches the start-up time \(\alpha\) dominates: \(160 \times 10\) µs = 1.6 ms per token, whatever the batch. That is fine inside a node with NVLink, and bad across nodes, where \(\alpha\) is larger and the link 9× slower.
PP has no all-reduce, but it does not make one token faster: the token must pass through all \(p\) stages one after the other. If \(B\) sequences are split into \(p\) micro-batches so that all stages stay busy, each GPU reads its weights once per micro-batch, so for the same \(B\) the weights are read \(p\) times more often than with TP. In the memory-bound regime PP therefore gives about \(p\) times lower throughput than TP for the same number of sequences; it catches up only when the micro-batches are large enough to be compute-bound. Figure 2 lets you check this.
How to read Figure 2
The top row shows the GPUs, colored by pipeline stage. The bars show the time per token and its parts. Start with TP8 at \(B = 32\): about 8.8 ms per token, of which 1.9 ms is all-reduce. Click PP8: same 8 GPUs, same memory per GPU, but about 44 ms per token and 5× lower throughput. Click TP2: the weights alone take 70.6 GB of each GPU’s 72, so even a few sequences of 4,096 tokens do not fit (the memory bar turns red); only \(B = 1\) fits. Click TP16 (2 nodes): the all-reduces now cross InfiniBand; at \(B = 256\) they cost more than the useful work, and TP8×PP2 over the same 16 GPUs is much better. This is the usual rule: TP inside a node, PP (or replicas) across nodes. [Likely]
Parallelism and queueing: the DistServe view of TTFT
For prefill there is one more effect: requests wait in a queue. DistServe models a prefill instance as a queue with random (Poisson) arrivals at rate \(R\) and a fixed service time \(D\) per request (an M/D/1 queue). Its average TTFT is the service time plus the average waiting time: [Certain]
- Read the symbols
- \(D\)
- time to prefill one request on one GPU
- \(R\)
- arrival rate of requests (per second); the queue is stable only if \(R D \lt 1\)
Now give the instance a second GPU. With 2-stage pipeline parallelism (“inter-op” in the paper) each stage needs \(D/2\), a request still needs \(D\) in total, but two requests can be in flight. With 2-way tensor parallelism (“intra-op”) one request takes \(D/K\), where \(1 \lt K \lt 2\) is the real speed-up (communication makes it less than 2). DistServe derives: [Certain]
At low load, waiting is small, and TP wins because it shortens \(D\). At high load, waiting dominates, and PP wins because it can accept up to \(2/D\) requests per second while TP accepts only \(K/D\). The best choice depends on the traffic, not only on the model. Figure 3 draws the three curves.
How to read Figure 3
With \(D = 100\) ms and \(K = 1.6\): at \(R = 2\) req/s, TP gives 67 ms and PP 103 ms, so TP wins. At \(R = 8\), TP gives 94 ms and PP 117 ms. Move \(R\) up: the TP curve shoots up near \(K/D = 16\) req/s while PP survives until 20. The crossing point moves left when \(K\) is small (poor TP scaling). Real arrivals are bursty, so real queues are worse than M/D/1, but the shape is the same.
Disaggregation: prefill GPUs and decode GPUs
In a colocated system (vLLM’s default, for example), every GPU group runs both phases. In a disaggregated system there are two pools. A request first goes to a prefill instance, which computes the prompt’s KV cache and the first token. Then the KV cache is sent to a decode instance, which generates the rest. Each pool can use its own parallelism, batch size and even its own GPU type.
- input: a request (prompt of \(s\) tokens, output limit); a prefill pool and a decode pool
- router picks a prefill instance // least loaded, or the one with the longest cached prefix
- router (or the prefill instance) picks a decode instance and reserves KV memory for \(s\) + expected output tokens
- prefill instance runs the prompt; after each layer \(\ell\), it starts sending that layer’s KV blocks to the decode instance // layer-wise transfer, overlapped with layer \(\ell+1\)
- prefill instance samples the first token and streams it to the user // TTFT ends here
- when all KV blocks have arrived, the decode instance adds the request to its running batch
- decode instance generates tokens until end-of-sequence; frees the KV blocks // TPOT is set here only
Two properties make this attractive. First, no interference: a decode step never waits for a prefill. Second, independent tuning: you can give prefill high TP to cut TTFT and give decode a big batch to raise throughput. The vLLM documentation lists exactly these two benefits and adds a warning: “Disaggregated prefill DOES NOT improve throughput.” [Certain] The gain is in meeting latency targets, and that needs a better metric than raw throughput.
Goodput
A service has a service-level objective (SLO), for example “TTFT under 0.25 s and TPOT under 0.1 s for 90% of requests”. DistServe defines per-GPU goodput as “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned”. [Certain] Throughput counts every finished request; goodput counts only requests served fast enough, divided by the number of GPUs.
With \(n_P\) prefill instances that can each serve \(r_P\) requests/s within the TTFT target, and \(n_D\) decode instances that can each serve \(r_D\) within the TPOT target, the system serves
- Read the symbols
- \(r_P, r_D\)
- goodput of one prefill instance and one decode instance (requests/s that meet their own SLO)
- \(g_P, g_D\)
- GPUs per prefill instance and per decode instance
- \(\min\)
- the slower pool sets the rate: requests cannot leave decode faster than they enter it
DistServe’s first experiment: a 13B model on one A100 80 GB, prompts of 512 tokens, 64 output tokens. A colocated GPU meets the SLOs up to about 1.6 requests/s. A GPU that only prefills reaches 5.6 requests/s; a GPU that only decodes reaches 10. [Certain]
With 2 prefill GPUs and 1 decode GPU: \(\min(2 \times 5.6,\ 1 \times 10) = \min(11.2,\ 10) = 10\) requests/s on 3 GPUs, which is 3.33 requests/s per GPU, or \(3.33 / 1.6 = 2.1\times\) the colocated system. The ratio rule gives \(n_P / n_D \approx 10 / 5.6 = 1.8\), and 2:1 is the closest small integer ratio.
How to read Figure 4
The two upper bars are the capacities of the two pools; the shorter one, outlined, sets the system rate. The lower bars compare goodput per GPU with the colocated system. With 2 prefill and 1 decode GPU you get 3.33 per GPU. Try 1:1: the prefill pool (5.6) limits, 2.8 per GPU. Try 3:1: decode limits at 10, 2.5 per GPU. With more GPUs you can get closer to the ideal ratio 1.79: the readout finds 7:4 (ratio 1.75), with 3.56 per GPU. Now model a workload with long prompts: lower \(r_P\) to 2. The best ratio moves to 5:1. Workloads change during the day, which is why Splitwise keeps a flexible mixed pool and TetriInfer can flip an instance between roles.
Splitwise: cheaper GPUs for decode
Splitwise (Patel et al., 2023) starts from a hardware observation: decode does not need the newest GPU. Its Table I compares one DGX-A100 and one DGX-H100 machine (8 GPUs each): the H100 machine has 3.43× the compute but only 1.64× the memory bandwidth, the same 80 GB per GPU, 1.75× the power and 2.16× the cost per hour ($38 versus $17.6). [Certain] Decode is memory-bound, so it gains only the 1.64×, at 2.16× the price.
Splitwise keeps three pools: a prompt pool, a token pool, and a mixed pool that grows and shrinks with load and runs normal mixed batching when one of the queues gets long. [Certain] It studies four cluster designs: Splitwise-AA (A100 for both phases), Splitwise-HH (H100 for both), Splitwise-HA (H100 for prompts, A100 for tokens), and Splitwise-HHcap (H100 for both, with the token machines power-capped). [Certain] The KV cache moves over InfiniBand with MSCCL++ one-sided writes; small prompts are sent after prefill, large ones layer by layer during prefill. [Certain] Its headline result: 1.4× higher throughput at 20% lower cost, or 2.35× more throughput at the same cost and power budget, compared with current designs. [Certain]
DistServe: search for the best placement
DistServe (Zhong et al., OSDI 2024) treats disaggregation as an optimization problem: for a given model, workload and SLO, choose the parallelism of each phase (TP and PP degrees), the number of instances of each, and where to put them in the cluster, to maximize per-GPU goodput. [Certain]
- input: model, GPUs per node, SLOs, a sample of the workload, the target request rate \(\lambda\)
- for each prefill config (intra-op degree, inter-op degree) that fits in memory do
- simulate the workload and binary-search the highest rate that meets the TTFT SLO; divide by its GPUs // per-GPU goodput of this config
- for each decode config do the same with the TPOT SLO
- keep the best prefill config and the best decode config // they are chosen independently
- replicate each until its pool can carry \(\lambda\): \(n_P = \lceil \lambda / r_P \rceil\), \(n_D = \lceil \lambda / r_D \rceil\)
The search is cheap: under 1.3 minutes in the largest setting of the paper. [Certain] A second algorithm handles clusters with slow links between nodes (25 Gbps in their testbed): it forces the prefill and decode parts of the same pipeline stage onto the same node, so that the KV cache moves over NVLink instead of the network. [Certain] We will see why in the KV-transfer section. DistServe reports that it can serve 7.4× more requests, or meet 12.6× tighter SLOs, than vLLM and DeepSpeed-MII while keeping more than 90% of requests within the latency limits, on OPT-13B to OPT-175B with chatbot, code-completion and summarization workloads. [Certain]
TetriInfer: chunks, predicted lengths and flipping roles
TetriInfer (Hu et al., 2024) adds three ideas to disaggregation. [Certain]
- Fixed-size chunked prefill. Prompts are cut and merged into chunks of exactly ChunkSize tokens (512 for OPT-13B on their V100 testbed), padding the last chunk, so each prefill step keeps the accelerator just at its saturation point.
- A length predictor and two-level scheduling. A small model (OPT-125M, fine-tuned as a classifier) predicts the output length of each request in buckets of 200 tokens, with 74.9% accuracy. A global scheduler sends prompts to the least-loaded prefill instance; the prefill side then picks a decode instance with power-of-two random choice among those with enough predicted free memory, so heavy decodes do not pile up on one instance.
- Instance flip. Prefill and decode are only roles. An idle instance can switch roles in about 5 to 7 ms (plus draining time), with no model reload.
The abstract reports up to 97% lower average TTFT, 47% lower average job completion time and 38% fewer resources than vLLM. These are the best case, a workload with light prefills and heavy decodes; for a mixed workload the paper reports 85%, 50% and 21%, and for heavy prefills with heavy decodes vLLM was 14% better in performance per dollar. [Certain] The prototype emulates the network transfer time instead of moving data over a real link. [Certain] So read TetriInfer as a study of scheduling, not of KV transport.
In production: Mooncake and DeepSeek-V3
Mooncake (Moonshot AI) is the serving platform of the Kimi chatbot. It separates prefill and decode clusters and uses the spare CPU memory and SSDs of the GPU servers as a large distributed KV-cache store, with a scheduler built around where KV caches already are, and early rejection of requests when overloaded. It reports that real Kimi traffic got 75% more requests served. [Certain] DeepSeek-V3 runs prefill on units of 4 nodes (32 GPUs: attention with TP4 and sequence parallelism plus 8-way data parallelism, experts with 32-way expert parallelism) and decode on units of 40 nodes (320 GPUs, 320-way expert parallelism). [Certain] For a mixture-of-experts model the two phases want very different expert layouts, which is another reason to separate them (session 96).
Moving the KV cache
Disaggregation has a price that colocation does not have: the KV cache of every prompt must cross from the prefill GPUs to the decode GPUs. How large is it? Per token, every layer stores one key and one value vector per KV head (session 80):
- Read the symbols
- \(2\)
- one key and one value
- \(L, n_{\text{kv}}, d_h\)
- layers, KV heads, head dimension
- \(b\)
- bytes per number: 2 for BF16, 1 for FP8
- \(s\)
- prompt length in tokens; \(M_{\text{KV}}\) is the bytes to move for one request
| Model (attention type) | \(L\), KV heads × dim | \(\kappa\) in BF16 | 4,096-token prompt |
|---|---|---|---|
| Llama 3.1 8B (GQA) | 32, 8 × 128 | 128 KiB | 0.54 GB |
| Llama 3.1 70B (GQA) | 80, 8 × 128 | 320 KiB | 1.34 GB |
| OPT-66B (MHA) | 64, 72 × 128 | 2.25 MiB | 9.66 GB |
| DeepSeek-V3 (MLA, 512 + 64 per layer) | 61, one 576-wide latent | 68.6 KiB | 0.29 GB |
The configurations are from the models’ config files. DistServe gives the same kind of number for OPT-66B: a 512-token request has about 1.13 GB of KV cache (we get \(512 \times 2.25\) MiB = 1.125 GiB), so 10 requests/s need about 11.3 GB/s, roughly 90 Gbps, of bandwidth between the pools. [Certain] Grouped-query attention, MLA (session 26) and KV quantization (session 110) shrink this a lot.
The time to move it is \(t_x = M_{\text{KV}} / B_{\text{net}}\). The useful comparison is with the prefill time \(t_p\). Ignoring the attention part of prefill FLOPs, both grow linearly with \(s\), so their ratio does not depend on the prompt length:
- Read the symbols
- \(B_{\text{net}}\)
- total bandwidth between the prefill instance and the decode instance
- \(N\)
- number of weights; prefill does about \(2N\) FLOPs per prompt token
- \(F_{\text{eff}}\)
- FLOP/s that the prefill instance really achieves (all its GPUs together)
For Llama 3.1 70B on 8 H100s at 50% of peak (\(F_{\text{eff}} = 3.96 \times 10^{15}\)) and 8 × 400 Gb/s InfiniBand (\(B_{\text{net}} = 400\) GB/s), one prompt token takes 35.7 µs of prefill and 0.82 µs of transfer: the transfer is about 2.3% of the prefill. A faster GPU makes the ratio worse (larger \(F_{\text{eff}}\)); a bigger model with GQA makes it better (larger \(N\), same \(\kappa\)).
Hide the transfer behind the prefill
The KV cache of layer \(\ell\) is final as soon as layer \(\ell\) finishes. So the prefill instance can send layer \(\ell\) while it computes layer \(\ell + 1\). Splitwise calls this layer-wise transfer, versus serialized transfer after the whole prefill. [Certain] If each layer takes \(t_p / L\) to compute and \(t_x / L\) to send, and the link sends one layer at a time, the delay added after the prefill ends is
When \(t_x \le t_p\), only the last layer’s transfer remains: the delay shrinks by a factor \(L\). When the link is too slow (\(t_x \gt t_p\)), the transfer cannot keep up and most of it shows. Splitwise measured an added end-to-end latency of 0.8% with its layer-wise scheme (up to 3% with serialized transfer for large prompts), and DistServe found the transfer below 0.1% of total latency for OPT-175B, with 95% of transfers under 30 ms. [Certain]
KV cache: \(4096 \times 327{,}680 = 1.34\) GB. Prefill on 8 H100s at 50% of peak: \((2 \times 70.55 \times 10^9 \times 4096 + 2 \times 80 \times 4096^2 \times 8192) / 3.96 \times 10^{15} = 0.152\) s.
Over 400 GB/s (8 InfiniBand ports): \(t_x = 3.36\) ms, 2.2% of the prefill. Layer-wise, the added delay is \(3.36 / 80 = 0.042\) ms. Over one 400 Gb/s port (50 GB/s): \(t_x = 26.8\) ms serialized, 0.34 ms layer-wise. Over a 25 Gb/s network (3.125 GB/s, the cross-node speed of the DistServe testbed): \(t_x = 429\) ms, more than the whole prefill; even layer-wise, \(429 - 0.9875 \times 152 = 280\) ms is added. That is why DistServe’s second algorithm keeps the KV transfer inside a node on such clusters.
How to read Figure 5
The top row is the prefill, one block per layer. The row below shows when each layer’s KV is on the wire. In layer-wise mode each transfer starts when its layer is done, so with a fast link the transfers finish just after the prefill. Switch to 25 Gb/s: the transfers fall behind and pile up after the prefill (red). Switch to OPT-66B: multi-head attention makes the KV cache 7× larger than Llama 3.1 70B’s, so even 50 GB/s is not enough to hide it. FP8 halves every transfer. Change \(s\): the ratio \(t_x / t_p\) barely moves at first (equation 137.6) and then falls slowly, because attention FLOPs grow like \(s^2\).
Practical details of KV transport
- Paged layout. Serving engines store the KV cache in fixed-size blocks scattered in memory (session 134). The transfer must gather many small blocks, so engines use RDMA (direct GPU-to-GPU writes over InfiniBand or RoCE, without the CPU) and batch the blocks. vLLM moves KV with pluggable “connectors”, for example NixlConnector and MooncakeConnector. [Certain]
- Different parallelism on each side. If prefill uses TP8 and decode TP4, each decode GPU must receive the KV heads of two prefill GPUs. The mapping of heads to GPUs must be planned. [Likely]
- Admission control. The decode instance must have free memory for the incoming KV cache and the future tokens before the transfer starts (step 3 of Algorithm 1); otherwise the request waits on the prefill side, holding memory there. [Likely]
- Prefix caching across the cluster. If the KV cache of a common prefix (a system prompt, a long document) already sits on some machine, routing the request there skips part of the prefill. Mooncake and Dynamo route by KV-cache overlap for this reason. [Certain]
Serving on cheap spot instances: SpotServe
Clouds rent out spare GPUs as spot (or preemptible) instances at up to 90% below the on-demand price. The catch: the provider can take an instance back at any time, after a short grace period (SpotServe uses 30 seconds; AWS documents a two-minute notice). [Certain for the paper; Likely for the AWS detail] For training, a preemption costs a restart from the last checkpoint (session 51). For serving, it is harder: requests are running right now, and the model may be split over the instance that disappears.
SpotServe (Miao et al., 2023) handles this with three parts. [Certain]
- Parallelization controller. When instances are lost or gained, or the request rate changes, it picks a new configuration \(C = (D, P, M, B)\): data-parallel degree, pipeline degree, tensor degree and maximum batch size. It minimizes the request latency subject to throughput at least the expected arrival rate; if no configuration can keep up, it maximizes throughput; ties go to fewer instances.
- Device mapper. A new configuration has “positions” (which pipeline stage, which tensor slice). Which surviving instance should take which position? The one that already holds most of the needed weights and KV cache. This is a bipartite matching problem, solved with the Kuhn–Munkres (Hungarian) algorithm.
- Migration planner and stateful recovery. It moves the missing weights and KV cache, front layers first, so the pipeline can restart early. Progress is committed per token, not per request, so an interrupted request continues from its last generated token instead of starting over. During the grace period, a just-in-time rule decides how many more decode steps to run before migrating.
Formally, with surviving instances \(i\) and new positions \(j\), let \(u_{ij}\) be the bytes of weights and KV cache that instance \(i\) already has for position \(j\). The device mapper solves
and the just-in-time rule for a preemption runs the largest number of decode steps \(S\) that still leaves time to migrate:
- Read the symbols
- \(\pi\)
- an assignment of instances to positions (each position gets at most one instance)
- \(u_{ij}\)
- reusable bytes if instance \(i\) takes position \(j\) (layers it already holds that the position needs)
- \(T_{\text{grace}}, T_{\text{mig}}, t_{\text{step}}\)
- grace period, estimated migration time, time of one decode step
- input: current config, the instance that will be preempted, \(T_{\text{grace}}\)
- controller: choose the new config \((D, P, M, B)\) for the surviving instances // latency model + arrival-rate estimate
- device mapper: build the reuse matrix \(u_{ij}\) and solve the matching (137.8) with Kuhn–Munkres // \(O(n^3)\)
- estimate \(T_{\text{mig}}\) from the bytes to move; keep decoding for \(S\) steps (137.9) // tokens are committed one by one
- migrate weights and KV cache, front layers first; switch to the new config
- resume every interrupted request from its last committed token
The paper evaluates OPT-6.7B, GPT-20B and LLaMA-30B on AWS g4dn.12xlarge instances (4 T4 GPUs each) with a real 12-hour spot availability trace. It reports 2.4× to 9.1× lower P99 latency than the best existing baselines, and 54% lower cost than using only on-demand instances. [Certain] Figure 6 shows the device-mapping step on a small example.
How to read Figure 6
Each box is an instance (4 GPUs) with the layers it holds. Preempt instance #1 (pipeline A, layers 0–23). With 3 stages of 16 layers, the naive rule “survivors in id order take stages in order” (dotted row) puts #2, which holds layers 24–47, on stage 0 (layers 0–15): it must load everything. Naive reuses 24 layers and must load 24 (20 GB, 3.2 s). The best matching reuses 40 and loads only 8 (6.7 GB, 1.07 s), leaving more of the 30 s grace period for decoding. Preempt #2 instead: here the naive order happens to be optimal. Switch to 2 stages: no weights need to move at all, but one instance is idle; the controller weighs this against the latency of each config.
PyTorch: hand the KV cache from a prefill worker to a decode worker
The code below builds a small GQA transformer and runs generation twice. First colocated: one model does prefill and decode. Then disaggregated: a “prefill worker” runs the prompt and pushes each layer’s KV cache through a hook as soon as that layer finishes (layer-wise transfer, as in Algorithm 1); a separate “decode worker” with its own copy of the weights receives the KV cache and continues. With the same sampling seed, both give exactly the same tokens, and the bytes sent match equation 137.5. Everything runs on the CPU and is deterministic.
import torch, torch.nn.functional as F torch.manual_seed(0) V, d, L, nh, nkv = 1000, 256, 4, 8, 2 # vocab, width, layers, query heads, KV heads (GQA) hd = d // nh class Layer(torch.nn.Module): def __init__(s): super().__init__() s.q = torch.nn.Linear(d, nh * hd, bias=False); s.kv = torch.nn.Linear(d, 2 * nkv * hd, bias=False) s.o = torch.nn.Linear(d, d, bias=False); s.ff = torch.nn.Sequential(torch.nn.Linear(d, 4 * d), torch.nn.GELU(), torch.nn.Linear(4 * d, d)) s.n1, s.n2 = torch.nn.LayerNorm(d), torch.nn.LayerNorm(d) def forward(s, x, cache): # x: (T, d); cache: dict with "k", "v" of shape (nkv, t, hd) T, h = x.shape[0], s.n1(x) q = s.q(h).view(T, nh, hd).transpose(0, 1) k, v = s.kv(h).view(T, 2, nkv, hd).permute(1, 2, 0, 3) if "k" in cache: k, v = torch.cat([cache["k"], k], 1), torch.cat([cache["v"], v], 1) cache["k"], cache["v"] = k, v k, v = k.repeat_interleave(nh // nkv, 0), v.repeat_interleave(nh // nkv, 0) a = F.scaled_dot_product_attention(q, k, v, is_causal=(T > 1)) # causal only in prefill x = x + s.o(a.transpose(0, 1).reshape(T, d)) return x + s.ff(s.n2(x)) class Model(torch.nn.Module): def __init__(s): super().__init__(); s.emb = torch.nn.Embedding(V, d); s.layers = torch.nn.ModuleList(Layer() for _ in range(L)) s.nf, s.head = torch.nn.LayerNorm(d), torch.nn.Linear(d, V, bias=False) def forward(s, ids, caches, on_layer_done=None): x = s.emb(ids) for i, layer in enumerate(s.layers): x = layer(x, caches[i]) if on_layer_done: on_layer_done(i, caches[i]) # hook: push this layer's KV as soon as it exists return s.head(s.nf(x[-1])) # logits of the last position model = Model().eval() prompt = torch.randint(0, V, (300,)) def sample(logits, g): # same seed on both paths, so same draws return torch.multinomial(F.softmax(logits, -1), 1, generator=g)[0] @torch.no_grad() def colocated(n_new): # one GPU does prefill and decode g = torch.Generator().manual_seed(1) caches = [dict() for _ in range(L)]; out = []; tok = sample(model(prompt, caches), g) for _ in range(n_new): out.append(tok.item()); tok = sample(model(tok.view(1), caches), g) return out @torch.no_grad() def disaggregated(n_new): prefill_worker = model # in real life: another GPU with its own weight copy decode_worker = Model().eval(); decode_worker.load_state_dict(model.state_dict()) link = [] # stands in for NVLink / RDMA; we count the bytes def send(i, c): # layer-wise transfer (Splitwise, DistServe) link.append((i, c["k"].clone(), c["v"].clone())) g = torch.Generator().manual_seed(1) first = sample(prefill_worker(prompt, [dict() for _ in range(L)], on_layer_done=send), g) sent = sum(k.numel() * k.element_size() + v.numel() * v.element_size() for _, k, v in link) caches = [dict(k=k, v=v) for _, k, v in sorted(link, key=lambda t: t[0])] # decode side receives out, tok = [], first for _ in range(n_new): out.append(tok.item()); tok = sample(decode_worker(tok.view(1), caches), g) return out, sent a = colocated(20) b, sent = disaggregated(20) print("same tokens:", a == b) print("first tokens:", a[:8]) formula = 2 * L * nkv * hd * 4 * len(prompt) # 2 (K and V) x layers x KV heads x head dim x 4 bytes (FP32) x tokens print(f"KV bytes sent: {sent:,} formula: {formula:,} per token: {sent // len(prompt):,}")
Output (PyTorch 2.8 on CPU):
same tokens: True first tokens: [510, 164, 118, 375, 325, 2, 297, 923] KV bytes sent: 614,400 formula: 614,400 per token: 2,048
Read it like this. The KV cache is the complete state of a request after prefill: with it and the weights, any GPU can continue the generation and get the same result. That is why disaggregation and migration (SpotServe) work at all. The per-token size, 2,048 bytes here, is \(2 \times 4 \times 2 \times 32 \times 4\) (FP32), equation 137.5. In a real system the send hook becomes an RDMA write of paged KV blocks, and the decode side must not start until the last layer has arrived.
A planner in the style of DistServe’s Algorithm 2, in plain Python. It uses our cost model (not the paper’s simulator) for Llama 3.1 70B on H100s, the M/D/1 formula 137.2 for TTFT, and a memory-bound decode step for TPOT. It finds the goodput of each TP degree for each phase, then the best pair and the ratio of instances (equation 137.4):
import math P, L, d, KV_TOK = 70.55e9, 80, 8192, 327_680 # Llama 3.1 70B, BF16 KV bytes per token FLOPS, HBM, MEM, MFU = 989e12, 3.35e12, 80e9, 0.5 # one H100 SXM; MFU is our assumption S_IN, S_OUT = 2048, 512 # prompt and output length of the workload SLO_TTFT, SLO_TPOT = 0.5, 0.040 # seconds (average TTFT, per-token time) TP_EFF = {1: 1.0, 2: 0.95, 4: 0.90, 8: 0.85} # assumed scaling efficiency of tensor parallelism def prefill_time(tp): # D in equation 137.2 flops = 2 * P * S_IN + 2 * L * S_IN ** 2 * d return flops / (tp * FLOPS * MFU * TP_EFF[tp]) def prefill_goodput(tp): # max rate R with D + R D^2 / (2(1 - R D)) <= SLO D = prefill_time(tp) if D > SLO_TTFT: return 0.0 lo, hi = 0.0, 1 / D for _ in range(60): R = (lo + hi) / 2 lo, hi = (R, hi) if D + R * D * D / (2 * (1 - R * D)) <= SLO_TTFT else (lo, R) return lo def decode_goodput(tp): # largest batch that meets TPOT and fits in memory best = 0.0 for b in range(1, 1025): kv = b * (S_IN + S_OUT / 2) * KV_TOK # average context during decoding if (2 * P + kv) / tp > 0.9 * MEM: break t = max((2 * P + kv) / (tp * HBM), 2 * P * b / (tp * FLOPS * MFU)) / TP_EFF[tp] if t <= SLO_TPOT: best = b / (S_OUT * t) # finished requests per second return best print(" TP | prefill D(s) req/s | decode req/s") for tp in (2, 4, 8): print(f" {tp:2d} | {prefill_time(tp):7.3f} {prefill_goodput(tp):7.2f} | {decode_goodput(tp):8.2f}") best = None for tp_p in (2, 4, 8): for tp_d in (2, 4, 8): rp, rd = prefill_goodput(tp_p), decode_goodput(tp_d) if rp == 0 or rd == 0: continue per_gpu = 1 / (tp_p / rp + tp_d / rd) # fluid model: GPUs needed per req/s, inverted if best is None or per_gpu > best[0]: best = (per_gpu, tp_p, tp_d, rd / rp) g, tp_p, tp_d, ratio = best print(f"best: prefill TP{tp_p}, decode TP{tp_d}, {ratio:.2f} prefill instances per decode instance, {g:.3f} req/s per GPU")
Output:
TP | prefill D(s) req/s | decode req/s 2 | 0.313 1.73 | 0.26 4 | 0.165 4.85 | 15.89 8 | 0.088 10.32 | 44.49 best: prefill TP8, decode TP8, 4.31 prefill instances per decode instance, 1.047 req/s per GPU
Three lessons. Decode on TP2 is useless here: the 141 GB of weights leave almost no room for KV cache in 2 × 72 GB, so the batch stays tiny. Prefill goodput grows faster than linearly with TP (1.73 → 4.85 → 10.32 req/s), because a shorter \(D\) also shortens the queue (equation 137.2). And with 2,048-token prompts and 512 output tokens, we need about 4 prefill instances per decode instance: the ratio is a property of the workload, so it must be re-planned when the traffic changes.
Disaggregate or not?
| Situation | Better choice | Why |
|---|---|---|
| Strict TPOT target (voice, streaming chat) with long prompts | Disaggregate | Prefills never stall decodes (Figure 1) |
| Short prompts, small cluster, one node | Colocate with chunked prefill | Little interference; no transfer, no ratio to tune |
| Traffic mix changes quickly | Colocate, or disaggregate with role flipping / a mixed pool | A fixed P:D ratio is wrong most of the day (Figure 4) |
| Slow network between nodes, MHA model | Colocate, or keep each P–D pair inside one node | KV transfer can exceed the prefill time (equation 137.6, Figure 5) |
| Older and newer GPU generations available | Disaggregate: new GPUs for prefill, old ones for decode | Decode is memory-bound and gains little from more FLOPs (Splitwise) |
| Huge MoE model at scale | Disaggregate | Each phase wants a different expert-parallel layout (DeepSeek-V3) |
Where this is used
| Application | What is distributed | Notes |
|---|---|---|
| Large chat services (Kimi, DeepSeek) | Separate prefill and decode clusters, KV store or KV transfer over RDMA | Mooncake; DeepSeek-V3’s 32-GPU prefill and 320-GPU decode units [Certain] |
| Open-source serving engines | TP inside a node, PP or replicas across nodes; optional PD disaggregation | vLLM (experimental connectors), SGLang, TensorRT-LLM, Dynamo [Likely] |
| Long-document RAG and code assistants | Many prefill GPUs per decode GPU | Long prompts, short answers: \(r_P\) is small, so the ratio \(n_P / n_D\) is large |
| Video question answering and Video-RAG with a VLM | Prefill pool also runs the vision encoder over frames; decode pool writes the answer | Thousands of visual tokens per clip make prefill heavy and the KV cache large, so the transfer cost (137.5) matters; frame tokens of a shared video can be cached and reused |
| Live video captioning or key-moment alerts | Decode pool with strict TPOT; prefill of each new clip on separate GPUs | New clips arrive all the time; colocated prefills would stall the running captions |
| Offline batch jobs (labeling, video classification at scale, synthetic data) | Spot instances, replicas, large batches | Latency is loose, so preemptions are cheap to absorb; SpotServe-style token-level recovery avoids redoing work |
Common mistakes
- Expecting disaggregation to raise throughput. It improves goodput under latency targets. If you only care about raw tokens per second, a colocated system with large batches is often as good or better; vLLM’s documentation says so directly.
- Using pipeline parallelism to cut latency. A token still passes through every stage. PP adds capacity and works over slow links; for decode latency, use TP inside a node (Figure 2).
- Running TP across nodes. Two all-reduces per layer over InfiniBand cost more than the useful work at large batches. Keep TP groups inside the NVLink domain.
- Fixing the prefill-to-decode ratio once. The best ratio is \(r_D / r_P\), and both depend on prompt and output lengths, which change with the time of day and the product. Re-plan, flip roles, or keep a mixed pool.
- Ignoring the KV transfer on slow networks. Check \(t_x / t_p\) with equation 137.6 before you design. For MHA models or 25 Gbps links, place prefill and decode in the same node.
- Sending the KV cache only after the prefill. Serialized transfer adds \(t_x\) to the second token. Layer-wise transfer hides all but \(t_x / L\) when the link is fast enough.
- Reporting average TPOT only. Interference shows in the tail: the worst gap between tokens, or P99 TPOT. Averages can look equal while users see pauses (Figure 1).
- Treating a spot preemption like a crash. There is a grace period, and the KV cache is the state of the request. Commit tokens, migrate state, and remap devices to reuse what survivors already hold.
Interview questions
What is prefill–decode disaggregation, and what problem does it solve?basic
Why are prefill and decode so different on a GPU?basic
You serve Llama 3.1 70B in BF16 on H100s. TP or PP? How many GPUs?core
Define goodput. Why is it better than throughput for serving?core
How do you choose the number of prefill and decode instances?core
Estimate the KV-cache transfer for a 4,096-token prompt on Llama 3.1 70B. Is it a problem?core
Why does the ratio of transfer time to prefill time not depend on prompt length?advanced
For a prefill instance, when is 2-way tensor parallelism better than 2-way pipeline parallelism?advanced
What does Splitwise add beyond splitting the phases?core
What is TetriInfer’s approach, and what would you check before trusting its numbers?advanced
How would you serve an LLM on spot GPUs? What does SpotServe do?advanced
Design question: a chat API with 4,000-token prompts, 300-token answers, TTFT under 1 s and TPOT under 50 ms. Sketch the serving layout.advanced
Exercises
- Compute \(\kappa\) (equation 137.5) for Llama 3.1 8B with an FP8 KV cache, and the bytes for a 10,000-token prompt.
- With DistServe’s rates \(r_P = 5.6\) and \(r_D = 10\) req/s per GPU, what is the goodput per GPU for 3 prefill and 2 decode GPUs? Is it better than 2:1?
- Use equation 137.2 with \(D = 0.2\) s. At which arrival rate does the average TTFT reach 0.5 s?
- In equation 137.7, take \(L = 80\), \(t_p = 150\) ms and \(t_x = 100\) ms. What is the added delay, serialized and layer-wise? Repeat for \(t_x = 300\) ms.
- In Figure 2’s model, why does TP8×PP2 beat TP16 at \(B = 256\)? Name the term of equation 137.1 that grows.
- SpotServe: with a 30 s grace period, an estimated migration time of 3.2 s and a decode step of 0.25 s, how many steps does the just-in-time rule allow (137.9)?
- A team says: “disaggregation doubled our throughput”. What would you ask?
Answerscheck
Key takeaways
- For serving, tensor parallelism makes each token faster at the cost of \(2L\) all-reduces per token (137.1); keep it inside the NVLink domain. Pipeline parallelism adds capacity, not speed; use it, or replicas, across nodes.
- Prefill is compute-bound and decode is memory-bound. On a shared GPU, prefills stall decodes; disaggregation removes this interference and lets each phase pick its own parallelism, batch and hardware.
- Measure designs by per-GPU goodput under SLOs, not raw throughput. Balance the pools: \(n_P / n_D \approx r_D / r_P\) (137.4); the ratio depends on the workload. Queueing (137.2, 137.3) decides whether TP or PP is better for prefill.
- Splitwise adds cheaper or power-capped GPUs for decode; DistServe searches placements for goodput (7.4× more requests or 12.6× tighter SLOs); TetriInfer adds fixed-size chunks, length prediction and role flipping.
- The KV cache is \(2 L n_{\text{kv}} d_h b\) bytes per token (137.5); its transfer time relative to prefill is \(\kappa F_{\text{eff}} / (2 N B_{\text{net}})\) (137.6), about 2% for Llama 3.1 70B on 400 GB/s, and layer-wise sending hides most of it (137.7).
- Spot GPUs are cheap but can vanish. SpotServe re-plans the config, maps devices with a bipartite matching to reuse state (137.8), commits tokens one by one and uses the grace period (137.9).
Key formulas
References for this session
- Patel et al. (2023). Splitwise: Efficient Generative LLM Inference Using Phase Splitting
- Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
- Hu et al. (2024). Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads (TetriInfer)
- Miao et al. (2023). SpotServe: Serving Generative Large Language Models on Preemptible Instances
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Qin, Li, He et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- vLLM project (2025). vLLM documentation: Disaggregated Prefilling (experimental)
- NVIDIA (2025). NVIDIA Dynamo: a datacenter-scale distributed inference serving framework