The serving stack and the life of a request
A model file is not a service. Between a user and the GPU sit an engine (vLLM, TensorRT-LLM, SGLang), an API server, a router and an autoscaler. This session follows one request through all of them: tokenize, queue, schedule, prefill, decode, detokenize, stream. You learn where each millisecond of latency comes from, how streaming, cancellation and timeouts work, and how one GPU can serve many models and hundreds of LoRA adapters.
What you will learn
- The four layers of a serving stack (engine, API server, router, autoscaler), what each one owns, and which real systems fill each layer
- The seven stages in the life of a request, and a cost model that splits TTFT and end-to-end latency into network, queue, prefill and decode parts
- Why one decode step costs almost the same for 1 and for 32 sequences, and how the batch size trades per-user speed against GPU throughput
- How streaming works (server-sent events, incremental detokenization that never cuts a UTF-8 character), and how cancellation and timeouts free GPU memory
- How a router spreads load and why prefix-aware routing can cut prefill work by a large factor, and how to size replicas with Little’s law
- How to serve many models and many LoRA adapters on one base model: adapter size, batched LoRA kernels (Punica, S-LoRA), and vLLM’s flags
Why this matters
Until now we made the model faster: better kernels, fewer bytes, fewer steps. But a user never talks to a model. A user sends an HTTP request to a service, and that service has many parts besides the GPU. Each part can add latency, waste GPU time, or drop the request.
Think about what happens when you type a question into a chat product. Your text travels over the network. A router picks one of many machines. A web server checks your request, applies the chat template and turns text into token ids. The request waits in a queue until the engine has room for it in GPU memory. The engine runs the prompt through the model (prefill), then makes one token at a time (decode). Each new token is turned back into text and streamed to you, piece by piece. Meanwhile an autoscaler watches the queues and adds or removes machines. If you close the tab, all of this must stop, or the GPU keeps working for nobody.
The serving survey by Miao et al. calls this the system side of LLM serving: memory management, batching, scheduling and distributed execution around the model. [Certain] The rest of this track goes deep into each piece: memory in session 134, batching in session 135, scheduling in session 136, multi-GPU serving in session 137, and real code in session 138. This session is the map that connects them. In session 132 you made one decode step fast. Now you see everything that happens around that step.
The four layers of a serving stack
A production LLM service has four layers. Each one has a different job and a different time scale.
- Engine. The program that owns the GPUs and runs the model. It keeps the KV cache, decides which requests run in each step (the scheduler), and calls the kernels. Examples: vLLM, TensorRT-LLM, SGLang. It works on a time scale of milliseconds: one decode step.
- API server. The front door of one engine. It speaks HTTP (or gRPC), checks inputs, applies the chat template, tokenizes, streams the output back, and reports metrics. The most common interface is the OpenAI-compatible API (
/v1/chat/completions,/v1/completions):vllm serve <model>starts one. [Certain] NVIDIA’s Triton Inference Server is a general server that can host many models with different backends, including TensorRT-LLM and vLLM. [Certain] - Router (also called gateway or load balancer). It sits in front of many engine replicas (copies of the same model on different GPUs) and picks one for each request. A simple router uses round robin; an LLM-aware router looks at queue lengths and at which replica already has the prompt’s prefix in its KV cache. It also handles authentication, rate limits and retries.
- Autoscaler. It watches metrics (queue length, KV-cache use, latency) and changes the number of replicas. It works on a time scale of tens of seconds to minutes, because starting a replica means loading many gigabytes of weights.
How to read Figure 1
There is one router for many replicas, and one API server per engine (vLLM runs it as a separate process that talks to the engine core). [Certain] The autoscaler is on the side because it is not in the path of a request: it only changes how many replicas exist. Notice which layer owns which resource: only the engine knows about GPU memory and the KV cache, so only the engine can decide whether a request fits.
Inside vLLM (version 1) these layers map to separate processes: one API server process handles HTTP, tokenization and streaming; one engine core process runs the scheduler and manages the KV cache; and one worker process per GPU runs the model. They talk over ZeroMQ. [Certain] Splitting them means that slow Python work in the server (tokenizing a long prompt, building JSON) does not stop the GPU loop.
The life of a request, step by step
Here are the seven stages every request goes through. We give names to the times, because every latency metric is a sum of them.
- Tokenize. The server checks the JSON, applies the chat template (session 9) and converts text to token ids. For a VLM it also decodes and resizes the images. Time \(t_{\text{tok}}\).
- Queue. The request waits in the engine’s waiting queue. Time \(t_{\text{queue}}\). This is the part that explodes when the service is overloaded.
- Schedule. At the start of every engine step, the scheduler picks which waiting requests to admit. A request is admitted only if there is enough free KV-cache memory for it. The policy (first come first served, priorities, preemption) is the topic of session 136.
- Prefill. The engine runs all prompt tokens through the model in one pass, fills the KV cache, and produces the first output token. Time \(t_{\text{prefill}}\). This is compute-bound (session 82).
- Decode. The engine makes one token per step for every running request. Each step reads all the weights and each request’s KV cache. The time of one step is the time per output token, TPOT.
- Detokenize. Each new token id is converted back to text, carefully, because one character can be split over several tokens.
- Stream. Each new piece of text is sent to the client right away, as a server-sent event. When a stop condition is met (end-of-sequence token,
max_tokens, a stop string), the engine frees the request’s KV cache and the server sends a final message.
From these stages we get the two latency numbers that users feel (session 81). The time to first token (TTFT) is everything until the first piece of text arrives:
and the end-to-end latency for \(N_{\text{out}}\) output tokens adds one decode step per remaining token:
- Read the symbols
- \(t_{\text{net}}^{\uparrow}, t_{\text{net}}^{\downarrow}\)
- network time from client to server and back (one way each)
- \(t_{\text{tok}}\)
- time to validate, apply the template and tokenize (plus image preprocessing for a VLM)
- \(t_{\text{queue}}\)
- time waiting in the engine queue until the scheduler admits the request
- \(t_{\text{prefill}}\)
- time of the forward pass over the whole prompt
- \(N_{\text{out}}\)
- number of output tokens
- TPOT
- time per output token: the duration of one decode step of the batch the request is in
The first output token comes out of the prefill pass itself, so only \(N_{\text{out}} - 1\) decode steps follow. Detokenization and sending each chunk overlap with the next decode step, so in a good server they do not add to TPOT.
A cost model for prefill and decode
To put numbers in (133.1) and (133.2), we use the roofline view from session 82. Prefill does about \(2P\) FLOPs per prompt token, where \(P\) is the number of parameters, and it is compute-bound:
A decode step for a batch of \(B\) sequences must read all weights once plus every sequence’s KV cache, and it does \(2PB\) FLOPs. It takes the larger of the memory time and the compute time, plus a fixed cost \(t_0\) for scheduling and kernel launches:
where the KV cache needs \(s_{kv}\) bytes per token (session 80):
- Read the symbols
- \(P, M_W\)
- number of parameters, and their size in bytes (\(2P\) in BF16)
- \(n_{\text{in}}\)
- number of prompt tokens
- \(F, u\)
- peak FLOP/s of the GPU, and the fraction of it we really get (model FLOPs utilization)
- \(\text{BW}, \eta\)
- peak memory bandwidth, and the fraction of it we really get
- \(B, \bar{L}\)
- batch size (sequences decoded together), and their average context length
- \(t_0\)
- fixed cost per step (CPU scheduling, launches, sampling)
- \(n_{kv}, d_{\text{head}}, b\)
- number of KV heads, size of one head, bytes per number; the 2 is for K and V
Our running example is Llama 3 8B in BF16 on one H100 SXM. From its config: 32 layers, hidden size 4096, FFN size 14,336, 8 KV heads of size 128, vocabulary 128,256. [Certain] That gives \(P = 8.03 \times 10^9\) parameters, \(M_W = 16.06\) GB, and \(s_{kv} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB) per token. The H100 SXM has \(F = 989\) TFLOP/s dense BF16 and \(\text{BW} = 3.35\) TB/s. [Certain] We assume \(u = 0.5\), \(\eta = 0.8\) and \(t_0 = 1\) ms. These three are assumptions, not measurements; real values depend on the engine and the kernels.
A request with \(n_{\text{in}} = 1000\) prompt tokens asks for \(N_{\text{out}} = 200\) tokens. It joins a batch of \(B = 32\) running requests. Assume 10 ms network each way, 1 ms to tokenize, and 50 ms in the queue.
Prefill (133.3): \(2 \cdot 8.03\times10^9 \cdot 1000 / (989\times10^{12} \cdot 0.5) = 32.5\) ms. So TTFT (133.1) \(= 10 + 1 + 50 + 32.5 + 10 = 103.5\) ms. Half of it is queueing; the GPU work is less than a third.
Decode (133.4) with \(\bar{L} = 1100\) (the average context: 1,000 prompt tokens plus half of the 200 output tokens): memory time \((16.06 + 32 \cdot 1100 \cdot 131{,}072 \times 10^{-9})\) GB \(/ (0.8 \cdot 3.35\) TB/s\() = 7.71\) ms; compute time \(2 \cdot 8.03\times10^9 \cdot 32 / (4.945\times10^{14}) = 1.04\) ms. The step is memory-bound: TPOT \(= 1 + 7.71 = 8.71\) ms, which is 115 tokens per second for this user.
End to end (133.2): \(103.5 + 199 \cdot 8.71 = 1{,}838\) ms. The 199 decode steps are 94% of the total. The GPU makes \(32 / 8.71\text{ ms} = 3{,}672\) tokens per second for the whole batch.
Step through the same request in Figure 2 to see which component holds it at each moment.
How to read Figure 2
The highlighted lane holds the request in this step. The readout gives the stage, what happens in it, and the clock time when it ends. Two things to notice. First, the API server is busy twice: before the engine (tokenize) and after every step (detokenize, send). Second, after prefill the request does not “own” the GPU: it is one row of a batch of 32, and every step serves all of them.
The batch size: per-user speed against GPU throughput
Look again at (133.4). At small batch sizes the step is memory-bound, and most of the bytes are the weights, which are read once per step no matter how many sequences are in the batch. So adding sequences is almost free: at \(B = 1\) the step takes 7.05 ms, at \(B = 32\) it takes 8.71 ms. The GPU makes 142 tokens per second at \(B = 1\) and 3,672 at \(B = 32\), 26 times more for 24% more latency per token. This is why every engine batches (session 84).
The GPU throughput in tokens per second is
But the free lunch ends. Each sequence adds its KV cache to the bytes read, so the step slowly gets longer: at \(B = 256\) it is 20.8 ms, and each user sees only 48 tokens per second. And the KV cache must fit in memory: with 90% of 80 GB usable, minus 16.06 GB of weights, about 55.9 GB is left, which holds 355 sequences of 1,200 tokens. Long prompts change everything: with 8,000-token prompts only 52 sequences fit. Explore it in Figure 3.
How to read Figure 3
Start at the defaults: decode dominates the bar. Move the batch slider from 1 to 256: the throughput curve rises steeply, then flattens, while the decode part of the bar grows. Now push the prompt to 8,000 tokens: prefill grows (TTFT goes over 300 ms) and the readout warns when the batch no longer fits in KV memory. Finally move the queue slider: queueing adds directly to TTFT, which is why overload shows up first as slow first tokens.
Streaming
A user reads about as fast as a few words per second, so waiting 1.8 s for the full answer feels slow, while seeing the first words after 0.1 s feels fast. That is why chat services stream: they send each new piece of text as soon as it exists.
The OpenAI-compatible API streams with server-sent events (SSE): one HTTP response stays open with content type text/event-stream, and the server writes lines of the form data: {...}, each followed by a blank line. Each chunk is a JSON object with "object": "chat.completion.chunk" and the new text in choices[0].delta.content. The last content chunk carries a finish_reason such as "stop" or "length", and the stream ends with data: [DONE]. If the client sets stream_options: {"include_usage": true}, one extra chunk with token counts and an empty choices list comes at the end. [Certain]
Incremental detokenization
Streaming has a trap. Byte-level BPE tokenizers (session 8) can split one character over several tokens. The emoji 👍 is four UTF-8 bytes (F0 9F 91 8D); if the model emits it as two tokens, the first token alone is not valid text. Decoding each token separately gives the replacement character � (U+FFFD). Also, some tokenizers add or remove spaces depending on the neighbors, so decoding one token alone can give the wrong spacing.
The fix is incremental detokenization: decode a short window of recent tokens together, compare with the text already sent, and send only the new part. If the new text ends with �, the last character is not finished yet, so send nothing and wait for the next token. vLLM does exactly this: its detokenize_incrementally keeps two offsets into the token list (a prefix_offset where the window starts and a read_offset where unsent text starts) and returns an empty string while the text ends with U+FFFD. [Certain] It also holds back the last few characters when the request has stop strings, so that a stop string that arrives in pieces is never shown to the user. [Certain]
- state per request: token list \(T\), offsets \(p\) (window start) and \(r\) (first unsent token), sent text \(S\)
- on each new token \(t\): append \(t\) to \(T\)
- \(a \leftarrow \text{decode}(T[p:r])\) // text already sent, decoded with its left context
- \(b \leftarrow \text{decode}(T[p:])\) // the same window plus the new tokens
- if \(b\) ends with U+FFFD or \(|b| \le |a|\): return nothing // unfinished character: wait
- \(\delta \leftarrow b[|a|:]\); \(\;p \leftarrow r\); \(\;r \leftarrow |T|\); append \(\delta\) to \(S\)
- if a stop string occurs in the new part of \(S\): cut \(S\) at it, finish with reason “stop”
- send \(S\) minus its last \(\ell_{\max} - 1\) characters (not yet sent) // \(\ell_{\max}\) = longest stop string
How to read Figure 4
The byte C3 starts a two-byte character (é), so alone it is not text: the naive column shows �, while the incremental column sends nothing and waits. One step later A9 completes the character and é is sent. The emoji needs four bytes spread over two tokens, so the same thing happens again. The user sees the same final text in both columns only if nothing is shown until the end; with streaming, only the right column is correct.
Two more streaming details matter in practice. Backpressure: if a client reads slowly, chunks pile up in the server’s buffers; a server should not let one slow reader block the engine loop, so the engine writes into a per-request queue and the HTTP layer drains it. Batching of chunks: sending one HTTP write per token costs CPU time for every token of every user; some servers can combine several tokens into one chunk to save it. [Likely]
Cancellation and timeouts
Requests end in four ways: the model emits the end-of-sequence token, the request reaches max_tokens, a stop string appears, or the request is cancelled. Cancellation is the one people forget, and it is common: users press “stop”, close the tab, or the client library gives up.
When a client disconnects, the engine does not know it. If nobody tells it, the request keeps decoding up to max_tokens, holding a batch slot and its KV cache for nobody. With \(N_{\text{out}} = 200\) and TPOT of 8.71 ms, that is 1.7 s of a batch slot wasted per abandoned request, and its KV cache (here 157 MB for 1,200 tokens) blocks a waiting request from being admitted. So the server must detect the disconnect and abort the request in the engine. In vLLM, the HTTP handler waits on two tasks at once, the real work and a listener for the http.disconnect message; when the client leaves, the work task is cancelled, and the engine’s generate() catches the cancellation and calls abort, which frees the request’s KV blocks at the next step. [Certain]
- input: request \(q\), deadline \(D\)
- tokenize \(q\); if the waiting queue is too long: reply HTTP 429 (“try later”) // admission control
- engine.add(\(q\)); start a task that waits for the client to disconnect
- try:
- loop: wait for the next output of \(q\), but not past \(D\)
- \(\delta \leftarrow\) incremental detokenize (Algorithm 1); if \(\delta\) is not empty: send
data: chunk - if \(q\) is finished: send the final chunk (finish reason, usage) and
data: [DONE]; break - on disconnect or deadline: stop waiting // both arrive as a cancellation
- finally: engine.abort(\(q\)) // always runs; aborting a finished request does nothing
- at its next step the engine drops \(q\) from the batch and returns its KV blocks to the free pool
Timeouts protect the system from requests that would take too long. Use separate limits for separate stages: a queue timeout (if a request waited 30 s, the user is gone anyway; reject it before spending GPU time on prefill), a limit on time to first token, and an overall deadline. Clients should also send a sensible max_tokens, which is the cheapest timeout of all. And a request that times out must be aborted in the engine, exactly like a disconnect, or the timeout only hides the waste.
Routing across replicas
One engine replica has a limit. To serve more traffic you run many replicas and put a router in front. The router must answer one question per request: which replica?
- Round robin: replica 1, 2, 3, 4, 1, 2, … Simple, but blind: LLM requests differ in cost by 1,000 times (a 50-token chat turn against a 50,000-token document).
- Least loaded: send to the replica with the fewest queued tokens or running requests. A cheap variant, “power of two choices”, picks two replicas at random and takes the less loaded one.
- Prefix-aware (cache-aware): many requests share a long prefix, such as a system prompt, a few-shot block, or an uploaded document. If a replica already has that prefix in its KV cache, it can skip that part of the prefill (session 134). So send the request to a replica that has the prefix, unless that replica is much busier than the others. The SGLang router has such a cache-aware policy, with a fallback to load balancing. [Certain]
Figure 5 simulates 600 requests from 16 “tenants” (for example, apps that each have their own 1,000-token system prompt) on 4 replicas. Each replica can keep a few prefixes in its cache.
How to read Figure 5
The bars show the total work each replica did; the orange part is prefill. Round robin and least loaded spread tenants over all replicas, so each replica must cache all of them and misses often. Prefix-aware routing gives each tenant a “home” replica, so the hit rate jumps and the orange parts shrink. The readout’s “offered load” is work per second divided by capacity: above 100%, queues grow without limit. Notice that at the default settings only the prefix-aware router keeps the load under 100%. Cache hits are capacity. Then move the balance slack (prefix-aware only). At 0 the router leaves a request’s home replica as soon as it is busier than the others: load is even, but the hit rate drops. Larger slack keeps requests at home: more hits, but the home replicas of popular tenants build queues. The best wait is in between.
The rule the simulation uses for the prefix-aware case is: among replicas that hold the prefix, take the least loaded one, but only if its queue is at most twice the shortest queue plus a slack (in units of one request’s work); otherwise take the least loaded replica. Real routers use the same idea with their own thresholds, and their flags change between versions, so check the documentation of the version you run. [Likely]
Autoscaling and capacity
How many replicas do you need? Two simple laws answer most of it.
Little’s law connects the average number of requests in the system \(L\), the arrival rate \(\lambda\), and the average time each request spends inside \(W\). It holds for any stable system, whatever the details of the scheduler:
GPU time per request. Each request needs its own prefill, and a \(1/B\) share of each of its \(N_{\text{out}}\) decode steps. If the replicas run at an operating batch size \(B\), one request costs \(s\) GPU-seconds, and with a target utilization \(\rho\) you need \(R\) replicas:
- Read the symbols
- \(\lambda\)
- arrival rate, requests per second
- \(W\), \(L\)
- average time a request spends in the system, and the average number of requests inside it
- \(s\)
- GPU-seconds one request uses
- \(\rho\)
- target utilization, below 1 so that bursts do not overflow the queue (0.6 to 0.8 is common [Likely])
- \(R\)
- number of replicas
Traffic: \(\lambda = 20\) requests per second, each with 1,000 prompt and 200 output tokens. We allow each replica a batch of at most \(B = 64\), so \(t_{\text{step}} = 10.44\) ms by (133.4).
GPU time per request (133.8): \(s = 32.5 + 200 \cdot 10.44 / 64 = 32.5 + 32.6 = 65.1\) ms. Half is prefill: long prompts are expensive even though decode takes most of the wall-clock time. Load \(\lambda s = 20 \cdot 0.0651 = 1.30\) GPUs. With \(\rho = 0.7\): \(R = \lceil 1.30 / 0.7 \rceil = \lceil 1.86 \rceil = 2\) replicas.
Check with Little’s law (133.7): \(W \approx 32.5 + 200 \cdot 10.44 = 2{,}120\) ms, so \(L = 20 \cdot 2.12 = 42\) requests are inside at any moment, about 21 per replica: below the batch limit of 64, so the plan is consistent. One prefill-only bound is also worth knowing: at \(u = 0.5\), one H100 prefills only about 30,800 tokens per second of this model, so 20,000 prompt tokens per second already use 65% of a GPU.
What to scale on. GPU utilization as reported by nvidia-smi is a poor signal: it only says a kernel was running, not how much work was done (session 123). Better signals come from the engine itself. vLLM exports Prometheus metrics such as vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc, and histograms of vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds. [Certain] A common policy: scale up when the waiting queue stays above a threshold or the p95 TTFT breaks its target, scale down slowly when the queue is empty.
Cold start. A new replica must get a GPU, start the container, load 16 GB of weights (at 1 GB/s from network storage that alone is 16 s), and warm up (for example capture CUDA graphs, session 127). So autoscaling reacts in tens of seconds to minutes, not milliseconds. Keep some headroom (the \(\rho\) in 133.8), cache weights on local disk, and scale on queue growth early rather than on latency after it is already bad.
Serving many models and many LoRA adapters
Real services rarely have one model. A platform may host ten base models, and hundreds of customers who each fine-tuned their own variant. Copying full models does not scale: one 8B model in BF16 is 16.06 GB, so one 80 GB GPU holds at most four copies, with no room left for KV cache. Moving a full model in from CPU memory over PCIe Gen5 x16 (about 64 GB/s per direction, theoretical [Likely]) takes at least 0.25 s, far too slow to do per request.
For many models of different families there are three options: give each its own replicas (simple, wasteful when traffic is uneven), put several small models on one GPU (Triton Inference Server can host many models and load or unload them through its model-control API [Certain]), or route by model name in the router to separate pools. For many fine-tuned variants of one base model, there is a much better option: LoRA.
One base model, many adapters
A LoRA adapter (session 57) changes a weight \(W\) by a low-rank product \(BA\) with rank \(r\). The base weights stay the same for every customer, so a server keeps one copy of the base model and many small adapters. Request \(i\) uses adapter \(a(i)\):
Each adapted layer with input size \(d_{\text{in}}\) and output size \(d_{\text{out}}\) adds \(r(d_{\text{in}} + d_{\text{out}})\) parameters, so for \(n_{\text{layers}}\) blocks with a set \(\mathcal{M}\) of adapted matrices:
- Read the symbols
- \(W\)
- the shared base weight of one layer (one copy for all requests)
- \(A_{a}, B_{a}\)
- the down-projection (\(r \times d_{\text{in}}\)) and up-projection (\(d_{\text{out}} \times r\)) of adapter \(a\)
- \(a(i)\)
- the adapter chosen by request \(i\) (in vLLM, by the
modelfield of the request) - \(\alpha / r\)
- the LoRA scale factor
- \(\mathcal{M}\)
- the adapted matrices, for example q, k, v, o only, or all seven linear layers of a block
Llama 3 8B, rank 16, all seven linear layers per block. Per block: q and o \(16 \cdot (4096 + 4096) = 131{,}072\) each; k and v \(16 \cdot (4096 + 1024) = 81{,}920\) each; gate, up and down \(16 \cdot (4096 + 14{,}336) = 294{,}912\) each. Sum: 1,310,720. Times 32 blocks: \(p_{\text{LoRA}} = 41{,}943{,}040\) parameters, 83.9 MB in BF16, which is 0.52% of the base model.
So 100 such adapters take 8.4 GB, and 1,000 take 84 GB: they fit in CPU memory, not all on the GPU. Loading one over PCIe at 64 GB/s takes about 1.3 ms. One hundred full fine-tuned copies would take 1,606 GB.
The hard part is not memory but batching. The batch now mixes requests for different adapters. The base product \(W\mathbf{x}\) is shared, so it stays one big GEMM for the whole batch. The adapter part is different for each row. Running one small GEMM per adapter is slow, and merging \(BA\) into \(W\) works only if the whole batch uses one adapter. Punica introduced a CUDA kernel that computes the adapter products for a batch where each row has its own adapter (Segmented Gather Matrix-Vector multiplication, SGMV), keeping one copy of the base model; the paper reports 12 times the throughput of previous systems on multi-LoRA workloads and 2 ms of extra latency per token. [Certain] S-LoRA keeps all adapters in CPU memory, moves the needed ones to the GPU, and stores adapter weights and KV cache in one paged memory pool (“unified paging”) to avoid fragmentation; it reports serving thousands of adapters on one GPU and up to 4 times the throughput of Hugging Face PEFT and vLLM’s early LoRA support. [Certain]
In vLLM you start the server with --enable-lora, register adapters with --lora-modules name=path, and the client picks one by putting its name in the model field. --max-loras sets how many adapters can be in one batch at the same time, --max-lora-rank must be at least the largest rank you serve (a much larger value wastes memory), and --max-cpu-loras sets how many are kept in CPU memory [Likely]. Adapters can also be loaded at runtime through /v1/load_lora_adapter when the environment variable VLLM_ALLOW_RUNTIME_LORA_UPDATING is set, which the docs warn against outside trusted environments. [Certain]
How to read Figure 6
At the defaults (rank 16, all layers, 8 slots) the adapter slots take 0.67 GB, which costs about 5,100 tokens of KV cache: almost nothing next to 55.9 GB. Raise the rank to 128 and the slots to 64: now adapters take 43 GB and the KV cache shrinks badly, so fewer requests fit in a batch. The number of adapters served only changes CPU memory, not GPU memory: that is the point of keeping a small set of slots on the GPU and the rest on the host.
Code: a tiny engine with streaming, cancellation and batched LoRA
The first program is a working miniature of the whole path: a request object, an engine that batches requests and runs one step at a time, incremental detokenization, a server function that yields SSE lines, a client that disconnects, and one that times out. The “model” is scripted so that the output is deterministic, but it runs one batched forward pass per step like a real one.
import asyncio, json, torch torch.manual_seed(0) EOS, VOCAB = 256, 257 # byte-level vocabulary: 256 bytes + an end token class ScriptedLM(torch.nn.Module): """Stands in for the real model: one batched forward pass per step, returns logits. Each request 'wants' to say a fixed reply, so the output is deterministic.""" def forward(self, reqs): nxt = [] for r in reqs: want = r.reply_ids + [EOS] nxt.append(want[min(len(r.out_ids), len(want) - 1)]) return torch.nn.functional.one_hot(torch.tensor(nxt), VOCAB).float() # [B, VOCAB] class Request: def __init__(self, rid, prompt, reply, max_tokens): self.rid, self.prompt_ids = rid, list(prompt.encode("utf-8")) # "tokenize" self.reply_ids, self.max_tokens = list(reply.encode("utf-8")), max_tokens self.out_ids, self.sent, self.aborted = [], 0, False self.queue = asyncio.Queue() # engine -> server def detok_delta(r): """Incremental detokenization: only emit text that is complete UTF-8.""" text = bytes(r.out_ids).decode("utf-8", errors="replace") if text.endswith("\ufffd"): # unfinished multi-byte character: hold it back return "" delta, r.sent = text[r.sent:], len(text) return delta class Engine: def __init__(self, model, max_batch=2): self.model, self.max_batch, self.waiting, self.running = model, max_batch, [], [] def add(self, r): self.waiting.append(r) def abort(self, r): r.aborted = True def step(self): self.running = [r for r in self.running if not r.aborted] # free KV of aborted requests self.waiting = [r for r in self.waiting if not r.aborted] while self.waiting and len(self.running) < self.max_batch: # FCFS admission self.running.append(self.waiting.pop(0)) # (prefill would run here) if not self.running: return nxt = self.model(self.running).argmax(-1).tolist() # one step for the whole batch still = [] for r, t in zip(self.running, nxt): if t != EOS: r.out_ids.append(t) done = t == EOS or len(r.out_ids) >= r.max_tokens r.queue.put_nowait((detok_delta(r), "stop" if t == EOS else ("length" if done else None))) if not done: still.append(r) self.running = still async def engine_loop(engine): while True: engine.step() await asyncio.sleep(0.01) # one "decode step" = 10 ms async def stream(engine, r, timeout): """Server side of one request: yields SSE lines; always aborts on exit.""" engine.add(r) deadline = asyncio.get_running_loop().time() + timeout try: while True: left = deadline - asyncio.get_running_loop().time() delta, finish = await asyncio.wait_for(r.queue.get(), max(left, 0)) if not delta and not finish: continue # nothing printable yet: send nothing chunk = {"id": r.rid, "object": "chat.completion.chunk", "choices": [{"index": 0, "delta": {"content": delta}, "finish_reason": finish}]} yield "data: " + json.dumps(chunk, ensure_ascii=False) if finish: yield "data: [DONE]" return finally: engine.abort(r) # runs on normal end, client disconnect, and timeout async def client(engine, r, timeout=5.0, read_max=None): got, n = [], 0 agen = stream(engine, r, timeout) try: async for line in agen: got.append(line); n += 1 if read_max and n >= read_max: break # client disconnects except asyncio.TimeoutError: got.append("timeout") finally: await agen.aclose() return got async def main(): eng = Engine(ScriptedLM(), max_batch=2) loop_task = asyncio.create_task(engine_loop(eng)) a = Request("a", "Hi", "Café 👍", max_tokens=50) b = Request("b", "Hello", "a long answer nobody reads", max_tokens=50) c = Request("c", "Hey", "this one is too slow", max_tokens=50) ra, rb, rc = await asyncio.gather(client(eng, a), client(eng, b, read_max=3), client(eng, c, timeout=0.08)) loop_task.cancel() print(ra[0]) print([json.loads(l[6:])["choices"][0]["delta"]["content"] for l in ra[:-1]], ra[-1]) print("b: client left after", len(rb), "chunks; engine stopped early:", len(b.out_ids) < len(b.reply_ids)) print("c:", rc[-1], "| aborted:", c.aborted, "| stopped early:", len(c.out_ids) < len(c.reply_ids)) asyncio.run(main())
The output: request a streams ['C', 'a', 'f', 'é', ' ', '👍', ''] and then data: [DONE], with no broken characters, because the half-finished é and 👍 are held back for one step. Request b reads 3 chunks and leaves, and the engine stops it early. Request c waits in the queue (the batch holds only 2), hits its deadline, and is aborted. The second program checks that batched multi-LoRA (133.9), with a gathered adapter per row and one shared base GEMM, gives the same result as merging each adapter separately, and computes the adapter size (133.10).
import torch torch.manual_seed(0) d_in, d_out, r, n_adapters = 64, 48, 8, 3 alpha = 16.0 W = torch.randn(d_out, d_in) / d_in ** 0.5 # shared base weight (one copy) A = torch.randn(n_adapters, r, d_in) * 0.1 # LoRA down-projections, one per adapter Bu = torch.randn(n_adapters, d_out, r) * 0.1 # LoRA up-projections x = torch.randn(5, d_in) # 5 requests in one batch idx = torch.tensor([0, 2, 2, -1, 1]) # adapter of each request; -1 = base model only def lora_batched(x, idx): y = x @ W.T # ONE base GEMM for the whole batch m = idx >= 0 h = torch.einsum("brd,bd->br", A[idx[m]], x[m]) # gather each row's A: shrink to rank r y[m] += (alpha / r) * torch.einsum("bor,br->bo", Bu[idx[m]], h) # expand back return y def lora_one_by_one(x, idx): # reference: merge weights per request rows = [] for xi, i in zip(x, idx.tolist()): Wi = W if i < 0 else W + (alpha / r) * Bu[i] @ A[i] rows.append(Wi @ xi) return torch.stack(rows) print((lora_batched(x, idx) - lora_one_by_one(x, idx)).abs().max().item()) # tiny: same math # size of one adapter for Llama 3 8B, rank 16 on all 7 linear layers of the 32 blocks shapes = {"q": (4096, 4096), "k": (4096, 1024), "v": (4096, 1024), "o": (4096, 4096), "gate": (4096, 14336), "up": (4096, 14336), "down": (14336, 4096)} p = 32 * sum(16 * (a + b) for a, b in shapes.values()) print(p, "parameters,", 2 * p / 1e6, "MB in BF16") # 41943040 parameters, 83.88608 MB
Where this is used
| Setting | What the stack must do | Main lever |
|---|---|---|
| Chat assistant | Low TTFT, steady streaming, many users; frequent “stop” presses | Streaming with incremental detokenization; abort on disconnect; scale on queue length and p95 TTFT |
| Coding assistant in an IDE | Very short deadlines; most completions are cancelled because the user keeps typing | Aggressive cancellation and small max_tokens; prefix-aware routing on the shared file context |
| Multi-tenant fine-tuning platform | Hundreds of customer variants of a few base models | Multi-LoRA serving (Punica, S-LoRA, vLLM --enable-lora), adapter cache on CPU |
| Agents and tool use | Many calls that share a long system prompt and history | Prefix-aware routing so the shared prefix is prefilled once per replica (session 141) |
| Video-RAG and video question answering | Very long multimodal prompts (frames as thousands of visual tokens) and often the same video asked many times | Route all questions about one video to the replica that has its prefix cached; long prefill makes TTFT the key metric |
| Batch jobs (video captioning, action-recognition labels, key-moment summaries) | Millions of requests, no user waiting | Large batch sizes for throughput (133.6), offline mode instead of the HTTP server, no streaming |
Common mistakes
- Not aborting on disconnect or timeout. The GPU keeps generating for nobody and the KV cache stays full. Test it: start a long request, kill the client, and watch
vllm:num_requests_running. - Decoding each token separately for streaming. Users see � in the middle of emoji and non-English text, or wrong spaces. Use the engine’s incremental detokenizer.
- Autoscaling on GPU utilization. A GPU decoding one sequence shows high utilization while doing a tiny fraction of the work it could. Scale on waiting requests, KV-cache use and latency percentiles.
- Round-robin routing for workloads with shared prefixes. Every replica must prefill every system prompt; prefix-aware routing can cut prefill work by several times (Figure 5).
- One timeout for everything. Without a queue timeout, an overloaded server spends GPU time on prefill for requests whose users already left.
- Setting
--max-lora-rankor--max-lorasfar above need. Slots are reserved GPU memory; that memory is taken from the KV cache and lowers the batch size (Figure 6). - Measuring latency only as an average. Users feel the tail. Track p50, p95 and p99 of TTFT and TPOT, separately (session 81).
Interview questions
Name the layers of an LLM serving stack and what each one owns.basic
Walk through the life of a request in a chat service.basic
/v1/chat/completions with stream: true. The router picks a replica. The API server validates, applies the chat template and tokenizes. The request waits in the engine’s queue until the scheduler admits it, which needs free KV-cache blocks. Prefill runs the prompt and produces the first token. Then decode adds one token per step, batched with other requests. Each token is detokenized incrementally and sent as an SSE chunk. On a stop condition the engine frees the KV cache, and the server sends a final chunk with the finish reason (and usage, if asked) and data: [DONE].Break down time to first token. Which part grows under overload?core
Why does going from batch 1 to batch 32 barely change TPOT?core
What happens when a streaming client disconnects, and what should happen?core
max_tokens, holding a batch slot and its KV cache. The server must detect the disconnect (vLLM races the handler against a listener for the http.disconnect message and cancels the handler), and the cancellation must reach the engine as an abort, which removes the request from the batch and frees its KV blocks at the next step. The same must happen on timeouts.Why can you not just decode each new token to text and send it?core
What metric would you autoscale an LLM service on, and why not GPU utilization?core
Size a deployment: 20 requests/s, 1,000 prompt and 200 output tokens, Llama 3 8B on H100s.advanced
How does prefix-aware routing work, and what is its risk?advanced
How do you serve 500 fine-tuned variants of the same 8B model?advanced
Why does vLLM split the API server, the engine core and the GPU workers into separate processes?advanced
Exercises
- With the cost model, compute TTFT for a 4,000-token prompt with 0 ms queue time (10 ms network each way, 1 ms tokenize).
- A request is abandoned after 20 output tokens but runs until
max_tokens= 1,000 at a TPOT of 10 ms. How many seconds of batch slot are wasted? If 5% of 20 requests per second are abandoned like this, how many batch slots are blocked on average? - Compute the size of a rank-8 adapter on q, k, v and o only for Llama 3 8B, in parameters and MB (BF16).
- In Figure 5, why does increasing “prefixes cached per replica” to 16 help round robin much more than it helps prefix-aware routing?
- A service has \(\lambda = 5\) requests per second and an average end-to-end time of 6 s. How many requests are in the system on average? If each replica can run 32 at once, what is the minimum number of replicas?
- The emoji “😀” is
F0 9F 98 80. If it arrives as three tokens (F0,9F 98,80), what does incremental detokenization send after each one?
Answerscheck
F0: nothing (the text ends with U+FFFD). After 9F 98: still nothing (three of four bytes). After 80: “😀”.Key takeaways
- A serving stack has four layers: engine (GPU, KV cache, scheduler), API server (protocol, tokenizer, streaming), router (which replica) and autoscaler (how many replicas).
- A request goes through tokenize, queue, schedule, prefill, decode, detokenize and stream. TTFT (133.1) is network + tokenize + queue + prefill; E2E (133.2) adds one TPOT per further token.
- Decode is memory-bound, so batching is almost free until KV reads, compute or KV memory become the limit (133.4, 133.6): 142 tokens/s at batch 1 against 3,672 at batch 32 in our model.
- Streaming needs incremental detokenization; cancellation and timeouts must reach the engine as aborts, or the GPU works for nobody.
- Prefix-aware routing turns cache hits into capacity; size replicas with GPU time per request and Little’s law (133.7, 133.8), and autoscale on queues, not on GPU utilization.
- Many variants of one base model are served as LoRA adapters (133.9, 133.10): 84 MB each for rank 16 on Llama 3 8B, batched together with shared base GEMMs.
Key formulas
References for this session
- Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
- vLLM project (docs). vLLM documentation
- Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
- vLLM project (docs). vLLM documentation
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
- NVIDIA (code). TensorRT-LLM
- NVIDIA (code). Triton Inference Server
- Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Sheng et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters
- Chen et al. (2023). Punica: Multi-Tenant LoRA Serving
- vLLM project (docs). vLLM documentation: LoRA adapters (--enable-lora, --max-loras, --max-lora-rank, dynamic loading)
- vLLM project (code). vLLM: with_cancellation and listen_for_disconnect (entrypoints/serve/utils/api_utils.py); abort on CancelledError (v1/engine/async_llm.py)
- vLLM project (code). vLLM: incremental detokenization (v1/engine/detokenizer.py, tokenizers/detokenizer_utils.py)
- SGLang project (code). SGLang router (sgl-router): cache-aware and power-of-two routing
- NVIDIA (2023). NVIDIA H100 Tensor Core GPU (specifications)