LLM & VLM courseBlog
133Track 5: LLM inference and serving · session 56 of 65

The serving stack and the life of a request

A model file is not a service. Between a user and the GPU sit an engine (vLLM, TensorRT-LLM, SGLang), an API server, a router and an autoscaler. This session follows one request through all of them: tokenize, queue, schedule, prefill, decode, detokenize, stream. You learn where each millisecond of latency comes from, how streaming, cancellation and timeouts work, and how one GPU can serve many models and hundreds of LoRA adapters.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 to 1.5 hours)Level: IntermediateBefore this: Sessions 78 (prefill and decode), 80 (the KV cache), 81 (inference metrics), 84 (batching basics) and 57 (LoRA)

What you will learn

  • The four layers of a serving stack (engine, API server, router, autoscaler), what each one owns, and which real systems fill each layer
  • The seven stages in the life of a request, and a cost model that splits TTFT and end-to-end latency into network, queue, prefill and decode parts
  • Why one decode step costs almost the same for 1 and for 32 sequences, and how the batch size trades per-user speed against GPU throughput
  • How streaming works (server-sent events, incremental detokenization that never cuts a UTF-8 character), and how cancellation and timeouts free GPU memory
  • How a router spreads load and why prefix-aware routing can cut prefill work by a large factor, and how to size replicas with Little’s law
  • How to serve many models and many LoRA adapters on one base model: adapter size, batched LoRA kernels (Punica, S-LoRA), and vLLM’s flags

Why this matters

Until now we made the model faster: better kernels, fewer bytes, fewer steps. But a user never talks to a model. A user sends an HTTP request to a service, and that service has many parts besides the GPU. Each part can add latency, waste GPU time, or drop the request.

Think about what happens when you type a question into a chat product. Your text travels over the network. A router picks one of many machines. A web server checks your request, applies the chat template and turns text into token ids. The request waits in a queue until the engine has room for it in GPU memory. The engine runs the prompt through the model (prefill), then makes one token at a time (decode). Each new token is turned back into text and streamed to you, piece by piece. Meanwhile an autoscaler watches the queues and adds or removes machines. If you close the tab, all of this must stop, or the GPU keeps working for nobody.

The serving survey by Miao et al. calls this the system side of LLM serving: memory management, batching, scheduling and distributed execution around the model. [Certain] The rest of this track goes deep into each piece: memory in session 134, batching in session 135, scheduling in session 136, multi-GPU serving in session 137, and real code in session 138. This session is the map that connects them. In session 132 you made one decode step fast. Now you see everything that happens around that step.

The four layers of a serving stack

A production LLM service has four layers. Each one has a different job and a different time scale.

  1. Engine. The program that owns the GPUs and runs the model. It keeps the KV cache, decides which requests run in each step (the scheduler), and calls the kernels. Examples: vLLM, TensorRT-LLM, SGLang. It works on a time scale of milliseconds: one decode step.
  2. API server. The front door of one engine. It speaks HTTP (or gRPC), checks inputs, applies the chat template, tokenizes, streams the output back, and reports metrics. The most common interface is the OpenAI-compatible API (/v1/chat/completions, /v1/completions): vllm serve <model> starts one. [Certain] NVIDIA’s Triton Inference Server is a general server that can host many models with different backends, including TensorRT-LLM and vLLM. [Certain]
  3. Router (also called gateway or load balancer). It sits in front of many engine replicas (copies of the same model on different GPUs) and picks one for each request. A simple router uses round robin; an LLM-aware router looks at queue lengths and at which replica already has the prompt’s prefix in its KV cache. It also handles authentication, rate limits and retries.
  4. Autoscaler. It watches metrics (queue length, KV-cache use, latency) and changes the number of replicas. It works on a time scale of tens of seconds to minutes, because starting a replica means loading many gigabytes of weights.
Figure 1. The serving stack. Requests flow from the clients at the top down to the GPUs, and tokens flow back up. Click a layer (or focus it and press Enter) to see what it owns, which real systems fill it, and which metric tells you it is the problem.
How to read Figure 1

There is one router for many replicas, and one API server per engine (vLLM runs it as a separate process that talks to the engine core). [Certain] The autoscaler is on the side because it is not in the path of a request: it only changes how many replicas exist. Notice which layer owns which resource: only the engine knows about GPU memory and the KV cache, so only the engine can decide whether a request fits.

Inside vLLM (version 1) these layers map to separate processes: one API server process handles HTTP, tokenization and streaming; one engine core process runs the scheduler and manages the KV cache; and one worker process per GPU runs the model. They talk over ZeroMQ. [Certain] Splitting them means that slow Python work in the server (tokenizing a long prompt, building JSON) does not stop the GPU loop.

The life of a request, step by step

Here are the seven stages every request goes through. We give names to the times, because every latency metric is a sum of them.

  1. Tokenize. The server checks the JSON, applies the chat template (session 9) and converts text to token ids. For a VLM it also decodes and resizes the images. Time \(t_{\text{tok}}\).
  2. Queue. The request waits in the engine’s waiting queue. Time \(t_{\text{queue}}\). This is the part that explodes when the service is overloaded.
  3. Schedule. At the start of every engine step, the scheduler picks which waiting requests to admit. A request is admitted only if there is enough free KV-cache memory for it. The policy (first come first served, priorities, preemption) is the topic of session 136.
  4. Prefill. The engine runs all prompt tokens through the model in one pass, fills the KV cache, and produces the first output token. Time \(t_{\text{prefill}}\). This is compute-bound (session 82).
  5. Decode. The engine makes one token per step for every running request. Each step reads all the weights and each request’s KV cache. The time of one step is the time per output token, TPOT.
  6. Detokenize. Each new token id is converted back to text, carefully, because one character can be split over several tokens.
  7. Stream. Each new piece of text is sent to the client right away, as a server-sent event. When a stop condition is met (end-of-sequence token, max_tokens, a stop string), the engine frees the request’s KV cache and the server sends a final message.

From these stages we get the two latency numbers that users feel (session 81). The time to first token (TTFT) is everything until the first piece of text arrives:

\[ \text{TTFT} = t_{\text{net}}^{\uparrow} + t_{\text{tok}} + t_{\text{queue}} + t_{\text{prefill}} + t_{\text{net}}^{\downarrow} \tag{133.1} \]

and the end-to-end latency for \(N_{\text{out}}\) output tokens adds one decode step per remaining token:

\[ \text{E2E} = \text{TTFT} + (N_{\text{out}} - 1)\cdot \text{TPOT} \tag{133.2} \]
Read the symbols
\(t_{\text{net}}^{\uparrow}, t_{\text{net}}^{\downarrow}\)
network time from client to server and back (one way each)
\(t_{\text{tok}}\)
time to validate, apply the template and tokenize (plus image preprocessing for a VLM)
\(t_{\text{queue}}\)
time waiting in the engine queue until the scheduler admits the request
\(t_{\text{prefill}}\)
time of the forward pass over the whole prompt
\(N_{\text{out}}\)
number of output tokens
TPOT
time per output token: the duration of one decode step of the batch the request is in

The first output token comes out of the prefill pass itself, so only \(N_{\text{out}} - 1\) decode steps follow. Detokenization and sending each chunk overlap with the next decode step, so in a good server they do not add to TPOT.

A cost model for prefill and decode

To put numbers in (133.1) and (133.2), we use the roofline view from session 82. Prefill does about \(2P\) FLOPs per prompt token, where \(P\) is the number of parameters, and it is compute-bound:

\[ t_{\text{prefill}} \approx \frac{2 P\, n_{\text{in}}}{F \cdot u} \tag{133.3} \]

A decode step for a batch of \(B\) sequences must read all weights once plus every sequence’s KV cache, and it does \(2PB\) FLOPs. It takes the larger of the memory time and the compute time, plus a fixed cost \(t_0\) for scheduling and kernel launches:

\[ \text{TPOT}(B) = t_{\text{step}}(B) \approx t_0 + \max\!\left( \frac{M_W + B\,\bar{L}\, s_{kv}}{\eta\, \text{BW}},\ \frac{2 P B}{F \cdot u} \right) \tag{133.4} \]

where the KV cache needs \(s_{kv}\) bytes per token (session 80):

\[ s_{kv} = 2 \cdot n_{\text{layers}} \cdot n_{kv} \cdot d_{\text{head}} \cdot b \tag{133.5} \]
Read the symbols
\(P, M_W\)
number of parameters, and their size in bytes (\(2P\) in BF16)
\(n_{\text{in}}\)
number of prompt tokens
\(F, u\)
peak FLOP/s of the GPU, and the fraction of it we really get (model FLOPs utilization)
\(\text{BW}, \eta\)
peak memory bandwidth, and the fraction of it we really get
\(B, \bar{L}\)
batch size (sequences decoded together), and their average context length
\(t_0\)
fixed cost per step (CPU scheduling, launches, sampling)
\(n_{kv}, d_{\text{head}}, b\)
number of KV heads, size of one head, bytes per number; the 2 is for K and V

Our running example is Llama 3 8B in BF16 on one H100 SXM. From its config: 32 layers, hidden size 4096, FFN size 14,336, 8 KV heads of size 128, vocabulary 128,256. [Certain] That gives \(P = 8.03 \times 10^9\) parameters, \(M_W = 16.06\) GB, and \(s_{kv} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB) per token. The H100 SXM has \(F = 989\) TFLOP/s dense BF16 and \(\text{BW} = 3.35\) TB/s. [Certain] We assume \(u = 0.5\), \(\eta = 0.8\) and \(t_0 = 1\) ms. These three are assumptions, not measurements; real values depend on the engine and the kernels.

Worked example: where the time of one request goes

A request with \(n_{\text{in}} = 1000\) prompt tokens asks for \(N_{\text{out}} = 200\) tokens. It joins a batch of \(B = 32\) running requests. Assume 10 ms network each way, 1 ms to tokenize, and 50 ms in the queue.

Prefill (133.3): \(2 \cdot 8.03\times10^9 \cdot 1000 / (989\times10^{12} \cdot 0.5) = 32.5\) ms. So TTFT (133.1) \(= 10 + 1 + 50 + 32.5 + 10 = 103.5\) ms. Half of it is queueing; the GPU work is less than a third.

Decode (133.4) with \(\bar{L} = 1100\) (the average context: 1,000 prompt tokens plus half of the 200 output tokens): memory time \((16.06 + 32 \cdot 1100 \cdot 131{,}072 \times 10^{-9})\) GB \(/ (0.8 \cdot 3.35\) TB/s\() = 7.71\) ms; compute time \(2 \cdot 8.03\times10^9 \cdot 32 / (4.945\times10^{14}) = 1.04\) ms. The step is memory-bound: TPOT \(= 1 + 7.71 = 8.71\) ms, which is 115 tokens per second for this user.

End to end (133.2): \(103.5 + 199 \cdot 8.71 = 1{,}838\) ms. The 199 decode steps are 94% of the total. The GPU makes \(32 / 8.71\text{ ms} = 3{,}672\) tokens per second for the whole batch.

Step through the same request in Figure 2 to see which component holds it at each moment.

Figure 2. The life of one request (1,000 prompt tokens, 200 output tokens, batch of 32, Llama 3 8B on one H100, cost model 133.3 and 133.4). Each lane is a component; the bar at the bottom is the clock, on a log-like scale so that short stages stay visible.
How to read Figure 2

The highlighted lane holds the request in this step. The readout gives the stage, what happens in it, and the clock time when it ends. Two things to notice. First, the API server is busy twice: before the engine (tokenize) and after every step (detokenize, send). Second, after prefill the request does not “own” the GPU: it is one row of a batch of 32, and every step serves all of them.

The batch size: per-user speed against GPU throughput

Look again at (133.4). At small batch sizes the step is memory-bound, and most of the bytes are the weights, which are read once per step no matter how many sequences are in the batch. So adding sequences is almost free: at \(B = 1\) the step takes 7.05 ms, at \(B = 32\) it takes 8.71 ms. The GPU makes 142 tokens per second at \(B = 1\) and 3,672 at \(B = 32\), 26 times more for 24% more latency per token. This is why every engine batches (session 84).

The GPU throughput in tokens per second is

\[ \Theta(B) = \frac{B}{t_{\text{step}}(B)} \tag{133.6} \]

But the free lunch ends. Each sequence adds its KV cache to the bytes read, so the step slowly gets longer: at \(B = 256\) it is 20.8 ms, and each user sees only 48 tokens per second. And the KV cache must fit in memory: with 90% of 80 GB usable, minus 16.06 GB of weights, about 55.9 GB is left, which holds 355 sequences of 1,200 tokens. Long prompts change everything: with 8,000-token prompts only 52 sequences fit. Explore it in Figure 3.

network + tokenizequeueprefilldecodeGPU tokens/s (right plot)
Figure 3. Latency budget of one request and GPU throughput, from the cost model (133.1) to (133.6) for Llama 3 8B BF16 on one H100 SXM (\(u = 0.5\), \(\eta = 0.8\), \(t_0 = 1\) ms, 10 ms network each way, 1 ms tokenize). Left: where the end-to-end time goes. Right: throughput against batch size, with the current batch marked.
How to read Figure 3

Start at the defaults: decode dominates the bar. Move the batch slider from 1 to 256: the throughput curve rises steeply, then flattens, while the decode part of the bar grows. Now push the prompt to 8,000 tokens: prefill grows (TTFT goes over 300 ms) and the readout warns when the batch no longer fits in KV memory. Finally move the queue slider: queueing adds directly to TTFT, which is why overload shows up first as slow first tokens.

Streaming

A user reads about as fast as a few words per second, so waiting 1.8 s for the full answer feels slow, while seeing the first words after 0.1 s feels fast. That is why chat services stream: they send each new piece of text as soon as it exists.

The OpenAI-compatible API streams with server-sent events (SSE): one HTTP response stays open with content type text/event-stream, and the server writes lines of the form data: {...}, each followed by a blank line. Each chunk is a JSON object with "object": "chat.completion.chunk" and the new text in choices[0].delta.content. The last content chunk carries a finish_reason such as "stop" or "length", and the stream ends with data: [DONE]. If the client sets stream_options: {"include_usage": true}, one extra chunk with token counts and an empty choices list comes at the end. [Certain]

Incremental detokenization

Streaming has a trap. Byte-level BPE tokenizers (session 8) can split one character over several tokens. The emoji 👍 is four UTF-8 bytes (F0 9F 91 8D); if the model emits it as two tokens, the first token alone is not valid text. Decoding each token separately gives the replacement character � (U+FFFD). Also, some tokenizers add or remove spaces depending on the neighbors, so decoding one token alone can give the wrong spacing.

The fix is incremental detokenization: decode a short window of recent tokens together, compare with the text already sent, and send only the new part. If the new text ends with �, the last character is not finished yet, so send nothing and wait for the next token. vLLM does exactly this: its detokenize_incrementally keeps two offsets into the token list (a prefix_offset where the window starts and a read_offset where unsent text starts) and returns an empty string while the text ends with U+FFFD. [Certain] It also holds back the last few characters when the request has stop strings, so that a stop string that arrives in pieces is never shown to the user. [Certain]

Algorithm 1: incremental detokenization with stop strings
  1. state per request: token list \(T\), offsets \(p\) (window start) and \(r\) (first unsent token), sent text \(S\)
  2. on each new token \(t\): append \(t\) to \(T\)
  3.   \(a \leftarrow \text{decode}(T[p:r])\) // text already sent, decoded with its left context
  4.   \(b \leftarrow \text{decode}(T[p:])\) // the same window plus the new tokens
  5.   if \(b\) ends with U+FFFD or \(|b| \le |a|\): return nothing // unfinished character: wait
  6.   \(\delta \leftarrow b[|a|:]\); \(\;p \leftarrow r\); \(\;r \leftarrow |T|\); append \(\delta\) to \(S\)
  7.   if a stop string occurs in the new part of \(S\): cut \(S\) at it, finish with reason “stop”
  8.   send \(S\) minus its last \(\ell_{\max} - 1\) characters (not yet sent) // \(\ell_{\max}\) = longest stop string
Figure 4. Streaming the text “ Café 👍 ok” from six byte-level tokens (an illustrative split; real tokenizers split differently). Each row is one decode step. The third column decodes each token alone, which is what a naive server would send. The fourth column is what incremental detokenization sends, and the last one is the text the client has so far. Text is shown in quotes; a space inside the quotes (for example the leading space in “ Caf”) is part of the text.
How to read Figure 4

The byte C3 starts a two-byte character (é), so alone it is not text: the naive column shows �, while the incremental column sends nothing and waits. One step later A9 completes the character and é is sent. The emoji needs four bytes spread over two tokens, so the same thing happens again. The user sees the same final text in both columns only if nothing is shown until the end; with streaming, only the right column is correct.

Two more streaming details matter in practice. Backpressure: if a client reads slowly, chunks pile up in the server’s buffers; a server should not let one slow reader block the engine loop, so the engine writes into a per-request queue and the HTTP layer drains it. Batching of chunks: sending one HTTP write per token costs CPU time for every token of every user; some servers can combine several tokens into one chunk to save it. [Likely]

Cancellation and timeouts

Requests end in four ways: the model emits the end-of-sequence token, the request reaches max_tokens, a stop string appears, or the request is cancelled. Cancellation is the one people forget, and it is common: users press “stop”, close the tab, or the client library gives up.

When a client disconnects, the engine does not know it. If nobody tells it, the request keeps decoding up to max_tokens, holding a batch slot and its KV cache for nobody. With \(N_{\text{out}} = 200\) and TPOT of 8.71 ms, that is 1.7 s of a batch slot wasted per abandoned request, and its KV cache (here 157 MB for 1,200 tokens) blocks a waiting request from being admitted. So the server must detect the disconnect and abort the request in the engine. In vLLM, the HTTP handler waits on two tasks at once, the real work and a listener for the http.disconnect message; when the client leaves, the work task is cancelled, and the engine’s generate() catches the cancellation and calls abort, which frees the request’s KV blocks at the next step. [Certain]

Algorithm 2: one streamed request on the server, with cancellation and a timeout
  1. input: request \(q\), deadline \(D\)
  2. tokenize \(q\); if the waiting queue is too long: reply HTTP 429 (“try later”) // admission control
  3. engine.add(\(q\)); start a task that waits for the client to disconnect
  4. try:
  5.   loop: wait for the next output of \(q\), but not past \(D\)
  6.     \(\delta \leftarrow\) incremental detokenize (Algorithm 1); if \(\delta\) is not empty: send data: chunk
  7.     if \(q\) is finished: send the final chunk (finish reason, usage) and data: [DONE]; break
  8. on disconnect or deadline: stop waiting // both arrive as a cancellation
  9. finally: engine.abort(\(q\)) // always runs; aborting a finished request does nothing
  10. at its next step the engine drops \(q\) from the batch and returns its KV blocks to the free pool

Timeouts protect the system from requests that would take too long. Use separate limits for separate stages: a queue timeout (if a request waited 30 s, the user is gone anyway; reject it before spending GPU time on prefill), a limit on time to first token, and an overall deadline. Clients should also send a sensible max_tokens, which is the cheapest timeout of all. And a request that times out must be aborted in the engine, exactly like a disconnect, or the timeout only hides the waste.

Routing across replicas

One engine replica has a limit. To serve more traffic you run many replicas and put a router in front. The router must answer one question per request: which replica?

  • Round robin: replica 1, 2, 3, 4, 1, 2, … Simple, but blind: LLM requests differ in cost by 1,000 times (a 50-token chat turn against a 50,000-token document).
  • Least loaded: send to the replica with the fewest queued tokens or running requests. A cheap variant, “power of two choices”, picks two replicas at random and takes the less loaded one.
  • Prefix-aware (cache-aware): many requests share a long prefix, such as a system prompt, a few-shot block, or an uploaded document. If a replica already has that prefix in its KV cache, it can skip that part of the prefill (session 134). So send the request to a replica that has the prefix, unless that replica is much busier than the others. The SGLang router has such a cache-aware policy, with a fallback to load balancing. [Certain]

Figure 5 simulates 600 requests from 16 “tenants” (for example, apps that each have their own 1,000-token system prompt) on 4 replicas. Each replica can keep a few prefixes in its cache.

prefill workdecode workcached prefixes (tenant number)
Figure 5. A toy router simulation (seeded, so every run is the same). Each request has a 1,000-token tenant prefix, 200 new prompt tokens and 200 output tokens; one output token is counted as 4 units of work and one prefill token as 1. A request arrives every 50 ms, and the 4 replicas together can do 1,500 units in 50 ms. A cache hit skips the 1,000 prefix tokens. Each replica keeps its most recently used prefixes.
How to read Figure 5

The bars show the total work each replica did; the orange part is prefill. Round robin and least loaded spread tenants over all replicas, so each replica must cache all of them and misses often. Prefix-aware routing gives each tenant a “home” replica, so the hit rate jumps and the orange parts shrink. The readout’s “offered load” is work per second divided by capacity: above 100%, queues grow without limit. Notice that at the default settings only the prefix-aware router keeps the load under 100%. Cache hits are capacity. Then move the balance slack (prefix-aware only). At 0 the router leaves a request’s home replica as soon as it is busier than the others: load is even, but the hit rate drops. Larger slack keeps requests at home: more hits, but the home replicas of popular tenants build queues. The best wait is in between.

The rule the simulation uses for the prefix-aware case is: among replicas that hold the prefix, take the least loaded one, but only if its queue is at most twice the shortest queue plus a slack (in units of one request’s work); otherwise take the least loaded replica. Real routers use the same idea with their own thresholds, and their flags change between versions, so check the documentation of the version you run. [Likely]

Autoscaling and capacity

How many replicas do you need? Two simple laws answer most of it.

Little’s law connects the average number of requests in the system \(L\), the arrival rate \(\lambda\), and the average time each request spends inside \(W\). It holds for any stable system, whatever the details of the scheduler:

\[ L = \lambda \, W \tag{133.7} \]

GPU time per request. Each request needs its own prefill, and a \(1/B\) share of each of its \(N_{\text{out}}\) decode steps. If the replicas run at an operating batch size \(B\), one request costs \(s\) GPU-seconds, and with a target utilization \(\rho\) you need \(R\) replicas:

\[ s = t_{\text{prefill}} + \frac{N_{\text{out}}\, t_{\text{step}}(B)}{B}, \qquad R = \left\lceil \frac{\lambda\, s}{\rho} \right\rceil \tag{133.8} \]
Read the symbols
\(\lambda\)
arrival rate, requests per second
\(W\), \(L\)
average time a request spends in the system, and the average number of requests inside it
\(s\)
GPU-seconds one request uses
\(\rho\)
target utilization, below 1 so that bursts do not overflow the queue (0.6 to 0.8 is common [Likely])
\(R\)
number of replicas
Worked example: sizing a service

Traffic: \(\lambda = 20\) requests per second, each with 1,000 prompt and 200 output tokens. We allow each replica a batch of at most \(B = 64\), so \(t_{\text{step}} = 10.44\) ms by (133.4).

GPU time per request (133.8): \(s = 32.5 + 200 \cdot 10.44 / 64 = 32.5 + 32.6 = 65.1\) ms. Half is prefill: long prompts are expensive even though decode takes most of the wall-clock time. Load \(\lambda s = 20 \cdot 0.0651 = 1.30\) GPUs. With \(\rho = 0.7\): \(R = \lceil 1.30 / 0.7 \rceil = \lceil 1.86 \rceil = 2\) replicas.

Check with Little’s law (133.7): \(W \approx 32.5 + 200 \cdot 10.44 = 2{,}120\) ms, so \(L = 20 \cdot 2.12 = 42\) requests are inside at any moment, about 21 per replica: below the batch limit of 64, so the plan is consistent. One prefill-only bound is also worth knowing: at \(u = 0.5\), one H100 prefills only about 30,800 tokens per second of this model, so 20,000 prompt tokens per second already use 65% of a GPU.

What to scale on. GPU utilization as reported by nvidia-smi is a poor signal: it only says a kernel was running, not how much work was done (session 123). Better signals come from the engine itself. vLLM exports Prometheus metrics such as vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc, and histograms of vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds. [Certain] A common policy: scale up when the waiting queue stays above a threshold or the p95 TTFT breaks its target, scale down slowly when the queue is empty.

Cold start. A new replica must get a GPU, start the container, load 16 GB of weights (at 1 GB/s from network storage that alone is 16 s), and warm up (for example capture CUDA graphs, session 127). So autoscaling reacts in tens of seconds to minutes, not milliseconds. Keep some headroom (the \(\rho\) in 133.8), cache weights on local disk, and scale on queue growth early rather than on latency after it is already bad.

Serving many models and many LoRA adapters

Real services rarely have one model. A platform may host ten base models, and hundreds of customers who each fine-tuned their own variant. Copying full models does not scale: one 8B model in BF16 is 16.06 GB, so one 80 GB GPU holds at most four copies, with no room left for KV cache. Moving a full model in from CPU memory over PCIe Gen5 x16 (about 64 GB/s per direction, theoretical [Likely]) takes at least 0.25 s, far too slow to do per request.

For many models of different families there are three options: give each its own replicas (simple, wasteful when traffic is uneven), put several small models on one GPU (Triton Inference Server can host many models and load or unload them through its model-control API [Certain]), or route by model name in the router to separate pools. For many fine-tuned variants of one base model, there is a much better option: LoRA.

One base model, many adapters

A LoRA adapter (session 57) changes a weight \(W\) by a low-rank product \(BA\) with rank \(r\). The base weights stay the same for every customer, so a server keeps one copy of the base model and many small adapters. Request \(i\) uses adapter \(a(i)\):

\[ \mathbf{y}_i = W \mathbf{x}_i + \frac{\alpha}{r}\, B_{a(i)} \left( A_{a(i)}\, \mathbf{x}_i \right) \tag{133.9} \]

Each adapted layer with input size \(d_{\text{in}}\) and output size \(d_{\text{out}}\) adds \(r(d_{\text{in}} + d_{\text{out}})\) parameters, so for \(n_{\text{layers}}\) blocks with a set \(\mathcal{M}\) of adapted matrices:

\[ p_{\text{LoRA}} = n_{\text{layers}} \sum_{m \in \mathcal{M}} r \left( d_{\text{in}}^{(m)} + d_{\text{out}}^{(m)} \right) \tag{133.10} \]
Read the symbols
\(W\)
the shared base weight of one layer (one copy for all requests)
\(A_{a}, B_{a}\)
the down-projection (\(r \times d_{\text{in}}\)) and up-projection (\(d_{\text{out}} \times r\)) of adapter \(a\)
\(a(i)\)
the adapter chosen by request \(i\) (in vLLM, by the model field of the request)
\(\alpha / r\)
the LoRA scale factor
\(\mathcal{M}\)
the adapted matrices, for example q, k, v, o only, or all seven linear layers of a block
Worked example: how big is one adapter?

Llama 3 8B, rank 16, all seven linear layers per block. Per block: q and o \(16 \cdot (4096 + 4096) = 131{,}072\) each; k and v \(16 \cdot (4096 + 1024) = 81{,}920\) each; gate, up and down \(16 \cdot (4096 + 14{,}336) = 294{,}912\) each. Sum: 1,310,720. Times 32 blocks: \(p_{\text{LoRA}} = 41{,}943{,}040\) parameters, 83.9 MB in BF16, which is 0.52% of the base model.

So 100 such adapters take 8.4 GB, and 1,000 take 84 GB: they fit in CPU memory, not all on the GPU. Loading one over PCIe at 64 GB/s takes about 1.3 ms. One hundred full fine-tuned copies would take 1,606 GB.

The hard part is not memory but batching. The batch now mixes requests for different adapters. The base product \(W\mathbf{x}\) is shared, so it stays one big GEMM for the whole batch. The adapter part is different for each row. Running one small GEMM per adapter is slow, and merging \(BA\) into \(W\) works only if the whole batch uses one adapter. Punica introduced a CUDA kernel that computes the adapter products for a batch where each row has its own adapter (Segmented Gather Matrix-Vector multiplication, SGMV), keeping one copy of the base model; the paper reports 12 times the throughput of previous systems on multi-LoRA workloads and 2 ms of extra latency per token. [Certain] S-LoRA keeps all adapters in CPU memory, moves the needed ones to the GPU, and stores adapter weights and KV cache in one paged memory pool (“unified paging”) to avoid fragmentation; it reports serving thousands of adapters on one GPU and up to 4 times the throughput of Hugging Face PEFT and vLLM’s early LoRA support. [Certain]

In vLLM you start the server with --enable-lora, register adapters with --lora-modules name=path, and the client picks one by putting its name in the model field. --max-loras sets how many adapters can be in one batch at the same time, --max-lora-rank must be at least the largest rank you serve (a much larger value wastes memory), and --max-cpu-loras sets how many are kept in CPU memory [Likely]. Adapters can also be loaded at runtime through /v1/load_lora_adapter when the environment variable VLLM_ALLOW_RUNTIME_LORA_UPDATING is set, which the docs warn against outside trusted environments. [Certain]

base weightsadapter slots on GPUKV cachefull copies (would need)
Figure 6. Multi-LoRA memory for Llama 3 8B in BF16 on one 80 GB GPU (90% usable). The top bar is one GPU serving the base model plus LoRA slots; the bottom bar is what one full fine-tuned copy per variant would need, on a log scale of GPUs.
How to read Figure 6

At the defaults (rank 16, all layers, 8 slots) the adapter slots take 0.67 GB, which costs about 5,100 tokens of KV cache: almost nothing next to 55.9 GB. Raise the rank to 128 and the slots to 64: now adapters take 43 GB and the KV cache shrinks badly, so fewer requests fit in a batch. The number of adapters served only changes CPU memory, not GPU memory: that is the point of keeping a small set of slots on the GPU and the rest on the host.

Code: a tiny engine with streaming, cancellation and batched LoRA

The first program is a working miniature of the whole path: a request object, an engine that batches requests and runs one step at a time, incremental detokenization, a server function that yields SSE lines, a client that disconnects, and one that times out. The “model” is scripted so that the output is deterministic, but it runs one batched forward pass per step like a real one.

import asyncio, json, torch
torch.manual_seed(0)
EOS, VOCAB = 256, 257                 # byte-level vocabulary: 256 bytes + an end token

class ScriptedLM(torch.nn.Module):
    """Stands in for the real model: one batched forward pass per step, returns logits.
    Each request 'wants' to say a fixed reply, so the output is deterministic."""
    def forward(self, reqs):
        nxt = []
        for r in reqs:
            want = r.reply_ids + [EOS]
            nxt.append(want[min(len(r.out_ids), len(want) - 1)])
        return torch.nn.functional.one_hot(torch.tensor(nxt), VOCAB).float()   # [B, VOCAB]

class Request:
    def __init__(self, rid, prompt, reply, max_tokens):
        self.rid, self.prompt_ids = rid, list(prompt.encode("utf-8"))      # "tokenize"
        self.reply_ids, self.max_tokens = list(reply.encode("utf-8")), max_tokens
        self.out_ids, self.sent, self.aborted = [], 0, False
        self.queue = asyncio.Queue()                                        # engine -> server

def detok_delta(r):
    """Incremental detokenization: only emit text that is complete UTF-8."""
    text = bytes(r.out_ids).decode("utf-8", errors="replace")
    if text.endswith("\ufffd"):        # unfinished multi-byte character: hold it back
        return ""
    delta, r.sent = text[r.sent:], len(text)
    return delta

class Engine:
    def __init__(self, model, max_batch=2):
        self.model, self.max_batch, self.waiting, self.running = model, max_batch, [], []
    def add(self, r):   self.waiting.append(r)
    def abort(self, r): r.aborted = True
    def step(self):
        self.running = [r for r in self.running if not r.aborted]       # free KV of aborted requests
        self.waiting = [r for r in self.waiting if not r.aborted]
        while self.waiting and len(self.running) < self.max_batch:      # FCFS admission
            self.running.append(self.waiting.pop(0))                    # (prefill would run here)
        if not self.running:
            return
        nxt = self.model(self.running).argmax(-1).tolist()             # one step for the whole batch
        still = []
        for r, t in zip(self.running, nxt):
            if t != EOS:
                r.out_ids.append(t)
            done = t == EOS or len(r.out_ids) >= r.max_tokens
            r.queue.put_nowait((detok_delta(r), "stop" if t == EOS else ("length" if done else None)))
            if not done:
                still.append(r)
        self.running = still

async def engine_loop(engine):
    while True:
        engine.step()
        await asyncio.sleep(0.01)                                       # one "decode step" = 10 ms

async def stream(engine, r, timeout):
    """Server side of one request: yields SSE lines; always aborts on exit."""
    engine.add(r)
    deadline = asyncio.get_running_loop().time() + timeout
    try:
        while True:
            left = deadline - asyncio.get_running_loop().time()
            delta, finish = await asyncio.wait_for(r.queue.get(), max(left, 0))
            if not delta and not finish:
                continue                       # nothing printable yet: send nothing
            chunk = {"id": r.rid, "object": "chat.completion.chunk",
                     "choices": [{"index": 0, "delta": {"content": delta}, "finish_reason": finish}]}
            yield "data: " + json.dumps(chunk, ensure_ascii=False)
            if finish:
                yield "data: [DONE]"
                return
    finally:
        engine.abort(r)          # runs on normal end, client disconnect, and timeout

async def client(engine, r, timeout=5.0, read_max=None):
    got, n = [], 0
    agen = stream(engine, r, timeout)
    try:
        async for line in agen:
            got.append(line); n += 1
            if read_max and n >= read_max:
                break                                                    # client disconnects
    except asyncio.TimeoutError:
        got.append("timeout")
    finally:
        await agen.aclose()
    return got

async def main():
    eng = Engine(ScriptedLM(), max_batch=2)
    loop_task = asyncio.create_task(engine_loop(eng))
    a = Request("a", "Hi", "Café 👍", max_tokens=50)
    b = Request("b", "Hello", "a long answer nobody reads", max_tokens=50)
    c = Request("c", "Hey", "this one is too slow", max_tokens=50)
    ra, rb, rc = await asyncio.gather(client(eng, a), client(eng, b, read_max=3),
                                      client(eng, c, timeout=0.08))
    loop_task.cancel()
    print(ra[0])
    print([json.loads(l[6:])["choices"][0]["delta"]["content"] for l in ra[:-1]], ra[-1])
    print("b: client left after", len(rb), "chunks; engine stopped early:", len(b.out_ids) < len(b.reply_ids))
    print("c:", rc[-1], "| aborted:", c.aborted, "| stopped early:", len(c.out_ids) < len(c.reply_ids))
asyncio.run(main())

The output: request a streams ['C', 'a', 'f', 'é', ' ', '👍', ''] and then data: [DONE], with no broken characters, because the half-finished é and 👍 are held back for one step. Request b reads 3 chunks and leaves, and the engine stops it early. Request c waits in the queue (the batch holds only 2), hits its deadline, and is aborted. The second program checks that batched multi-LoRA (133.9), with a gathered adapter per row and one shared base GEMM, gives the same result as merging each adapter separately, and computes the adapter size (133.10).

import torch
torch.manual_seed(0)
d_in, d_out, r, n_adapters = 64, 48, 8, 3
alpha = 16.0
W = torch.randn(d_out, d_in) / d_in ** 0.5            # shared base weight (one copy)
A = torch.randn(n_adapters, r, d_in) * 0.1            # LoRA down-projections, one per adapter
Bu = torch.randn(n_adapters, d_out, r) * 0.1          # LoRA up-projections
x = torch.randn(5, d_in)                              # 5 requests in one batch
idx = torch.tensor([0, 2, 2, -1, 1])                  # adapter of each request; -1 = base model only

def lora_batched(x, idx):
    y = x @ W.T                                       # ONE base GEMM for the whole batch
    m = idx >= 0
    h = torch.einsum("brd,bd->br", A[idx[m]], x[m])   # gather each row's A: shrink to rank r
    y[m] += (alpha / r) * torch.einsum("bor,br->bo", Bu[idx[m]], h)   # expand back
    return y

def lora_one_by_one(x, idx):                          # reference: merge weights per request
    rows = []
    for xi, i in zip(x, idx.tolist()):
        Wi = W if i < 0 else W + (alpha / r) * Bu[i] @ A[i]
        rows.append(Wi @ xi)
    return torch.stack(rows)

print((lora_batched(x, idx) - lora_one_by_one(x, idx)).abs().max().item())   # tiny: same math

# size of one adapter for Llama 3 8B, rank 16 on all 7 linear layers of the 32 blocks
shapes = {"q": (4096, 4096), "k": (4096, 1024), "v": (4096, 1024), "o": (4096, 4096),
          "gate": (4096, 14336), "up": (4096, 14336), "down": (14336, 4096)}
p = 32 * sum(16 * (a + b) for a, b in shapes.values())
print(p, "parameters,", 2 * p / 1e6, "MB in BF16")   # 41943040 parameters, 83.88608 MB

Where this is used

SettingWhat the stack must doMain lever
Chat assistantLow TTFT, steady streaming, many users; frequent “stop” pressesStreaming with incremental detokenization; abort on disconnect; scale on queue length and p95 TTFT
Coding assistant in an IDEVery short deadlines; most completions are cancelled because the user keeps typingAggressive cancellation and small max_tokens; prefix-aware routing on the shared file context
Multi-tenant fine-tuning platformHundreds of customer variants of a few base modelsMulti-LoRA serving (Punica, S-LoRA, vLLM --enable-lora), adapter cache on CPU
Agents and tool useMany calls that share a long system prompt and historyPrefix-aware routing so the shared prefix is prefilled once per replica (session 141)
Video-RAG and video question answeringVery long multimodal prompts (frames as thousands of visual tokens) and often the same video asked many timesRoute all questions about one video to the replica that has its prefix cached; long prefill makes TTFT the key metric
Batch jobs (video captioning, action-recognition labels, key-moment summaries)Millions of requests, no user waitingLarge batch sizes for throughput (133.6), offline mode instead of the HTTP server, no streaming

Common mistakes

  • Not aborting on disconnect or timeout. The GPU keeps generating for nobody and the KV cache stays full. Test it: start a long request, kill the client, and watch vllm:num_requests_running.
  • Decoding each token separately for streaming. Users see � in the middle of emoji and non-English text, or wrong spaces. Use the engine’s incremental detokenizer.
  • Autoscaling on GPU utilization. A GPU decoding one sequence shows high utilization while doing a tiny fraction of the work it could. Scale on waiting requests, KV-cache use and latency percentiles.
  • Round-robin routing for workloads with shared prefixes. Every replica must prefill every system prompt; prefix-aware routing can cut prefill work by several times (Figure 5).
  • One timeout for everything. Without a queue timeout, an overloaded server spends GPU time on prefill for requests whose users already left.
  • Setting --max-lora-rank or --max-loras far above need. Slots are reserved GPU memory; that memory is taken from the KV cache and lowers the batch size (Figure 6).
  • Measuring latency only as an average. Users feel the tail. Track p50, p95 and p99 of TTFT and TPOT, separately (session 81).

Interview questions

Name the layers of an LLM serving stack and what each one owns.basic
The engine (vLLM, TensorRT-LLM, SGLang) owns the GPUs: model weights, KV cache, the scheduler and the kernels. The API server (OpenAI-compatible server, Triton Inference Server) owns the protocol: validation, chat template, tokenization, detokenization, streaming, metrics. The router owns the choice of replica, plus authentication, rate limits and retries. The autoscaler owns the number of replicas. Only the engine knows GPU memory, so only it decides if a request fits.
Walk through the life of a request in a chat service.basic
The client sends a POST to /v1/chat/completions with stream: true. The router picks a replica. The API server validates, applies the chat template and tokenizes. The request waits in the engine’s queue until the scheduler admits it, which needs free KV-cache blocks. Prefill runs the prompt and produces the first token. Then decode adds one token per step, batched with other requests. Each token is detokenized incrementally and sent as an SSE chunk. On a stop condition the engine frees the KV cache, and the server sends a final chunk with the finish reason (and usage, if asked) and data: [DONE].
Break down time to first token. Which part grows under overload?core
TTFT = network up + tokenize + queue + prefill + network down (133.1). Prefill grows with prompt length (about \(2Pn_{\text{in}}/(Fu)\), 32.5 ms for 1,000 tokens of an 8B model on an H100 at 50% MFU). The queue time is the part that grows without limit under overload, because when arrivals exceed capacity the waiting queue grows every second. That is why TTFT percentiles and queue length are the first signals of overload.
Why does going from batch 1 to batch 32 barely change TPOT?core
A decode step is memory-bound: it must read all 16 GB of weights once per step, whatever the batch size. Extra sequences add only their KV cache bytes and some FLOPs that are far below the compute limit. In the cost model the step goes from 7.05 ms at batch 1 to 8.71 ms at batch 32, while throughput rises from 142 to 3,672 tokens per second. The gain stops when KV reads dominate the bytes, when compute becomes the limit, or when KV memory is full.
What happens when a streaming client disconnects, and what should happen?core
The engine does not see the TCP connection, so by default the request keeps running until EOS or max_tokens, holding a batch slot and its KV cache. The server must detect the disconnect (vLLM races the handler against a listener for the http.disconnect message and cancels the handler), and the cancellation must reach the engine as an abort, which removes the request from the batch and frees its KV blocks at the next step. The same must happen on timeouts.
Why can you not just decode each new token to text and send it?core
With byte-level BPE, one character can span several tokens (an emoji is 4 UTF-8 bytes), so a single token can be an incomplete byte sequence that decodes to U+FFFD. Some tokenizers also change spaces depending on neighbors. Incremental detokenization decodes a small window of tokens, compares it with the text already sent, sends only the new suffix, and waits while the text ends with U+FFFD. With stop strings it also holds back the last few characters so a partial stop string is never shown.
What metric would you autoscale an LLM service on, and why not GPU utilization?core
On engine-level signals: number of waiting requests, KV-cache usage, and latency percentiles such as p95 TTFT against the SLO. GPU utilization from nvidia-smi only says a kernel was running during the sample interval; a GPU decoding batch 1 looks busy while delivering a small fraction of its possible throughput. Also account for cold start (loading 16 GB of weights, warm-up), which means scaling must start early and keep headroom.
Size a deployment: 20 requests/s, 1,000 prompt and 200 output tokens, Llama 3 8B on H100s.advanced
With the cost model and a max batch of 64: decode step 10.44 ms, prefill 32.5 ms. GPU time per request \(s = 32.5 + 200 \cdot 10.44/64 = 65.1\) ms (133.8), load \(\lambda s = 1.30\) GPUs, so 2 replicas at a 70% target. Little’s law: \(W \approx 2.12\) s, \(L = 42\) requests inside, about 21 per replica, under the batch limit. Then check memory: 21 sequences of 1,200 tokens need 3.3 GB of KV cache, far below 55.9 GB. Mention that real numbers must be measured and that p99 targets need more headroom.
How does prefix-aware routing work, and what is its risk?advanced
The router keeps track (exactly or approximately) of which prefixes each replica has in its KV cache, for example with a radix tree of token sequences. For a new request it prefers the replica with the longest cached match, so the shared system prompt or document is not prefilled again. The risk is hot spots: a popular prefix sends most traffic to one replica. So the router falls back to the least loaded replica when the preferred one is much busier, trading some hit rate for balance.
How do you serve 500 fine-tuned variants of the same 8B model?advanced
Fine-tune them as LoRA adapters and serve one base model with multi-LoRA. Each rank-16 adapter on all linear layers is 84 MB (0.52% of the base), so 500 take 42 GB of CPU memory, versus 8 TB for full copies. The engine keeps a few adapter slots on the GPU, loads adapters on demand (about 1 ms each over PCIe), and batches requests of different adapters together: one shared base GEMM plus a gathered per-row LoRA product (Punica’s SGMV, S-LoRA’s kernels). Limit max rank and slots, because they take memory from the KV cache.
Why does vLLM split the API server, the engine core and the GPU workers into separate processes?advanced
Python work in the server (HTTP parsing, JSON, tokenization, detokenization) is CPU-bound and would otherwise share the interpreter with the scheduling loop, adding gaps between GPU steps. In separate processes, talking over ZeroMQ, the engine core can schedule the next step while the server is still formatting the last chunk, and one worker per GPU runs the model. The number of processes grows with tensor, pipeline and data parallelism.

Exercises

  1. With the cost model, compute TTFT for a 4,000-token prompt with 0 ms queue time (10 ms network each way, 1 ms tokenize).
  2. A request is abandoned after 20 output tokens but runs until max_tokens = 1,000 at a TPOT of 10 ms. How many seconds of batch slot are wasted? If 5% of 20 requests per second are abandoned like this, how many batch slots are blocked on average?
  3. Compute the size of a rank-8 adapter on q, k, v and o only for Llama 3 8B, in parameters and MB (BF16).
  4. In Figure 5, why does increasing “prefixes cached per replica” to 16 help round robin much more than it helps prefix-aware routing?
  5. A service has \(\lambda = 5\) requests per second and an average end-to-end time of 6 s. How many requests are in the system on average? If each replica can run 32 at once, what is the minimum number of replicas?
  6. The emoji “😀” is F0 9F 98 80. If it arrives as three tokens (F0, 9F 98, 80), what does incremental detokenization send after each one?
Answerscheck
1. Prefill \(= 2 \cdot 8.03\times10^9 \cdot 4000 / 4.945\times10^{14} = 129.9\) ms; TTFT \(= 10 + 1 + 0 + 129.9 + 10 = 150.9\) ms. 2. \(980 \times 10\) ms \(= 9.8\) s wasted per request. Abandoned rate \(= 1\) per second, each holding a slot for 9.8 s, so by Little’s law about 9.8 slots are blocked on average. 3. Per block: \(8\cdot(8192) \cdot 2 + 8 \cdot (5120) \cdot 2 = 131{,}072 + 81{,}920 = 212{,}992\); times 32 \(= 6{,}815{,}744\) parameters, 13.6 MB. 4. With 16 slots, every replica can cache all 16 tenants, so round robin stops missing after the warm-up; prefix-aware routing already gave each tenant a home replica, so it needed few slots. 5. \(L = 5 \cdot 6 = 30\) requests; one replica of 32 is the minimum, but with no headroom for bursts, so 2 would be safer. 6. After F0: nothing (the text ends with U+FFFD). After 9F 98: still nothing (three of four bytes). After 80: “😀”.

Key takeaways

  • A serving stack has four layers: engine (GPU, KV cache, scheduler), API server (protocol, tokenizer, streaming), router (which replica) and autoscaler (how many replicas).
  • A request goes through tokenize, queue, schedule, prefill, decode, detokenize and stream. TTFT (133.1) is network + tokenize + queue + prefill; E2E (133.2) adds one TPOT per further token.
  • Decode is memory-bound, so batching is almost free until KV reads, compute or KV memory become the limit (133.4, 133.6): 142 tokens/s at batch 1 against 3,672 at batch 32 in our model.
  • Streaming needs incremental detokenization; cancellation and timeouts must reach the engine as aborts, or the GPU works for nobody.
  • Prefix-aware routing turns cache hits into capacity; size replicas with GPU time per request and Little’s law (133.7, 133.8), and autoscale on queues, not on GPU utilization.
  • Many variants of one base model are served as LoRA adapters (133.9, 133.10): 84 MB each for rank 16 on Llama 3 8B, batched together with shared base GEMMs.

Key formulas

\[ \text{TTFT} = t_{\text{net}} + t_{\text{tok}} + t_{\text{queue}} + t_{\text{prefill}}, \quad \text{E2E} = \text{TTFT} + (N_{\text{out}} - 1)\,\text{TPOT} \]
\[ t_{\text{step}}(B) \approx t_0 + \max\!\left(\frac{M_W + B \bar{L} s_{kv}}{\eta\,\text{BW}}, \frac{2PB}{Fu}\right), \quad L = \lambda W, \quad R = \left\lceil \frac{\lambda s}{\rho} \right\rceil \]
\[ \mathbf{y}_i = W\mathbf{x}_i + \tfrac{\alpha}{r} B_{a(i)} A_{a(i)} \mathbf{x}_i, \quad p_{\text{LoRA}} = n_{\text{layers}} \sum_m r\,(d_{\text{in}} + d_{\text{out}}) \]

References for this session

  1. Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  2. vLLM project (docs). vLLM documentation
  3. Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  4. vLLM project (docs). vLLM documentation
  5. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  6. Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
  7. NVIDIA (code). TensorRT-LLM
  8. NVIDIA (code). Triton Inference Server
  9. Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
  10. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  11. Sheng et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  12. Chen et al. (2023). Punica: Multi-Tenant LoRA Serving
  13. vLLM project (docs). vLLM documentation: LoRA adapters (--enable-lora, --max-loras, --max-lora-rank, dynamic loading)
  14. vLLM project (code). vLLM: with_cancellation and listen_for_disconnect (entrypoints/serve/utils/api_utils.py); abort on CancelledError (v1/engine/async_llm.py)
  15. vLLM project (code). vLLM: incremental detokenization (v1/engine/detokenizer.py, tokenizers/detokenizer_utils.py)
  16. SGLang project (code). SGLang router (sgl-router): cache-aware and power-of-two routing
  17. NVIDIA (2023). NVIDIA H100 Tensor Core GPU (specifications)