LLM & VLM courseBlog
81Track 5: LLM inference and serving · session 4 of 65

Inference metrics: latency, memory and throughput

Before you can make inference faster, you must say what “faster” means. This session defines the numbers every serving team watches: time to the first token, time per output token, end-to-end latency, model size, KV-cache size, peak memory, tokens per second and requests per second. It shows why one user’s speed and the whole system’s throughput pull in opposite directions, why only requests that meet a latency target (goodput) should count, and how Little’s law links request rate, latency and the number of requests in memory.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 78 and 80 (the generation loop, the KV cache)

What you will learn

  • The three latency metrics: time to first token (TTFT), time per output token (TPOT) and end-to-end latency, how they relate, and how inter-token latency differs from TPOT
  • A best-case model of where the time goes: prefill limited by compute, decode limited by memory bandwidth, with real numbers for Llama 3.1 8B on an H100
  • The three memory metrics: model size, KV-cache size and peak memory, and how to find the largest batch that fits
  • Throughput as tokens per second and requests per second, and why per-user speed falls while system throughput rises as the batch grows
  • Why serving teams report P50, P90 and P99 and not the mean
  • Goodput: throughput that counts only requests that meet a latency target (SLO)
  • Little’s law L = λW, and how to use it for capacity planning and to check a benchmark

Why this matters

“Make the model faster” is not a goal you can work on. Faster for whom? The user who waits for the first word, the user who reads a long answer, or the company that pays for the GPUs? Each of these is a different number, and improving one often makes another worse.

In the last two sessions you saw how a model writes text: one big prefill pass over the prompt, then a decode loop that makes one token per step and keeps the keys and values of all earlier tokens in the KV cache (session 78, session 80). This session gives you the words and the formulas to measure that loop. They come in three groups, and the efficient-inference survey uses the same three groups [Certain]:

  • Latency (how long one request waits): time to first token, time per output token, and end-to-end latency.
  • Memory (what must fit on the GPU): model size, KV-cache size, and peak memory.
  • Throughput (how much work the whole system does): tokens per second and requests per second.

Then we add the two ideas that real serving teams use every day: goodput, which counts only the requests that meet a latency target, and Little’s law, which tells you how many requests are inside your system at once, and so how much KV-cache memory you need.

Every serving system you will meet later in this track (vLLM, SGLang, TensorRT-LLM, TGI) reports these metrics, and every optimization you will study (batching, quantization, speculative decoding, disaggregation) is judged by how it moves them. Learn them once, precisely, and the rest of Track 5 becomes much easier to read.

One request on a clock

Follow one chat request from the moment it reaches the server. Four moments matter:

  1. \(t_{\text{arr}}\): the request arrives. It may wait in a queue if the GPU is busy.
  2. The server starts the prefill: one forward pass over all \(P\) prompt tokens. At its end the model has the KV cache of the prompt and the first output token.
  3. \(t_1\): the first output token is ready and is sent (streamed) to the user.
  4. \(t_2, t_3, \dots, t_n\): each decode step makes one more token. After token \(n\) (or an end-of-sequence token), the request is done.

From these timestamps we define all latency metrics.

\[ \text{TTFT} = t_1 - t_{\text{arr}} = T_{\text{queue}} + T_{\text{prefill}} \tag{81.1} \]
\[ \text{ITL}_i = t_i - t_{i-1}, \qquad i = 2, \dots, n \tag{81.2} \]
\[ \text{TPOT} = \frac{t_n - t_1}{n - 1} = \frac{1}{n-1}\sum_{i=2}^{n} \text{ITL}_i \tag{81.3} \]
\[ \text{E2E} = t_n - t_{\text{arr}} = \text{TTFT} + (n - 1)\cdot \text{TPOT} \tag{81.4} \]
Read the symbols
TTFT
time to first token, also called first-token latency: how long the user stares at an empty screen. It includes waiting in the queue and the prefill.
\(\text{ITL}_i\)
inter-token latency: the gap between two output tokens that follow each other. There is one gap per token, so it has a whole distribution inside one request.
TPOT
time per output token, also called per-output-token latency: the average gap after the first token, for one request. The first token is left out because its time is already in TTFT.
E2E
end-to-end latency (the survey calls it generation latency): the time from arrival to the last token. It is what a user of a non-streaming API feels.
\(n\)
the number of output tokens of this request.

Two numbers you will often see are simple inverses: per-user speed \(= 1/\text{TPOT}\) tokens per second (TPOT = 20 ms means 50 tokens/s for that user), and the normalized latency E2E \(/\, n\), which the vLLM paper uses to compare systems across requests of different length [Likely].

Which number matters depends on the product. For a chat app that streams, users judge TTFT first (“is it working?”) and then TPOT (“can I read along?”). Adults read roughly 200 to 300 words per minute, about 4 to 5 words per second [Likely], so a TPOT far below the reading speed brings little extra value to a person reading along [Speculation]. For an agent that calls the model ten times in a row, or a non-streaming API that returns JSON, only E2E matters. For a code-completion box, TTFT and a short E2E matter most.

waiting in the queueprefill (whole prompt, one pass)decode steps (one token each)
Figure 1. The life of one request for Llama 3.1 8B in BF16 on one H100, using the best-case model of formulas (81.5) and (81.6). Each orange tick is one output token reaching the user. The times are lower bounds: a real server is slower, but the shape is the same.
How to read Figure 1

Start with the defaults: a 2,048-token prompt costs about 33 ms of prefill, and each of the 499 decode steps costs about 4.9 ms, so the decode part dominates E2E. Now drag the prompt to 16,384 tokens: TTFT grows 8 times, but TPOT barely moves, because a decode step is dominated by reading the 16 GB of weights. Now raise the batch: TPOT rises, because each step must also read the KV cache of every other request in the batch. Finally add queue time: it adds directly to TTFT and E2E, and the GPU did no extra work. Users do not care why they wait.

Four details that trip people up

  • TPOT vs ITL. TPOT is one number per request (an average). ITL is one number per token. A request can have a good TPOT of 15 ms but one gap of 300 ms, for example when the server stops all decoding to run a long prefill for a new request. The user sees the text freeze. This is why systems such as Sarathi-Serve report the tail of the ITL distribution, not only TPOT [Certain]. You will meet the fix, chunked prefill, in session 135.
  • Where you measure. The client sees network time, tokenization and detokenization on top of the server’s numbers. Always say which clock you used.
  • What a “token” is. Different tokenizers cut the same text into different numbers of tokens (session 8). Two models with the same tokens per second can produce text at different speeds. Compare in words or characters per second when tokenizers differ.
  • Averages over what. “Mean TPOT” can mean the mean over requests (each request counts once) or over tokens (long requests count more). They differ when long and short requests see different batch sizes.

Where the time goes: a best-case model

We do not need a profiler to estimate these numbers. Two facts from session 25 and session 42 are enough. A forward pass of a model with \(N\) parameters costs about \(2N\) FLOPs (floating-point operations) per token. And every forward pass must read all the weights from GPU memory (HBM) at least once, whatever the number of tokens in the pass.

Prefill processes \(P\) tokens in one pass. That is a lot of math per byte of weights read, so the GPU’s compute speed \(F\) (FLOPs per second) sets the limit:

\[ T_{\text{prefill}} \gtrsim \frac{2 N P}{F} \tag{81.5} \]

Decode processes only one token per request per step. With \(B\) requests in the batch, the step does \(2NB\) FLOPs but must read the weights (\(M_w\) bytes) and the KV cache of every request. Each request has a context of \(c\) tokens and \(s_{\text{kv}}\) bytes of cache per token. The step takes at least the larger of the memory time and the compute time:

\[ T_{\text{step}}(B) \gtrsim \max\!\left( \frac{M_w + B\, c\, s_{\text{kv}}}{\text{BW}},\ \ \frac{2 N B}{F} \right) \tag{81.6} \]
Read the symbols
\(N\)
number of parameters (8.03 billion for Llama 3.1 8B)
\(F\)
peak compute speed of the GPU in FLOPs per second (H100 SXM, BF16, dense: about \(989 \times 10^{12}\))
\(\text{BW}\)
memory bandwidth in bytes per second (H100 SXM: \(3.35 \times 10^{12}\))
\(M_w\)
bytes of weights: \(N\) times the bytes per weight (2 for BF16)
\(s_{\text{kv}}\)
KV-cache bytes per token: \(2 \cdot L \cdot n_{kv} \cdot d_h \cdot\) bytes per value (from session 80)
\(B, c\)
batch size (requests decoding together) and context length of each request

The “\(\gtrsim\)” is important. These are lower bounds: they assume the GPU runs at 100% of its peak, which never happens, and they ignore attention FLOPs (small for short prompts, large for long ones). Real numbers are higher. But the bounds tell you which resource limits each phase, and that is what decides which optimization helps. The next session, session 82, turns this into the roofline picture.

Worked example: Llama 3.1 8B on one H100, one user

Weights: \(8.03 \times 10^9 \times 2 = 16.06\) GB. KV cache: \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes = 128 KiB per token.

Prefill of a 2,000-token prompt: \(2 \cdot 8.03\times10^9 \cdot 2000 / 989.5\times10^{12} = 32.5\) ms. So TTFT \(\ge 32.5\) ms (with an empty queue).

One decode step with \(B = 1\) and 2,000 tokens of context: memory time \((16.06 + 0.26)\) GB \(/\, 3.35\) TB/s \(= 4.87\) ms; compute time \(2 \cdot 8.03\times10^9 / 989.5\times10^{12} = 0.016\) ms. Memory wins by about 300 times. So TPOT \(\ge 4.87\) ms, or at most about 205 tokens/s for this user.

E2E for 500 output tokens: \(32.5 + 499 \times 4.87 \approx 2{,}464\) ms. Decode is 99% of the time, even though prefill handled 4 times more tokens. This is the typical shape of a chat request.

Memory: model size, KV cache, and peak memory

Latency is about time; memory is about whether the job can run at all. Three numbers:

  • Model size \(M_w\): bytes to store the weights. It is fixed once you choose the model and its precision.
  • KV-cache size \(M_{kv}\): bytes to store keys and values of every token of every request in flight. It grows with batch and context.
  • Peak memory: the most memory used at any moment during generation. It is approximately weights plus KV cache, plus a smaller amount for activations, temporary buffers and the framework itself.
\[ M_{\text{peak}} \approx \underbrace{N \cdot b_w}_{M_w} \;+\; \underbrace{2 L\, n_{kv}\, d_h\, b_{kv} \cdot \textstyle\sum_{r} c_r}_{M_{kv}} \;+\; M_{\text{act}} \tag{81.7} \]
Read the symbols
\(b_w, b_{kv}\)
bytes per weight and per cached value (2 for BF16, 1 for 8-bit, 0.5 for 4-bit)
\(L, n_{kv}, d_h\)
number of layers, number of KV heads, and head size
\(\sum_r c_r\)
total tokens in the cache over all requests \(r\) in flight; with \(B\) requests of \(c\) tokens each it is \(B c\)
\(M_{\text{act}}\)
activations and work buffers; for inference this is small next to the others, except for very long prefills

Serving engines use this budget directly. vLLM, for example, takes a fraction of the GPU memory (by default 0.9) [Likely], loads the weights, measures the activation peak with a test pass, and gives everything that is left to the KV cache ([[D:vllm]]). The KV space, divided by the bytes per token, is the number of tokens the server can hold at once. That number, not compute, usually decides the largest batch.

Worked example: the largest batch that fits

Llama 3.1 8B in BF16 on an 80 GB H100, keeping 90% (72 GB) for the model and cache. KV space \(= 72 - 16.06 = 55.94\) GB. One request with 4,096 tokens of context needs \(4096 \times 131{,}072\) bytes \(= 0.537\) GB. So at most \(\lfloor 55.94 / 0.537 \rfloor = 104\) such requests fit at once. With 32 requests the peak is \(16.06 + 17.18 = 33.24\) GB.

The same GPU with one request at the full 128K context (131,072 tokens): the cache is 17.18 GB, more than the weights. Long context is a memory problem before it is a speed problem.

Model
WeightsKV cache
GPU memory
weights (model size)KV cachefree
Figure 2. The memory budget of formula (81.7). The dashed line is the usable 90% of the GPU memory; the rest is kept for activations, buffers and the framework. Model shapes are read from each model’s config.json.
How to read Figure 2

With Llama 3.1 8B, the weights are a fixed block and the KV cache grows with \(B \times c\). Set the context to 128K (the right end of the slider): about three requests fill the GPU. Switch to Qwen2.5 7B: it has only 4 KV heads and 28 layers, so its cache per token is 56 KiB instead of 128 KiB, and more than twice as many requests fit. Switch to Llama 3.1 70B: 141 GB of BF16 weights do not fit on one 80 GB GPU at all; 4-bit weights (35 GB) do, and leave room for about 112K tokens of cache at 320 KiB per token.

Throughput: per user and for the whole system

Throughput is the rate of work done by the whole server. Two units are common:

  • Token throughput: generated (output) tokens per second, summed over all requests. Some benchmarks also count prompt tokens (“total tokens per second”); always check which one a report means, because prompt tokens are much cheaper.
  • Request throughput: completed requests per second. It depends on how long the requests are, so it only makes sense together with a description of the workload.

In the decode phase, one step of formula (81.6) produces one token for each of the \(B\) requests. So

\[ \text{per-user speed} = \frac{1}{T_{\text{step}}(B)}, \qquad \text{system throughput} = \frac{B}{T_{\text{step}}(B)} \tag{81.8} \]

Here is the central trade-off of LLM serving. When \(B\) is small, \(T_{\text{step}}\) is almost all weight reading, which costs the same for 1 request or 64. So adding requests is nearly free: system throughput grows almost linearly and each user barely slows down. When \(B\) is large, the KV-cache reads (and, much later, the compute) grow with \(B\), each step gets longer, and every user gets slower. Throughput keeps rising, but more and more slowly.

system throughput (tokens/s)TPOT (ms)over the target or out of memory
Figure 3. Decode throughput and TPOT against batch size (log scale) for Llama 3.1 8B in BF16 on one H100, from formulas (81.6) and (81.8). The shaded part on the right is where the KV cache no longer fits in the usable 72 GB. The best batch is the largest one that is under the TPOT target and inside memory.
How to read Figure 3

At \(c = 2{,}048\) and \(B = 64\), TPOT is about 10 ms (100 tokens/s per user) and the system makes about 6,500 tokens/s: about 31 times the single-user rate, for only a 2× slowdown per user. Lower the TPOT target to 10 ms: the best batch drops, and so does throughput. That is the price of a stricter latency promise. Raise the context: the memory wall moves left and the TPOT curve bends up earlier, because each request brings a bigger cache.

Worked example: what a 15 ms TPOT target buys

With 2,000 tokens of context per request, formula (81.6) gives TPOT = 9.80 ms at \(B = 64\) (6,529 tokens/s), and 14.97 ms at \(B = 130\) (8,686 tokens/s). \(B = 130\) is the largest batch under 15 ms, and its cache (34.1 GB) fits in the 55.9 GB of KV space. A 10 ms target allows only \(B = 66\) (6,627 tokens/s); a 20 ms target allows \(B = 194\) (9,712 tokens/s, 50.9 GB of cache). Doubling the allowed TPOT from 10 to 20 ms gives about 47% more tokens per GPU. These are best-case numbers from the model; the trend is what you should remember.

Prefill changes the picture. A prefill of 2,000 tokens takes the GPU for about 32 ms of full compute, during which, in a simple server, no one decodes. With many new requests per second, prefill can use most of the GPU, and the decoding users see the ITL spikes described above. This interaction between new and running requests is why throughput, latency and arrival rate must be studied together, which is the job of the next two sections.

Tail latency: P50, P90, P99

A server handles thousands of requests, and each has its own TTFT and TPOT. We describe them with percentiles: P50 (the median) is the value half of the requests beat; P99 is the value 99% of requests beat. The tail (P90, P99) is what your unhappiest users feel, and at scale “1% of users” is a lot of people.

The mean is a bad summary for latency. Latency distributions have long right tails: most requests are fast, a few wait behind long prompts or a full batch. A mean of 200 ms can hide a P99 of 2 seconds. In the request log of the code below, the P50 TTFT is 0.18 s but the P99 is 0.96 s, more than 5 times higher. Service targets are therefore written on percentiles: “P99 TTFT under 1 s”.

Goodput: throughput that keeps its promise

A service-level objective (SLO) is a latency target that the service promises, for example “\(\text{TTFT} \le 1\) s and \(\text{TPOT} \le 15\) ms”. A request that misses it is, for the product, almost as bad as a failed request: the user has already left. Goodput counts only the requests that met the SLO:

\[ \text{goodput} = \frac{1}{T}\, \Big|\{\, r : \text{TTFT}_r \le \tau_{\text{ttft}} \ \text{and}\ \text{TPOT}_r \le \tau_{\text{tpot}} \,\}\Big| \tag{81.9} \]
Read the symbols
\(T\)
the length of the measurement window in seconds
\(r\)
a request completed in the window
\(\tau_{\text{ttft}}, \tau_{\text{tpot}}\)
the SLO thresholds
\(|\{\dots\}|\)
the number of requests that met both targets

The fraction of requests that meet the SLO is the SLO attainment. DistServe, which made goodput popular for LLM serving, uses a related system-level definition: the largest request rate the system can sustain while keeping attainment above a target such as 90%, per GPU [Certain]. Its result is reported in these units: up to 7.4 times more requests, or a 12.6 times tighter SLO, than earlier systems [Certain]. Both definitions say the same thing: throughput beyond the point where users suffer is worthless.

Why does this matter so much? Because throughput and goodput separate exactly when a system is pushed hard. As the request rate grows, raw throughput keeps going up (the batch is fuller), but latency goes up too, and at some point most requests miss the SLO. Goodput then falls, sometimes to almost zero, while the throughput chart still looks great. You will see this in Figure 4.

Little’s law: how many requests are inside

Here is a law that holds for any stable system where things arrive, stay for a while, and leave: a shop, a call center, a GPU server.

\[ L = \lambda\, W \tag{81.10} \]
Read the symbols
\(L\)
the average number of requests inside the system (waiting plus running): the concurrency
\(\lambda\)
the average arrival rate (requests per second); in a stable system it equals the completion rate
\(W\)
the average time a request spends inside: for us, the mean E2E latency

John Little proved in 1961 that this holds for long-run averages with almost no assumptions: any arrival pattern, any service order, any distribution of service times [Certain]. The only requirement is that the system is stable: on average, requests leave as fast as they arrive. If the arrival rate is above what the server can handle, the queue grows forever, \(W\) has no limit, and the law no longer describes a steady state.

For LLM serving, Little’s law connects the three groups of metrics. Concurrency \(L\) is the batch the server must hold, so it sets the KV-cache memory. \(W\) is the latency. \(\lambda\) is the request throughput. Fix any two and the third follows.

Worked example: capacity planning with Little’s law

A product expects \(\lambda = 20\) requests/s at peak, with a mean E2E of \(W = 4\) s. Then \(L = 20 \times 4 = 80\) requests are in flight on average. If an average request holds about 1,500 tokens in the cache, Llama 3.1 8B needs \(80 \times 1500 \times 131{,}072\) bytes \(= 15.7\) GB of KV cache on average, plus the 16.06 GB of weights. One H100 has room for that; but check the peaks, not only the average, and check that the GPU can also do the compute for 20 prefills per second.

The law also checks a benchmark. If your load test reports 9.6 requests/s, a mean E2E of 4.7 s and an average of 25 requests running, something is wrong: \(\lambda W = 45\), not 25. Either requests were dropped, or the clocks measured different things.

request throughputgoodput (\(\text{TTFT} \le 1\) s and \(\text{TPOT} \le\) target)
Figure 4. A simulated server: Llama 3.1 8B in BF16 on one H100 with the best-case step times of (81.5) and (81.6), continuous batching (new requests join at the next step, and their whole prefill runs in that step), first come first served. Requests arrive at random (Poisson) with prompts of 256 to 2,048 tokens and answers of 64 to 512 tokens. Each point is a 60-second measurement after a 10-second warm-up. The random numbers are seeded, so every run gives the same result.
How to read Figure 4

At the defaults (12 requests/s, max batch 128, TPOT ≤ 15 ms), every request meets the SLO, and the measured concurrency matches \(\lambda W\). Move \(\lambda\) up slowly. Throughput keeps rising until the GPU is full, near 23 requests/s. Goodput follows it up to about 18 requests/s, then crashes: the batch gets so full and prefills so frequent that TPOT passes 15 ms for most requests. Loosen the TPOT target to 30 ms and goodput follows throughput up to about 22 requests/s. Now set max batch to 16: the server cannot hold enough requests, the queue explodes (huge TTFT), and Little’s law “breaks” because the system is not stable any more.

Three lessons from Figure 4 that hold for real servers too. First, the knee is sharp: a server that is fine at 80% load can be terrible at 95%. Capacity plans keep a safety margin. Second, the limit can come from different places: memory (max batch too small, so the queue grows and TTFT explodes), or compute and bandwidth (the batch is large, so TPOT grows). Third, the best operating point depends on the SLO, not only on the hardware. Splitwise and DistServe go one step further and run prefill and decode on different GPUs, so prefills stop hurting the TPOT of decoding requests [Certain]; you will study this in session 137.

Measuring it right

A good benchmark sends requests the way real users do and records a timestamp for every token. Two kinds of load generator exist. A closed-loop generator keeps a fixed number of users, each sending a new request only after the last one finished. An open-loop generator sends requests at a fixed average rate (often Poisson arrivals), whether or not the server keeps up. A closed loop hides overload: when the server slows down, the users slow down too, and by Little’s law the arrival rate falls to \(L/W\) by itself. Real traffic does not wait for your server, so measure SLOs with an open loop.

Algorithm 1: measure the latency metrics of one load level
  1. input: server, request set (prompts and output lengths from real traffic), arrival rate \(\lambda\), duration \(T\), warm-up \(T_0\)
  2. send a few requests and throw away their timings // warm-up: compilation, caches, CUDA graphs
  3. draw arrival times with gaps \(\sim \text{Exponential}(\lambda)\) // open loop
  4. for each request \(r\), in its own task: record \(t_{\text{arr}}\); stream the answer; record \(t_1, \dots, t_n\)
  5. keep only requests with \(T_0 \le t_{\text{arr}} \le T_0 + T\)
  6. \(\text{TTFT}_r \leftarrow t_1 - t_{\text{arr}}\); \(\text{TPOT}_r \leftarrow (t_n - t_1)/(n_r - 1)\); \(\text{E2E}_r \leftarrow t_n - t_{\text{arr}}\) // (81.1), (81.3), (81.4)
  7. collect all gaps \(t_i - t_{i-1}\) over all requests // ITL distribution, (81.2)
  8. report P50, P90, P99 of TTFT, TPOT, ITL and E2E; tokens/s and requests/s; SLO attainment and goodput (81.9)
  9. check \(\bar L \approx \lambda \cdot \overline{\text{E2E}}\) // Little’s law (81.10): a sanity test of the log
Algorithm 2: find the goodput of a deployment (largest rate that keeps the SLO)
  1. input: SLO \((\tau_{\text{ttft}}, \tau_{\text{tpot}})\), attainment target \(a\) (for example 0.9), rate range \([\lambda_{\text{lo}}, \lambda_{\text{hi}}]\)
  2. while \(\lambda_{\text{hi}} - \lambda_{\text{lo}} \gt \delta\):
  3. \(\lambda \leftarrow (\lambda_{\text{lo}} + \lambda_{\text{hi}})/2\)
  4. run Algorithm 1 at rate \(\lambda\); attainment \(\leftarrow\) fraction of requests meeting both targets
  5. if attainment \(\ge a\): \(\lambda_{\text{lo}} \leftarrow \lambda\) else \(\lambda_{\text{hi}} \leftarrow \lambda\)
  6. return \(\lambda_{\text{lo}}\) // requests/s per deployment; divide by GPUs for per-GPU goodput

Binary search works because attainment falls as the rate grows (more load never helps latency). Repeat the whole search for each candidate setting (max batch, chunk size, quantization, number of GPUs) and compare goodput per GPU, or per dollar.

The PyTorch code below does the three things of this session on a small scale. It times prefill and decode of a tiny model with a KV cache; it computes model and KV memory for Llama 3.1 8B; and it computes throughput, goodput, percentiles and a Little’s-law check from a request log. The log is synthetic and seeded, so those numbers are exactly reproducible; the timings depend on your machine.

import time, torch
torch.manual_seed(0)

# 1) a tiny decoder with a KV cache, so we can time prefill and decode
class Block(torch.nn.Module):
    def __init__(s, d, h):
        super().__init__(); s.h = h
        s.qkv = torch.nn.Linear(d, 3 * d); s.o = torch.nn.Linear(d, d)
        s.ff = torch.nn.Sequential(torch.nn.Linear(d, 4 * d), torch.nn.GELU(), torch.nn.Linear(4 * d, d))
        s.n1 = torch.nn.LayerNorm(d); s.n2 = torch.nn.LayerNorm(d)
    def forward(s, x, cache):
        B, T, d = x.shape
        q, k, v = s.qkv(s.n1(x)).view(B, T, 3, s.h, d // s.h).permute(2, 0, 3, 1, 4)
        if cache is not None:                       # append the new K, V to the cache
            k = torch.cat([cache[0], k], 2); v = torch.cat([cache[1], v], 2)
        a = torch.nn.functional.scaled_dot_product_attention(q, k, v, is_causal=cache is None)
        x = x + s.o(a.transpose(1, 2).reshape(B, T, d))
        return x + s.ff(s.n2(x)), (k, v)

class TinyLM(torch.nn.Module):
    def __init__(s, V=1000, d=256, h=8, L=4):
        super().__init__()
        s.emb = torch.nn.Embedding(V, d); s.blocks = torch.nn.ModuleList(Block(d, h) for _ in range(L))
        s.head = torch.nn.Linear(d, V)
    def forward(s, ids, caches=None):
        x = s.emb(ids); new = []
        for i, b in enumerate(s.blocks):
            x, c = b(x, None if caches is None else caches[i]); new.append(c)
        return s.head(x), new

@torch.no_grad()
def generate_timed(model, prompt, n_new):
    """Greedy generation. Returns the ids and the time (s) at which each output token was ready."""
    t0 = time.perf_counter(); stamps = []
    logits, caches = model(prompt)                          # prefill: the whole prompt in one pass
    tok = logits[:, -1].argmax(-1, keepdim=True); out = [tok]; stamps.append(time.perf_counter() - t0)
    for _ in range(n_new - 1):                              # decode: one token per pass
        logits, caches = model(tok, caches)
        tok = logits[:, -1].argmax(-1, keepdim=True); out.append(tok); stamps.append(time.perf_counter() - t0)
    return torch.cat(out, 1), stamps

# 2) latency metrics from token timestamps, formulas (81.1) to (81.4)
def latency_metrics(queue_s, stamps):
    ttft = queue_s + stamps[0]
    itl = [b - a for a, b in zip(stamps, stamps[1:])]       # inter-token gaps
    e2e = queue_s + stamps[-1]
    tpot = (e2e - ttft) / (len(stamps) - 1)                 # mean gap after the first token
    return dict(ttft=ttft, tpot=tpot, e2e=e2e, itl_max=max(itl))

model = TinyLM().eval()
prompt = torch.randint(0, 1000, (1, 512))
generate_timed(model, prompt, 8)                            # warm-up: the first call is much slower
ids, stamps = generate_timed(model, prompt, 64)
m = latency_metrics(0.0, stamps)
print({k: round(v * 1e3, 2) for k, v in m.items()}, "ms  (depends on your machine)")
print("per-user speed:", round(1 / m["tpot"]), "tokens/s")
_, s1 = generate_timed(model, prompt[:, :64], 64)         # 8x shorter prompt: much shorter TTFT
print("TTFT with 512 vs 64 prompt tokens (ms):", round(stamps[0] * 1e3, 2), round(s1[0] * 1e3, 2))

# 3) memory: weights + KV cache, formula (81.7), for Llama 3.1 8B in BF16
def peak_memory_gb(n_params, L, n_kv, d_head, batch, ctx, bytes_w=2, bytes_kv=2):
    weights = n_params * bytes_w
    kv = 2 * L * n_kv * d_head * bytes_kv * batch * ctx
    return weights / 1e9, kv / 1e9, (weights + kv) / 1e9
print("8B, batch 32, 4096 tokens:", [round(x, 2) for x in peak_memory_gb(8.03e9, 32, 8, 128, 32, 4096)])
# [16.06, 17.18, 33.24]

# 4) throughput, goodput, percentiles and Little's law from a request log (deterministic)
g = torch.Generator().manual_seed(1)
N, rate = 2000, 10.0                                         # 10 requests per second on average
arrive = torch.cumsum(torch.empty(N).exponential_(rate, generator=g), 0)
n_out = torch.randint(64, 513, (N,), generator=g).float()
ttft = 0.05 + torch.empty(N).exponential_(5.0, generator=g)  # mean 0.25 s, long right tail
tpot = 0.012 + 0.006 * torch.rand(N, generator=g)
e2e = ttft + (n_out - 1) * tpot                              # formula (81.4)
T = (arrive + e2e).max() - arrive.min()                      # wall time of the whole log
print("request throughput:", round(N / T.item(), 2), "req/s   output tokens/s:", round(n_out.sum().item() / T.item()))
ok = (ttft <= 0.5) & (tpot <= 0.015)                         # SLO: TTFT <= 0.5 s and TPOT <= 15 ms
print("SLO attainment:", round(ok.float().mean().item(), 3), "  goodput:", round(ok.sum().item() / T.item(), 2), "req/s")
lam = (N / (arrive[-1] - arrive[0])).item()                  # arrival rate
W = e2e.mean().item()                                        # mean time in the system
t = torch.linspace(arrive[100].item(), arrive[-100].item(), 20000)   # stay away from the edges
L_meas = ((arrive[None] <= t[:, None]) & (arrive[None] + e2e[None] > t[:, None])).sum(1).float().mean().item()
print(f"Little: lambda*W = {lam * W:.1f}, measured concurrency = {L_meas:.1f}")
for q in (0.5, 0.9, 0.99):
    print(f"P{int(q*100)} TTFT = {torch.quantile(ttft, q).item():.3f} s")
# request throughput: 9.6 req/s   output tokens/s: 2756
# SLO attainment: 0.435   goodput: 4.18 req/s
# Little: lambda*W = 45.2, measured concurrency = 45.4
# P50 TTFT = 0.181 s,  P90 TTFT = 0.516 s,  P99 TTFT = 0.963 s

Notice in the output: throughput is 9.6 requests/s but goodput is only 4.18, because in this synthetic log half of the requests have a TPOT above 15 ms and about 11% have a TTFT above 0.5 s. And \(\lambda W\) matches the measured concurrency within half a percent. In production you get the same signals from the server’s own metrics. vLLM, for example, exports histograms named vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:e2e_request_latency_seconds and vllm:request_queue_time_seconds, and gauges such as vllm:num_requests_running and vllm:kv_cache_usage_perc [Likely] (names change between versions; check the docs of your version).

Where this is used

SettingMetric that matters mostWhy
Streaming chat assistantP99 TTFT and P99 TPOT (or ITL), goodput per GPUUsers judge the first word and the reading flow; cost is GPUs per user
Agents and tool calls (many model calls in a chain)E2E per call, summed over the chainNo one reads the middle steps; the total wait is what counts
Code completion in an editorTTFT and E2E of short outputs, tight P99A suggestion that comes after the user typed on is useless
Offline batch jobs (labeling, synthetic data, evaluation)Token throughput per dollar; latency hardly mattersRun the largest batch that fits in memory
Long-document and RAG question answeringTTFT (long prefill) and KV-cache memoryPrompts of tens of thousands of tokens make prefill and cache the bottleneck
Video question answering and Video-RAG with a VLMTTFT, because of many visual tokens\(64\) frames \(\times\ 196\) tokens is \(12{,}544\) tokens before the question; ideal prefill on an H100 for an 8B model is already 0.2 s
Live video: real-time key-moment alerts or live captionsE2E per clip must be shorter than the clip intervalIf a 2-second clip takes 3 seconds to describe, the backlog grows forever (an unstable queue)
Video retrieval, classification and fingerprinting over an archiveRequest throughput (clips per second) of the encoderOffline indexing; no user is waiting, so batch for throughput
Capacity planning for a new productLittle’s law with peak \(\lambda\) and target \(W\)Gives concurrency, hence KV memory and the number of GPUs

Common mistakes

  • Reporting only tokens per second. Without the batch size, the latency at that load and the workload, the number says almost nothing. Report throughput together with P99 TTFT and TPOT at the same load.
  • Using the mean instead of percentiles. The mean hides the tail that users complain about.
  • Counting the first token in TPOT. TPOT is \((t_n - t_1)/(n-1)\); including the first token mixes prefill into the decode metric.
  • Benchmarking with a closed loop and claiming SLOs. A fixed number of users waiting on each other cannot overload the server, so queueing delay never shows.
  • No warm-up. The first requests pay for compilation, memory allocation and CUDA graph capture; the code above shows a first call many times slower.
  • Forgetting the KV cache in memory plans. “The 16 GB model fits on a 24 GB GPU” says nothing about how many 8K-token users fit (about 5 when 90% of the memory is usable).
  • Applying Little’s law to an overloaded system. If the queue grows during the test, there is no steady state and \(L\), \(W\) depend on the test length.
  • Comparing tokens per second across tokenizers. Compare words or characters per second, or the same task end to end.

Interview questions

Define TTFT, TPOT and end-to-end latency. How are they related?basic
TTFT is the time from the request’s arrival until the first output token: queue time plus prefill. TPOT is the average time between output tokens after the first: \((t_n - t_1)/(n-1)\). End-to-end latency is arrival to last token. They are linked by \(\text{E2E} = \text{TTFT} + (n-1)\,\text{TPOT}\). Per-user speed is \(1/\text{TPOT}\).
Why is TTFT usually limited by compute while TPOT is limited by memory bandwidth?core
Prefill runs all \(P\) prompt tokens through the model in one pass: about \(2NP\) FLOPs for one read of the weights, so hundreds or thousands of FLOPs per byte, which keeps the tensor cores busy. A decode step handles only one token per request: \(2NB\) FLOPs but it must still read all weights plus the KV cache. For small \(B\) that is about 1 FLOP per byte, far below what the GPU could do, so the step time is bytes divided by bandwidth. For Llama 3.1 8B on an H100, one step at \(B = 1\) needs about 4.9 ms for memory and 0.016 ms for compute.
Explain the throughput–latency trade-off of batching in decode.core
A decode step reads the weights once whatever the batch, so adding requests shares that cost: system throughput \(B / T_{\text{step}}(B)\) rises almost linearly at first. But each request also adds its KV-cache reads (and, at large \(B\), compute), so \(T_{\text{step}}\) grows and each user’s speed \(1/T_{\text{step}}\) drops. Example from the best-case model with 2,000-token contexts: \(B=1\) gives 205 tokens/s per user and 205 total; \(B=64\) gives 102 per user and about 6,500 total. You choose the largest batch that keeps TPOT under the SLO and the KV cache inside memory.
What is goodput, and why is it a better target than throughput?core
Goodput counts only completed requests that met the SLO (for example \(\text{TTFT} \le 1\) s and \(\text{TPOT} \le 50\) ms), per second; or, as in DistServe, the largest request rate at which at least 90% of requests meet the SLO, often per GPU. Throughput keeps rising as you push more load, even when most users get bad latency, so optimizing throughput can make the product worse. Goodput rises with load only while latency stays acceptable, then falls, so its peak is the real capacity.
State Little’s law and use it for capacity planning.core
\(L = \lambda W\): average number of requests in the system equals arrival rate times average time in the system, for any stable system. With 20 requests/s and 4 s mean E2E, 80 requests are in flight. Multiply by the average tokens per request and the KV bytes per token to get cache memory: \(80 \times 1{,}500 \times 128\ \text{KiB} \approx 15.7\) GB for Llama 3.1 8B. Add the weights and compare to GPU memory; then check compute for the prefills. If \(W\) must drop to 2 s, only 40 slots are needed, but each must run faster.
Why report P99 and not the mean latency?basic
Latency distributions have long right tails (queueing, long prompts, prefill stalls). The mean is pulled by both the many fast and the few slow requests and describes neither well. P99 tells you what the worst 1% experience, and at scale that is many users; SLOs are therefore written on percentiles. Also, percentiles of sums are not sums of percentiles, so compute each one from per-request data.
P99 TTFT got much worse after a traffic increase, but TPOT looks fine. What is happening and what do you check?advanced
Most likely requests are waiting in the queue: the server is at its concurrency limit (KV-cache slots or max batch), so new requests cannot be admitted even though running ones decode fine. Check queue time (for example vLLM’s request queue time histogram), the number of waiting requests, and KV-cache usage near 100%. Other causes: a few very long prompts that hold the GPU in prefill, or preemption and recomputation. Fixes: more KV space (quantize weights or cache, more GPUs), prefix caching, chunked prefill, admission control, or routing long prompts to separate replicas.
TPOT vs inter-token latency: why might a team track the ITL tail separately?advanced
TPOT averages all gaps of a request, so a single long stall barely changes it. ITL keeps every gap. In servers that run a new request’s full prefill in the same step as ongoing decodes, all decoding users freeze for the length of that prefill: tens to hundreds of ms for long prompts. Users see this as stuttering. P99 ITL exposes it; chunked prefill (Sarathi-Serve) and prefill–decode disaggregation (Splitwise, DistServe) are designed to remove it.
How do you estimate the largest batch that fits on a GPU?core
Usable memory (say 90% of the GPU) minus weights minus an activation reserve gives the KV space. Divide by the KV bytes per token \(2 L n_{kv} d_h b_{kv}\) to get the token capacity, then by the expected tokens per request. Llama 3.1 8B BF16 on 80 GB: \(72 - 16.06 = 55.9\) GB; at 128 KiB per token that is about 427K tokens, so 104 requests of 4,096 tokens. With paging (vLLM) you can plan with the average length instead of the maximum.
What is wrong with benchmarking a server using 32 concurrent clients in a loop?advanced
That is a closed loop: each client waits for its answer before sending the next request. When the server slows down, the arrival rate falls by itself (by Little’s law, \(\lambda = 32/W\)), so queues never build and TTFT looks good. Real users arrive independently. Use an open-loop generator with Poisson arrivals at a fixed rate, sweep the rate, and report latency percentiles and goodput at each rate. Closed loops are fine for measuring maximum throughput, not for SLOs.
For a VLM that answers questions about videos, which metrics would you set targets on?core
Video adds thousands of visual tokens to every prompt (for example \(64 \times 196 = 12{,}544\) for 64 frames), so TTFT and KV-cache memory dominate: the prefill is long and each request holds a big cache. Set targets on P99 TTFT (including frame decoding and the vision encoder), P99 TPOT for streamed answers, and goodput per GPU. Watch the token budget per request, because it decides how many requests fit. For live video the E2E per clip must stay below the clip interval or the system becomes unstable.

Exercises

  1. A streamed request has TTFT 0.4 s, 201 output tokens and E2E 4.4 s. Find TPOT and the per-user speed.
  2. Llama 3.1 70B has 80 layers, 8 KV heads and head size 128. Find the BF16 KV-cache bytes per token. How much memory do 16 requests of 8,192 tokens need, cache plus BF16 weights (70.55B parameters)? Does it fit on two 80 GB GPUs?
  3. Your server can hold at most 50 requests at once and the mean E2E is 5 s. What is the largest stable arrival rate?
  4. In 100 s, a server completes 1,000 requests; 930 of them meet the SLO. Give the request throughput, goodput and attainment. Does it pass a 90% attainment target?
  5. Using formula (81.6) and the H100 numbers, compute TPOT, per-user speed and system throughput for Llama 3.1 8B at \(B = 128\) and \(c = 4{,}096\). Does the cache fit in 55.9 GB?
  6. A VLM prompt has 64 frames of 196 visual tokens and 100 text tokens. Give the best-case prefill time for an 8.03B-parameter language model on one H100 with formula (81.5). What does this say about TTFT for video QA?
Answerscheck
1. \(\text{TPOT} = (4.4 - 0.4)/200 = 0.020\) s = 20 ms, so 50 tokens/s per user. 2. \(2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327{,}680\) bytes = 320 KiB per token. Cache: \(16 \times 8192 \times 327{,}680 = 42.9\) GB; weights \(70.55\times10^9 \times 2 = 141.1\) GB; total about 184 GB, which does not fit in 160 GB (and even less in the usable 90%). Four GPUs, or 8-bit weights (\(70.6 + 42.9 = 113.5\) GB), would work. 3. \(\lambda = L/W = 50/5 = 10\) requests/s; above this the queue grows without limit. 4. Throughput 10 req/s; goodput 9.3 req/s; attainment 93%, so it passes. 5. Memory time \((16.06 + 128 \cdot 4096 \cdot 131072/10^9)/3350 = (16.06 + 68.72)/3350\) s = 25.3 ms; compute \(128 \times 0.016 = 2.1\) ms; so \(\text{TPOT} \approx 25.3\) ms, 39.5 tokens/s per user, about 5,060 tokens/s in total. The cache is 68.7 GB, which does not fit in 55.9 GB: memory caps the batch at about 104 here, before the latency does. 6. \(12{,}644\) tokens; \(2 \cdot 8.03\times10^9 \cdot 12{,}644 / 989.5\times10^{12} = 0.205\) s, before the vision encoder runs and with a perfect GPU. A real TTFT will be several times that, so video QA needs fewer frames, token merging, or caching of visual tokens to meet a sub-second TTFT.

Key takeaways

  • Latency has three faces: TTFT (queue plus prefill, 81.1), TPOT (average gap after the first token, 81.3) and E2E (81.4). ITL (81.2) is the per-gap version and shows stalls that TPOT hides.
  • Prefill is limited by compute (81.5); decode is limited by reading weights and KV cache (81.6). For Llama 3.1 8B on an H100 the best case is about 32 ms of prefill per 2,000 tokens and about 4.9 ms per decode step.
  • Peak memory \(\approx\) weights + KV cache (81.7). The KV space left after the weights decides how many requests fit: 104 requests of 4,096 tokens for the 8B model on 80 GB.
  • Batching raises system throughput \(B/T_{\text{step}}\) and lowers per-user speed \(1/T_{\text{step}}\) (81.8). The best batch is the largest one under the TPOT target and inside memory.
  • Report percentiles (P50, P99), not means. Judge systems by goodput (81.9): requests per second that meet the SLO.
  • Little’s law \(L = \lambda W\) (81.10) links request rate, latency and concurrency; use it to size the KV cache and to sanity-check benchmarks, but only for stable systems.

Key formulas

\[ \text{TTFT} = T_{\text{queue}} + T_{\text{prefill}}, \qquad \text{E2E} = \text{TTFT} + (n-1)\,\text{TPOT} \]
\[ T_{\text{prefill}} \gtrsim \frac{2NP}{F}, \qquad T_{\text{step}} \gtrsim \max\!\left(\frac{M_w + B c\, s_{kv}}{\text{BW}}, \frac{2NB}{F}\right) \]
\[ M_{\text{peak}} \approx N b_w + 2 L n_{kv} d_h b_{kv} \textstyle\sum_r c_r, \qquad \text{throughput} = \frac{B}{T_{\text{step}}}, \qquad L = \lambda W \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  3. vLLM project (docs). vLLM documentation
  4. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  5. Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  6. vLLM project (docs). vLLM documentation
  7. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  8. Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
  9. Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
  10. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  11. Patel et al. (2023). Splitwise: Efficient Generative LLM Inference Using Phase Splitting
  12. Little (1961). A Proof for the Queuing Formula: L = λW
  13. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures