Why inference is slow: three root causes and the roofline
Running a trained model costs three things: arithmetic (compute), moving bytes from memory (memory access), and space to hold weights and the KV cache (memory size). This session finds three root causes behind these costs: the model is big, attention grows with the square of the prompt length, and decoding makes one token at a time. With the roofline model you will see why the prefill phase is limited by compute while the decode phase is limited by memory bandwidth, and you will work it all out for one 8B model on an NVIDIA L4 and on an H100.
What you will learn
- The three costs of inference: compute (FLOPs), memory access (bytes moved) and memory size (bytes stored), and how each one turns into time or into a hard limit
- The three root causes: model size, attention that grows with the square of the prompt length, and token-by-token decoding
- How to write the FLOPs and bytes of one prefill pass and one decode step for a real model (Llama 3.1 8B)
- Why prefill sits near the compute roof and decode sits on the bandwidth slope, and why batching helps the weight matrices but not decode attention
- A full worked example: Llama 3.1 8B on an NVIDIA L4 (24 GB, 300 GB/s) and an H100 SXM (80 GB, 3.35 TB/s): first-token time, time per token, KV-cache room and batch size
- How to read a speed problem as one of the three causes, which prepares the map of optimizations in the next session
Why this matters
Take Llama 3.1 8B in BF16 on one NVIDIA L4 GPU. It reads a 2,048-token prompt in about a quarter of a second. Then it writes a 256-token answer in about 13 seconds. The same chip that read 2,048 tokens in 0.26 s needs 0.05 s for each single new token. This session explains that gap.
In session 81 you learned to measure speed: time to first token (TTFT), time per output token (TPOT), memory, throughput. Now we ask why the numbers are what they are. The efficient-inference survey of Zhou et al. (section 2.3) gives a clean frame that we follow. [Certain] There are three costs (compute, memory access, memory size) and three root causes that make them large (model size, attention, and token-by-token decoding). Every optimization in the rest of this track attacks one of these causes.
The tool that connects them is the roofline model from session 42. With it, a few lines of arithmetic tell you, before you run anything, whether a phase of inference is limited by the GPU’s arithmetic or by its memory. That arithmetic is also what serving teams do when they choose a GPU, a batch size and a quantization format.
Three costs: compute, memory access, memory size
When a GPU runs one step of a model, three different things cost us something.
- Compute: the number of floating-point operations (FLOPs) the step does. A multiply plus an add counts as 2 FLOPs. Time spent on compute is FLOPs divided by the GPU’s peak speed \(P\) (FLOP per second).
- Memory access: the number of bytes the step moves between the GPU’s main memory (HBM or GDDR, the large off-chip memory) and the chip. Time spent on it is bytes divided by the memory bandwidth \(B\) (bytes per second).
- Memory size: the number of bytes we must keep in GPU memory: the weights, the KV cache (the stored keys and values of past tokens, see session 80), and temporary activations. This is not a time. It is a hard limit: if it does not fit, it does not run (or it must use more GPUs).
The GPU can overlap math and memory traffic. So, as in session 42, the time of one step is at least the larger of the two times, and the stored bytes must fit in the capacity \(C\):
- Read the symbols
- \(F\)
- FLOPs done in the step (compute cost)
- \(Q\)
- bytes moved to and from GPU main memory in the step (memory-access cost)
- \(P\)
- peak dense compute of the GPU for the number format used, in FLOP/s
- \(B\)
- main-memory bandwidth, in bytes per second
- \(M_{\text{weights}}, M_{\text{KV}}, M_{\text{act}}\)
- bytes stored for weights, for the KV cache, and for activations and workspace (memory-size cost)
- \(C\)
- memory capacity of the GPU, in bytes
If \(F/P\) is the larger term, the step is compute-bound: only faster arithmetic (or fewer FLOPs) helps. If \(Q/B\) is larger, the step is memory-bound: only fewer bytes or faster memory helps. Memory size works differently: it does not slow a single step, but it caps how many requests (and how long a context) can be served at once, and so it limits throughput.
Here are the two GPUs we use throughout. Both are real, current data-center GPUs. The L4 is a small, low-power card (72 W) used for cheap serving; the H100 is the large training and serving GPU.
| GPU | Dense BF16 peak \(P\) | Bandwidth \(B\) | Capacity \(C\) | Ridge point \(P/B\) |
|---|---|---|---|---|
| NVIDIA L4 (GDDR6) | 121 TFLOP/s | 300 GB/s | 24 GB | 403 FLOP/byte |
| NVIDIA H100 SXM (HBM3) | 989 TFLOP/s | 3.35 TB/s | 80 GB | 295 FLOP/byte |
NVIDIA’s product pages list the tensor-core numbers “with sparsity” (242 TFLOP/s for the L4, 1,979 for the H100 SXM); the dense numbers are half of that. [Certain] The H100 has about 8.2 times the compute of the L4, but about 11.2 times the bandwidth. Keep these two ratios in mind: they will reappear as the speed-ups of prefill and decode.
The model we use: Llama 3.1 8B
From its config.json: hidden size \(d = 4096\), \(L = 32\) layers, 32 query heads and \(H_{kv} = 8\) key-value heads (grouped-query attention, GQA) of size \(d_h = 128\), MLP width 14,336, vocabulary 128,256, input and output embeddings not shared. [Certain] Counting as in session 25 gives \(N = 8{,}030{,}261{,}248\) parameters. Of these, \(N_{mm} = 7{,}504{,}658{,}432\) sit in matrices that are multiplied in every step (the 7 projections of each layer and the output head). The input embedding table (525 million numbers) is only a lookup: each token reads one row of it.
- Read the symbols
- \(s\)
- bytes per stored number: 2 for BF16, 1 for FP8 or INT8, 0.5 for INT4
- \(N\)
- all parameters (what must be stored)
- \(N_{mm}\)
- parameters in the matrices multiplied for every token (what must be read and used)
- \(F_{\text{token}}\)
- forward FLOPs per token in the linear layers: one multiply-add (2 FLOPs) per weight
In BF16: \(M_{\text{weights}} = 16.06\) GB (14.96 GiB), and \(F_{\text{token}} = 15.0\) GFLOP. Attention scores add more FLOPs on top; we count them next.
Root cause 1: model size
The first cause is simple: the model has billions of weights, and every token needs all of them. This one fact hits all three costs at once:
- Compute: 15.0 GFLOP per token (formula 82.2). Fine for an H100 if it can process many tokens together.
- Memory access: to make one step, the GPU must read all 15.0 GB of multiplied weights from main memory into the chip. The chip’s fast on-chip memory (tens of MB) is far too small to keep them. [Certain]
- Memory size: 16.06 GB must stay resident. On a 24 GB L4 that leaves 7.94 GB for everything else. A 70B model in BF16 needs about 140 GB: at least two 80 GB GPUs. [Certain]
The survey makes the same point with Llama 2 70B: 140 GB in FP16, so at least 2 A100 80 GB GPUs, and about 100 ms per generated token on them. [Certain] By formula (82.1) the floor is \(140 / (2 \times 2.039) = 34\) ms per token, so the setup the survey describes reaches roughly a third of the bandwidth roof. Real systems never hit the roof exactly. [Likely]
Root cause 2: attention grows with length
The second cause is attention. In the linear layers, each token costs the same no matter how long the text is. In attention, each token compares itself with all earlier tokens. So the work for one token grows with the context, and the work for a whole prompt grows with the square of its length.
Count it for a prompt of \(T\) tokens (prefill). Per layer, the query of token \(i\) takes a dot product with \(i\) keys (\(2 d\, i\) FLOPs over all heads) and then mixes \(i\) values (another \(2 d\, i\)). Sum over \(i = 1 \dots T\) with a causal mask, and a kernel that skips the masked half (FlashAttention does), and you get about \(2 d T^2\) FLOPs per layer:
The linear part grows like \(T\); the attention part like \(T^2\). They are equal at
- Read the symbols
- \(T\)
- prompt length in tokens
- \(L, d\)
- number of layers and hidden size (\(d\) = number of query heads \(\times\) head size)
- \(T_{\times}\)
- the prompt length where attention FLOPs equal linear-layer FLOPs
For Llama 3.1 8B, \(T_{\times} = 7.50 \times 10^9 / (32 \times 4096) = 57{,}256\) tokens. Attention is 3.5% of prefill FLOPs at 2,048 tokens, 12.5% at 8,192, 36% at 32,768 and 70% at 131,072 (the model’s maximum). So for short chats, attention FLOPs are small. For long documents, long videos, and agents with long histories, they take over.
Attention also hurts the other two costs. A naive implementation writes the full \(T \times T\) score matrix for every head to main memory: in BF16 that is \(32 \times T^2 \times 2\) bytes per layer, 4.3 GB at \(T = 8192\) and 69 GB at \(T = 32768\). FlashAttention avoids writing it by computing attention in tiles in fast on-chip memory (session 124). [Certain] And during decode, the KV cache that attention reads grows with every token (root cause 3).
How to read Figure 1
On log-log axes a line with slope 1 grows like \(T\) and a line with slope 2 grows like \(T^2\). The orange attention line is steeper, so it always wins in the end. Move the slider: the readout gives the share of attention, the prefill time on each GPU if it ran at peak, the size a naive score matrix would need, and, for comparison, what the same context costs a single decode step: there the KV cache is read but the FLOPs are tiny.
Root cause 3: token-by-token decoding
The third cause is how text is produced. The model makes token \(t+1\) only after it knows token \(t\) (see session 78). So the answer is built by a loop of decode steps, and each step processes one new token per sequence. Each step still runs all 32 layers, and so it must read all the weights again. It also reads the whole KV cache of that sequence, which grows by one token every step.
The KV cache stores, for every past token and every layer, one key and one value vector per KV head:
For Llama 3.1 8B in BF16: \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes = 128 KiB per token. (Without GQA, with 32 KV heads, it would be 4 times more.) Now write the cost of one decode step for a batch of \(b\) sequences, each with \(c\) tokens of context:
- Read the symbols
- \(b\)
- batch size: how many sequences make one token each in this step
- \(c\)
- context length: tokens already in each sequence’s KV cache
- \(4Ldc\)
- attention FLOPs for one new token: scores and value mixing against \(c\) past tokens, in all layers
- \(sN_{mm}\)
- bytes of weights, read once per step and shared by all \(b\) sequences
- \(b\,c\,\text{kv}_{\text{tok}}\)
- bytes of KV cache, read separately for each sequence
With one sequence (\(b = 1\)) and a short context, the memory term is about \(15\) GB and the compute term is about \(15\) GFLOP: one FLOP per byte. Both GPUs can do hundreds of FLOPs in the time one byte arrives. So the step time is set by bandwidth alone:
This is the most useful formula in LLM serving. For Llama 3.1 8B in BF16 with \(c = 2048\): 15.28 GB per step, so 50.9 ms on an L4 (19.6 tokens/s) and 4.56 ms on an H100 (219 tokens/s). The math units are busy less than 0.4% of the time. [Certain, as a bound]
Decoding has two more problems. First, the steps are sequential: no amount of hardware lets you compute token 200 before token 199. Second, the KV cache grows and shrinks request by request, which leads to wasted, fragmented memory if it is not managed carefully (session 134). [Certain]
How to read Figure 2
Step through. In the prefill step, the blue compute bar is longer than the orange memory bar: 2,048 tokens share one read of the weights. In every decode step it flips: the orange bar is the same 15 GB read, but now it buys only one token, and the blue bar shrinks to almost nothing. At the end, look at the clock: decode is about 98% of the total on both GPUs. Switch GPUs: prefill speeds up by the compute ratio (8.2×), decode by the bandwidth ratio (11.2×).
The roofline: prefill near the roof, decode on the slope
Now put each phase on the roofline. The arithmetic intensity \(I = F / Q\) is the number of FLOPs done per byte moved. The roofline says the best possible speed is \(\min(P, I \cdot B)\), and the ridge point \(I^* = P / B\) separates the two regimes (session 42).
Prefill. One pass over \(T\) prompt tokens reads the weights once and uses each weight \(T\) times. Ignoring the small KV writes:
In BF16 that is about \(T\) FLOP per byte (attention adds a bit more). The ridge is at 403 on the L4 and 295 on the H100. So a prompt longer than a few hundred tokens makes prefill compute-bound: exactly 402 tokens or more on the L4, 295 or more on the H100 with our counts. Very short prompts are memory-bound even in prefill.
Decode. From formula (82.6), split the step into its two parts:
- Read the symbols
- \(I_{\text{weights}}\)
- intensity of the linear layers in decode: grows with the batch, because all \(b\) sequences share one read of the weights
- \(I_{\text{attn}}\)
- intensity of decode attention: each sequence reads its own KV cache, so the batch cancels out
- \(g\)
- the GQA group size: query heads per KV head (4 for Llama 3.1 8B; 1 for plain multi-head attention)
This is the heart of the session. The linear layers in decode have intensity \(b\) in BF16: at batch 1 they sit deep on the bandwidth slope, 300 to 400 times below the ridge, and batching moves them up the slope for free. Decode attention has intensity \(2g/s = 4\) FLOP/byte whatever the batch. It stays on the slope forever; only fewer KV bytes (GQA, MLA, KV quantization) or fewer past tokens (sparse attention) raise it. The whole decode step mixes the two:
With short contexts it grows like \(b\). With long contexts and big batches the KV bytes dominate and it falls back toward 4. Yuan et al. run this kind of analysis layer by layer for Llama 2 7B on an A6000 GPU with their LLM-Viewer tool, and find the same picture: most prefill operations are compute-bound, and all decode operations are memory-bound. [Certain]
How to read Figure 3
Points left of the ridge are memory-bound; their height is \(I \cdot B\). Start with batch 1: the decode points sit near \(I = 1\) to \(4\), and the prefill point is on the flat roof. Raise the batch: the green linear-layer point climbs the slope and reaches the roof near \(b = 403\) on the L4. The orange attention point does not move. Raise the context too and watch the full decode step (teal) fall back toward the attention point. The readout also warns you when the KV cache of that batch does not fit in memory; on the L4 this happens long before the ridge.
Worked example: one 8B model on an L4 and an H100
Memory size. Weights: 16.06 GB. On the L4 (24 GB) that leaves at most 7.94 GB, so at most \(7.94 \times 10^9 / 131{,}072 = 60{,}573\) tokens of KV cache in total (before activations, the CUDA context and fragmentation take their share). On the H100: 63.94 GB, 487,820 tokens.
Prefill (TTFT). \(F = 2 \times 7.50 \times 10^9 \times 2048 + 2 \times 32 \times 4096 \times 2048^2 = 31.84\) TFLOP. On the L4: \(31.84 / 121 = 263\) ms of compute against 51 ms of memory traffic: compute-bound, TTFT at least 263 ms. On the H100: 32.2 ms against 4.6 ms. Speed-up 8.2×, the compute ratio.
Decode (TPOT). First step: \(Q = 15.01 + 2048 \times 131{,}072 / 10^9 = 15.28\) GB, \(F = 16.1\) GFLOP. On the L4: 50.9 ms of memory against 0.13 ms of compute. On the H100: 4.56 ms against 0.016 ms. Speed-up 11.2×, the bandwidth ratio.
The whole answer. Adding all 256 steps (the context grows from 2,048 to 2,303): 13.05 s of decode on the L4, total 13.3 s; 1.17 s on the H100, total 1.20 s. Decode is 98% (L4) and 97% (H100) of the time.
Batching. At 4,096 tokens of context each sequence’s KV cache is 0.537 GB, so the L4 holds at most 14 sequences. At \(b = 14\): \(Q = 22.5\) GB, 75.1 ms per step, 186 tokens/s in total, compute still only 2.0 ms. Memory size, not compute, stops the L4. The H100 holds 119 such sequences: 23.6 ms per step, about 5,050 tokens/s in total, 42 tokens/s per user.
Smaller weights. With INT4 weights (0.5 bytes, scales ignored) the L4 stores 4.0 GB of weights and a batch-1 decode step at \(c = 2048\) reads 3.75 GB of weights plus 0.27 GB of KV cache: 13.4 ms, about 3.8 times faster, with no change in compute. This is why weight-only quantization is so popular for decode (session 104).
All of these numbers are lower bounds from formula (82.1). Real kernels reach a large fraction of peak bandwidth in decode and a smaller fraction of peak compute in prefill, so measured TTFT is usually further from the bound than measured TPOT. [Likely] Use Figure 4 to try other settings.
How to read Figure 4
Each row is one quantity from this section. Red means the batch does not fit in memory. Try: batch 1, then raise the batch until the L4 runs out of memory; then switch to INT4 and see the memory freed for the KV cache, and the shorter decode steps. Notice that TTFT does not change with the weight format, because prefill is compute-bound and the math is still BF16.
From numbers to a diagnosis
The three costs and three causes give a simple way to read any speed problem. First find the phase; then find which cost limits it; then the cause tells you which family of fixes can help.
- input: model shapes (\(N, N_{mm}, L, d, H_{kv}, d_h\)), bytes per weight \(s\), prompt \(T\), answer length \(n\), batch \(b\), GPU (\(P, B, C\))
- \(\text{kv}_{\text{tok}} \leftarrow 2 L H_{kv} d_h \cdot 2\) // formula (82.5), BF16 cache
- if \(sN + b\,(T + n)\,\text{kv}_{\text{tok}} \gt C\) then report “does not fit” stop // memory size
- \(F \leftarrow b\,(2N_{mm}T + 2LdT^2)\); \(Q \leftarrow sN_{mm} + bT\,\text{kv}_{\text{tok}}\)
- \(\text{TTFT} \leftarrow \max(F/P,\ Q/B)\) // formula (82.1), usually compute-bound
- \(t \leftarrow 0\)
- for \(k = 0, 1, \dots, n-1\) do
- \(c \leftarrow T + k\)
- \(F \leftarrow b\,(2N_{mm} + 4Ldc)\); \(Q \leftarrow sN_{mm} + b\,c\,\text{kv}_{\text{tok}}\) // formula (82.6)
- \(t \leftarrow t + \max(F/P,\ Q/B)\) // usually memory-bound
- return TTFT, mean TPOT \(= t / n\), throughput \(= b\,n / (\text{TTFT} + t)\)
- input: a measured slow phase (TTFT too high, TPOT too high, or throughput too low)
- if TTFT is high then // prefill: compute-bound
- if \(T\) is close to or above \(T_\times\) then cause = attention: FlashAttention, sparse or linear attention, shorter prompts (prompt compression, RAG)
- else cause = model size: fewer FLOPs (smaller or distilled model, MoE, FP8 math), more GPUs (tensor parallelism), chunked prefill
- if TPOT is high then // decode: memory-bound
- if \(sN_{mm} \gt b\,c\,\text{kv}_{\text{tok}}\) then cause = model size: weight quantization, smaller model, speculative decoding (more tokens per weight read)
- else cause = attention + decoding: GQA or MLA, KV quantization, KV eviction, FlashDecoding
- if throughput is low but TPOT is fine then cause = memory size limits \(b\): PagedAttention, KV quantization, more GPUs, continuous batching
- apply one fix, re-measure TTFT, TPOT and throughput // session 81
This split (change the input, change the model, change the system) is exactly the map of the next session, session 83.
In code
The script below counts the weights of Llama 3.1 8B from its shapes (on PyTorch’s meta device, which stores no data), computes the roofline numbers of this page, and then measures a \(4096 \times 4096\) layer on your own device with 1 token and with 1,024 tokens. The first part is exact and deterministic; the timings depend on your hardware.
import time, torch, torch.nn as nn torch.manual_seed(0) # 1) Llama 3.1 8B shapes (from its config.json); "meta" tensors use no memory C = dict(d=4096, L=32, H=32, Hkv=8, dh=128, ff=14336, V=128256) def lin(i, o): return nn.Linear(i, o, bias=False, device="meta") def block(c): d, dh = c["d"], c["dh"] return nn.ModuleDict(dict(q=lin(d, c["H"] * dh), k=lin(d, c["Hkv"] * dh), v=lin(d, c["Hkv"] * dh), o=lin(c["H"] * dh, d), gate=lin(d, c["ff"]), up=lin(d, c["ff"]), down=lin(c["ff"], d), n1=nn.RMSNorm(d, device="meta"), n2=nn.RMSNorm(d, device="meta"))) model = nn.ModuleDict(dict(emb=nn.Embedding(C["V"], C["d"], device="meta"), layers=nn.ModuleList(block(C) for _ in range(C["L"])), norm=nn.RMSNorm(C["d"], device="meta"), head=lin(C["d"], C["V"]))) N = sum(p.numel() for p in model.parameters()) N_mm = sum(m.weight.numel() for m in model.modules() if isinstance(m, nn.Linear)) kv_tok = 2 * C["L"] * C["Hkv"] * C["dh"] * 2 # bytes per token, BF16 (82.5) print(f"N = {N:,} matmul weights = {N_mm:,} KV/token = {kv_tok} B") # 2) the roofline: time >= max(compute time, memory time) (82.1) GPUS = {"L4": (121e12, 300e9, 24e9), "H100 SXM": (989e12, 3.35e12, 80e9)} def step_time(F, Q, P, B): return max(F / P, Q / B), ("compute" if F / P > Q / B else "memory") def prefill(T, s=2): # one prompt of T tokens, causal attention (82.3) F = 2 * N_mm * T + 2 * C["L"] * C["d"] * T * T Q = s * N_mm + T * kv_tok # read weights, write KV return F, Q def decode(b, c, s=2): # one new token for each of b sequences (82.6) F = b * (2 * N_mm + 4 * C["L"] * C["d"] * c) Q = s * N_mm + b * c * kv_tok # weights once, KV per sequence return F, Q for g, (P, B, cap) in GPUS.items(): free = cap - 2 * N t_pre, bound_pre = step_time(*prefill(2048), P, B) t_dec, bound_dec = step_time(*decode(1, 2048), P, B) print(f"{g:8s} ridge {P/B:5.0f} FLOP/B | prefill 2048: {1e3*t_pre:6.1f} ms ({bound_pre})" f" | decode b=1: {1e3*t_dec:5.2f} ms ({bound_dec}) = {1/t_dec:5.1f} tok/s" f" | KV room {free/kv_tok:,.0f} tokens") # 3) see it on your own device: one 4096 x 4096 layer, 1 token vs 1024 tokens dev = "cuda" if torch.cuda.is_available() else "cpu" dt = torch.bfloat16 if dev == "cuda" else torch.float32 W = torch.randn(4096, 4096, device=dev, dtype=dt) for M in (1, 1024): x = torch.randn(M, 4096, device=dev, dtype=dt) for _ in range(3): y = x @ W # warm-up if dev == "cuda": torch.cuda.synchronize() t0 = time.perf_counter() for _ in range(20): y = x @ W if dev == "cuda": torch.cuda.synchronize() t = (time.perf_counter() - t0) / 20 print(f"M={M:5d}: {2*M*4096*4096/t/1e12:7.3f} TFLOP/s, " f"{W.numel()*W.element_size()/t/1e9:7.1f} GB/s of weight reads") # N = 8,030,261,248 matmul weights = 7,504,658,432 KV/token = 131072 B # L4 ridge 403 FLOP/B | prefill 2048: 263.1 ms (compute) | decode b=1: 50.93 ms (memory) = 19.6 tok/s | KV room 60,573 tokens # H100 SXM ridge 295 FLOP/B | prefill 2048: 32.2 ms (compute) | decode b=1: 4.56 ms (memory) = 219.3 tok/s | KV room 487,820 tokens
On a laptop CPU, the 1-token case shows high “GB/s of weight reads” and low TFLOP/s, and the 1,024-token case shows the opposite: the same weights, used 1,024 times per read. On a GPU the gap is much larger. Your numbers will differ; the direction will not.
Where this is used
| Setting | Which phase and cost dominate | What people do |
|---|---|---|
| Chat assistant, one user per GPU | Decode, memory access (weights). \(\text{TPOT} \approx \text{weight bytes} / \text{bandwidth}\) | Weight quantization (INT4, FP8), speculative decoding, a GPU with more bandwidth |
| Busy chat service | Decode, memory size (KV cache limits the batch), then KV bandwidth | Continuous batching, PagedAttention, GQA models, KV quantization (session 84) |
| RAG and long documents | Prefill, compute; attention share grows with \(T\) | FlashAttention, prefix caching, prompt compression, chunked prefill |
| Video understanding with a VLM (classification, action recognition, key-moment detection) | Prefill of many visual tokens: for example \(64 \times 256 = 16{,}384\) tokens (64 frames of 256), where attention is already 22% of the FLOPs for an 8B model | Fewer tokens per frame (token merging or pooling), frame sampling, sparse attention |
| Video-RAG over long recordings | Prefill of retrieved clips (compute) plus a KV cache of many tokens (memory size) | Retrieve fewer, shorter clips; cache encoded clips; KV compression |
| Offline batch jobs (captioning a video archive, labeling data) | Throughput; large \(b\) moves decode up the slope | Largest batch that fits; long-context requests grouped together |
| Edge and small GPUs (L4, laptops, phones) | Memory size first (does it fit at all?), then bandwidth | 4-bit weights, small models, offloading (session 130) |
Common mistakes
- Using the “with sparsity” peak. Product pages show tensor-core numbers with 2:4 sparsity. Dense LLM math gets half of that: 121, not 242, TFLOP/s on the L4.
- Expecting a GPU with more FLOPs to decode faster by the same factor. Decode follows bandwidth. From L4 to H100, prefill gets 8.2× faster and decode 11.2× faster, because those are the compute and bandwidth ratios.
- Forgetting the KV cache when choosing the batch. Weights fit, so people assume a big batch fits. On an L4 with 4,096-token contexts, 14 sequences fill the free memory.
- Thinking batching makes decode free. It shares the weight reads, not the KV reads. With long contexts the KV term in formula (82.6) grows with \(b\) and the step slows down.
- Counting attention FLOPs as negligible always. True at 2,048 tokens (3.5%), false at 32,768 (36%) and above.
- Treating the roofline as a prediction. It is a lower bound on time. Kernel launch overhead, imperfect overlap, small matrices and communication make real steps slower.
- Mixing GB and GiB. 16.06 GB is 14.96 GiB. Bandwidths are quoted in decimal GB/s; memory tools often print GiB.
- Timing GPU code without synchronizing. CUDA calls return before the work ends. Warm up, call
torch.cuda.synchronize(), and average many runs.
Interview questions
What are the three costs of LLM inference, and how does each one turn into time?basic
Name the three root causes of slow inference and the cost each one raises.basic
Why is decoding memory-bound? Give the number.core
Estimate the decode speed of an 8B BF16 model at batch 1 on an H100.core
Why is prefill compute-bound, and when is it not?core
Batching raises decode throughput a lot. Why does it not help decode attention?advanced
At what prompt length do attention FLOPs equal linear-layer FLOPs for Llama 3.1 8B?advanced
You move a service from L4 to H100. How much faster are TTFT and TPOT?core
Why does 4-bit weight-only quantization speed up decode even though the math stays in BF16?core
Decode is sequential. Which techniques get around that?advanced
Your measured TPOT is 3× the roofline bound. What do you check?advanced
Exercises
- Llama 2 70B in FP16 needs 140 GB. On 2 A100 80 GB GPUs (2.039 TB/s each, weights split across them), what is the roofline bound per token at batch 1? Compare with the survey’s “about 100 ms”.
- What is the KV cache per token of Llama 3.1 8B if the cache is stored in FP8? How many tokens fit in 7.94 GB?
- From which prompt length is prefill compute-bound on the L4, using \(I \approx T\)?
- Ignoring the KV cache, at which batch size do the decode linear layers of a BF16 model reach the ridge point of an H100?
- What is the decode-attention intensity for a model with plain multi-head attention (\(H_{kv} = H\)) in BF16? And with a group size of 8?
- Use Figure 4: with INT4 weights on the L4 and a 2,048-token prompt with 256-token answers, what is the largest batch that fits?
Answerscheck
Key takeaways
- Three costs: compute \(F/P\), memory access \(Q/B\), and memory size \(\le C\). A step takes at least \(\max(F/P, Q/B)\) (82.1); memory size caps batch and context.
- Three root causes: model size (all three costs, 82.2), attention (\(T^2\) FLOPs in prefill, 82.3, and a growing KV cache, 82.5), and token-by-token decoding (all weights read for every token, 82.6 and 82.7).
- Prefill has intensity about \(T\) and sits near the compute roof (82.8). Decode has intensity about \(b\) for the weights and a fixed \(2g/s\) for attention (82.9), so it sits on the bandwidth slope.
- Llama 3.1 8B BF16, 2,048-token prompt: \(\text{TTFT} \ge 263\) ms on an L4 and 32 ms on an H100; \(\text{TPOT} \ge 50.9\) ms and 4.56 ms. Prefill scales with compute (8.2×), decode with bandwidth (11.2×).
- On a 24 GB L4, memory size limits the batch long before compute does; INT4 weights free memory and cut decode time by about 3.8×.
Key formulas
References for this session
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- NVIDIA (docs). NVIDIA L4 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
- Meta (mirrored by Unsloth on Hugging Face) (2024). Llama 3.1 8B Instruct: config.json
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding