LLM & VLM courseBlog
82Track 5: LLM inference and serving · session 5 of 65

Why inference is slow: three root causes and the roofline

Running a trained model costs three things: arithmetic (compute), moving bytes from memory (memory access), and space to hold weights and the KV cache (memory size). This session finds three root causes behind these costs: the model is big, attention grows with the square of the prompt length, and decoding makes one token at a time. With the roofline model you will see why the prefill phase is limited by compute while the decode phase is limited by memory bandwidth, and you will work it all out for one 8B model on an NVIDIA L4 and on an H100.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 25, 42 and 78 to 81 (counting FLOPs; GPU architecture and the roofline; prefill and decode; the KV cache; inference metrics)

What you will learn

  • The three costs of inference: compute (FLOPs), memory access (bytes moved) and memory size (bytes stored), and how each one turns into time or into a hard limit
  • The three root causes: model size, attention that grows with the square of the prompt length, and token-by-token decoding
  • How to write the FLOPs and bytes of one prefill pass and one decode step for a real model (Llama 3.1 8B)
  • Why prefill sits near the compute roof and decode sits on the bandwidth slope, and why batching helps the weight matrices but not decode attention
  • A full worked example: Llama 3.1 8B on an NVIDIA L4 (24 GB, 300 GB/s) and an H100 SXM (80 GB, 3.35 TB/s): first-token time, time per token, KV-cache room and batch size
  • How to read a speed problem as one of the three causes, which prepares the map of optimizations in the next session

Why this matters

Take Llama 3.1 8B in BF16 on one NVIDIA L4 GPU. It reads a 2,048-token prompt in about a quarter of a second. Then it writes a 256-token answer in about 13 seconds. The same chip that read 2,048 tokens in 0.26 s needs 0.05 s for each single new token. This session explains that gap.

In session 81 you learned to measure speed: time to first token (TTFT), time per output token (TPOT), memory, throughput. Now we ask why the numbers are what they are. The efficient-inference survey of Zhou et al. (section 2.3) gives a clean frame that we follow. [Certain] There are three costs (compute, memory access, memory size) and three root causes that make them large (model size, attention, and token-by-token decoding). Every optimization in the rest of this track attacks one of these causes.

The tool that connects them is the roofline model from session 42. With it, a few lines of arithmetic tell you, before you run anything, whether a phase of inference is limited by the GPU’s arithmetic or by its memory. That arithmetic is also what serving teams do when they choose a GPU, a batch size and a quantization format.

Three costs: compute, memory access, memory size

When a GPU runs one step of a model, three different things cost us something.

  • Compute: the number of floating-point operations (FLOPs) the step does. A multiply plus an add counts as 2 FLOPs. Time spent on compute is FLOPs divided by the GPU’s peak speed \(P\) (FLOP per second).
  • Memory access: the number of bytes the step moves between the GPU’s main memory (HBM or GDDR, the large off-chip memory) and the chip. Time spent on it is bytes divided by the memory bandwidth \(B\) (bytes per second).
  • Memory size: the number of bytes we must keep in GPU memory: the weights, the KV cache (the stored keys and values of past tokens, see session 80), and temporary activations. This is not a time. It is a hard limit: if it does not fit, it does not run (or it must use more GPUs).

The GPU can overlap math and memory traffic. So, as in session 42, the time of one step is at least the larger of the two times, and the stored bytes must fit in the capacity \(C\):

\[ t_{\text{step}} \ \ge\ \max\!\left(\frac{F}{P},\ \frac{Q}{B}\right), \qquad M_{\text{weights}} + M_{\text{KV}} + M_{\text{act}} \ \le\ C \tag{82.1} \]
Read the symbols
\(F\)
FLOPs done in the step (compute cost)
\(Q\)
bytes moved to and from GPU main memory in the step (memory-access cost)
\(P\)
peak dense compute of the GPU for the number format used, in FLOP/s
\(B\)
main-memory bandwidth, in bytes per second
\(M_{\text{weights}}, M_{\text{KV}}, M_{\text{act}}\)
bytes stored for weights, for the KV cache, and for activations and workspace (memory-size cost)
\(C\)
memory capacity of the GPU, in bytes

If \(F/P\) is the larger term, the step is compute-bound: only faster arithmetic (or fewer FLOPs) helps. If \(Q/B\) is larger, the step is memory-bound: only fewer bytes or faster memory helps. Memory size works differently: it does not slow a single step, but it caps how many requests (and how long a context) can be served at once, and so it limits throughput.

Here are the two GPUs we use throughout. Both are real, current data-center GPUs. The L4 is a small, low-power card (72 W) used for cheap serving; the H100 is the large training and serving GPU.

GPUDense BF16 peak \(P\)Bandwidth \(B\)Capacity \(C\)Ridge point \(P/B\)
NVIDIA L4 (GDDR6)121 TFLOP/s300 GB/s24 GB403 FLOP/byte
NVIDIA H100 SXM (HBM3)989 TFLOP/s3.35 TB/s80 GB295 FLOP/byte

NVIDIA’s product pages list the tensor-core numbers “with sparsity” (242 TFLOP/s for the L4, 1,979 for the H100 SXM); the dense numbers are half of that. [Certain] The H100 has about 8.2 times the compute of the L4, but about 11.2 times the bandwidth. Keep these two ratios in mind: they will reappear as the speed-ups of prefill and decode.

The model we use: Llama 3.1 8B

From its config.json: hidden size \(d = 4096\), \(L = 32\) layers, 32 query heads and \(H_{kv} = 8\) key-value heads (grouped-query attention, GQA) of size \(d_h = 128\), MLP width 14,336, vocabulary 128,256, input and output embeddings not shared. [Certain] Counting as in session 25 gives \(N = 8{,}030{,}261{,}248\) parameters. Of these, \(N_{mm} = 7{,}504{,}658{,}432\) sit in matrices that are multiplied in every step (the 7 projections of each layer and the output head). The input embedding table (525 million numbers) is only a lookup: each token reads one row of it.

\[ M_{\text{weights}} = s \cdot N, \qquad F_{\text{token}} \approx 2\, N_{mm} \tag{82.2} \]
Read the symbols
\(s\)
bytes per stored number: 2 for BF16, 1 for FP8 or INT8, 0.5 for INT4
\(N\)
all parameters (what must be stored)
\(N_{mm}\)
parameters in the matrices multiplied for every token (what must be read and used)
\(F_{\text{token}}\)
forward FLOPs per token in the linear layers: one multiply-add (2 FLOPs) per weight

In BF16: \(M_{\text{weights}} = 16.06\) GB (14.96 GiB), and \(F_{\text{token}} = 15.0\) GFLOP. Attention scores add more FLOPs on top; we count them next.

Root cause 1: model size

The first cause is simple: the model has billions of weights, and every token needs all of them. This one fact hits all three costs at once:

  • Compute: 15.0 GFLOP per token (formula 82.2). Fine for an H100 if it can process many tokens together.
  • Memory access: to make one step, the GPU must read all 15.0 GB of multiplied weights from main memory into the chip. The chip’s fast on-chip memory (tens of MB) is far too small to keep them. [Certain]
  • Memory size: 16.06 GB must stay resident. On a 24 GB L4 that leaves 7.94 GB for everything else. A 70B model in BF16 needs about 140 GB: at least two 80 GB GPUs. [Certain]

The survey makes the same point with Llama 2 70B: 140 GB in FP16, so at least 2 A100 80 GB GPUs, and about 100 ms per generated token on them. [Certain] By formula (82.1) the floor is \(140 / (2 \times 2.039) = 34\) ms per token, so the setup the survey describes reaches roughly a third of the bandwidth roof. Real systems never hit the roof exactly. [Likely]

Root cause 2: attention grows with length

The second cause is attention. In the linear layers, each token costs the same no matter how long the text is. In attention, each token compares itself with all earlier tokens. So the work for one token grows with the context, and the work for a whole prompt grows with the square of its length.

Count it for a prompt of \(T\) tokens (prefill). Per layer, the query of token \(i\) takes a dot product with \(i\) keys (\(2 d\, i\) FLOPs over all heads) and then mixes \(i\) values (another \(2 d\, i\)). Sum over \(i = 1 \dots T\) with a causal mask, and a kernel that skips the masked half (FlashAttention does), and you get about \(2 d T^2\) FLOPs per layer:

\[ F_{\text{lin}}(T) = 2\, N_{mm}\, T, \qquad F_{\text{attn}}(T) \approx 2\, L\, d\, T^2 \tag{82.3} \]

The linear part grows like \(T\); the attention part like \(T^2\). They are equal at

\[ T_{\times} = \frac{N_{mm}}{L\, d} \tag{82.4} \]
Read the symbols
\(T\)
prompt length in tokens
\(L, d\)
number of layers and hidden size (\(d\) = number of query heads \(\times\) head size)
\(T_{\times}\)
the prompt length where attention FLOPs equal linear-layer FLOPs

For Llama 3.1 8B, \(T_{\times} = 7.50 \times 10^9 / (32 \times 4096) = 57{,}256\) tokens. Attention is 3.5% of prefill FLOPs at 2,048 tokens, 12.5% at 8,192, 36% at 32,768 and 70% at 131,072 (the model’s maximum). So for short chats, attention FLOPs are small. For long documents, long videos, and agents with long histories, they take over.

Attention also hurts the other two costs. A naive implementation writes the full \(T \times T\) score matrix for every head to main memory: in BF16 that is \(32 \times T^2 \times 2\) bytes per layer, 4.3 GB at \(T = 8192\) and 69 GB at \(T = 32768\). FlashAttention avoids writing it by computing attention in tiles in fast on-chip memory (session 124). [Certain] And during decode, the KV cache that attention reads grows with every token (root cause 3).

linear layers \(2N_{mm}T\)attention \(2LdT^2\)your prompt length
Figure 1. Prefill FLOPs of Llama 3.1 8B, split into linear layers and attention (formula 82.3), on log-log axes. The two lines cross at \(T_\times = 57{,}256\) tokens (formula 82.4).
How to read Figure 1

On log-log axes a line with slope 1 grows like \(T\) and a line with slope 2 grows like \(T^2\). The orange attention line is steeper, so it always wins in the end. Move the slider: the readout gives the share of attention, the prefill time on each GPU if it ran at peak, the size a naive score matrix would need, and, for comparison, what the same context costs a single decode step: there the KV cache is read but the FLOPs are tiny.

Root cause 3: token-by-token decoding

The third cause is how text is produced. The model makes token \(t+1\) only after it knows token \(t\) (see session 78). So the answer is built by a loop of decode steps, and each step processes one new token per sequence. Each step still runs all 32 layers, and so it must read all the weights again. It also reads the whole KV cache of that sequence, which grows by one token every step.

The KV cache stores, for every past token and every layer, one key and one value vector per KV head:

\[ \text{kv}_{\text{tok}} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot s \tag{82.5} \]

For Llama 3.1 8B in BF16: \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes = 128 KiB per token. (Without GQA, with 32 KV heads, it would be 4 times more.) Now write the cost of one decode step for a batch of \(b\) sequences, each with \(c\) tokens of context:

\[ F_{\text{dec}} = b\,\big(2N_{mm} + 4\,L\,d\,c\big), \qquad Q_{\text{dec}} = s\,N_{mm} + b\,c\,\text{kv}_{\text{tok}} \tag{82.6} \]
Read the symbols
\(b\)
batch size: how many sequences make one token each in this step
\(c\)
context length: tokens already in each sequence’s KV cache
\(4Ldc\)
attention FLOPs for one new token: scores and value mixing against \(c\) past tokens, in all layers
\(sN_{mm}\)
bytes of weights, read once per step and shared by all \(b\) sequences
\(b\,c\,\text{kv}_{\text{tok}}\)
bytes of KV cache, read separately for each sequence

With one sequence (\(b = 1\)) and a short context, the memory term is about \(15\) GB and the compute term is about \(15\) GFLOP: one FLOP per byte. Both GPUs can do hundreds of FLOPs in the time one byte arrives. So the step time is set by bandwidth alone:

\[ t_{\text{token}} \ \gtrsim\ \frac{s\,N_{mm} + c\,\text{kv}_{\text{tok}}}{B} \qquad (b = 1) \tag{82.7} \]

This is the most useful formula in LLM serving. For Llama 3.1 8B in BF16 with \(c = 2048\): 15.28 GB per step, so 50.9 ms on an L4 (19.6 tokens/s) and 4.56 ms on an H100 (219 tokens/s). The math units are busy less than 0.4% of the time. [Certain, as a bound]

Decoding has two more problems. First, the steps are sequential: no amount of hardware lets you compute token 200 before token 199. Second, the KV cache grows and shrinks request by request, which leads to wasted, fragmented memory if it is not managed carefully (session 134). [Certain]

weightsKV cache / memory-access timecompute timefree memory
Figure 2. The life of one request on one GPU: Llama 3.1 8B in BF16, a 2,048-token prompt, a 256-token answer, batch size 1. Top: what is stored (memory size). Middle: the compute time and the memory-access time of the current step (formula 82.1). Bottom: the clock. All times are roofline lower bounds.
How to read Figure 2

Step through. In the prefill step, the blue compute bar is longer than the orange memory bar: 2,048 tokens share one read of the weights. In every decode step it flips: the orange bar is the same 15 GB read, but now it buys only one token, and the blue bar shrinks to almost nothing. At the end, look at the clock: decode is about 98% of the total on both GPUs. Switch GPUs: prefill speeds up by the compute ratio (8.2×), decode by the bandwidth ratio (11.2×).

The roofline: prefill near the roof, decode on the slope

Now put each phase on the roofline. The arithmetic intensity \(I = F / Q\) is the number of FLOPs done per byte moved. The roofline says the best possible speed is \(\min(P, I \cdot B)\), and the ridge point \(I^* = P / B\) separates the two regimes (session 42).

Prefill. One pass over \(T\) prompt tokens reads the weights once and uses each weight \(T\) times. Ignoring the small KV writes:

\[ I_{\text{prefill}} \approx \frac{2\,N_{mm}\,T}{s\,N_{mm}} = \frac{2T}{s} \tag{82.8} \]

In BF16 that is about \(T\) FLOP per byte (attention adds a bit more). The ridge is at 403 on the L4 and 295 on the H100. So a prompt longer than a few hundred tokens makes prefill compute-bound: exactly 402 tokens or more on the L4, 295 or more on the H100 with our counts. Very short prompts are memory-bound even in prefill.

Decode. From formula (82.6), split the step into its two parts:

\[ I_{\text{weights}} = \frac{2\,b\,N_{mm}}{s\,N_{mm}} = \frac{2b}{s}, \qquad I_{\text{attn}} = \frac{4\,L\,d\,c}{c \cdot 2L H_{kv} d_h s} = \frac{2\,g}{s}, \quad g = \frac{d}{H_{kv}\,d_h} \tag{82.9} \]
Read the symbols
\(I_{\text{weights}}\)
intensity of the linear layers in decode: grows with the batch, because all \(b\) sequences share one read of the weights
\(I_{\text{attn}}\)
intensity of decode attention: each sequence reads its own KV cache, so the batch cancels out
\(g\)
the GQA group size: query heads per KV head (4 for Llama 3.1 8B; 1 for plain multi-head attention)

This is the heart of the session. The linear layers in decode have intensity \(b\) in BF16: at batch 1 they sit deep on the bandwidth slope, 300 to 400 times below the ridge, and batching moves them up the slope for free. Decode attention has intensity \(2g/s = 4\) FLOP/byte whatever the batch. It stays on the slope forever; only fewer KV bytes (GQA, MLA, KV quantization) or fewer past tokens (sparse attention) raise it. The whole decode step mixes the two:

\[ I_{\text{dec}} = \frac{F_{\text{dec}}}{Q_{\text{dec}}} = \frac{b\,(2N_{mm} + 4Ldc)}{s\,N_{mm} + b\,c\,\text{kv}_{\text{tok}}} \tag{82.10} \]

With short contexts it grows like \(b\). With long contexts and big batches the KV bytes dominate and it falls back toward 4. Yuan et al. run this kind of analysis layer by layer for Llama 2 7B on an A6000 GPU with their LLM-Viewer tool, and find the same picture: most prefill operations are compute-bound, and all decode operations are memory-bound. [Certain]

prefill passfull decode stepdecode: linear layers onlydecode: attention only
Figure 3. The roofline of the chosen GPU (solid) and of the other one (dashed), on log-log axes, with four points for Llama 3.1 8B in BF16: a prefill pass (formula 82.8), the two parts of a decode step (formula 82.9), and the full decode step (formula 82.10).
How to read Figure 3

Points left of the ridge are memory-bound; their height is \(I \cdot B\). Start with batch 1: the decode points sit near \(I = 1\) to \(4\), and the prefill point is on the flat roof. Raise the batch: the green linear-layer point climbs the slope and reaches the roof near \(b = 403\) on the L4. The orange attention point does not move. Raise the context too and watch the full decode step (teal) fall back toward the attention point. The readout also warns you when the KV cache of that batch does not fit in memory; on the L4 this happens long before the ridge.

Worked example: one 8B model on an L4 and an H100

Llama 3.1 8B in BF16, prompt 2,048 tokens, answer 256 tokens

Memory size. Weights: 16.06 GB. On the L4 (24 GB) that leaves at most 7.94 GB, so at most \(7.94 \times 10^9 / 131{,}072 = 60{,}573\) tokens of KV cache in total (before activations, the CUDA context and fragmentation take their share). On the H100: 63.94 GB, 487,820 tokens.

Prefill (TTFT). \(F = 2 \times 7.50 \times 10^9 \times 2048 + 2 \times 32 \times 4096 \times 2048^2 = 31.84\) TFLOP. On the L4: \(31.84 / 121 = 263\) ms of compute against 51 ms of memory traffic: compute-bound, TTFT at least 263 ms. On the H100: 32.2 ms against 4.6 ms. Speed-up 8.2×, the compute ratio.

Decode (TPOT). First step: \(Q = 15.01 + 2048 \times 131{,}072 / 10^9 = 15.28\) GB, \(F = 16.1\) GFLOP. On the L4: 50.9 ms of memory against 0.13 ms of compute. On the H100: 4.56 ms against 0.016 ms. Speed-up 11.2×, the bandwidth ratio.

The whole answer. Adding all 256 steps (the context grows from 2,048 to 2,303): 13.05 s of decode on the L4, total 13.3 s; 1.17 s on the H100, total 1.20 s. Decode is 98% (L4) and 97% (H100) of the time.

Batching. At 4,096 tokens of context each sequence’s KV cache is 0.537 GB, so the L4 holds at most 14 sequences. At \(b = 14\): \(Q = 22.5\) GB, 75.1 ms per step, 186 tokens/s in total, compute still only 2.0 ms. Memory size, not compute, stops the L4. The H100 holds 119 such sequences: 23.6 ms per step, about 5,050 tokens/s in total, 42 tokens/s per user.

Smaller weights. With INT4 weights (0.5 bytes, scales ignored) the L4 stores 4.0 GB of weights and a batch-1 decode step at \(c = 2048\) reads 3.75 GB of weights plus 0.27 GB of KV cache: 13.4 ms, about 3.8 times faster, with no change in compute. This is why weight-only quantization is so popular for decode (session 104).

All of these numbers are lower bounds from formula (82.1). Real kernels reach a large fraction of peak bandwidth in decode and a smaller fraction of peak compute in prefill, so measured TTFT is usually further from the bound than measured TPOT. [Likely] Use Figure 4 to try other settings.

Figure 4. A roofline calculator for Llama 3.1 8B on the two GPUs. Weights can be stored in BF16, 8-bit or 4-bit (weight-only quantization: the math stays BF16, the KV cache stays BF16). All prompts in the batch are prefilled together, then decoded together. Times are lower bounds from formula (82.1).
How to read Figure 4

Each row is one quantity from this section. Red means the batch does not fit in memory. Try: batch 1, then raise the batch until the L4 runs out of memory; then switch to INT4 and see the memory freed for the KV cache, and the shorter decode steps. Notice that TTFT does not change with the weight format, because prefill is compute-bound and the math is still BF16.

From numbers to a diagnosis

The three costs and three causes give a simple way to read any speed problem. First find the phase; then find which cost limits it; then the cause tells you which family of fixes can help.

Algorithm 1: roofline estimate of one request
  1. input: model shapes (\(N, N_{mm}, L, d, H_{kv}, d_h\)), bytes per weight \(s\), prompt \(T\), answer length \(n\), batch \(b\), GPU (\(P, B, C\))
  2. \(\text{kv}_{\text{tok}} \leftarrow 2 L H_{kv} d_h \cdot 2\) // formula (82.5), BF16 cache
  3. if \(sN + b\,(T + n)\,\text{kv}_{\text{tok}} \gt C\) then report “does not fit” stop // memory size
  4. \(F \leftarrow b\,(2N_{mm}T + 2LdT^2)\); \(Q \leftarrow sN_{mm} + bT\,\text{kv}_{\text{tok}}\)
  5. \(\text{TTFT} \leftarrow \max(F/P,\ Q/B)\) // formula (82.1), usually compute-bound
  6. \(t \leftarrow 0\)
  7. for \(k = 0, 1, \dots, n-1\) do
  8. \(c \leftarrow T + k\)
  9. \(F \leftarrow b\,(2N_{mm} + 4Ldc)\); \(Q \leftarrow sN_{mm} + b\,c\,\text{kv}_{\text{tok}}\) // formula (82.6)
  10. \(t \leftarrow t + \max(F/P,\ Q/B)\) // usually memory-bound
  11. return TTFT, mean TPOT \(= t / n\), throughput \(= b\,n / (\text{TTFT} + t)\)
Algorithm 2: which root cause is it?
  1. input: a measured slow phase (TTFT too high, TPOT too high, or throughput too low)
  2. if TTFT is high then // prefill: compute-bound
  3. if \(T\) is close to or above \(T_\times\) then cause = attention: FlashAttention, sparse or linear attention, shorter prompts (prompt compression, RAG)
  4. else cause = model size: fewer FLOPs (smaller or distilled model, MoE, FP8 math), more GPUs (tensor parallelism), chunked prefill
  5. if TPOT is high then // decode: memory-bound
  6. if \(sN_{mm} \gt b\,c\,\text{kv}_{\text{tok}}\) then cause = model size: weight quantization, smaller model, speculative decoding (more tokens per weight read)
  7. else cause = attention + decoding: GQA or MLA, KV quantization, KV eviction, FlashDecoding
  8. if throughput is low but TPOT is fine then cause = memory size limits \(b\): PagedAttention, KV quantization, more GPUs, continuous batching
  9. apply one fix, re-measure TTFT, TPOT and throughput // session 81

This split (change the input, change the model, change the system) is exactly the map of the next session, session 83.

In code

The script below counts the weights of Llama 3.1 8B from its shapes (on PyTorch’s meta device, which stores no data), computes the roofline numbers of this page, and then measures a \(4096 \times 4096\) layer on your own device with 1 token and with 1,024 tokens. The first part is exact and deterministic; the timings depend on your hardware.

import time, torch, torch.nn as nn
torch.manual_seed(0)

# 1) Llama 3.1 8B shapes (from its config.json); "meta" tensors use no memory
C = dict(d=4096, L=32, H=32, Hkv=8, dh=128, ff=14336, V=128256)
def lin(i, o): return nn.Linear(i, o, bias=False, device="meta")
def block(c):
    d, dh = c["d"], c["dh"]
    return nn.ModuleDict(dict(q=lin(d, c["H"] * dh), k=lin(d, c["Hkv"] * dh),
        v=lin(d, c["Hkv"] * dh), o=lin(c["H"] * dh, d), gate=lin(d, c["ff"]),
        up=lin(d, c["ff"]), down=lin(c["ff"], d),
        n1=nn.RMSNorm(d, device="meta"), n2=nn.RMSNorm(d, device="meta")))
model = nn.ModuleDict(dict(emb=nn.Embedding(C["V"], C["d"], device="meta"),
    layers=nn.ModuleList(block(C) for _ in range(C["L"])),
    norm=nn.RMSNorm(C["d"], device="meta"), head=lin(C["d"], C["V"])))
N = sum(p.numel() for p in model.parameters())
N_mm = sum(m.weight.numel() for m in model.modules() if isinstance(m, nn.Linear))
kv_tok = 2 * C["L"] * C["Hkv"] * C["dh"] * 2            # bytes per token, BF16 (82.5)
print(f"N = {N:,}  matmul weights = {N_mm:,}  KV/token = {kv_tok} B")

# 2) the roofline: time >= max(compute time, memory time)   (82.1)
GPUS = {"L4": (121e12, 300e9, 24e9), "H100 SXM": (989e12, 3.35e12, 80e9)}
def step_time(F, Q, P, B):
    return max(F / P, Q / B), ("compute" if F / P > Q / B else "memory")

def prefill(T, s=2):        # one prompt of T tokens, causal attention (82.3)
    F = 2 * N_mm * T + 2 * C["L"] * C["d"] * T * T
    Q = s * N_mm + T * kv_tok                          # read weights, write KV
    return F, Q
def decode(b, c, s=2):      # one new token for each of b sequences (82.6)
    F = b * (2 * N_mm + 4 * C["L"] * C["d"] * c)
    Q = s * N_mm + b * c * kv_tok                      # weights once, KV per sequence
    return F, Q

for g, (P, B, cap) in GPUS.items():
    free = cap - 2 * N
    t_pre, bound_pre = step_time(*prefill(2048), P, B)
    t_dec, bound_dec = step_time(*decode(1, 2048), P, B)
    print(f"{g:8s} ridge {P/B:5.0f} FLOP/B | prefill 2048: {1e3*t_pre:6.1f} ms ({bound_pre})"
          f" | decode b=1: {1e3*t_dec:5.2f} ms ({bound_dec}) = {1/t_dec:5.1f} tok/s"
          f" | KV room {free/kv_tok:,.0f} tokens")

# 3) see it on your own device: one 4096 x 4096 layer, 1 token vs 1024 tokens
dev = "cuda" if torch.cuda.is_available() else "cpu"
dt = torch.bfloat16 if dev == "cuda" else torch.float32
W = torch.randn(4096, 4096, device=dev, dtype=dt)
for M in (1, 1024):
    x = torch.randn(M, 4096, device=dev, dtype=dt)
    for _ in range(3): y = x @ W                       # warm-up
    if dev == "cuda": torch.cuda.synchronize()
    t0 = time.perf_counter()
    for _ in range(20): y = x @ W
    if dev == "cuda": torch.cuda.synchronize()
    t = (time.perf_counter() - t0) / 20
    print(f"M={M:5d}: {2*M*4096*4096/t/1e12:7.3f} TFLOP/s, "
          f"{W.numel()*W.element_size()/t/1e9:7.1f} GB/s of weight reads")

# N = 8,030,261,248  matmul weights = 7,504,658,432  KV/token = 131072 B
# L4       ridge   403 FLOP/B | prefill 2048:  263.1 ms (compute) | decode b=1: 50.93 ms (memory) =  19.6 tok/s | KV room 60,573 tokens
# H100 SXM ridge   295 FLOP/B | prefill 2048:   32.2 ms (compute) | decode b=1:  4.56 ms (memory) = 219.3 tok/s | KV room 487,820 tokens

On a laptop CPU, the 1-token case shows high “GB/s of weight reads” and low TFLOP/s, and the 1,024-token case shows the opposite: the same weights, used 1,024 times per read. On a GPU the gap is much larger. Your numbers will differ; the direction will not.

Where this is used

SettingWhich phase and cost dominateWhat people do
Chat assistant, one user per GPUDecode, memory access (weights). \(\text{TPOT} \approx \text{weight bytes} / \text{bandwidth}\)Weight quantization (INT4, FP8), speculative decoding, a GPU with more bandwidth
Busy chat serviceDecode, memory size (KV cache limits the batch), then KV bandwidthContinuous batching, PagedAttention, GQA models, KV quantization (session 84)
RAG and long documentsPrefill, compute; attention share grows with \(T\)FlashAttention, prefix caching, prompt compression, chunked prefill
Video understanding with a VLM (classification, action recognition, key-moment detection)Prefill of many visual tokens: for example \(64 \times 256 = 16{,}384\) tokens (64 frames of 256), where attention is already 22% of the FLOPs for an 8B modelFewer tokens per frame (token merging or pooling), frame sampling, sparse attention
Video-RAG over long recordingsPrefill of retrieved clips (compute) plus a KV cache of many tokens (memory size)Retrieve fewer, shorter clips; cache encoded clips; KV compression
Offline batch jobs (captioning a video archive, labeling data)Throughput; large \(b\) moves decode up the slopeLargest batch that fits; long-context requests grouped together
Edge and small GPUs (L4, laptops, phones)Memory size first (does it fit at all?), then bandwidth4-bit weights, small models, offloading (session 130)

Common mistakes

  • Using the “with sparsity” peak. Product pages show tensor-core numbers with 2:4 sparsity. Dense LLM math gets half of that: 121, not 242, TFLOP/s on the L4.
  • Expecting a GPU with more FLOPs to decode faster by the same factor. Decode follows bandwidth. From L4 to H100, prefill gets 8.2× faster and decode 11.2× faster, because those are the compute and bandwidth ratios.
  • Forgetting the KV cache when choosing the batch. Weights fit, so people assume a big batch fits. On an L4 with 4,096-token contexts, 14 sequences fill the free memory.
  • Thinking batching makes decode free. It shares the weight reads, not the KV reads. With long contexts the KV term in formula (82.6) grows with \(b\) and the step slows down.
  • Counting attention FLOPs as negligible always. True at 2,048 tokens (3.5%), false at 32,768 (36%) and above.
  • Treating the roofline as a prediction. It is a lower bound on time. Kernel launch overhead, imperfect overlap, small matrices and communication make real steps slower.
  • Mixing GB and GiB. 16.06 GB is 14.96 GiB. Bandwidths are quoted in decimal GB/s; memory tools often print GiB.
  • Timing GPU code without synchronizing. CUDA calls return before the work ends. Warm up, call torch.cuda.synchronize(), and average many runs.

Interview questions

What are the three costs of LLM inference, and how does each one turn into time?basic
Compute (FLOPs), memory access (bytes moved between main memory and the chip) and memory size (bytes stored). The first two give times \(F/P\) and \(Q/B\); the step takes at least the larger one. Memory size is not a time but a limit: weights plus KV cache plus activations must fit in the GPU, and this limit caps the batch size and context, and so the throughput.
Name the three root causes of slow inference and the cost each one raises.basic
(1) Model size: billions of weights raise all three costs (FLOPs per token, bytes read per step, bytes stored). (2) Attention: in prefill its FLOPs and naive memory grow with \(T^2\); in decode its KV reads grow with the context. (3) Token-by-token decoding: each step makes one token per sequence but must read all weights and the whole KV cache, and steps cannot run in parallel. This is the framing of the Zhou et al. survey.
Why is decoding memory-bound? Give the number.core
At batch 1, each step does about \(2N_{mm}\) FLOPs while reading \(sN_{mm}\) bytes of weights, so the intensity is \(2/s = 1\) FLOP/byte in BF16. The ridge point is about 295 FLOP/byte on an H100 and 403 on an L4, so the GPU waits for memory almost all the time: compute is under 0.4% of the step time for Llama 3.1 8B.
Estimate the decode speed of an 8B BF16 model at batch 1 on an H100.core
Bytes per step \(\approx 15\) GB of multiplied weights (plus 0.27 GB of KV cache at 2,048 tokens). \(15.28\text{ GB} / 3.35\text{ TB/s} = 4.56\) ms, so at most about 219 tokens/s. Real engines get a good fraction of this. On an L4 (300 GB/s) the same bound is 50.9 ms, about 20 tokens/s.
Why is prefill compute-bound, and when is it not?core
Prefill processes all \(T\) prompt tokens in one pass, so each weight read is used \(T\) times: intensity \(\approx\) \(2T/s = T\) FLOP/byte in BF16. Above the ridge point (295 tokens on an H100, about 403 on an L4) the pass is compute-bound. Very short prompts at batch 1 are memory-bound even in prefill; batching several short prompts fixes that.
Batching raises decode throughput a lot. Why does it not help decode attention?advanced
The weights are shared: one read serves all \(b\) sequences, so the linear layers have intensity \(2b/s\). But each sequence has its own KV cache, read once and used only by that sequence’s queries. Attention intensity is \(2g/s\), where \(g\) is the number of query heads per KV head: 4 FLOP/byte for Llama 3.1 8B in BF16, whatever \(b\) is. Only fewer KV bytes (GQA/MQA/MLA, KV quantization) or fewer attended tokens raise it.
At what prompt length do attention FLOPs equal linear-layer FLOPs for Llama 3.1 8B?advanced
Linear: \(2N_{mm}T\). Causal attention with masked blocks skipped: \(2LdT^2\). Equal at \(T = N_{mm}/(Ld) = 7.50\times10^9/(32 \times 4096) \approx 57{,}000\) tokens. Without skipping the masked half, it would be about 28,600. Larger models with bigger \(N_{mm}/(Ld)\) cross later. Note this is FLOPs only; naive attention memory is a problem much earlier.
You move a service from L4 to H100. How much faster are TTFT and TPOT?core
TTFT (prefill, compute-bound) scales with dense compute: \(989/121 \approx 8.2\times\). TPOT (decode, memory-bound) scales with bandwidth: \(3.35/0.30 \approx 11.2\times\). And the H100 has 80 GB instead of 24 GB, so the free memory for the KV cache grows from about 7.9 GB to about 64 GB for an 8B BF16 model: about 8× more concurrent sequences, which raises throughput further.
Why does 4-bit weight-only quantization speed up decode even though the math stays in BF16?core
Decode time is bytes divided by bandwidth. INT4 weights cut the weight bytes by 4× (plus small scale overhead); the weights are dequantized on the chip, where compute is idle anyway. For Llama 3.1 8B on an L4 at 2,048 tokens of context, the bound drops from 50.9 to 13.4 ms. Prefill does not speed up, because it is compute-bound and the math is unchanged.
Decode is sequential. Which techniques get around that?advanced
Speculative decoding: a cheap draft proposes several tokens and the big model checks them all in one pass, so one read of the weights can yield several tokens. Multi-token heads (Medusa-style) and lookahead decoding do similar things. Batching does not make one sequence faster, but it shares weight reads across users. Output-organization methods (for example Skeleton-of-Thought) make independent parts of the answer in parallel. All of them raise the useful tokens per weight read.
Your measured TPOT is 3× the roofline bound. What do you check?advanced
Achieved bandwidth of the matmul kernels (GEMV kernels at batch 1 should reach a large share of peak); kernel launch overhead between many small kernels (CUDA graphs, fusion); Python and scheduler overhead per step; extra copies (dequantization to a temporary buffer, KV cache layout); communication if tensor parallel; and whether the bound used the right bytes (KV cache, embedding head). A profiler breakdown (session 123) tells you which.

Exercises

  1. Llama 2 70B in FP16 needs 140 GB. On 2 A100 80 GB GPUs (2.039 TB/s each, weights split across them), what is the roofline bound per token at batch 1? Compare with the survey’s “about 100 ms”.
  2. What is the KV cache per token of Llama 3.1 8B if the cache is stored in FP8? How many tokens fit in 7.94 GB?
  3. From which prompt length is prefill compute-bound on the L4, using \(I \approx T\)?
  4. Ignoring the KV cache, at which batch size do the decode linear layers of a BF16 model reach the ridge point of an H100?
  5. What is the decode-attention intensity for a model with plain multi-head attention (\(H_{kv} = H\)) in BF16? And with a group size of 8?
  6. Use Figure 4: with INT4 weights on the L4 and a 2,048-token prompt with 256-token answers, what is the largest batch that fits?
Answerscheck
1. Each GPU reads half of the weights in parallel: \(140 / (2 \times 2.039) = 34.3\) ms per token. The reported 100 ms is about 3× slower than the bound: real frameworks lose time to kernel overheads and communication. 2. \(2 \times 32 \times 8 \times 128 \times 1 = 65{,}536\) bytes = 64 KiB; \(7.94\times10^9 / 65{,}536 \approx 121{,}000\) tokens, twice as many as in BF16. 3. Ridge 403 FLOP/byte, so \(T \gtrsim 403\) (402 with the exact count including attention). 4. \(I = b\), so \(b \approx 295\). 5. \(2g/s\): with \(g = 1\), 1 FLOP/byte; with \(g = 8\), 8 FLOP/byte. 6. Free memory \(24 - 4.02 = 19.98\) GB; each sequence needs \(2304 \times 131{,}072 = 0.302\) GB, so 66 sequences.

Key takeaways

  • Three costs: compute \(F/P\), memory access \(Q/B\), and memory size \(\le C\). A step takes at least \(\max(F/P, Q/B)\) (82.1); memory size caps batch and context.
  • Three root causes: model size (all three costs, 82.2), attention (\(T^2\) FLOPs in prefill, 82.3, and a growing KV cache, 82.5), and token-by-token decoding (all weights read for every token, 82.6 and 82.7).
  • Prefill has intensity about \(T\) and sits near the compute roof (82.8). Decode has intensity about \(b\) for the weights and a fixed \(2g/s\) for attention (82.9), so it sits on the bandwidth slope.
  • Llama 3.1 8B BF16, 2,048-token prompt: \(\text{TTFT} \ge 263\) ms on an L4 and 32 ms on an H100; \(\text{TPOT} \ge 50.9\) ms and 4.56 ms. Prefill scales with compute (8.2×), decode with bandwidth (11.2×).
  • On a 24 GB L4, memory size limits the batch long before compute does; INT4 weights free memory and cut decode time by about 3.8×.

Key formulas

\[ t \ge \max\!\left(\frac{F}{P}, \frac{Q}{B}\right), \qquad \text{kv}_{\text{tok}} = 2LH_{kv}d_h s, \qquad t_{\text{token}} \gtrsim \frac{sN_{mm} + c\,\text{kv}_{\text{tok}}}{B} \]
\[ I_{\text{prefill}} \approx \frac{2T}{s}, \qquad I_{\text{weights}} = \frac{2b}{s}, \qquad I_{\text{attn}} = \frac{2g}{s}, \qquad T_\times = \frac{N_{mm}}{Ld} \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights
  3. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  4. Austin et al., Google DeepMind (2025). How to Scale Your Model
  5. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  6. Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights
  7. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  8. Austin et al., Google DeepMind (2025). How to Scale Your Model
  9. Llama Team, Meta (2024). The Llama 3 Herd of Models
  10. NVIDIA (docs). NVIDIA L4 Tensor Core GPU: product specifications
  11. NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
  12. Meta (mirrored by Unsloth on Hugging Face) (2024). Llama 3.1 8B Instruct: config.json
  13. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  14. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding