LLM & VLM courseBlog
123Track 5: LLM inference and serving · session 46 of 65

Runtime profiling: where the time goes

Before you optimize an LLM, you must know where its time goes. This session reads the survey’s profiling result (attention and linear operators often take more than 75% of the time), builds a small cost model of every kernel in one decode step, and shows why a GPU can sit idle while a model runs: too many tiny kernels, each waiting for the CPU to launch it. Then you learn to read a PyTorch profiler table and an Nsight Systems trace, and to name the top 3 time sinks of a real model.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 to 1.5 hours)Level: IntermediateBefore this: Sessions 81 (inference metrics), 82 (why inference is slow), 85 (CUDA basics) and 50 (CPU and GPU running side by side)

What you will learn

  • What the survey’s runtime profiling found, and how to read its claim that attention and linear operators often take more than 75% of the time
  • A one-line cost model for any kernel (bytes, FLOPs, a fixed floor), and how to use it to predict a profile before you run one
  • Why a decode step of an 8B model launches about 1,300 kernels in eager PyTorch, and why three quarters of them are tiny element-wise kernels
  • Launch overhead and CPU gaps: the timeline model that tells you when a faster kernel makes the model no faster at all
  • How to read a PyTorch profiler table: self time vs total time, CPU vs CUDA time, calls, and why operator names are not categories
  • How to read an Nsight Systems trace: NVTX ranges, the CUDA API row, the kernel row, gaps, and the sync at the end of each step
  • A repeatable procedure to find the top 3 time sinks of a real model, rank them with Amdahl’s law, and pick the session that fixes each one

Why this matters

Every optimization in the rest of this track makes one part of the model faster. But a part that takes 5% of the time can never give you more than a 5% gain. So the first job is always the same: measure where the time goes, then attack the biggest piece.

This sounds obvious, but LLM inference hides the answer well. A GPU kernel is launched by the CPU and runs later, so the time you see in Python is not the time the GPU spends. A decode step of an 8-billion-parameter model in plain PyTorch runs more than a thousand kernels. Most of them are tiny, and the GPU can be idle between them while the CPU prepares the next one. A faster matrix multiplication then changes nothing, because the GPU was not the slow part.

The efficient-inference survey (Zhou et al., Section 6.1.1) starts its system-level part with exactly this step. It profiles LLMs and finds two facts that organize the next sessions: attention and linear operators take most of the time, and many small operators break the timeline into pieces and add CPU launch cost. FlashAttention and FlashDecoding (session 124, session 125) attack attention, session 126 attacks the matrix multiplications, and session 127 attacks the small kernels. This session gives you the tools to decide which of these your model needs. In session 122 you measured speed for one model and one method; now you learn to explain the number.

What a profile measures

Three facts about how PyTorch runs on a GPU decide how to read every profile.

  1. Kernels are asynchronous. When Python calls torch.matmul on GPU tensors, the CPU puts a kernel (one GPU program) into a queue called a CUDA stream and returns at once. The GPU takes kernels from the queue in order. So CPU and GPU work at the same time, on different things.
  2. Each launch costs CPU time. Before a kernel is in the queue, Python runs the module code, the PyTorch dispatcher picks the right implementation, and the CUDA driver writes the launch. Together this is the launch overhead \(t_L\). In eager PyTorch it is on the order of a few to about ten microseconds per operator. [Likely]
  3. A sync drains the queue. Some calls make the CPU wait until the GPU has finished everything in the queue: .item(), .cpu(), printing a GPU tensor, torch.cuda.synchronize(). A generation loop has one at every step, because the CPU must know the new token (or at least whether all sequences are finished) before it starts the next step.

So a profile must answer three different questions, and different tools answer them:

QuestionWhat you look atTool
What does the GPU spend its busy time on?Kernel time, grouped by operatorPyTorch profiler table (self_cuda_time_total), nsys stats
Is the GPU waiting?Gaps in the GPU row of a timelineTrace viewer (Perfetto), Nsight Systems
Why is it waiting?What the CPU is doing during the gapCPU rows and CUDA API row of the same timeline

The survey’s profile

The survey profiles the inference runtime of several LLMs with the Hugging Face implementation, at different context lengths. Its Figure 15 is titled “Inference runtime breakdown over multiple LLMs”. Three findings: [Certain]

  • “Attention operators and linear operators collectively dominate runtime, with their combined duration often exceeding 75% of the inference duration.”
  • The share of the attention operator grows as the context gets longer.
  • For Mixtral (a mixture-of-experts model), the linear operator dominates, which the survey attributes to the non-optimized FFN code in the Hugging Face implementation.

And one more sentence, which is the second half of this session: there are many operators that each take a small share of the runtime, which “fragments the operator execution timeline and increases the cost of kernel launch on the CPU side”.

Note what the survey does not give: it does not state the GPU or split prefill from decode in that section. So treat “more than 75%” as a typical result for Hugging Face code, not as a law. The share depends on the model, the phase, the batch size, the context length and the code. The rest of this session builds a model that shows exactly how it depends on each of them.

A cost model for one kernel

You can predict most of a profile before you run it. A kernel must move some bytes between HBM (the GPU’s main memory) and the chip, and do some floating-point operations. Whichever takes longer sets its time, as in the roofline of session 82. Real kernels also have a fixed cost: even a kernel that does almost nothing takes a couple of microseconds to start, run and finish. So:

\[ d \;=\; t_0 \;+\; \max\!\left( \frac{Q}{\eta_B\, \mathrm{BW}},\ \frac{W}{\eta_F\, F} \right) \tag{123.1} \]
Read the symbols
\(d\)
the GPU time of one kernel
\(Q\)
bytes the kernel reads and writes in HBM
\(W\)
floating-point operations (FLOPs) the kernel does
\(\mathrm{BW}, F\)
peak memory bandwidth and peak compute; for an H100 SXM, 3.35 TB/s and about 989 TFLOP/s in dense BF16 (from session 81)
\(\eta_B, \eta_F\)
the fraction of peak a good kernel reaches; we use 0.8 for bandwidth and 0.7 for matmul compute (0.5 for attention in prefill)
\(t_0\)
the fixed floor of a kernel; we use 2 µs

The values of \(\eta_B\), \(\eta_F\) and \(t_0\) are our assumptions, chosen to be typical, not measured on one machine. [Likely] Every number in this session’s figures comes from formula (123.1) with these values, for Llama 3.1 8B (32 layers, hidden size 4096, 32 query heads, 8 KV heads of size 128, FFN size 14336, vocabulary 128,256, from its config.json [Certain]) on an H100 SXM in BF16. They show the shape of a real profile; your measured numbers will differ.

Worked example: a big kernel and a tiny kernel in one decode step

The query projection at batch 1 multiplies a 4096-vector by a \(4096 \times 4096\) BF16 matrix. It reads \(Q = 2 \cdot 4096 \cdot 4096 = 33.55\) MB of weights (plus 16 KB of activations), and does \(W = 2 \cdot 4096^2 = 33.6\) MFLOP. Memory time: \(33.57\ \text{MB} / (0.8 \times 3.35\ \text{TB/s}) = 12.5\ \mu s\). Compute time: \(0.05\ \mu s\). So \(d = 2 + 12.5 = 14.5\ \mu s\). It is a GEMV (matrix times vector), limited by memory. It reaches \(33.57\ \text{MB} / 14.5\ \mu s = 2.31\) TB/s, 69% of peak.

A multiply of two 4096-vectors in BF16 moves 24 KB. Memory time: \(0.009\ \mu s\). So \(d \approx t_0 = 2.0\ \mu s\). The kernel does almost no work; its time is all fixed cost.

Now compare with the launch cost. If the CPU needs \(t_L = 8\ \mu s\) to launch each kernel, the tiny multiply is finished 6 µs before the next kernel even arrives. The GPU waits. That is the core of the launch-overhead problem.

Where the GPU time goes in one decode step

Let us list the kernels of one decoder layer, written the way the Hugging Face Llama code runs in eager mode. Each PyTorch operator that computes something is one kernel. (Views and transposes are free; they only change the shape bookkeeping.) [Likely]

Part of the layerOperators (eager)KernelsClass
RMSNorm before attentioncast to FP32, pow, mean, + eps, rsqrt, mul, cast back, mul by weight8other
Q, K, V projectionsthree mm3linear
RoPE on Q and on Kq*cos, neg, cat (rotate half), *sin, add; the same for K10other
KV-cache updatetorch.cat for K and for V (the dynamic cache)2attention
Attentionscaled_dot_product_attention1attention
Output projection, residualmm, add2linear, other
RMSNorm before the MLPas above8other
MLP (SwiGLU)gate mm, up mm, silu, mul, down mm, residual add6linear, other
One layer407 linear, 3 attention, 30 other

The RMSNorm, RoPE and MLP lines follow the current modeling_llama.py, and the dynamic cache really appends with torch.cat while the static cache writes in place with index_copy_. [Certain] With 32 layers, plus the embedding, the final norm, the LM head, a cast of the logits to FP32 and an argmax, one decode step launches 1,292 kernels: 225 linear, 96 attention and 971 other. Three quarters of the kernels are tiny element-wise kernels.

The share of a class \(g\) (linear, attention or other) is its part of the total kernel time:

\[ p_g = \frac{\sum_{i \in g} d_i}{\sum_i d_i} \tag{123.2} \]

This is what the survey’s breakdown shows, and what a profiler table sorted by CUDA time gives you. Note that it is a share of GPU busy time, not of wall-clock time. We come back to that difference in the next section.

Why the attention share grows with context

In decode, both big classes are limited by memory, so compare their bytes per layer. The linear layers read all the layer’s weights once per step, whatever the batch. Attention reads the KV cache of every sequence in the batch:

\[ Q_{\text{lin}} \approx 2 P_\ell, \qquad Q_{\text{attn}} \approx B\, c\, s_\ell \qquad \Longrightarrow \qquad c^{*} = \frac{2 P_\ell}{B\, s_\ell} \tag{123.3} \]
Read the symbols
\(P_\ell\)
weights in one layer (218.1 million for Llama 3.1 8B); times 2 bytes in BF16
\(s_\ell\)
KV-cache bytes per token in one layer: \(2 \cdot 8 \cdot 128 \cdot 2 = 4{,}096\) bytes (keys and values, 8 KV heads, size 128, BF16)
\(B, c\)
batch size and context length
\(c^{*}\)
the context length at which attention reads as many bytes as the linear layers
Worked example: the crossover context

\(2 P_\ell = 436{,}207{,}616\) bytes. So \(c^{*} = 436{,}207{,}616 / (4{,}096\, B) = 106{,}496 / B\). For one sequence, attention reads as much as the weights only at about 106k tokens. For a batch of 32, already at 3,328 tokens. Batching shares the weight reads across sequences but not the KV-cache reads, so at large batch attention becomes the top sink.

Real eager code reaches the crossover much earlier, for two reasons. The dynamic cache copies the whole cache with torch.cat at every step (read it and write it again, for K and V), so the cache bytes are moved about 3 times instead of once. And a decode attention kernel with one query per head gives work to only \(B \cdot 32\) thread blocks; at \(B = 1\) that is 32 blocks for 132 SMs, so most of the chip is idle. Our model scales attention bandwidth by \(\min(1, 32B/132)\) for this. [Likely] With both effects, the model gives attention 24.9% of GPU time at 8k tokens and 55.6% at 32k, for one sequence. This low parallelism is exactly what FlashDecoding fixes by splitting the sequence across blocks (session 125).

Figure 1 runs this model. Try the default first (decode, batch 1, 1,024 tokens of context), then move the context, then the batch, then switch to prefill.

linear (matrix multiplications)attention (SDPA and cache update)other (norm, RoPE, activation, add, cast)
Figure 1. Share of GPU kernel time by operator class for Llama 3.1 8B on an H100, from the cost model (123.1). Each bar is one context length (128 to 128k tokens); the outlined bar is the context chosen with the slider. This is a model of a profile, not a measurement.
How to read Figure 1

At batch 1 and short context, the linear class is about 70% and “other” is about 23% of GPU time: the 971 tiny kernels add up, at 2 µs each. Linear plus attention is 77%, close to the survey’s “often more than 75%”. Move the context up: attention grows and passes linear between 16k and 32k tokens. Move the batch up at a fixed context: the weight reads are shared, the cache reads are not, so attention grows again. Turn on the fused mode: “other” almost disappears (423 kernels instead of 1,292), and attention time drops from 0.50 to 0.40 ms per step because the cache is no longer copied. In prefill, the matmuls do a lot of math per byte and “other” stays large, because the FP32 RMSNorm reads and writes big activations several times.

Many tiny kernels: launch overhead and CPU gaps

Figure 1 shows how the GPU spends its busy time. It says nothing about the time the GPU is idle. To see that, follow the CPU and the GPU on one timeline. The CPU launches kernel \(i\) when it has finished launching kernel \(i-1\). The GPU starts kernel \(i\) when the kernel has arrived and the previous kernel is done:

\[ c_i = c_{i-1} + t_L, \qquad s_i = \max(c_i,\ e_{i-1}), \qquad e_i = s_i + d_i \tag{123.4} \]
Read the symbols
\(c_i\)
the time at which the CPU has put kernel \(i\) in the queue
\(s_i, e_i\)
start and end of kernel \(i\) on the GPU
\(t_L\)
CPU time per launched operator (Python, dispatcher, driver)

When \(c_i \gt e_{i-1}\), the GPU waits: that is a gap. The step ends with the sync, when both are done. Its time, the GPU busy fraction and the idle time are:

\[ T_{\text{step}} = \max(c_n,\ e_n), \qquad U = \frac{\sum_i d_i}{T_{\text{step}}}, \qquad T_{\text{idle}} = T_{\text{step}} - \sum_i d_i \tag{123.5} \]

A simple rule follows. Over a long run of kernels, the GPU falls behind the CPU only if the kernels are longer than the launches. Let \(\bar d\) be the average kernel time over a stretch of the step. Then that stretch is

\[ \text{launch-bound if } \ t_L \gt \bar d, \qquad \text{GPU-bound if } \ t_L \lt \bar d \tag{123.6} \]

In a decode step the stretches alternate: a 46 µs MLP matmul lets the CPU run ahead and fill the queue; then a run of eight 2 µs RMSNorm kernels drains it, and the GPU waits for each new launch. Figure 2 shows this on a real timeline.

Worked example: one decode step, eager, batch 1, 1,024 tokens

With \(t_L = 8\ \mu s\): the CPU needs \(1{,}292 \times 8 = 10.34\) ms to launch the step. The GPU busy time is 8.50 ms. The timeline (123.4) gives \(T_{\text{step}} = 10.77\) ms, so \(U = 79\%\), and the GPU is idle for 2.26 ms (21% of the step). That is 92.9 tokens per second for this user.

Of the wall-clock step: linear kernels 56.2%, idle 21.0%, other kernels 18.1%, attention 4.6%. The idle time is the second biggest “operator”, and no profiler table lists it.

The lower bound for comparison: the step must read 15.0 GB of weights (7.50 billion parameters in the layers and the LM head; the embedding is only looked up). At the full 3.35 TB/s that is 4.48 ms; at 80% of it, 5.60 ms. With fused kernels and a CUDA graph (one launch for the whole step), the model gives 423 kernels and 6.67 ms, 150.0 tokens per second: 1.61× faster than eager.

CPU launching an operatorlinear kernelattention kernelother kernelGPU idle (hatched)
Figure 2. The timeline (123.4) of one decode step of Llama 3.1 8B at 1,024 tokens of context. Top: the first decoder layer, zoomed in, with the CPU lane above the GPU lane. Bottom: the whole step, with idle time hatched in red. Thin lines between the lanes connect each launch to its kernel.
How to read Figure 2

In eager mode the CPU lane is solid: the CPU is always busy launching. Under the RMSNorm and RoPE kernels the GPU lane is mostly hatched (idle), because each 2 µs kernel waits for an 8 µs launch. Under the long matmuls the GPU catches up and the CPU runs ahead. Now tick “make every linear kernel 2× faster”: in eager mode the step barely changes (10.77 to 10.52 ms), because the CPU still needs 10.34 ms to launch everything. Tick fused and CUDA graph first, then the 2× box: now the step goes from 6.67 to 3.64 ms. The same kernel speedup is worth 2% in one setting and 83% in the other. Raise the batch to 32 or 64: the kernels get longer, \(\bar d\) passes \(t_L\), and the hatched gaps close even in eager mode.

Reading a PyTorch profiler table

The PyTorch profiler (torch.profiler) records every operator the CPU runs and, with ProfilerActivity.CUDA, every kernel the GPU runs. prof.key_averages().table(sort_by=...) prints one row per operator name. The columns are easy to misread:

ColumnWhat it means
Self CPUCPU time inside this operator, not counting operators it calls. For a kernel launch, this is the launch overhead.
CPU totalCPU time including all children. aten::linear includes aten::t and aten::mm. Sorting by it puts wrapper ops and your record_function ranges on top.
Self CUDAGPU kernel time launched directly by this operator. This is the column for “what does the GPU spend its busy time on”.
# of CallsHow many times the operator ran. Divide every total by it: a big total can be many tiny calls or one slow call.
CPU time avgCPU total divided by calls. For tiny ops it shows you \(t_L\) directly.

Two traps. First, operator names are not categories. aten::cat appears both in RoPE (rotate half) and in the KV-cache update; aten::mul appears in RMSNorm, RoPE and SwiGLU. To get the survey’s classes you must group by where the op is called: wrap code in record_function("attn") or look at the call stack (group_by_stack_n). Second, the table has no row for idle time. The sum of self CUDA times is the busy time, not the step time. Always compare it with the measured step time.

When you have found a candidate, ask what fixing it is worth. If a part with share \(p\) of the time becomes \(s\) times faster, the total becomes faster by

\[ S = \frac{1}{(1 - p) + p / s} \tag{123.7} \]

This is Amdahl’s law (also used in session 50). It holds for the time the share was measured on. Apply it to GPU busy time and you get the best case; the real step gains less if the step is launch-bound, as Figure 2 showed. Figure 3 is a profiler table produced by our model of the same decode step. Click a row, then choose a speedup for it.

Figure 3. A profiler table for one decode step (batch 1, 1,024 tokens of context, \(t_L = 8\ \mu s\)), in the format of key_averages().table(). The times come from the cost model, not from a real GPU. Self CPU is \(\text{calls} \times t_L\).
How to read Figure 3

Sorted by Self CUDA, aten::mm is on top with 71% of GPU time from 225 calls. Sort by calls: aten::mul (290 calls) and aten::add (193) lead, with about 2 µs of GPU time per call but 8 µs of CPU time per call. Sort by Self CPU: the tiny ops dominate the CPU side, and that CPU time is what keeps the GPU idle. Select aten::mm with \(s = 2\): Amdahl on GPU time promises 1.55×, the timeline gives only 1.02× in eager. Switch to compiled and do the same: both are close to 1.83×. The readout also shows how much of the step is idle; that line is not in the real table.

Reading an Nsight Systems trace

Nsight Systems (nsys) records the whole process with low overhead: CPU threads, CUDA API calls, GPU kernels, memory copies and your own NVTX ranges (named time ranges you mark in code, for example with torch.cuda.nvtx.range_push("layer 3")). You open the report in the Nsight Systems GUI. The PyTorch profiler can also export a timeline (export_chrome_trace) that you open in Perfetto; it has similar rows. A trace has one horizontal row per thread or stream, with time going right. Read it from top to bottom, as Algorithm 1 says, and step through Figure 4.

Algorithm 1: reading one decode step in a trace
  1. Skip the first steps (compilation, memory allocator warmup, cuBLAS heuristics); pick one steady step between two syncs
  2. Measure the step on the GPU row: \(T_{\text{step}}\) and the busy time \(\sum d_i\), so \(U\) by (123.5) // \(U\) well below 1 means look at gaps before kernels
  3. Use the NVTX row to find the layers; check that all layers look the same // one slow layer means a different code path
  4. Zoom into one layer. If the CPU (CUDA API) row is solid and the GPU row has holes \(\rightarrow\) launch-bound, rule (123.6)
  5. If the CPU row has holes too, read what the CPU does there: Python work, allocator, a hidden sync (cudaStreamSynchronize, cudaMemcpy device to host)
  6. Only then look at the long kernels: compute bytes or FLOPs per µs and compare with the peak // formula (123.1)
  7. At the end of the step, check the sync: how long does the CPU wait, and is the GPU idle while the CPU prepares the next step?
NVTX range (CPU time)CUDA API call (launch)linear kernelattention kernelother kernelidle or waiting (hatched)
Figure 4. A drawn Nsight Systems timeline of one eager decode step, built from the timeline model (123.4) with \(t_L = 8\ \mu s\). Rows from top: NVTX ranges, the CUDA API calls of the Python thread, and the GPU kernels. Step through it in the order of Algorithm 1.
How to read Figure 4

The first view is the whole step: from far away the kernel row looks almost full, which is why people miss the problem. The zoomed views are one layer. The highlighted parts and the numbers in the text below the drawing are the measurements you would take in the real GUI: the length of a gap, the length of a launch, the bytes per microsecond of a long kernel. The last view shows the same step captured in a CUDA graph: one cudaGraphLaunch and a kernel row with no holes.

The commands, for a script that marks the decode loop with NVTX and starts the capture with torch.cuda.profiler.start():

nsys profile -t cuda,nvtx,osrt --capture-range=cudaProfilerApi --capture-range-end=stop \
     -o decode python generate.py
nsys stats -r cuda_gpu_kern_sum,cuda_api_sum,nvtx_sum decode.nsys-rep   # text summaries
# with CUDA graphs, add --cuda-graph-trace=node to see the kernels inside each graph (more overhead)

The trace, capture-range and graph-trace options are in the Nsight Systems user guide [Certain]; the names of the nsys stats reports have changed between versions, so check nsys stats --help-reports on your install. [Likely]

Finding the top 3 time sinks of a real model

Now put the pieces together into a procedure you can run on any model. A time sink is anything you can name and fix: an operator class, a group of kernels, or the idle time itself. The goal is a ranked list of three, each with its share of the wall-clock step and a candidate fix.

Algorithm 2: top 3 time sinks
  1. input: model, real input shapes (batch, prompt length, output length), the real dtype and the real attention backend
  2. Time prefill and one steady decode step separately with CUDA events after warmup // they have different sinks
  3. For each phase, profile a few steps with torch.profiler (CPU + CUDA), with record_function around attention, MLP, norms, sampling
  4. \(T_{\text{busy}} \leftarrow \sum\) self CUDA time per step; \(T_{\text{idle}} \leftarrow T_{\text{step}} - T_{\text{busy}}\) // formula (123.5)
  5. Group kernel time into classes (linear, attention, KV-cache update, norms and element-wise, sampling, communication); add “idle” as a class
  6. Rank the classes by share of \(T_{\text{step}}\); keep the top 3
  7. For each, compare with its bound from (123.1): a class near its bound needs less work (quantization, fewer bytes); a class far from it needs a better kernel
  8. Estimate the gain of each fix with Amdahl (123.7) on the right time base; confirm with a trace (Algorithm 1)
  9. output: three sinks, each with share, bound, fix, and expected gain

For the default decode step of this session (eager, batch 1, 1,024 tokens), the model’s answer is:

#SinkShare of the stepWhyFix and session
1Linear GEMVs (225 kernels)56.2%reading 15 GB of weights per token; already near the bandwidth boundfewer bytes: weight-only quantization (session 104), or more tokens per weight read: batching, speculative decoding (session 128)
2GPU idle (launch gaps)21.0%971 kernels of 2 µs launched at 8 µs eachCUDA graphs, torch.compile (session 127)
3Element-wise and norm kernels18.1%each tiny kernel pays the 2 µs floor; in prefill they also re-read activationsfusion: RMSNorm + residual, RoPE, SiLU × mul (session 127, session 86)

Change the setting and the list changes. At 32k context, attention (with the copied cache) moves to the top. At batch 32 and 8k context, attention is 82% of GPU time in eager mode. In a prefill of 4,096 tokens, idle time is near zero and the list is linear, other, attention. That is why you profile your shapes, not someone else’s.

Measuring correctly

A wrong measurement leads to the wrong fix. Four rules:

  • Synchronize before you read a clock. time.time() around GPU code measures how long the CPU took to queue the work. Use CUDA events (torch.cuda.Event(enable_timing=True)) or call torch.cuda.synchronize() before each clock read.
  • Warm up. The first calls include CUDA context setup, memory allocation, cuBLAS algorithm choice, and for torch.compile the compilation itself. Use a profiler schedule(wait=1, warmup=2, active=3) or run a few steps first.
  • Know the observer effect. The profiler adds CPU work per operator, so it makes launch-bound code look even more launch-bound. Recording shapes and stacks adds more. Take step times from a run without the profiler, and use the profile for shares.
  • Profile the real thing. The same model with SDPA or eager attention, BF16 or FP32, static or dynamic cache, batch 1 or 64 is a different program. Write the versions of PyTorch, the CUDA driver and the library next to every profile.

PyTorch code

A small Llama-style decoder written like the Hugging Face code (FP32 RMSNorm, rotate-half RoPE, torch.cat cache, SwiGLU), profiled for one decode step. It runs on CPU or GPU. The operator counts are deterministic; the times are not, and they depend on your machine.

import torch, torch.nn as nn, torch.nn.functional as Fn
from collections import Counter
from torch.profiler import profile, ProfilerActivity, record_function
torch.manual_seed(0)
dev = "cuda" if torch.cuda.is_available() else "cpu"
d, nh, nkv, dh, ff, L, V = 256, 8, 2, 32, 512, 4, 1000

class RMSNorm(nn.Module):                      # like LlamaRMSNorm: 8 kernels in eager mode
    def __init__(s, d): super().__init__(); s.w = nn.Parameter(torch.ones(d))
    def forward(s, x):
        h = x.to(torch.float32)
        h = h * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + 1e-6)
        return s.w * h.to(x.dtype)

def rot_half(x): x1, x2 = x.chunk(2, -1); return torch.cat((-x2, x1), -1)

class Layer(nn.Module):
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = RMSNorm(d), RMSNorm(d)
        s.q, s.k, s.v = (nn.Linear(d, n * dh, bias=False) for n in (nh, nkv, nkv))
        s.o = nn.Linear(nh * dh, d, bias=False)
        s.g, s.u, s.dn = nn.Linear(d, ff, bias=False), nn.Linear(d, ff, bias=False), nn.Linear(ff, d, bias=False)
    def forward(s, x, cos, sin, cache):
        B, T, _ = x.shape
        with record_function("attn"):            # a named range: groups ops by where they are called
            h = s.n1(x)
            q = s.q(h).view(B, T, nh, dh).transpose(1, 2)
            k = s.k(h).view(B, T, nkv, dh).transpose(1, 2)
            v = s.v(h).view(B, T, nkv, dh).transpose(1, 2)
            q, k = q * cos + rot_half(q) * sin, k * cos + rot_half(k) * sin
            k, v = torch.cat([cache[0], k], 2), torch.cat([cache[1], v], 2)   # copies the whole cache
            a = Fn.scaled_dot_product_attention(q, k, v, enable_gqa=True)
            x = x + s.o(a.transpose(1, 2).reshape(B, T, nh * dh))
        with record_function("mlp"):
            h = s.n2(x)
            x = x + s.dn(Fn.silu(s.g(h)) * s.u(h))
        return x, (k, v)

emb, layers = nn.Embedding(V, d).to(dev), nn.ModuleList(Layer() for _ in range(L)).to(dev)
norm, head = RMSNorm(d).to(dev), nn.Linear(d, V, bias=False).to(dev)

@torch.no_grad()
def step(tok, pos, caches):                    # one decode step for one new token
    x = emb(tok)
    inv = 1.0 / (10000 ** (torch.arange(0, dh, 2, device=dev) / dh))
    ang = pos * inv; ang = torch.cat([ang, ang]); cos, sin = ang.cos(), ang.sin()
    new = []
    for l, lay in enumerate(layers):
        x, c = lay(x, cos, sin, caches[l]); new.append(c)
    return head(norm(x[:, -1])).float().argmax(-1, keepdim=True), new

ctx = 64
caches = [(torch.randn(1, nkv, ctx, dh, device=dev), torch.randn(1, nkv, ctx, dh, device=dev)) for _ in range(L)]
tok = torch.tensor([[1]], device=dev)
for _ in range(3): step(tok, ctx, caches)       # warmup: never profile the first calls

acts = [ProfilerActivity.CPU] + ([ProfilerActivity.CUDA] if dev == "cuda" else [])
with profile(activities=acts) as prof:
    with record_function("decode_step"):
        nxt, _ = step(tok, ctx, caches)
        nxt.item()                              # the sync every generation loop has
key = "self_cuda_time_total" if dev == "cuda" else "self_cpu_time_total"
print(prof.key_averages().table(sort_by=key, row_limit=12))

# group the top-level operators into the survey's classes
cls = {"linear": ("aten::linear", "aten::mm", "aten::addmm", "aten::matmul"),
       "attention": ("aten::scaled_dot_product_attention",)}
top = [e for e in prof.events() if e.name.startswith("aten::") and e.cpu_parent is not None
       and not e.cpu_parent.name.startswith("aten::")]
count = Counter(next((k for k, v in cls.items() if e.name.startswith(v)), "other") for e in top)
print(dict(count), "top-level ops:", len(top))   # linear 29, attention 4, other 191: 224 ops
print(Counter(e.name for e in top).most_common(5))   # aten::mul 40, aten::linear 29, aten::add 25, ...

# time a GPU step the right way: CUDA events, after warmup
if dev == "cuda":
    s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    s.record()
    for _ in range(20): step(tok, ctx, caches)
    e.record(); torch.cuda.synchronize()
    print("ms per step:", s.elapsed_time(e) / 20)

We ran this on CPU with PyTorch 2.8: 224 top-level operators for 4 layers (29 linear, 4 attention, 191 other), the same every run. Some of the “other” ops are views that launch no kernel; on a GPU, the CUDA rows of the table tell you which ones really ran. To mark the same regions for Nsight Systems, replace record_function with torch.cuda.nvtx.range_push("attn") and range_pop(), or run the profiler with NVTX emission on.

Where this is used

SettingWhat the profile usually showsWhat people do
Chat model at batch 1 on a local GPU (Hugging Face eager)GEMVs on top, a large share of GPU idle time from launch gaps, many norm and RoPE kernelsstatic cache + torch.compile with CUDA graphs, or a dedicated engine (llama.cpp, vLLM) [Likely]
Serving engine at large batch (vLLM, SGLang, TensorRT-LLM)linear GEMMs and decode attention; attention grows with batch × contextpaged attention kernels, FlashDecoding, CUDA graphs for decode, quantized KV cache (session 138) [Likely]
Long-context summarization or RAG (32k+ tokens)attention dominates prefill and decodeFlashAttention for prefill, split-K decode attention, KV-cache compression
MoE models in reference codelinear dominates, many small per-expert matmuls and index ops (the survey’s Mixtral case)fused or grouped MoE kernels (session 95)
VLM answering about an image or a videothe vision encoder and the long prefill of image tokens appear as their own blocks before the first tokenprofile time to first token separately; cache image features; fewer visual tokens [Likely]
Video retrieval and Video-RAG indexing (embedding many frames)the encoder at large batch is GPU-bound; decode and resize on the CPU can leave the GPU idlecheck the CPU rows first: GPU video decoding, more loader workers, pinned memory
Live key-moment detection or action recognition on a streamsmall batches per frame window: launch gaps and host-to-device copiesCUDA graphs for the fixed-shape model, batching across streams

Common mistakes

  • Timing GPU code with a CPU clock and no sync. You measure how fast the CPU queues work. Use CUDA events or synchronize first.
  • Profiling the first step. It contains allocation, autotuning and compilation. Use a schedule with warmup.
  • Reading the profiler table as the whole truth. The sum of kernel times is GPU busy time. Idle time is invisible in the table; compare with the measured step time or look at the trace.
  • Sorting by CPU total. Wrappers and record_function ranges come first because they include their children. Use self time, or the CUDA column, for “who uses the time”.
  • Treating an operator name as one thing. aten::mul or aten::cat can come from five different places. Group by call site before you draw conclusions.
  • Mixing prefill and decode in one profile. One long prefill hides the decode pattern, or the other way round. Profile them separately.
  • Optimizing the biggest kernel in a launch-bound step. Figure 2: a 2× faster matmul gave 1.02×. Remove the gaps first.
  • Generalizing from one implementation. The survey’s Mixtral result came from unoptimized reference code. A profile describes a code version and a shape, not a model architecture.

Interview questions

What did the survey’s runtime profiling find, and how would you use it?basic
Profiling Hugging Face implementations of several LLMs, the survey found that attention and linear operators together often take more than 75% of the inference time, that the attention share grows with context length, and that Mixtral was dominated by linear operators because of non-optimized FFN code. It also found many small operators that fragment the timeline and add CPU launch cost. I would use it as a prior, not an answer: it says where to look first (matmuls, attention, small-kernel overhead), but I would profile my own model, phase and shapes before choosing a fix.
Why is a GPU often idle during decode at batch 1 in eager PyTorch?core
Each operator costs CPU time to launch (Python, dispatcher, driver), on the order of several microseconds. A decode step of an 8B model has more than a thousand kernels, and most of them (norm, RoPE, add, cast) take only about 2 µs on the GPU. When the launch takes longer than the kernel, the GPU finishes and waits for the next launch. In our model of Llama 3.1 8B at batch 1, the CPU needs 10.3 ms to launch a step whose kernels take 8.5 ms, and the GPU is idle about 21% of the step.
You made all matmuls 2× faster and the model got 2% faster. What happened?core
The step is launch-bound. The step time is about max(CPU launch time, GPU time). The CPU still has to launch the same number of kernels, so shortening GPU kernels only creates more idle time. Amdahl on GPU time predicted about 1.55×; the timeline gives about 1.02×. Fix the launch problem first (CUDA graphs, torch.compile, fusion), and then the same matmul speedup gives about 1.8× in our model.
In a PyTorch profiler table, what is the difference between Self CPU, CPU total and Self CUDA?basic
Self CPU is CPU time spent in the operator itself, without its children. CPU total includes the children, so wrappers like aten::linear or your record_function ranges look large. Self CUDA is GPU kernel time launched directly by the operator; this is the column for GPU busy time. For tiny ops, the CPU time per call is close to the launch overhead, and comparing it with the CUDA time per call tells you if the op is launch-bound.
How do you see launch overhead in an Nsight Systems trace?core
Zoom into one layer. The CUDA API row of the Python thread is a solid sequence of cudaLaunchKernel calls with Python work between them, and the GPU kernel row has short kernels separated by gaps. The connections from each launch to its kernel show that each kernel starts right after its launch ends, not right after the previous kernel. In contrast, a GPU-bound stretch has a full kernel row and a CPU that runs ahead, so kernels start long after their launch.
Why does attention take a bigger share at long context, and even more at large batch?core
In decode, linear layers read all weights once per step regardless of batch, while attention reads the KV cache of every sequence. Per layer the weights are \(2P\) bytes and the cache is \(B \cdot c \cdot s\) bytes, so attention passes linear at \(c^{*} = 2P/(B s)\). For Llama 3.1 8B that is \(106{,}496/B\) tokens if the cache is read once. Batching divides the weight cost per token but not the cache cost, so attention becomes the top sink at large batch × context. Eager code reaches it earlier because the dynamic cache is copied with torch.cat and decode attention has little parallelism at small batch.
How do you time a GPU operation correctly?basic
Warm up first, then record CUDA events around many iterations and synchronize before reading elapsed_time, or call torch.cuda.synchronize() before each clock read. Never time with time.time() alone, because kernel launches are asynchronous. Report the median of several runs, and measure without the profiler, which adds its own overhead.
nvidia-smi shows 100% GPU utilization. Is the GPU well used?core
Not necessarily. That metric is the fraction of time at least one kernel was running, not how much of the chip was used. A decode attention kernel using 32 of 132 SMs, or a chain of tiny element-wise kernels, can show high utilization while reaching a small fraction of peak bandwidth or compute. Compare achieved bytes/s or FLOP/s per kernel with the peaks, and look for idle gaps in a trace.
Walk me through finding the top 3 time sinks of a model you have never seen.advanced
Fix the real shapes and dtype. Time prefill and a steady decode step separately with CUDA events. Profile a few steps with CPU and CUDA activities and named ranges. Compute busy time from kernel times and idle time as step time minus busy time. Group kernels by call site into classes (linear, attention, cache update, norms and element-wise, sampling, communication) and add idle as a class. Rank by share of wall time and take three. For each, compare with its roofline bound to decide between less work and a better kernel, estimate the gain with Amdahl on the correct time base, and confirm in a trace. For eager batch-1 decode of an 8B model, I would expect linear GEMVs, launch gaps and small element-wise kernels.
Why can Amdahl’s law on profiler shares overestimate a speedup?advanced
Profiler shares are shares of GPU busy time. If the step is limited by CPU launches, the wall time is not the sum of kernel times, so a faster kernel does not shorten the step by its own saving. Also, speeding one kernel can change overlap: the CPU may no longer be able to run ahead, or the next kernel may hit a cold cache. Apply Amdahl to wall-clock shares, including idle time, and confirm with a measurement. The model in this session gives 1.55× predicted versus 1.02× real for 2× faster matmuls in eager decode.
Why does prefill have a very different profile from decode?core
Prefill processes many tokens per weight read, so the matmuls are compute-bound GEMMs and every kernel is long; launch overhead is hidden. Element-wise ops are no longer tiny: they read and write large activations, so FP32 norms and unfused RoPE cost real bandwidth. Attention grows with the square of the prompt length. Decode is one token per sequence: memory-bound GEMVs, tiny element-wise kernels, launch gaps, and attention that grows with the cache size.

Exercises

  1. With the cost model (123.1) and our constants, compute the GPU time of the MLP down projection (\(14336 \times 4096\), BF16) in a decode step at batch 1.
  2. A stretch of 30 kernels of 2 µs each is launched with \(t_L = 8\ \mu s\), starting with an empty queue. How long does the stretch take, and what is \(U\) for it?
  3. Compute the crossover context \(c^{*}\) of (123.3) for batch 16 if the cache is read once. What if the eager cache moves its bytes 3 times?
  4. Element-wise kernels take 18.1% of a decode step and fusion makes them 4× faster. What does Amdahl give? Why may the real gain be larger?
  5. A profiler table shows aten::mul with 290 calls, 2.3 ms self CPU and 0.58 ms self CUDA. What is the CPU time per call and the GPU time per call? What do you conclude?
  6. A colleague reports “the model runs at 100 tokens/s” measured with time.time() around a loop of 100 calls to model(x), with no sync. Name two ways this number can be wrong.
Answerscheck
1. \(Q = 2 \cdot 14336 \cdot 4096 = 117.4\) MB (plus tiny activations); memory time \(117.48\ \text{MB} / 2.68\ \text{TB/s} = 43.8\ \mu s\); \(d = 45.8\ \mu s\). 2. Each kernel waits for its launch: kernel \(i\) starts at \(8i\) µs and ends at \(8i + 2\), so the stretch takes \(30 \times 8 + 2 = 242\) µs while the GPU is busy 60 µs: \(U = 60/242 \approx 25\%\). 3. \(106{,}496 / 16 = 6{,}656\) tokens; with 3× the bytes, about 2,219 tokens. 4. \(1/(0.819 + 0.181/4) = 1.16\times\). Fusion also removes launches, so it shrinks the idle time (21%) too; the real gain can be larger than Amdahl on the element-wise share alone. 5. 7.9 µs of CPU and 2.0 µs of GPU per call: the op is launch-bound, and the CPU side is the cost. Fuse it or capture it in a graph. 6. Without a sync, the timer can stop before the GPU has finished (too fast), and the loop includes the first calls with warmup costs (too slow); also, a forward call on a fixed input is not a generation step (no cache growth, no sampling sync).

Key takeaways

  • Measure before you optimize. The survey finds attention plus linear often above 75% of the time in Hugging Face code, attention growing with context, and many small operators that add launch cost.
  • Each kernel costs about \(t_0 + \max(Q/\text{BW}, W/F)\) (123.1). In decode at batch 1, matmuls are memory-bound GEMVs and most other kernels are only the 2 µs floor.
  • Attention passes linear at \(c^{*} = 2P_\ell/(B s_\ell)\) (123.3): about 106k/B tokens for Llama 3.1 8B, much earlier in eager code.
  • Launch overhead: when \(t_L \gt \bar d\) the GPU waits (123.4–123.6). In our eager decode model, 21% of the step is idle, and a 2× faster matmul gives only 1.02×.
  • A profiler table shows busy time by operator name; a trace shows gaps and why. Rank sinks by share of wall time, including idle, and estimate fixes with Amdahl (123.7).

Key formulas

\[ d = t_0 + \max\!\Big(\frac{Q}{\eta_B \mathrm{BW}}, \frac{W}{\eta_F F}\Big), \quad s_i = \max(c_i, e_{i-1}), \quad U = \frac{\sum_i d_i}{T_{\text{step}}}, \quad c^{*} = \frac{2P_\ell}{B s_\ell}, \quad S = \frac{1}{(1-p) + p/s} \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
  3. NVIDIA (docs). CUDA C++ Programming Guide
  4. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  5. Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
  6. NVIDIA (docs). CUDA C++ Programming Guide
  7. PyTorch (docs). torch.profiler: PyTorch Profiler API reference
  8. PyTorch (docs). PyTorch Profiler recipe (torch.profiler: schedule, key_averages, export_chrome_trace)
  9. NVIDIA (docs). Nsight Systems User Guide
  10. Nguyen, Carilli, Eryilmaz et al. (NVIDIA and Meta) (2021). Accelerating PyTorch with CUDA Graphs (PyTorch blog)
  11. Hugging Face (code). transformers: modeling_llama.py (LlamaRMSNorm, apply_rotary_pos_emb, LlamaMLP)
  12. Hugging Face (code). transformers: cache_utils.py (DynamicLayer uses torch.cat, StaticLayer uses index_copy_)
  13. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  14. Dao et al. (2023). Flash-Decoding for long-context inference (blog)
  15. Hong et al. (2023). FlashDecoding++: Faster Large Language Model Inference on GPUs
  16. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  17. Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights