LLM & VLM courseBlog
50Track 3: Pretraining and scale · session 20 of 22

Making training fast: compile, CUDA graphs, fused kernels, profiling

A training step is thousands of GPU kernels launched from Python. Speed is lost in three places: the CPU cannot launch kernels fast enough, small kernels move the same data in and out of memory again and again, and the GPU waits for data, copies or synchronizations. This session shows the tools that fix each one (torch.compile, CUDA graphs, fused kernels for attention, RMSNorm and cross-entropy) and how to find the problem first by reading a profiler trace.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 37, 38 and 42 (numbers on GPUs, memory math for training, GPU architecture for ML)

What you will learn

  • Where training time goes: launch-bound, memory-bound and compute-bound work, and why the step time is roughly the larger of the CPU time and the GPU time
  • How torch.compile works: TorchDynamo captures an FX graph with guards, AOTAutograd builds the backward graph, TorchInductor fuses operations into Triton kernels
  • Graph breaks and recompiles: what causes them, how to count them, and how to remove them
  • CUDA graphs: capture once, replay with one launch; the rules (static shapes, static addresses, no CPU synchronization) and mode="reduce-overhead"
  • Fused kernels and why they help: FlashAttention, RMSNorm and SwiGLU fusion, and fused or chunked cross-entropy that never stores the full logits
  • How to read a PyTorch profiler or Nsight Systems trace: idle gaps, data-loader stalls, pageable memory copies, hidden synchronizations and exposed communication
  • How to estimate the benefit of a fix before you write it, with bytes, launch counts and Amdahl's law

Why this matters

Two teams train the same model on the same GPUs. One reaches 45% of the GPU’s peak arithmetic, the other 20%. The second team pays more than twice as much for the same model. The difference is rarely the model. It is the code around it: how kernels are launched, how often data is moved, and where the GPU sits idle.

In session 42 you learned the roofline: an operation is either compute-bound (limited by tensor-core FLOPs) or memory-bound (limited by HBM bytes per second). Real training adds a third limit that the roofline does not show: the CPU. PyTorch runs your Python code on the CPU, and every operation becomes one or more kernels (small GPU programs) that the CPU must launch one at a time. If launching takes longer than running, the GPU waits.

So there are three ways a training step is slow, and each has a standard fix:

ProblemWhat you see in a traceStandard fix
Launch-bound: the CPU cannot launch kernels fast enoughMany tiny kernels with gaps between them; the CPU thread is 100% busyCUDA graphs; fewer, bigger kernels (torch.compile); bigger batches
Memory-bound: kernels read and write the same tensor again and againMany element-wise kernels (add, mul, norm, activation) that together take a large share of the stepKernel fusion: torch.compile, or hand-written fused kernels (FlashAttention, fused RMSNorm, fused cross-entropy)
Starved: the GPU waits for something elseLarge idle blocks on the GPU: data loading, memory copies, .item() synchronizations, communicationFix the input pipeline, pinned memory, remove syncs, overlap communication

The rule of this session: measure first, then fix. A profiler trace tells you which of the three problems you have. Fixing the wrong one wastes days. This session builds the mental models, then shows the tools, then teaches you to read a trace. The previous session, session 49, spread one model over many GPUs. This one makes each GPU do its share faster. The next session, session 51, measures the result with MFU.

A model of the training step: CPU and GPU run side by side

A CUDA kernel launch is asynchronous. The CPU puts the kernel into a queue (a CUDA stream) and returns at once, without waiting for the GPU. The GPU takes kernels from the queue and runs them in order. So the CPU and the GPU work at the same time, like two people on a production line.

Let \(n\) be the number of kernels in one step, \(t_L\) the CPU time to launch one kernel (Python, the PyTorch dispatcher and the CUDA driver together), and \(d_i\) the GPU time of kernel \(i\). As long as the queue never fills up, the step time is about the larger of the two workloads:

\[ T_{\text{step}} \approx \max\!\Big( n\, t_L,\ \sum_{i=1}^{n} d_i \Big), \qquad U_{\text{GPU}} = \frac{\sum_i d_i}{T_{\text{step}}} \tag{50.1} \]
Read the symbols
\(n\, t_L\)
the total CPU time spent launching kernels in one step
\(\sum_i d_i\)
the total time the GPU is really computing
\(U_{\text{GPU}}\)
the fraction of the step the GPU is busy; 1 means no idle gaps

This is a simplification: it ignores that a kernel can only start after its launch, and that real kernels have different sizes. Figure 3 simulates the exact version. But it shows the key fact: when the kernels are small (\(d_i \lt t_L\)), the GPU time does not matter; the step is limited by launches. A kernel launch from eager PyTorch costs on the order of a few microseconds to about ten microseconds of CPU time. [Likely] A large matrix multiplication runs for hundreds of microseconds or more, so big models with big batches are rarely launch-bound. Small models, small batches per GPU (common with heavy parallelism), and long chains of element-wise operations are.

Worked example: a launch-bound step

Assume a step launches \(n = 1{,}000\) kernels, each runs for \(d = 4\ \mu s\) on the GPU, and each launch costs \(t_L = 10\ \mu s\) on the CPU (illustrative numbers). Then \(n t_L = 10\) ms and \(\sum d_i = 4\) ms. By (50.1), the step takes about 10 ms and the GPU is busy \(4/10 = 40\%\) of the time.

A GPU twice as fast changes nothing: still 10 ms. Removing the launch cost (a CUDA graph, next sections) gives about 4 ms, a 2.5× speedup. Fusing pairs of kernels to halve \(n\) gives \(\max(5, 4) = 5\) ms, 2×, and also less memory traffic.

torch.compile: from Python to fused kernels

In eager mode (normal PyTorch), each line of Python runs as soon as it is reached, and each operation launches its own kernel. This is easy to debug but leaves speed on the table: PyTorch never sees more than one operation at a time, so it cannot combine them. torch.compile (PyTorch 2) looks at a whole region of code at once, turns it into a graph of operations, and generates faster code for the graph. The PyTorch 2 paper reports a geometric-mean speedup of 2.27× for inference and 1.41× for training over eager mode, on an NVIDIA A100 across more than 180 real models. [Certain] Your own speedup can be anywhere from none to several times; measure it.

It has three parts (Figure 1):

  1. TorchDynamo (graph capture). It hooks into CPython’s frame evaluation, reads the bytecode of your function before it runs, and records every PyTorch operation into an FX graph (a simple list of operations and their inputs). Plain Python (loops over a fixed range, if-statements on Python values, attribute lookups) is run during tracing and disappears from the graph. Dynamo also writes guards: cheap checks on the inputs (shapes, dtypes, devices, values of Python numbers) that must be true for the compiled code to be reused.
  2. AOTAutograd (ahead-of-time autograd). It traces the backward pass of the captured graph too, so the compiler sees the forward and the backward together. It then decides which intermediate values to save for backward and which to recompute, trading cheap recomputation of element-wise results for memory.
  3. TorchInductor (the code generator, the default backend). It groups operations, fuses chains of element-wise and reduction operations into single kernels, and writes them in Triton (a Python-like language for GPU kernels) on GPUs, or C++ on CPUs. Large matrix multiplications usually still call cuBLAS, or Triton templates when you ask for autotuning.
Figure 1. The torch.compile stack. Dynamo turns Python bytecode into an FX graph plus guards. AOTAutograd adds the backward graph. Inductor fuses operations and emits Triton kernels. The compiled code is cached and reused while the guards hold.

The first call of a compiled function is slow: it traces and compiles, which can take seconds to minutes for a large model. Later calls check the guards (microseconds) and run the cached kernels. So torch.compile pays off only when the same code runs many times with the same kind of inputs, which is exactly what a training loop does.

Algorithm 1: what happens when you call a compiled function
  1. input: function \(f\), arguments \(a\), cache of (guards, compiled code) pairs for \(f\)
  2. for each cached entry: if all its guards pass on \(a\): return its compiled code run on \(a\) // fast path
  3. if the number of entries has reached the recompile limit (8 by default): run \(f\) eagerly and return
  4. Dynamo traces the bytecode of \(f\) on \(a\), recording PyTorch operations into an FX graph and writing guards
  5. if it meets something it cannot trace: end the graph here (graph break), run that piece in plain Python, start a new graph after it
  6. AOTAutograd builds the joint forward and backward graph; Inductor fuses it and generates kernels
  7. store (guards, compiled code) in the cache; run it on \(a\)

Graph breaks and recompiles

Two things quietly destroy the speedup of torch.compile.

A graph break happens when Dynamo meets code it cannot put into a graph. It then compiles the part before, runs the problem in normal Python, and starts a new graph after it. Each break means: two smaller graphs (less to fuse), a return to Python (launch overhead again), and sometimes a CPU-GPU synchronization. Typical causes:

  • Branching on a tensor value: if loss > 0: or if mask.any():. The value is on the GPU and is not known during tracing.
  • Reading a value to the CPU: .item(), .tolist(), .numpy(). By default .item() is a graph break (the setting capture_scalar_outputs can change this).
  • Calls Dynamo cannot see into: print, logging, some C extensions and third-party libraries.

A recompile happens when a guard fails: the input shape, dtype or a Python number changed. Dynamo then compiles again for the new case. Integer and float arguments are treated as constants by default, so passing a changing Python float (for example the learning rate, or a step counter) into a compiled function recompiles every time. For shapes, the default (dynamic=None) is: compile the first call for the exact shape; if a later call has a different size, recompile once with that dimension made dynamic (symbolic), which then works for all sizes. [Certain] After 8 compiles of the same function (the default recompile_limit), Dynamo gives up and runs it eagerly. [Certain]

Figure 2 lets you add these problems to a small forward function and see how many graphs and compiles you get. The counts follow what torch._dynamo.explain and a counting backend report in PyTorch 2.8 for the same patterns (checked by running the code at the end of this session).

compiled graphgraph break: back to Python
Figure 2. Graph breaks and recompiles for a small block run for 4 training steps. Each break splits the code into one more graph; each graph is compiled once per distinct guard set.
How to read Figure 2

With no box ticked, the whole block is one graph, compiled once. Each break you add cuts the code: the blue boxes are separate graphs that Inductor optimizes separately, and each orange cut is a return to Python. Ticking “sequence length changes” makes each graph compile twice: once for length 128 and once more, with a dynamic length, at length 256; lengths 384 and 512 reuse the dynamic version. With fullgraph=True, any break becomes an error at the first call, which is the fastest way to find them.

How to remove the common breaks:

  • Replace data-dependent Python branches with tensor operations: torch.where(cond, a, b) computes both sides and selects, with no break.
  • Move .item(), printing and logging out of the compiled region. Log a detached tensor every \(k\) steps, outside the model.
  • Keep shapes stable: pack sequences to a fixed length (session 18), or bucket them into a few fixed lengths, or mark the dimension dynamic from the start with torch._dynamo.mark_dynamic(x, 1).
  • Pass changing numbers as 0-dimensional tensors, not as Python floats.
  • For large models, compile each transformer block (block.compile() or torch.compile(block) per layer) instead of the whole model. All blocks have the same code, so the compiled code is reused and compile time drops a lot; this is what torchtitan does. [Likely]

CUDA graphs: one launch for the whole step

torch.compile reduces the number of kernels. A CUDA graph removes the launch cost of the kernels that remain. The idea: run the step once in capture mode. Kernels issued to a capturing stream do not run; they are recorded, with all their arguments, into a graph. After that, one call (cudaGraphLaunch) replays the whole recorded sequence. The replay skips Python, the PyTorch dispatcher and most of the driver work for every kernel. [Certain]

The price is flexibility. A replay runs the same kernels with the same arguments, and for pointer arguments that means the same memory addresses. [Certain] So:

  • Static shapes. A new batch shape needs a new capture.
  • Static memory. Inputs must be copied into the same “static” input tensors before each replay; outputs appear in the same static output tensors. PyTorch gives each graph a private memory pool for its intermediate tensors.
  • No CPU synchronization inside. No .item(), no printing of tensor values, no data-dependent Python branches, no operations whose output size depends on the data (such as nonzero or boolean indexing).
  • No CPU-side logic per step inside the graph. Everything the CPU decides (for example a Python learning-rate value) is frozen at capture time, unless it lives in a GPU tensor that you update in place.
Algorithm 2: capture and replay a training step with a CUDA graph
  1. input: model, optimizer, static input tensor \(X_s\), static target tensor \(Y_s\)
  2. warm up: run 3 normal steps on a side stream // lets lazy initialization and the allocator settle
  3. set gradients to None, then begin capture on the stream
  4. \(\hat Y_s \leftarrow\) model(\(X_s\)); \(L_s \leftarrow\) loss(\(\hat Y_s, Y_s\)); backward(\(L_s\)); optimizer step // recorded, not run
  5. end capture \(\rightarrow\) graph \(G\)
  6. for each new batch \((X, Y)\): copy \(X \to X_s\), \(Y \to Y_s\) (in place); replay \(G\) // one launch; gradients are refilled in place, no zero_grad needed
  7. read the loss from \(L_s\) only when you need it (reading it is a sync)

How much does it help? NVIDIA’s MLPerf v1.0 training submissions in PyTorch used CUDA graphs. For Mask R-CNN on 272 GPUs (batch 1 per GPU), the graphed backbone ran in 6 ms instead of 31 ms, and the whole step was about 1.70× faster. For BERT on 4,096 GPUs, the gain was 1.12×. [Certain] The pattern is clear: the gain is largest when per-GPU work is small, which is exactly what strong scaling to many GPUs produces. For a big model with a big per-GPU batch, the GPU is already the bottleneck and a CUDA graph gives little.

You rarely capture graphs by hand. torch.compile(model, mode="reduce-overhead") uses CUDA graphs for the compiled regions; mode="max-autotune" also enables them on GPU, and "max-autotune-no-cudagraphs" autotunes without them. [Certain] For inference with a KV cache, CUDA graphs are standard for the decode step, because each step is small (session 127).

CPU: launching a kernelGPU: matrix multiplicationGPU: element-wise kernelGPU idle
Figure 3. A simulated block of 60 kernels (10 repeats of: matmul, 2 element-wise, a norm, matmul, element-wise; base durations 6, 2, 2, 3, 6, 2 µs). A kernel starts when its launch is done and the previous kernel has finished. With a CUDA graph there is one launch, then the kernels run back to back. These are model numbers to build intuition, not measurements.
How to read Figure 3

Start at batch scale 1 and launch cost 8 µs. The red marks under the GPU lane are idle time: the GPU has finished and waits for the next launch. Switch to the CUDA graph: the kernels pack tightly and the step is about twice as fast. Now raise the batch scale to 4 or more. The kernels become longer than the launches, the eager timeline also packs tightly, and the CUDA graph gains almost nothing. That is formula (50.1): once \(\sum d_i\) is larger than \(n t_L\), launches are hidden.

Kernel fusion: stop moving the same bytes

Most operations in a transformer that are not matrix multiplications are element-wise or small reductions: residual adds, normalization, activation functions, dropout, scaling, masking, softmax. Each one does a handful of FLOPs per element but must read its inputs from HBM and write its output back. They are deeply memory-bound. By the roofline, their time is set by bytes, not FLOPs:

\[ t_{\text{kernel}} \approx \max\!\left( \frac{\text{FLOPs}}{P_{\text{peak}}},\ \frac{\text{bytes moved}}{B_{\text{HBM}}} \right) \tag{50.2} \]

Fusion merges a chain of such operations into one kernel. The fused kernel reads the inputs once, keeps the intermediate values in registers or shared memory, and writes only the final output. The FLOPs are the same; the bytes drop. For a chain of memory-bound kernels:

\[ S_{\text{fuse}} \approx \frac{\sum_{k} (\text{bytes read}_k + \text{bytes written}_k)}{\text{bytes read once} + \text{bytes written once}} \tag{50.3} \]

Take RMSNorm (session 22), used in Llama and most current LLMs:

\[ \operatorname{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum_{j=1}^{d} x_j^2 + \epsilon}} \odot \mathbf{g} \tag{50.4} \]
Read the symbols
\(\mathbf{x} \in \R^d\)
one token’s hidden vector (\(d = 4{,}096\) in Llama 3 8B)
\(\epsilon\)
a small constant for safety (\(10^{-5}\) in Llama 3 8B)
\(\mathbf{g}\)
the learned gain vector; \(\odot\) is element-wise multiplication

Written with plain PyTorch operations, x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * g launches several kernels: pow reads \(x\) and writes \(x^2\); mean reads \(x^2\); the small per-token steps (add \(\epsilon\), rsqrt) touch only one number per token; x * r reads \(x\) and writes a new tensor; * g reads that and writes the output. Counting only the big tensors of \(N = T \cdot d\) elements, that is about \(7N\) element reads and writes. A fused kernel reads \(x\) once and writes \(y\) once: \(2N\). So fusion moves 3.5× fewer bytes.

Worked example: RMSNorm on an H100

A micro-batch of 4 sequences of 4,096 tokens gives \(T = 16{,}384\) tokens; with \(d = 4{,}096\), \(N = 67{,}108{,}864\) elements, 2 bytes each in BF16.

Unfused: \(7 \times N \times 2 = 939.5\) MB moved. At the H100’s 3.35 TB/s, that is at least 280 µs. Fused: \(2 \times N \times 2 = 268.4\) MB, at least 80 µs. One RMSNorm saves 200 µs. A Llama-style model has two RMSNorms per layer, and the backward pass has its own memory-bound kernels, so across 32 layers the savings add up to many milliseconds per step. Real eager code often moves even more, because it upcasts to FP32 inside the norm.

Figure 4 does the same count for three common chains. Fusion helps most when a long chain of cheap operations works on the same tensor.

bytes readbytes written
Figure 4. HBM traffic of a chain of element-wise kernels, unfused (one bar per kernel) and fused (one bar). BF16 values, H100 bandwidth 3.35 TB/s. Only tensors with one value per element are counted; per-token values are tiny. For SwiGLU the hidden size is the FFN width.
How to read Figure 4

Each bar is one kernel; its length is the bytes it reads (blue) and writes (orange). The time in the readout is the lower bound bytes ÷ bandwidth from formula (50.2). The ratio does not depend on the sizes, only on the chain: 3.5× for RMSNorm, 2.5× for add + RMSNorm, 1.67× for SwiGLU. The absolute savings grow with tokens × hidden size.

Fusion has a second benefit in training: memory. Each unfused intermediate that autograd needs for the backward pass is stored until the backward reaches it. A fused kernel with its own backward can save only what it needs (for RMSNorm, the input and one number per token) and recompute the rest. That is the same idea as activation checkpointing (session 39), done inside one kernel.

Three fused kernels every LLM trainer uses

Fused attention (FlashAttention)

Plain attention computes the score matrix \(S = QK^\top\), writes it to HBM, reads it back for the softmax, writes \(P\), and reads it again for \(PV\). For a sequence of \(T\) tokens and \(h\) heads the score matrix alone is

\[ M_S = B \cdot h \cdot T^2 \cdot b \quad \text{bytes, versus} \quad O(B \cdot h \cdot T \cdot d_h) \ \text{for FlashAttention} \tag{50.5} \]

where \(B\) is the batch, \(d_h\) the head dimension and \(b\) the bytes per number. With \(h = 32\), \(T = 8{,}192\) and BF16, \(M_S = 4.29\) GB per sequence per layer, and backward needs it again. FlashAttention fuses the whole attention into one kernel: it loads tiles of \(Q\), \(K\), \(V\) into shared memory, computes the softmax online (keeping a running maximum and running sum per row), and never writes \(S\) or \(P\) to HBM. In backward it recomputes the tiles instead of loading a stored \(P\). The FLOPs are about the same (a little more, because of recomputation), but the memory traffic is far lower and the memory is linear in \(T\). The FlashAttention paper reports a 3× speedup on GPT-2 at sequence length 1K, and 15% on BERT-large at length 512 against the MLPerf 1.1 record. [Certain] In PyTorch you get it from F.scaled_dot_product_attention, which picks a fused backend when it can. The algorithm itself is the topic of session 124.

Fused RMSNorm, RoPE and SwiGLU

These are the memory-bound chains of Figure 4. torch.compile fuses them automatically when there are no graph breaks. Libraries also ship hand-written Triton versions with a matching backward. Liger Kernel (from LinkedIn) provides fused RMSNorm, LayerNorm, RoPE, SwiGLU, GeGLU, cross-entropy and fused linear cross-entropy, and reports about 20% higher training throughput and 60% less GPU memory than the Hugging Face implementations of popular LLMs. [Certain]

Fused (chunked) cross-entropy

The last layer is often the biggest memory problem in training. The output projection turns every token’s hidden vector into \(V\) logits, one per vocabulary entry:

\[ M_{\text{logits}} = T \cdot V \cdot b \tag{50.6} \]

Llama 3 has \(V = 128{,}256\). [Certain] With 8 sequences of 8,192 tokens (\(T = 65{,}536\)) per GPU, the BF16 logits are \(65{,}536 \times 128{,}256 \times 2 = 16.8\) GB. Many implementations upcast logits to FP32 for a stable softmax: another 33.6 GB. And the gradient of the logits has the same size again. A single loss can need more memory than the model’s weights.

The fix uses one fact about cross-entropy. For one token with logits \(\mathbf{z}\) and target \(y\), the loss is \(\ell = \log \sum_j e^{z_j} - z_y\), and its gradient is

\[ \frac{\partial \ell}{\partial \mathbf{z}} = \softmax(\mathbf{z}) - \mathbf{e}_y \tag{50.7} \]

where \(\mathbf{e}_y\) is the one-hot vector of the target. So the gradient can be computed in the same pass as the loss, right after the softmax, without keeping the logits around. A fused linear cross-entropy kernel does this chunk by chunk: take a chunk of \(C\) tokens, compute their \(C \times V\) logits, the loss, and the gradient with respect to the hidden states and the output weight, then throw the logits away. Peak memory is \(C \cdot V\) instead of \(T \cdot V\). Cut Cross-Entropy goes further and also splits the vocabulary into blocks, merging partial results with the log-sum-exp rule:

\[ m = \max(m_a, m_b), \qquad \operatorname{LSE} = m + \log\!\big( s_a e^{m_a - m} + s_b e^{m_b - m} \big) \tag{50.8} \]

where block \(a\) has maximum logit \(m_a\) and sum \(s_a = \sum_{j \in a} e^{z_j - m_a}\), and likewise for block \(b\). This is the same online-softmax trick FlashAttention uses. For Gemma 2 (2B), the paper reports that the loss computation’s memory falls from 24 GB to 1 MB, and the classifier head’s total training memory from 28 GB to 1 GB. [Certain]

Algorithm 3: chunked (fused) linear cross-entropy, forward and backward in one pass
  1. input: hidden states \(H \in \R^{T \times d}\), output weight \(W \in \R^{V \times d}\), targets \(y\), chunk size \(C\)
  2. \(L \leftarrow 0\); \(\nabla H \leftarrow 0\); \(\nabla W \leftarrow 0\)
  3. for each chunk \(c\) of \(C\) rows: \(Z_c \leftarrow H_c W^\top\) // \(C \times V\) logits, computed in FP32
  4. \(\quad P_c \leftarrow \softmax(Z_c)\) row by row; \(L \mathrel{+}= \sum_t (\operatorname{LSE}(Z_{c,t}) - Z_{c,t,y_t})\)
  5. \(\quad G_c \leftarrow (P_c - \text{onehot}(y_c)) / T\) // formula (50.7)
  6. \(\quad \nabla H_c \leftarrow G_c W\); \(\;\nabla W \mathrel{+}= G_c^\top H_c\); free \(Z_c, P_c, G_c\)
  7. return \(L / T\), and save \(\nabla H, \nabla W\) for the backward call // logits never all in memory
Figure 5. Peak memory of the loss computation on one GPU. Naive: BF16 logits, an FP32 copy for the softmax, and the BF16 logit gradient, all \(T \times V\) (8 bytes per logit). Chunked: one FP32 chunk of \(C \times V\) logits plus its FP32 gradient (8 bytes per logit). The dashed line is 80 GB, the memory of one H100.
How to read Figure 5

Move the number of sequences: the naive bar grows linearly with tokens and crosses the whole GPU memory at about 10 sequences for Llama 3’s vocabulary. The chunked bar does not depend on the number of tokens at all, only on \(C\) and \(V\). Bigger chunks use more memory but fewer, larger matrix multiplications; a few thousand tokens per chunk is a common middle ground. [Likely]

Reading a profiler trace

All of the above are fixes. The profiler tells you which one you need. Two tools are standard:

  • PyTorch profiler (torch.profiler). Records CPU operations and CUDA kernels with their times. prof.key_averages().table() gives a sorted summary; prof.export_chrome_trace("trace.json") writes a timeline you open in Perfetto (ui.perfetto.dev) or chrome://tracing. A schedule(wait, warmup, active) skips the first, unusual steps. Wrap parts of your code in record_function("name") so they show up as labeled blocks.
  • Nsight Systems (nsys), from NVIDIA. Records the whole process with low overhead: CPU threads, CUDA API calls, kernels, memory copies, NCCL kernels, and your own NVTX ranges (torch.cuda.nvtx.range_push("forward")). Typical use: nsys profile -t cuda,nvtx,osrt -o step python train.py, then open the report in the Nsight Systems GUI.

A trace is a set of horizontal lanes, one per CPU thread and one per GPU stream, with time going right. Look at it in this order:

Algorithm 4: a checklist for reading a training trace
  1. Skip the first steps (compilation, allocator warmup); look at one steady step from start to end
  2. Measure GPU busy time: \(U_{\text{GPU}} = \sum\) kernel time \(/\) step time // formula (50.1); below about 80% means there is something to find
  3. Big idle block at the start of the step, CPU in the data loader \(\rightarrow\) input pipeline is too slow: more workers, prefetching
  4. Memcpy HtoD with the word “Pageable”, CPU blocked during it \(\rightarrow\) use pin_memory=True and .to(device, non_blocking=True)
  5. CPU in cudaStreamSynchronize or cudaMemcpy DtoH, then a GPU gap \(\rightarrow\) a hidden sync (.item(), printing, nonzero); remove or do it every \(k\) steps
  6. Many kernels of a few µs with gaps, CPU lane fully busy \(\rightarrow\) launch-bound: torch.compile, CUDA graphs, larger micro-batch
  7. Long element-wise and norm kernels between the matmuls \(\rightarrow\) memory-bound: fusion
  8. NCCL kernels with nothing running beside them on the compute stream \(\rightarrow\) exposed communication: check overlap (DDP buckets, FSDP prefetch)
  9. Only then look at the biggest kernels (matmuls, attention) and their efficiency

Figure 6 is a toy trace of one training step with all of these problems at once. Turn on the fixes one by one and watch the idle time disappear.

CPU workCPU blocked (waiting)GPU kernelscopy / NCCLGPU compute idle
Figure 6. A toy trace of one data-parallel training step. The durations are invented to show the patterns, not measured. Click any block to see what it is.
How to read Figure 6

With no fix, the GPU compute lane is idle for about half of the step: first while the CPU loads data, then during a blocking pageable copy, then in many small gaps between kernels, then while waiting for the all-reduce, and once more after the .item() sync drains the queue. Each fix removes one red region. Notice that torch.compile alone helps only a little at first, because most idle time is elsewhere. That is why you read the trace before choosing a fix.

To decide whether a fix is worth the work, use Amdahl’s law. If a part that takes a fraction \(p\) of the step becomes \(s\) times faster, the whole step becomes

\[ S_{\text{total}} = \frac{1}{(1 - p) + p / s} \tag{50.9} \]

times faster. If element-wise kernels take 20% of the step and fusion makes them 3× faster, the step gets \(1/(0.8 + 0.2/3) = 1.15\)× faster. Even infinitely fast element-wise kernels give only 1.25×. Spend your effort on the largest \(p\) in the trace.

Worked example: what Amdahl says about the Mask R-CNN result

The graphed backbone went from 31 ms to 6 ms, saving 25 ms, and the step became 1.70× faster. If nothing else changed, the old step time \(X\) satisfies \(X / (X - 25) = 1.70\), so \(X = 25 \times 1.70 / 0.70 = 60.7\) ms and the new step is about 35.7 ms. The backbone was about half of the step (\(p = 31/60.7 = 0.51\)), with \(s = 31/6 = 5.2\). This is our own back-calculation under that assumption, not a number from the blog post.

PyTorch code

The code below runs on a CPU in a few seconds (PyTorch 2.x; tested with 2.8). It checks that a compiled RMSNorm matches eager mode, counts graph breaks and recompiles, verifies that chunked cross-entropy gives the same loss and gradients as the full one, checks formula (50.7), and profiles a few steps.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# 1) RMSNorm written with plain ops: eager runs several kernels, torch.compile fuses them
def rmsnorm(x, w, eps=1e-5):
    r = torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps)    # formula (50.4)
    return x * r * w
x, w = torch.randn(64, 512), torch.ones(512)
rms_c = torch.compile(rmsnorm)
print("rmsnorm max diff:", (rms_c(x, w) - rmsnorm(x, w)).abs().max().item())   # about 5e-7

# 2) Find graph breaks before you trust the speed-up
def f(x):
    y = torch.sin(x) * 2
    if y.sum() > 0:          # branch on a tensor value -> graph break
        y = y + 1
    print("step done")       # print -> graph break
    return torch.cos(y)
torch._dynamo.reset()
ex = torch._dynamo.explain(f)(torch.ones(8))
print("graphs:", ex.graph_count, "breaks:", ex.graph_break_count)           # 3 graphs, 2 breaks

# 3) Shape changes: the first new shape recompiles, then Dynamo marks the dim dynamic
torch._dynamo.reset()
n_compiles = 0
def counting_backend(gm, example_inputs):
    global n_compiles; n_compiles += 1
    return gm.forward
g = torch.compile(lambda t: torch.relu(t) * 2, backend=counting_backend)
for L in [128, 256, 384, 512]:
    g(torch.randn(L))
print("compiles for 4 lengths:", n_compiles)                                # 2

# 4) Chunked cross-entropy: never hold all T x V logits; same loss, same gradients
T, d, V = 256, 64, 1000
h = torch.randn(T, d, requires_grad=True)
W = (0.05 * torch.randn(V, d)).requires_grad_(True)
y = torch.randint(0, V, (T,))
full = F.cross_entropy(h @ W.T, y)
gh, gW = torch.autograd.grad(full, (h, W))
def chunked_ce(h, W, y, chunk=64):
    total = h.new_zeros(())
    for s in range(0, h.shape[0], chunk):
        logits = h[s:s + chunk] @ W.T                    # only chunk x V logits at a time
        total = total + F.cross_entropy(logits.float(), y[s:s + chunk], reduction="sum")
    return total / h.shape[0]
ch = chunked_ce(h, W, y)
ch_gh, ch_gW = torch.autograd.grad(ch, (h, W))
print("loss diff:", abs(full.item() - ch.item()),
      "grad diff:", max((gh - ch_gh).abs().max().item(), (gW - ch_gW).abs().max().item()))

# 5) Formula (50.7): d loss / d logits = (softmax - one_hot) / T
z = torch.randn(4, 10, requires_grad=True); t = torch.tensor([1, 2, 3, 4])
gz, = torch.autograd.grad(F.cross_entropy(z, t), z)
print("softmax - onehot check:", (gz - (z.softmax(-1) - F.one_hot(t, 10)) / 4).abs().max().item())

# 6) Profile a few steps (CPU here; add ProfilerActivity.CUDA on a GPU)
from torch.profiler import profile, ProfilerActivity, record_function, schedule
model = torch.nn.Sequential(torch.nn.Linear(512, 2048), torch.nn.GELU(), torch.nn.Linear(2048, 512))
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
with profile(activities=[ProfilerActivity.CPU],
             schedule=schedule(wait=1, warmup=1, active=2)) as prof:
    for step in range(4):
        with record_function("train_step"):
            loss = model(torch.randn(32, 512)).pow(2).mean()
            loss.backward(); opt.step(); opt.zero_grad(set_to_none=True)
        prof.step()                                       # tell the schedule a step ended
print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=5))
# prof.export_chrome_trace("trace.json")  -> open in https://ui.perfetto.dev

Output: the RMSNorm difference is about \(5 \times 10^{-7}\); graphs: 3 breaks: 2; compiles for 4 lengths: 2; the chunked loss differs by about \(5 \times 10^{-7}\) and the gradients by about \(10^{-8}\); the softmax-minus-one-hot check is about \(10^{-8}\). The profiler table lists aten::mm and aten::addmm on top; its times vary from machine to machine.

On a GPU, the CUDA graph and reduce-overhead versions look like this (this part needs a CUDA GPU and was not run for this page; the capture pattern follows the PyTorch CUDA graphs blog post):

# needs a CUDA GPU
model = model.cuda(); opt = torch.optim.SGD(model.parameters(), lr=0.1)
static_x = torch.randn(32, 512, device="cuda"); static_y = torch.randn(32, 512, device="cuda")
s = torch.cuda.Stream(); s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):                        # warmup on a side stream
    for _ in range(3):
        opt.zero_grad(set_to_none=True)
        F.mse_loss(model(static_x), static_y).backward(); opt.step()
torch.cuda.current_stream().wait_stream(s)
graph = torch.cuda.CUDAGraph()
opt.zero_grad(set_to_none=True)
with torch.cuda.graph(graph):                     # capture: recorded, not run
    static_loss = F.mse_loss(model(static_x), static_y)
    static_loss.backward(); opt.step()
for xb, yb in batches:                           # same shapes every step
    static_x.copy_(xb); static_y.copy_(yb)       # new data into the same addresses
    graph.replay()                               # one launch: forward + backward + step

# or let the compiler do it
fast_model = torch.compile(model, mode="reduce-overhead")

Where this is used

SettingMain bottleneckTypical tools
Large LLM pretraining (big micro-batches, long sequences)Matmuls and attention; memory for activations and logits; exposed communicationFlashAttention, per-block torch.compile, fused or chunked cross-entropy, communication overlap (torchtitan, Megatron-LM) [Likely]
Fine-tuning on a few GPUs with large vocabulariesMemory of the logits and of activationsLiger Kernel or similar fused kernels; chunked cross-entropy
Small models or heavy parallelism (small per-GPU work)Launch overheadCUDA graphs, mode="reduce-overhead", fewer kernels
Video classification and action recognition trainingOften the input pipeline: decoding and resizing video frames on the CPU starves the GPUMore data-loader workers, GPU video decoding, pinned memory, pre-extracted frames or features [Likely]
Video retrieval and fingerprinting with contrastive lossesThe \(B \times B\) similarity logits with very large batches, the same memory problem as formula (50.6)Chunked or fused contrastive loss, gradient caching [Likely]
Key-moment detection and segmentation on clips of different lengthsRecompiles and dynamic shapesBucketing clip lengths, mark_dynamic, padding to a few fixed sizes
Video-RAG indexing (embedding millions of clips) and LLM decodingSmall, repeated inference steps: launch-boundCUDA graphs, torch.compile, static batch shapes (session 127)

Common mistakes

  • Timing without synchronizing. Kernel launches are asynchronous, so time.time() around a GPU call measures only the launch. Call torch.cuda.synchronize() before reading the clock, or use CUDA events, and skip the first (compiling) steps.
  • Including compile time in the benchmark. The first call can take minutes. Measure steady steps only.
  • Ignoring graph breaks and recompiles. The model “is compiled” but runs as dozens of small graphs or falls back to eager after 8 recompiles. Check with TORCH_LOGS="graph_breaks,recompiles".
  • A hidden sync every step. loss.item(), printing a tensor, if (x > 0).any():, or boolean indexing force the CPU to wait for the GPU and drain the launch queue. Log every \(k\) steps.
  • Reading CUDA graph outputs after the next replay. The static output tensors are overwritten by every replay. Copy what you need (.clone()) before replaying again.
  • Fusing the wrong thing. Writing a custom kernel for an operation that is 3% of the step gives at most 3% (Amdahl, formula 50.9). Read the trace first.
  • Profiling the first steps or too many steps. The first steps include compilation and memory allocation; a long trace is huge and slow to open. Use schedule(wait=…, warmup=…, active=2 or 3).
  • Pageable memory for input batches. Without pin_memory=True, the copy to the GPU goes through an extra CPU buffer and non_blocking=True cannot overlap it.

Interview questions

What does torch.compile do, step by step?basic
TorchDynamo reads the Python bytecode of the function and records the PyTorch operations into an FX graph, with guards on shapes, dtypes and Python values. AOTAutograd traces the backward pass too, giving joint forward and backward graphs and choosing what to save or recompute. TorchInductor fuses element-wise and reduction chains and generates Triton kernels (GPU) or C++ (CPU), calling cuBLAS or autotuned templates for big matmuls. The result is cached and reused while the guards hold. The PyTorch 2 paper reports 1.41× geometric-mean training speedup over eager on an A100 across 180+ models.
What is a graph break, what causes it and why does it hurt?core
Dynamo ends the current graph when it meets code it cannot trace: branching on a tensor value, .item() or .tolist(), print, unsupported libraries. It runs that code in Python and starts a new graph. Each break makes graphs smaller (less fusion), returns to Python (launch overhead), and may force a CPU-GPU sync. Find them with torch._dynamo.explain, TORCH_LOGS="graph_breaks" or fullgraph=True; remove them with torch.where, moving logging outside, and keeping values on the GPU.
Why does a compiled model recompile, and what happens after many recompiles?core
A guard failed: a new shape, dtype, device, or a changed Python int or float (these are treated as constants). With dynamic=None, the first changed size triggers one recompile with that dimension made symbolic, which then covers other sizes. After 8 entries (the default recompile limit) Dynamo stops compiling that function and runs it eagerly, so speed silently drops. Fix: stable shapes (packing, bucketing), mark_dynamic, and passing changing numbers as tensors.
When do CUDA graphs help, and what are their restrictions?core
They help when the step is launch-bound: many short kernels and the CPU cannot keep the GPU queue full, typical for small per-GPU batches, small models, decoding, and strongly scaled training (MLPerf Mask R-CNN gained 1.70×, BERT on 4,096 GPUs 1.12×). A graph replays the same kernels with the same arguments and memory addresses, so it needs static shapes, static input and output buffers, no CPU syncs or data-dependent control flow inside, and no per-step CPU logic unless it lives in GPU tensors. Captured graphs also hold a private memory pool.
FlashAttention does about the same FLOPs as standard attention. Why is it faster?core
Standard attention is memory-bound in its softmax part: it writes the \(T \times T\) scores to HBM, reads them for softmax, writes probabilities, reads them again. FlashAttention fuses everything into one kernel with tiles in shared memory and an online softmax, so the \(T^2\) matrices never go to HBM; backward recomputes them from \(Q, K, V\). Far fewer bytes moved, memory linear in \(T\). The paper reports 3× on GPT-2 at length 1K.
Why is the cross-entropy layer a memory problem for LLMs, and how is it fixed?core
Logits are \(T \times V\): with \(V = 128{,}256\) and 65,536 tokens per GPU, that is 16.8 GB in BF16, plus an FP32 copy and the gradient of the same size. Because \(\partial\ell/\partial z = \softmax(z) - e_y\), the gradient can be computed in the same pass as the loss. Fused linear cross-entropy processes chunks of tokens, computes loss and gradients for each, and discards the logits, so memory is \(C \times V\). Cut Cross-Entropy also blocks the vocabulary and merges with log-sum-exp; it reports 24 GB to 1 MB for the loss of Gemma 2 2B.
Your training job shows 40% GPU utilization. How do you find out why?core
Profile a few steady steps (torch.profiler with a schedule, or nsys with NVTX ranges). On the timeline, find where the GPU compute stream is idle and what the CPU is doing at that moment: data loader (more workers, prefetch), blocking pageable copies (pin memory, non_blocking), cudaStreamSynchronize from .item() (remove syncs), dense tiny kernels with a saturated CPU (launch-bound: compile, CUDA graphs, bigger micro-batch), or NCCL waits (overlap, bucket sizes). Then estimate each fix with Amdahl’s law and start with the biggest.
Estimate the time of an unfused vs fused RMSNorm for 16,384 tokens × 4,096 on an H100.advanced
\(N = 67.1\)M elements, BF16. Unfused: pow (read+write), mean (read), x·r (read+write), ·g (read+write) is about \(7N\) element accesses, 939.5 MB, at least 280 µs at 3.35 TB/s. Fused: read once, write once, \(2N\), 268.4 MB, at least 80 µs. The FLOPs are tiny (a few per element), so bandwidth decides. Ratio 3.5×.
What is the difference between mode="default", "reduce-overhead" and "max-autotune" in torch.compile?basic
Default balances compile time and speed with Inductor fusion. “reduce-overhead” adds CUDA graphs to cut Python and launch overhead, useful for small batches; it costs extra memory for the graph pools and needs static shapes. “max-autotune” benchmarks Triton or template matmul variants to pick the fastest and also enables CUDA graphs on GPU; it compiles much longer. “max-autotune-no-cudagraphs” autotunes without graphs.
Fusion saves memory in training, not only time. Explain.advanced
In eager mode, autograd stores the inputs each operation needs for its backward: a chain of five element-wise ops may store several full-size intermediates. A fused operator with its own backward saves only its inputs (and small per-token statistics such as the RMS) and recomputes the rest during backward, which is cheap for element-wise math. AOTAutograd’s partitioner makes the same save-or-recompute choice automatically in compiled graphs. FlashAttention is the extreme case: it never stores the \(T \times T\) probabilities.
Why can a faster GPU fail to speed up a training step?advanced
If the step is launch-bound, \(T_{\text{step}} \approx n t_L\) (formula 50.1) does not depend on GPU speed; a faster GPU only makes the idle gaps longer. The same holds when the GPU is starved by data loading, syncs or exposed communication. As GPUs get faster, more workloads become CPU- or input-bound, which is why CUDA graphs, fusion and asynchronous input pipelines matter more with each GPU generation.

Exercises

  1. A step launches 2,400 kernels, each 3 µs on the GPU, and each launch costs 6 µs. Estimate the step time and GPU busy fraction with formula (50.1). What do CUDA graphs give?
  2. Count the HBM traffic of “residual add + RMSNorm” unfused and fused (in units of \(N\) elements). What is the ratio?
  3. Llama 2 has \(V = 32{,}000\). How much memory do the BF16 logits of 65,536 tokens take? Compare with Llama 3.
  4. Chunked cross-entropy with \(C = 1{,}024\) tokens and \(V = 128{,}256\): how big is one FP32 chunk of logits?
  5. Attention, element-wise kernels and the rest take 30%, 25% and 45% of a step. Fusing the element-wise kernels makes them 2.5× faster. What is the overall speedup? And if attention also becomes 2× faster?
  6. Rewrite if x.sum() > 0: y = x * 2 else: y = x - 1 without a graph break.
Answerscheck
1. \(n t_L = 2{,}400 \times 6 = 14.4\) ms, \(\sum d = 7.2\) ms, so the step is about 14.4 ms and the GPU is busy 50%. With one graph launch the step is close to 7.2 ms: about 2×. 2. Unfused: add reads 2N, writes N (3N); RMSNorm 7N; total 10N. Fused: read x and the residual (2N), write the new residual and the normed output (2N): 4N. Ratio 2.5×. 3. \(65{,}536 \times 32{,}000 \times 2 = 4.19\) GB, versus 16.8 GB for Llama 3 (4× larger vocabulary, 4× the memory). 4. \(1{,}024 \times 128{,}256 \times 4 = 525\) MB. 5. \(1/(0.75 + 0.25/2.5)\) \(= 1/0.85 = 1.18\)×. With attention too: \(1/(0.45 + 0.15 + 0.10)\) \(= 1/0.70 = 1.43\)×. 6. y = torch.where(x.sum() > 0, x * 2, x - 1): both sides are computed and one is selected on the GPU, no Python branch.

Key takeaways

  • A step is launch-bound, memory-bound or starved. The step time is roughly the larger of CPU launch time and GPU work (50.1); a profiler trace tells you which case you are in.
  • torch.compile = Dynamo (bytecode to FX graph plus guards) + AOTAutograd (forward and backward together) + Inductor (fusion, Triton kernels). Graph breaks and recompiles silently remove the gain; count them.
  • CUDA graphs replay a captured sequence of kernels with one launch. They need static shapes and addresses and no syncs, and help most when per-GPU work is small (Mask R-CNN 1.70×, BERT 1.12× in MLPerf v1.0).
  • Fusion cuts HBM traffic of memory-bound chains (50.2, 50.3): 3.5× for RMSNorm. FlashAttention never writes the \(T^2\) scores (50.5). Fused cross-entropy uses \(\softmax - e_y\) (50.7) to avoid storing \(T \times V\) logits (50.6).
  • Read traces in order: steady step, GPU busy fraction, data loader, copies, syncs, launch gaps, element-wise time, exposed communication. Estimate every fix with Amdahl’s law (50.9) first.

Key formulas

\[ T_{\text{step}} \approx \max\Big(n t_L, \sum_i d_i\Big), \quad t \approx \max\Big(\tfrac{\text{FLOPs}}{P}, \tfrac{\text{bytes}}{B}\Big), \quad \frac{\partial \ell}{\partial \mathbf{z}} = \softmax(\mathbf{z}) - \mathbf{e}_y, \quad S = \frac{1}{(1-p) + p/s} \]

References for this session

  1. Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
  2. NVIDIA (docs). CUDA C++ Programming Guide
  3. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  4. Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
  5. PyTorch (docs). torch.compile API reference (modes, fullgraph, dynamic) and the recompilation guide
  6. Nguyen, Carilli, Eryilmaz et al. (NVIDIA and Meta) (2021). Accelerating PyTorch with CUDA Graphs (PyTorch blog)
  7. NVIDIA (docs). CUDA C++ Programming Guide
  8. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  9. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  10. Zhang & Sennrich (2019). Root Mean Square Layer Normalization
  11. Hsu et al. (2024). Liger Kernel: Efficient Triton Kernels for LLM Training
  12. Wijmans et al. (2024). Cut Your Losses in Large-Vocabulary Language Models
  13. PyTorch (docs). torch.profiler: PyTorch Profiler API reference
  14. NVIDIA (docs). Nsight Systems User Guide
  15. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters