Making training fast: compile, CUDA graphs, fused kernels, profiling
A training step is thousands of GPU kernels launched from Python. Speed is lost in three places: the CPU cannot launch kernels fast enough, small kernels move the same data in and out of memory again and again, and the GPU waits for data, copies or synchronizations. This session shows the tools that fix each one (torch.compile, CUDA graphs, fused kernels for attention, RMSNorm and cross-entropy) and how to find the problem first by reading a profiler trace.
What you will learn
- Where training time goes: launch-bound, memory-bound and compute-bound work, and why the step time is roughly the larger of the CPU time and the GPU time
- How torch.compile works: TorchDynamo captures an FX graph with guards, AOTAutograd builds the backward graph, TorchInductor fuses operations into Triton kernels
- Graph breaks and recompiles: what causes them, how to count them, and how to remove them
- CUDA graphs: capture once, replay with one launch; the rules (static shapes, static addresses, no CPU synchronization) and mode="reduce-overhead"
- Fused kernels and why they help: FlashAttention, RMSNorm and SwiGLU fusion, and fused or chunked cross-entropy that never stores the full logits
- How to read a PyTorch profiler or Nsight Systems trace: idle gaps, data-loader stalls, pageable memory copies, hidden synchronizations and exposed communication
- How to estimate the benefit of a fix before you write it, with bytes, launch counts and Amdahl's law
Why this matters
Two teams train the same model on the same GPUs. One reaches 45% of the GPU’s peak arithmetic, the other 20%. The second team pays more than twice as much for the same model. The difference is rarely the model. It is the code around it: how kernels are launched, how often data is moved, and where the GPU sits idle.
In session 42 you learned the roofline: an operation is either compute-bound (limited by tensor-core FLOPs) or memory-bound (limited by HBM bytes per second). Real training adds a third limit that the roofline does not show: the CPU. PyTorch runs your Python code on the CPU, and every operation becomes one or more kernels (small GPU programs) that the CPU must launch one at a time. If launching takes longer than running, the GPU waits.
So there are three ways a training step is slow, and each has a standard fix:
| Problem | What you see in a trace | Standard fix |
|---|---|---|
| Launch-bound: the CPU cannot launch kernels fast enough | Many tiny kernels with gaps between them; the CPU thread is 100% busy | CUDA graphs; fewer, bigger kernels (torch.compile); bigger batches |
| Memory-bound: kernels read and write the same tensor again and again | Many element-wise kernels (add, mul, norm, activation) that together take a large share of the step | Kernel fusion: torch.compile, or hand-written fused kernels (FlashAttention, fused RMSNorm, fused cross-entropy) |
| Starved: the GPU waits for something else | Large idle blocks on the GPU: data loading, memory copies, .item() synchronizations, communication | Fix the input pipeline, pinned memory, remove syncs, overlap communication |
The rule of this session: measure first, then fix. A profiler trace tells you which of the three problems you have. Fixing the wrong one wastes days. This session builds the mental models, then shows the tools, then teaches you to read a trace. The previous session, session 49, spread one model over many GPUs. This one makes each GPU do its share faster. The next session, session 51, measures the result with MFU.
A model of the training step: CPU and GPU run side by side
A CUDA kernel launch is asynchronous. The CPU puts the kernel into a queue (a CUDA stream) and returns at once, without waiting for the GPU. The GPU takes kernels from the queue and runs them in order. So the CPU and the GPU work at the same time, like two people on a production line.
Let \(n\) be the number of kernels in one step, \(t_L\) the CPU time to launch one kernel (Python, the PyTorch dispatcher and the CUDA driver together), and \(d_i\) the GPU time of kernel \(i\). As long as the queue never fills up, the step time is about the larger of the two workloads:
- Read the symbols
- \(n\, t_L\)
- the total CPU time spent launching kernels in one step
- \(\sum_i d_i\)
- the total time the GPU is really computing
- \(U_{\text{GPU}}\)
- the fraction of the step the GPU is busy; 1 means no idle gaps
This is a simplification: it ignores that a kernel can only start after its launch, and that real kernels have different sizes. Figure 3 simulates the exact version. But it shows the key fact: when the kernels are small (\(d_i \lt t_L\)), the GPU time does not matter; the step is limited by launches. A kernel launch from eager PyTorch costs on the order of a few microseconds to about ten microseconds of CPU time. [Likely] A large matrix multiplication runs for hundreds of microseconds or more, so big models with big batches are rarely launch-bound. Small models, small batches per GPU (common with heavy parallelism), and long chains of element-wise operations are.
Assume a step launches \(n = 1{,}000\) kernels, each runs for \(d = 4\ \mu s\) on the GPU, and each launch costs \(t_L = 10\ \mu s\) on the CPU (illustrative numbers). Then \(n t_L = 10\) ms and \(\sum d_i = 4\) ms. By (50.1), the step takes about 10 ms and the GPU is busy \(4/10 = 40\%\) of the time.
A GPU twice as fast changes nothing: still 10 ms. Removing the launch cost (a CUDA graph, next sections) gives about 4 ms, a 2.5× speedup. Fusing pairs of kernels to halve \(n\) gives \(\max(5, 4) = 5\) ms, 2×, and also less memory traffic.
torch.compile: from Python to fused kernels
In eager mode (normal PyTorch), each line of Python runs as soon as it is reached, and each operation launches its own kernel. This is easy to debug but leaves speed on the table: PyTorch never sees more than one operation at a time, so it cannot combine them. torch.compile (PyTorch 2) looks at a whole region of code at once, turns it into a graph of operations, and generates faster code for the graph. The PyTorch 2 paper reports a geometric-mean speedup of 2.27× for inference and 1.41× for training over eager mode, on an NVIDIA A100 across more than 180 real models. [Certain] Your own speedup can be anywhere from none to several times; measure it.
It has three parts (Figure 1):
- TorchDynamo (graph capture). It hooks into CPython’s frame evaluation, reads the bytecode of your function before it runs, and records every PyTorch operation into an FX graph (a simple list of operations and their inputs). Plain Python (loops over a fixed range, if-statements on Python values, attribute lookups) is run during tracing and disappears from the graph. Dynamo also writes guards: cheap checks on the inputs (shapes, dtypes, devices, values of Python numbers) that must be true for the compiled code to be reused.
- AOTAutograd (ahead-of-time autograd). It traces the backward pass of the captured graph too, so the compiler sees the forward and the backward together. It then decides which intermediate values to save for backward and which to recompute, trading cheap recomputation of element-wise results for memory.
- TorchInductor (the code generator, the default backend). It groups operations, fuses chains of element-wise and reduction operations into single kernels, and writes them in Triton (a Python-like language for GPU kernels) on GPUs, or C++ on CPUs. Large matrix multiplications usually still call cuBLAS, or Triton templates when you ask for autotuning.
The first call of a compiled function is slow: it traces and compiles, which can take seconds to minutes for a large model. Later calls check the guards (microseconds) and run the cached kernels. So torch.compile pays off only when the same code runs many times with the same kind of inputs, which is exactly what a training loop does.
- input: function \(f\), arguments \(a\), cache of (guards, compiled code) pairs for \(f\)
- for each cached entry: if all its guards pass on \(a\): return its compiled code run on \(a\) // fast path
- if the number of entries has reached the recompile limit (8 by default): run \(f\) eagerly and return
- Dynamo traces the bytecode of \(f\) on \(a\), recording PyTorch operations into an FX graph and writing guards
- if it meets something it cannot trace: end the graph here (graph break), run that piece in plain Python, start a new graph after it
- AOTAutograd builds the joint forward and backward graph; Inductor fuses it and generates kernels
- store (guards, compiled code) in the cache; run it on \(a\)
Graph breaks and recompiles
Two things quietly destroy the speedup of torch.compile.
A graph break happens when Dynamo meets code it cannot put into a graph. It then compiles the part before, runs the problem in normal Python, and starts a new graph after it. Each break means: two smaller graphs (less to fuse), a return to Python (launch overhead again), and sometimes a CPU-GPU synchronization. Typical causes:
- Branching on a tensor value:
if loss > 0:orif mask.any():. The value is on the GPU and is not known during tracing. - Reading a value to the CPU:
.item(),.tolist(),.numpy(). By default.item()is a graph break (the settingcapture_scalar_outputscan change this). - Calls Dynamo cannot see into:
print, logging, some C extensions and third-party libraries.
A recompile happens when a guard fails: the input shape, dtype or a Python number changed. Dynamo then compiles again for the new case. Integer and float arguments are treated as constants by default, so passing a changing Python float (for example the learning rate, or a step counter) into a compiled function recompiles every time. For shapes, the default (dynamic=None) is: compile the first call for the exact shape; if a later call has a different size, recompile once with that dimension made dynamic (symbolic), which then works for all sizes. [Certain] After 8 compiles of the same function (the default recompile_limit), Dynamo gives up and runs it eagerly. [Certain]
Figure 2 lets you add these problems to a small forward function and see how many graphs and compiles you get. The counts follow what torch._dynamo.explain and a counting backend report in PyTorch 2.8 for the same patterns (checked by running the code at the end of this session).
How to read Figure 2
With no box ticked, the whole block is one graph, compiled once. Each break you add cuts the code: the blue boxes are separate graphs that Inductor optimizes separately, and each orange cut is a return to Python. Ticking “sequence length changes” makes each graph compile twice: once for length 128 and once more, with a dynamic length, at length 256; lengths 384 and 512 reuse the dynamic version. With fullgraph=True, any break becomes an error at the first call, which is the fastest way to find them.
How to remove the common breaks:
- Replace data-dependent Python branches with tensor operations:
torch.where(cond, a, b)computes both sides and selects, with no break. - Move
.item(), printing and logging out of the compiled region. Log a detached tensor every \(k\) steps, outside the model. - Keep shapes stable: pack sequences to a fixed length (session 18), or bucket them into a few fixed lengths, or mark the dimension dynamic from the start with
torch._dynamo.mark_dynamic(x, 1). - Pass changing numbers as 0-dimensional tensors, not as Python floats.
- For large models, compile each transformer block (
block.compile()ortorch.compile(block)per layer) instead of the whole model. All blocks have the same code, so the compiled code is reused and compile time drops a lot; this is what torchtitan does. [Likely]
CUDA graphs: one launch for the whole step
torch.compile reduces the number of kernels. A CUDA graph removes the launch cost of the kernels that remain. The idea: run the step once in capture mode. Kernels issued to a capturing stream do not run; they are recorded, with all their arguments, into a graph. After that, one call (cudaGraphLaunch) replays the whole recorded sequence. The replay skips Python, the PyTorch dispatcher and most of the driver work for every kernel. [Certain]
The price is flexibility. A replay runs the same kernels with the same arguments, and for pointer arguments that means the same memory addresses. [Certain] So:
- Static shapes. A new batch shape needs a new capture.
- Static memory. Inputs must be copied into the same “static” input tensors before each replay; outputs appear in the same static output tensors. PyTorch gives each graph a private memory pool for its intermediate tensors.
- No CPU synchronization inside. No
.item(), no printing of tensor values, no data-dependent Python branches, no operations whose output size depends on the data (such asnonzeroor boolean indexing). - No CPU-side logic per step inside the graph. Everything the CPU decides (for example a Python learning-rate value) is frozen at capture time, unless it lives in a GPU tensor that you update in place.
- input: model, optimizer, static input tensor \(X_s\), static target tensor \(Y_s\)
- warm up: run 3 normal steps on a side stream // lets lazy initialization and the allocator settle
- set gradients to None, then begin capture on the stream
- \(\hat Y_s \leftarrow\) model(\(X_s\)); \(L_s \leftarrow\) loss(\(\hat Y_s, Y_s\)); backward(\(L_s\)); optimizer step // recorded, not run
- end capture \(\rightarrow\) graph \(G\)
- for each new batch \((X, Y)\): copy \(X \to X_s\), \(Y \to Y_s\) (in place); replay \(G\) // one launch; gradients are refilled in place, no zero_grad needed
- read the loss from \(L_s\) only when you need it (reading it is a sync)
How much does it help? NVIDIA’s MLPerf v1.0 training submissions in PyTorch used CUDA graphs. For Mask R-CNN on 272 GPUs (batch 1 per GPU), the graphed backbone ran in 6 ms instead of 31 ms, and the whole step was about 1.70× faster. For BERT on 4,096 GPUs, the gain was 1.12×. [Certain] The pattern is clear: the gain is largest when per-GPU work is small, which is exactly what strong scaling to many GPUs produces. For a big model with a big per-GPU batch, the GPU is already the bottleneck and a CUDA graph gives little.
You rarely capture graphs by hand. torch.compile(model, mode="reduce-overhead") uses CUDA graphs for the compiled regions; mode="max-autotune" also enables them on GPU, and "max-autotune-no-cudagraphs" autotunes without them. [Certain] For inference with a KV cache, CUDA graphs are standard for the decode step, because each step is small (session 127).
How to read Figure 3
Start at batch scale 1 and launch cost 8 µs. The red marks under the GPU lane are idle time: the GPU has finished and waits for the next launch. Switch to the CUDA graph: the kernels pack tightly and the step is about twice as fast. Now raise the batch scale to 4 or more. The kernels become longer than the launches, the eager timeline also packs tightly, and the CUDA graph gains almost nothing. That is formula (50.1): once \(\sum d_i\) is larger than \(n t_L\), launches are hidden.
Kernel fusion: stop moving the same bytes
Most operations in a transformer that are not matrix multiplications are element-wise or small reductions: residual adds, normalization, activation functions, dropout, scaling, masking, softmax. Each one does a handful of FLOPs per element but must read its inputs from HBM and write its output back. They are deeply memory-bound. By the roofline, their time is set by bytes, not FLOPs:
Fusion merges a chain of such operations into one kernel. The fused kernel reads the inputs once, keeps the intermediate values in registers or shared memory, and writes only the final output. The FLOPs are the same; the bytes drop. For a chain of memory-bound kernels:
Take RMSNorm (session 22), used in Llama and most current LLMs:
- Read the symbols
- \(\mathbf{x} \in \R^d\)
- one token’s hidden vector (\(d = 4{,}096\) in Llama 3 8B)
- \(\epsilon\)
- a small constant for safety (\(10^{-5}\) in Llama 3 8B)
- \(\mathbf{g}\)
- the learned gain vector; \(\odot\) is element-wise multiplication
Written with plain PyTorch operations, x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * g launches several kernels: pow reads \(x\) and writes \(x^2\); mean reads \(x^2\); the small per-token steps (add \(\epsilon\), rsqrt) touch only one number per token; x * r reads \(x\) and writes a new tensor; * g reads that and writes the output. Counting only the big tensors of \(N = T \cdot d\) elements, that is about \(7N\) element reads and writes. A fused kernel reads \(x\) once and writes \(y\) once: \(2N\). So fusion moves 3.5× fewer bytes.
A micro-batch of 4 sequences of 4,096 tokens gives \(T = 16{,}384\) tokens; with \(d = 4{,}096\), \(N = 67{,}108{,}864\) elements, 2 bytes each in BF16.
Unfused: \(7 \times N \times 2 = 939.5\) MB moved. At the H100’s 3.35 TB/s, that is at least 280 µs. Fused: \(2 \times N \times 2 = 268.4\) MB, at least 80 µs. One RMSNorm saves 200 µs. A Llama-style model has two RMSNorms per layer, and the backward pass has its own memory-bound kernels, so across 32 layers the savings add up to many milliseconds per step. Real eager code often moves even more, because it upcasts to FP32 inside the norm.
Figure 4 does the same count for three common chains. Fusion helps most when a long chain of cheap operations works on the same tensor.
How to read Figure 4
Each bar is one kernel; its length is the bytes it reads (blue) and writes (orange). The time in the readout is the lower bound bytes ÷ bandwidth from formula (50.2). The ratio does not depend on the sizes, only on the chain: 3.5× for RMSNorm, 2.5× for add + RMSNorm, 1.67× for SwiGLU. The absolute savings grow with tokens × hidden size.
Fusion has a second benefit in training: memory. Each unfused intermediate that autograd needs for the backward pass is stored until the backward reaches it. A fused kernel with its own backward can save only what it needs (for RMSNorm, the input and one number per token) and recompute the rest. That is the same idea as activation checkpointing (session 39), done inside one kernel.
Three fused kernels every LLM trainer uses
Fused attention (FlashAttention)
Plain attention computes the score matrix \(S = QK^\top\), writes it to HBM, reads it back for the softmax, writes \(P\), and reads it again for \(PV\). For a sequence of \(T\) tokens and \(h\) heads the score matrix alone is
where \(B\) is the batch, \(d_h\) the head dimension and \(b\) the bytes per number. With \(h = 32\), \(T = 8{,}192\) and BF16, \(M_S = 4.29\) GB per sequence per layer, and backward needs it again. FlashAttention fuses the whole attention into one kernel: it loads tiles of \(Q\), \(K\), \(V\) into shared memory, computes the softmax online (keeping a running maximum and running sum per row), and never writes \(S\) or \(P\) to HBM. In backward it recomputes the tiles instead of loading a stored \(P\). The FLOPs are about the same (a little more, because of recomputation), but the memory traffic is far lower and the memory is linear in \(T\). The FlashAttention paper reports a 3× speedup on GPT-2 at sequence length 1K, and 15% on BERT-large at length 512 against the MLPerf 1.1 record. [Certain] In PyTorch you get it from F.scaled_dot_product_attention, which picks a fused backend when it can. The algorithm itself is the topic of session 124.
Fused RMSNorm, RoPE and SwiGLU
These are the memory-bound chains of Figure 4. torch.compile fuses them automatically when there are no graph breaks. Libraries also ship hand-written Triton versions with a matching backward. Liger Kernel (from LinkedIn) provides fused RMSNorm, LayerNorm, RoPE, SwiGLU, GeGLU, cross-entropy and fused linear cross-entropy, and reports about 20% higher training throughput and 60% less GPU memory than the Hugging Face implementations of popular LLMs. [Certain]
Fused (chunked) cross-entropy
The last layer is often the biggest memory problem in training. The output projection turns every token’s hidden vector into \(V\) logits, one per vocabulary entry:
Llama 3 has \(V = 128{,}256\). [Certain] With 8 sequences of 8,192 tokens (\(T = 65{,}536\)) per GPU, the BF16 logits are \(65{,}536 \times 128{,}256 \times 2 = 16.8\) GB. Many implementations upcast logits to FP32 for a stable softmax: another 33.6 GB. And the gradient of the logits has the same size again. A single loss can need more memory than the model’s weights.
The fix uses one fact about cross-entropy. For one token with logits \(\mathbf{z}\) and target \(y\), the loss is \(\ell = \log \sum_j e^{z_j} - z_y\), and its gradient is
where \(\mathbf{e}_y\) is the one-hot vector of the target. So the gradient can be computed in the same pass as the loss, right after the softmax, without keeping the logits around. A fused linear cross-entropy kernel does this chunk by chunk: take a chunk of \(C\) tokens, compute their \(C \times V\) logits, the loss, and the gradient with respect to the hidden states and the output weight, then throw the logits away. Peak memory is \(C \cdot V\) instead of \(T \cdot V\). Cut Cross-Entropy goes further and also splits the vocabulary into blocks, merging partial results with the log-sum-exp rule:
where block \(a\) has maximum logit \(m_a\) and sum \(s_a = \sum_{j \in a} e^{z_j - m_a}\), and likewise for block \(b\). This is the same online-softmax trick FlashAttention uses. For Gemma 2 (2B), the paper reports that the loss computation’s memory falls from 24 GB to 1 MB, and the classifier head’s total training memory from 28 GB to 1 GB. [Certain]
- input: hidden states \(H \in \R^{T \times d}\), output weight \(W \in \R^{V \times d}\), targets \(y\), chunk size \(C\)
- \(L \leftarrow 0\); \(\nabla H \leftarrow 0\); \(\nabla W \leftarrow 0\)
- for each chunk \(c\) of \(C\) rows: \(Z_c \leftarrow H_c W^\top\) // \(C \times V\) logits, computed in FP32
- \(\quad P_c \leftarrow \softmax(Z_c)\) row by row; \(L \mathrel{+}= \sum_t (\operatorname{LSE}(Z_{c,t}) - Z_{c,t,y_t})\)
- \(\quad G_c \leftarrow (P_c - \text{onehot}(y_c)) / T\) // formula (50.7)
- \(\quad \nabla H_c \leftarrow G_c W\); \(\;\nabla W \mathrel{+}= G_c^\top H_c\); free \(Z_c, P_c, G_c\)
- return \(L / T\), and save \(\nabla H, \nabla W\) for the backward call // logits never all in memory
How to read Figure 5
Move the number of sequences: the naive bar grows linearly with tokens and crosses the whole GPU memory at about 10 sequences for Llama 3’s vocabulary. The chunked bar does not depend on the number of tokens at all, only on \(C\) and \(V\). Bigger chunks use more memory but fewer, larger matrix multiplications; a few thousand tokens per chunk is a common middle ground. [Likely]
Reading a profiler trace
All of the above are fixes. The profiler tells you which one you need. Two tools are standard:
- PyTorch profiler (
torch.profiler). Records CPU operations and CUDA kernels with their times.prof.key_averages().table()gives a sorted summary;prof.export_chrome_trace("trace.json")writes a timeline you open in Perfetto (ui.perfetto.dev) or chrome://tracing. Aschedule(wait, warmup, active)skips the first, unusual steps. Wrap parts of your code inrecord_function("name")so they show up as labeled blocks. - Nsight Systems (
nsys), from NVIDIA. Records the whole process with low overhead: CPU threads, CUDA API calls, kernels, memory copies, NCCL kernels, and your own NVTX ranges (torch.cuda.nvtx.range_push("forward")). Typical use:nsys profile -t cuda,nvtx,osrt -o step python train.py, then open the report in the Nsight Systems GUI.
A trace is a set of horizontal lanes, one per CPU thread and one per GPU stream, with time going right. Look at it in this order:
- Skip the first steps (compilation, allocator warmup); look at one steady step from start to end
- Measure GPU busy time: \(U_{\text{GPU}} = \sum\) kernel time \(/\) step time // formula (50.1); below about 80% means there is something to find
- Big idle block at the start of the step, CPU in the data loader \(\rightarrow\) input pipeline is too slow: more workers, prefetching
Memcpy HtoDwith the word “Pageable”, CPU blocked during it \(\rightarrow\) usepin_memory=Trueand.to(device, non_blocking=True)- CPU in
cudaStreamSynchronizeorcudaMemcpy DtoH, then a GPU gap \(\rightarrow\) a hidden sync (.item(), printing,nonzero); remove or do it every \(k\) steps - Many kernels of a few µs with gaps, CPU lane fully busy \(\rightarrow\) launch-bound: torch.compile, CUDA graphs, larger micro-batch
- Long element-wise and norm kernels between the matmuls \(\rightarrow\) memory-bound: fusion
- NCCL kernels with nothing running beside them on the compute stream \(\rightarrow\) exposed communication: check overlap (DDP buckets, FSDP prefetch)
- Only then look at the biggest kernels (matmuls, attention) and their efficiency
Figure 6 is a toy trace of one training step with all of these problems at once. Turn on the fixes one by one and watch the idle time disappear.
How to read Figure 6
With no fix, the GPU compute lane is idle for about half of the step: first while the CPU loads data, then during a blocking pageable copy, then in many small gaps between kernels, then while waiting for the all-reduce, and once more after the .item() sync drains the queue. Each fix removes one red region. Notice that torch.compile alone helps only a little at first, because most idle time is elsewhere. That is why you read the trace before choosing a fix.
To decide whether a fix is worth the work, use Amdahl’s law. If a part that takes a fraction \(p\) of the step becomes \(s\) times faster, the whole step becomes
times faster. If element-wise kernels take 20% of the step and fusion makes them 3× faster, the step gets \(1/(0.8 + 0.2/3) = 1.15\)× faster. Even infinitely fast element-wise kernels give only 1.25×. Spend your effort on the largest \(p\) in the trace.
The graphed backbone went from 31 ms to 6 ms, saving 25 ms, and the step became 1.70× faster. If nothing else changed, the old step time \(X\) satisfies \(X / (X - 25) = 1.70\), so \(X = 25 \times 1.70 / 0.70 = 60.7\) ms and the new step is about 35.7 ms. The backbone was about half of the step (\(p = 31/60.7 = 0.51\)), with \(s = 31/6 = 5.2\). This is our own back-calculation under that assumption, not a number from the blog post.
PyTorch code
The code below runs on a CPU in a few seconds (PyTorch 2.x; tested with 2.8). It checks that a compiled RMSNorm matches eager mode, counts graph breaks and recompiles, verifies that chunked cross-entropy gives the same loss and gradients as the full one, checks formula (50.7), and profiles a few steps.
import torch, torch.nn.functional as F torch.manual_seed(0) # 1) RMSNorm written with plain ops: eager runs several kernels, torch.compile fuses them def rmsnorm(x, w, eps=1e-5): r = torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) # formula (50.4) return x * r * w x, w = torch.randn(64, 512), torch.ones(512) rms_c = torch.compile(rmsnorm) print("rmsnorm max diff:", (rms_c(x, w) - rmsnorm(x, w)).abs().max().item()) # about 5e-7 # 2) Find graph breaks before you trust the speed-up def f(x): y = torch.sin(x) * 2 if y.sum() > 0: # branch on a tensor value -> graph break y = y + 1 print("step done") # print -> graph break return torch.cos(y) torch._dynamo.reset() ex = torch._dynamo.explain(f)(torch.ones(8)) print("graphs:", ex.graph_count, "breaks:", ex.graph_break_count) # 3 graphs, 2 breaks # 3) Shape changes: the first new shape recompiles, then Dynamo marks the dim dynamic torch._dynamo.reset() n_compiles = 0 def counting_backend(gm, example_inputs): global n_compiles; n_compiles += 1 return gm.forward g = torch.compile(lambda t: torch.relu(t) * 2, backend=counting_backend) for L in [128, 256, 384, 512]: g(torch.randn(L)) print("compiles for 4 lengths:", n_compiles) # 2 # 4) Chunked cross-entropy: never hold all T x V logits; same loss, same gradients T, d, V = 256, 64, 1000 h = torch.randn(T, d, requires_grad=True) W = (0.05 * torch.randn(V, d)).requires_grad_(True) y = torch.randint(0, V, (T,)) full = F.cross_entropy(h @ W.T, y) gh, gW = torch.autograd.grad(full, (h, W)) def chunked_ce(h, W, y, chunk=64): total = h.new_zeros(()) for s in range(0, h.shape[0], chunk): logits = h[s:s + chunk] @ W.T # only chunk x V logits at a time total = total + F.cross_entropy(logits.float(), y[s:s + chunk], reduction="sum") return total / h.shape[0] ch = chunked_ce(h, W, y) ch_gh, ch_gW = torch.autograd.grad(ch, (h, W)) print("loss diff:", abs(full.item() - ch.item()), "grad diff:", max((gh - ch_gh).abs().max().item(), (gW - ch_gW).abs().max().item())) # 5) Formula (50.7): d loss / d logits = (softmax - one_hot) / T z = torch.randn(4, 10, requires_grad=True); t = torch.tensor([1, 2, 3, 4]) gz, = torch.autograd.grad(F.cross_entropy(z, t), z) print("softmax - onehot check:", (gz - (z.softmax(-1) - F.one_hot(t, 10)) / 4).abs().max().item()) # 6) Profile a few steps (CPU here; add ProfilerActivity.CUDA on a GPU) from torch.profiler import profile, ProfilerActivity, record_function, schedule model = torch.nn.Sequential(torch.nn.Linear(512, 2048), torch.nn.GELU(), torch.nn.Linear(2048, 512)) opt = torch.optim.AdamW(model.parameters(), lr=1e-3) with profile(activities=[ProfilerActivity.CPU], schedule=schedule(wait=1, warmup=1, active=2)) as prof: for step in range(4): with record_function("train_step"): loss = model(torch.randn(32, 512)).pow(2).mean() loss.backward(); opt.step(); opt.zero_grad(set_to_none=True) prof.step() # tell the schedule a step ended print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=5)) # prof.export_chrome_trace("trace.json") -> open in https://ui.perfetto.dev
Output: the RMSNorm difference is about \(5 \times 10^{-7}\); graphs: 3 breaks: 2; compiles for 4 lengths: 2; the chunked loss differs by about \(5 \times 10^{-7}\) and the gradients by about \(10^{-8}\); the softmax-minus-one-hot check is about \(10^{-8}\). The profiler table lists aten::mm and aten::addmm on top; its times vary from machine to machine.
On a GPU, the CUDA graph and reduce-overhead versions look like this (this part needs a CUDA GPU and was not run for this page; the capture pattern follows the PyTorch CUDA graphs blog post):
# needs a CUDA GPU model = model.cuda(); opt = torch.optim.SGD(model.parameters(), lr=0.1) static_x = torch.randn(32, 512, device="cuda"); static_y = torch.randn(32, 512, device="cuda") s = torch.cuda.Stream(); s.wait_stream(torch.cuda.current_stream()) with torch.cuda.stream(s): # warmup on a side stream for _ in range(3): opt.zero_grad(set_to_none=True) F.mse_loss(model(static_x), static_y).backward(); opt.step() torch.cuda.current_stream().wait_stream(s) graph = torch.cuda.CUDAGraph() opt.zero_grad(set_to_none=True) with torch.cuda.graph(graph): # capture: recorded, not run static_loss = F.mse_loss(model(static_x), static_y) static_loss.backward(); opt.step() for xb, yb in batches: # same shapes every step static_x.copy_(xb); static_y.copy_(yb) # new data into the same addresses graph.replay() # one launch: forward + backward + step # or let the compiler do it fast_model = torch.compile(model, mode="reduce-overhead")
Where this is used
| Setting | Main bottleneck | Typical tools |
|---|---|---|
| Large LLM pretraining (big micro-batches, long sequences) | Matmuls and attention; memory for activations and logits; exposed communication | FlashAttention, per-block torch.compile, fused or chunked cross-entropy, communication overlap (torchtitan, Megatron-LM) [Likely] |
| Fine-tuning on a few GPUs with large vocabularies | Memory of the logits and of activations | Liger Kernel or similar fused kernels; chunked cross-entropy |
| Small models or heavy parallelism (small per-GPU work) | Launch overhead | CUDA graphs, mode="reduce-overhead", fewer kernels |
| Video classification and action recognition training | Often the input pipeline: decoding and resizing video frames on the CPU starves the GPU | More data-loader workers, GPU video decoding, pinned memory, pre-extracted frames or features [Likely] |
| Video retrieval and fingerprinting with contrastive losses | The \(B \times B\) similarity logits with very large batches, the same memory problem as formula (50.6) | Chunked or fused contrastive loss, gradient caching [Likely] |
| Key-moment detection and segmentation on clips of different lengths | Recompiles and dynamic shapes | Bucketing clip lengths, mark_dynamic, padding to a few fixed sizes |
| Video-RAG indexing (embedding millions of clips) and LLM decoding | Small, repeated inference steps: launch-bound | CUDA graphs, torch.compile, static batch shapes (session 127) |
Common mistakes
- Timing without synchronizing. Kernel launches are asynchronous, so
time.time()around a GPU call measures only the launch. Calltorch.cuda.synchronize()before reading the clock, or use CUDA events, and skip the first (compiling) steps. - Including compile time in the benchmark. The first call can take minutes. Measure steady steps only.
- Ignoring graph breaks and recompiles. The model “is compiled” but runs as dozens of small graphs or falls back to eager after 8 recompiles. Check with
TORCH_LOGS="graph_breaks,recompiles". - A hidden sync every step.
loss.item(), printing a tensor,if (x > 0).any():, or boolean indexing force the CPU to wait for the GPU and drain the launch queue. Log every \(k\) steps. - Reading CUDA graph outputs after the next replay. The static output tensors are overwritten by every replay. Copy what you need (
.clone()) before replaying again. - Fusing the wrong thing. Writing a custom kernel for an operation that is 3% of the step gives at most 3% (Amdahl, formula 50.9). Read the trace first.
- Profiling the first steps or too many steps. The first steps include compilation and memory allocation; a long trace is huge and slow to open. Use
schedule(wait=…, warmup=…, active=2 or 3). - Pageable memory for input batches. Without
pin_memory=True, the copy to the GPU goes through an extra CPU buffer andnon_blocking=Truecannot overlap it.
Interview questions
What does torch.compile do, step by step?basic
What is a graph break, what causes it and why does it hurt?core
.item() or .tolist(), print, unsupported libraries. It runs that code in Python and starts a new graph. Each break makes graphs smaller (less fusion), returns to Python (launch overhead), and may force a CPU-GPU sync. Find them with torch._dynamo.explain, TORCH_LOGS="graph_breaks" or fullgraph=True; remove them with torch.where, moving logging outside, and keeping values on the GPU.Why does a compiled model recompile, and what happens after many recompiles?core
dynamic=None, the first changed size triggers one recompile with that dimension made symbolic, which then covers other sizes. After 8 entries (the default recompile limit) Dynamo stops compiling that function and runs it eagerly, so speed silently drops. Fix: stable shapes (packing, bucketing), mark_dynamic, and passing changing numbers as tensors.When do CUDA graphs help, and what are their restrictions?core
FlashAttention does about the same FLOPs as standard attention. Why is it faster?core
Why is the cross-entropy layer a memory problem for LLMs, and how is it fixed?core
Your training job shows 40% GPU utilization. How do you find out why?core
cudaStreamSynchronize from .item() (remove syncs), dense tiny kernels with a saturated CPU (launch-bound: compile, CUDA graphs, bigger micro-batch), or NCCL waits (overlap, bucket sizes). Then estimate each fix with Amdahl’s law and start with the biggest.Estimate the time of an unfused vs fused RMSNorm for 16,384 tokens × 4,096 on an H100.advanced
What is the difference between mode="default", "reduce-overhead" and "max-autotune" in torch.compile?basic
Fusion saves memory in training, not only time. Explain.advanced
Why can a faster GPU fail to speed up a training step?advanced
Exercises
- A step launches 2,400 kernels, each 3 µs on the GPU, and each launch costs 6 µs. Estimate the step time and GPU busy fraction with formula (50.1). What do CUDA graphs give?
- Count the HBM traffic of “residual add + RMSNorm” unfused and fused (in units of \(N\) elements). What is the ratio?
- Llama 2 has \(V = 32{,}000\). How much memory do the BF16 logits of 65,536 tokens take? Compare with Llama 3.
- Chunked cross-entropy with \(C = 1{,}024\) tokens and \(V = 128{,}256\): how big is one FP32 chunk of logits?
- Attention, element-wise kernels and the rest take 30%, 25% and 45% of a step. Fusing the element-wise kernels makes them 2.5× faster. What is the overall speedup? And if attention also becomes 2× faster?
- Rewrite
if x.sum() > 0: y = x * 2 else: y = x - 1without a graph break.
Answerscheck
y = torch.where(x.sum() > 0, x * 2, x - 1): both sides are computed and one is selected on the GPU, no Python branch.Key takeaways
- A step is launch-bound, memory-bound or starved. The step time is roughly the larger of CPU launch time and GPU work (50.1); a profiler trace tells you which case you are in.
- torch.compile = Dynamo (bytecode to FX graph plus guards) + AOTAutograd (forward and backward together) + Inductor (fusion, Triton kernels). Graph breaks and recompiles silently remove the gain; count them.
- CUDA graphs replay a captured sequence of kernels with one launch. They need static shapes and addresses and no syncs, and help most when per-GPU work is small (Mask R-CNN 1.70×, BERT 1.12× in MLPerf v1.0).
- Fusion cuts HBM traffic of memory-bound chains (50.2, 50.3): 3.5× for RMSNorm. FlashAttention never writes the \(T^2\) scores (50.5). Fused cross-entropy uses \(\softmax - e_y\) (50.7) to avoid storing \(T \times V\) logits (50.6).
- Read traces in order: steady step, GPU busy fraction, data loader, copies, syncs, launch gaps, element-wise time, exposed communication. Estimate every fix with Amdahl’s law (50.9) first.
Key formulas
References for this session
- Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
- NVIDIA (docs). CUDA C++ Programming Guide
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
- PyTorch (docs). torch.compile API reference (modes, fullgraph, dynamic) and the recompilation guide
- Nguyen, Carilli, Eryilmaz et al. (NVIDIA and Meta) (2021). Accelerating PyTorch with CUDA Graphs (PyTorch blog)
- NVIDIA (docs). CUDA C++ Programming Guide
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Zhang & Sennrich (2019). Root Mean Square Layer Normalization
- Hsu et al. (2024). Liger Kernel: Efficient Triton Kernels for LLM Training
- Wijmans et al. (2024). Cut Your Losses in Large-Vocabulary Language Models
- PyTorch (docs). torch.profiler: PyTorch Profiler API reference
- NVIDIA (docs). Nsight Systems User Guide
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters