LLM & VLM courseBlog
127Track 5: LLM inference and serving · session 50 of 65

Graph-level optimization: fusion, CUDA graphs, torch.compile

A transformer layer in plain PyTorch runs as dozens of small GPU programs, and each one costs memory traffic and a launch from the CPU. This session shows how fusion merges them, how CUDA graphs replay a whole decode step with one launch, how torch.compile does both for you (and what a graph break is), and how inference engines like TensorRT-LLM and DeepSpeed fuse whole layers.

Time: 3 to 3.5 hours (reading 2 hours, exercises 1 to 1.5 hours)Level: IntermediateBefore this: Sessions 82, 85, 123 and 126 (the roofline; CUDA basics; runtime profiling; linear operators)

What you will learn

  • Why small elementwise and normalization ops are memory-bound, and how to count the bytes that fusion saves
  • Which ops can be fused (pointwise, reductions, GEMM epilogues, merged GEMMs) and why a cross-block dependency stops fusion
  • Why decode at small batch is often CPU-bound, and how CUDA graphs remove launch overhead with capture and replay
  • The rules of CUDA graphs: static shapes, static memory addresses, no host sync, and batch-size buckets with padding
  • How torch.compile works: TorchDynamo bytecode capture with guards, AOTAutograd, TorchInductor fusion into Triton kernels, and the modes default, reduce-overhead and max-autotune
  • What causes graph breaks and recompilations, how to find them, and how to remove them
  • How TensorRT, TensorRT-LLM, DeepSpeed Inference and ByteTransformer fuse a whole transformer layer into a few kernels

Why this matters

Run one decode step of a 7B model in plain PyTorch and the GPU does not get one big job. It gets more than a thousand small ones: a matrix product here, an add there, a square root, a multiply, a copy into the KV cache. Each small job is a separate GPU program, a kernel. Each kernel reads its inputs from GPU memory and writes its outputs back, and each one must be started by the CPU. This session is about the cost of “many small kernels”, and about the three tools that remove it.

  • Kernel fusion: merge several small operations into one kernel, so that intermediate results stay in registers instead of going to memory and back.
  • CUDA graphs: record the whole sequence of kernels of one decode step once, then replay it with a single launch from the CPU.
  • torch.compile: a compiler inside PyTorch that captures your Python model as a graph, fuses it automatically into generated kernels, and (in one mode) wraps the result in CUDA graphs.

How large is the effect? The PyTorch team took a simple Llama-7B implementation on an A100 at batch size 1. Eager PyTorch gave 25.5 tokens per second. Adding torch.compile(mode="reduce-overhead", fullgraph=True) and a static KV cache gave 107.0 tokens per second, more than 4 times faster, with the same weights and the same math. [Certain] Their trace showed why: the slow version was CPU-overhead bound. The CPU could not tell the GPU what to do fast enough. The PyTorch 2 paper reports a 2.27× inference and 1.41× training geometric-mean speed-up for TorchInductor over eager PyTorch on an A100, across more than 180 real models. [Certain]

In session 126 you made each matrix product fast on its own. In session 124 you saw FlashAttention, which is itself a fusion: it merges the score, softmax and weighted-sum steps of attention into one kernel. Here we zoom out from single operators to the graph: the whole list of operations of a layer, a step, a model. In session 123 you learned to see these small kernels and the idle gaps between them in a trace. Now we remove them.

The cost of one small kernel

Look at RMSNorm, the normalization in Llama-style models. For each token vector \(\mathbf{h}\) of width \(d\), it computes \(\mathbf{y} = \mathbf{h} \cdot \text{rsqrt}(\text{mean}(\mathbf{h}^2) + \epsilon) \cdot \mathbf{w}\). Written in eager PyTorch, this is six operations, and each runs as its own kernel: pow, mean, add (the \(\epsilon\)), rsqrt, mul (by the scale), mul (by \(\mathbf{w}\)). [Certain] “Eager” means PyTorch runs each line of Python as soon as it reaches it, one operation at a time, without looking ahead.

Each of these kernels does very little math: one or two operations per number. So they are memory-bound (see session 82): their time is set by the bytes they move, not by the FLOPs. Each kernel also has a fixed cost that does not depend on its size: the GPU must start it, fill the streaming multiprocessors with thread blocks, and wait for the last block to finish. A simple and useful model of one kernel \(k\) is

\[ t_k = t_0 + \frac{Q_k}{B} \tag{127.1} \]
Read the symbols
\(t_k\)
time of kernel \(k\) on the GPU
\(t_0\)
fixed cost per kernel: start, ramp-up and tail; a few microseconds (we use 3 µs as an example value)
\(Q_k\)
bytes the kernel reads from and writes to HBM (the GPU’s main memory)
\(B\)
memory bandwidth: 3.35 TB/s on an H100 SXM

Now chain \(K\) such kernels. Without fusion, every intermediate result is written to HBM by one kernel and read back by the next. With fusion, one kernel reads only the true inputs (those that come from outside the group) and writes only the true outputs (those that are used outside the group). Everything in between stays in registers or shared memory:

\[ T_{\text{unfused}} = K\,t_0 + \frac{1}{B}\sum_{k=1}^{K} Q_k, \qquad T_{\text{fused}} = t_0 + \frac{Q_{\text{ext}}}{B} \tag{127.2} \]
Read the symbols
\(K\)
number of kernels in the unfused chain
\(Q_{\text{ext}}\)
external bytes: inputs read from outside the group plus outputs that someone else needs

Divide the two and look at the two extreme cases:

\[ S_{\text{fuse}} = \frac{K\,t_0 + \sum_k Q_k / B}{t_0 + Q_{\text{ext}} / B} \;\longrightarrow\; \begin{cases} K & \text{small tensors (fixed cost dominates)} \\[2pt] \dfrac{\sum_k Q_k}{Q_{\text{ext}}} & \text{large tensors (bandwidth dominates)} \end{cases} \tag{127.3} \]

This is the key picture of fusion. In decode, tensors are tiny and fusion wins by removing kernels. In prefill, tensors are large and fusion wins by removing bytes. [Certain] (Both follow from 127.2; the exact numbers depend on \(t_0\), which varies by GPU and kernel.)

Worked example: fusing RMSNorm, prefill and decode, on an H100

Width \(d = 4{,}096\), BF16 (2 bytes per number). Prefill with \(T = 2{,}048\) tokens. One activation tensor is \(2{,}048 \times 4{,}096 \times 2 = 16.8\) MB. The six eager kernels read and write 117.5 MB in total (the pow and the two multiplies each read and write a full tensor; mean reads one; the small per-token tensors add almost nothing). The fused kernel reads \(\mathbf{h}\) once and writes \(\mathbf{y}\) once: 33.6 MB. At 3.35 TB/s that is 35.1 µs against 10.0 µs of memory time. With \(t_0 = 3\) µs per kernel: \(6 \cdot 3 + 35.1 = 53.1\) µs unfused against \(3 + 10.0 = 13.0\) µs fused, a 4.08× speed-up. The byte ratio alone is 3.5×.

Decode with \(T = 1\). Now one tensor is 8 KB and the whole chain moves 57 KB: 0.017 µs of memory time. The time is \(6 \cdot 3 = 18.0\) µs unfused against \(3.0\) µs fused: 6.0×, which is just \(K\). In decode, fusion is about kernel count. [Certain]

Figure 1 runs this RMSNorm for real, on a toy input of 2 tokens with width 6, so you can watch the numbers move. In eager mode each of the six kernels reads its inputs from HBM, computes, and writes its result back to HBM. In fused mode the same six steps happen in registers, and HBM is touched only twice. The counters at the bottom add the bytes of the toy, and the time of the same step at full size with formulas 127.1 and 127.2.

read from HBMwritten to HBMkept in registersfixed cost \(t_0\)
Eager PyTorch: six kernels
    Figure 1. RMSNorm computed live on 2 tokens of width 6 (\(\epsilon = 10^{-5}\)). Top: the tensors in HBM. A tensor box is filled when a kernel has written it; in fused mode the intermediate tensors are never created. Middle: the chip, with the inputs and the output of the current step, all values computed in your browser. Bottom: the bytes moved for this toy (BF16, 2 bytes per number), and the time of the same step at full size (width 4,096, \(t_0 = 3\) µs, 3.35 TB/s) using the byte counts of Figure 2. The weight vector \(\mathbf{w}\) is read by both versions and is left out of the counts, as in the worked example.
    How to read Figure 1

    Press Play in eager mode. Each kernel takes three steps: launch and read (blue packets go down), compute (the new values appear on the chip), write (a green packet goes up and fills a box in HBM). Token 0 is \(\mathbf{h} = (0.5, -1.2, 0.8, 2.0, -0.3, 1.1)\): kernel 2 computes its mean square 1.2717, kernel 4 gives \(1/\sqrt{1.27168} = 0.8868\), and kernel 6 gives \(\mathbf{y} = (0.443, -0.958, 0.780, 2.128, -0.213, 0.975)\). The toy moves 192 bytes, and at full prefill size the six kernels take 53.06 µs. Switch to fused and play again: the same numbers appear, but only \(\mathbf{h}\) goes down and only \(\mathbf{y}\) goes up: 48 bytes, and 13.02 µs at full size (4.08× faster). Choose decode: now the memory part of each bar is almost zero and the time is just the fixed costs, 18.02 µs against 3.00 µs.

    fixed cost \(t_0\)memory time \(Q/B\)
    Figure 2. Fusion of a chain of small operations on an H100 (3.35 TB/s), using formulas 127.1 to 127.3. Top: the eager kernels, each box sized by the bytes it moves, and the single fused kernel. Bottom: the time of both versions, split into fixed cost and memory time. Widths: 16,384 for bias + GELU (a 4× MLP of a 4,096-wide model), 4,096 for the norms, 11,008 for SwiGLU (Llama 2 7B).
    How to read Figure 2

    Start with RMSNorm at \(T = 2{,}048\): the bars show that memory time dominates, and fusion saves 3.5× of the bytes. Move \(T\) down to 1: the memory part almost disappears and the bars become pure fixed cost, so the speed-up becomes the number of kernels (6). Set \(t_0 = 0\) to see the “bytes only” limit. Try SwiGLU: it is only two kernels and saves fewer bytes, so the gain is smaller. Residual add + RMSNorm has 7 kernels; the fused version must write two outputs (the new residual stream and the normed input of the next layer).

    What can be fused, and what stops fusion

    A kernel runs as many thread blocks in parallel. Thread blocks cannot wait for each other inside one kernel (there is no cheap global barrier). So the rule for fusing two operations is about tiles: cut the work of each operation into pieces that thread blocks own. DeepSpeed Inference states the rule as follows: two operators can be fused if each tile of the second depends on exactly one output tile of the first. [Certain] Written as a formula, with \(\text{dep}(j)\) the set of tiles of the first op that tile \(j\) of the second op needs:

    \[ \text{fusible}(A \to B) \iff \exists \text{ a tiling such that } \ \forall j:\ \big|\text{dep}_{A}(j)\big| = 1 \tag{127.4} \]

    If the data a block needs was produced by a different block, a global synchronization is needed, and on a GPU that means a new kernel. [Certain] With this rule we can sort the common cases:

    PatternExample in a transformerFusible?
    Pointwise → pointwise (vertical)bias add → GELU; SiLU(gate) × upYes, always: tile \(j\) needs only element \(j\)
    Reduction with its pointwise neighboursRMSNorm, LayerNorm, softmax over one rowYes, if one block owns a whole row (tile along tokens, reduce inside the tile)
    GEMM epilogueGEMM → bias, activation, residual add, quantize, castYes: the output tile is still in registers when the GEMM ends
    Horizontal (siblings)Q, K and V projections of the same input; gate and up projectionsYes: concatenate the weights and run one bigger GEMM
    GEMM prologueRMSNorm → QKV GEMMOnly if each block can compute the norm of its own rows; easy at small batch, hard at large
    Attention chain\(QK^\top\) → softmax → \(\times V\)Yes with online softmax: FlashAttention (session 124)
    GEMM → GEMMMLP up → downUsually no: each output tile of the down GEMM needs a full row of the up output, made by many blocks

    The last row is why a fused transformer layer still has a few kernels and not one. The second GEMM needs results from many blocks of the first. [Certain] At very small batch, DeepSpeed goes around this for the norm before a GEMM: every block recomputes the same small norm on its own copy of the input. This wastes some math, but it is still faster than a separate kernel at very small batch sizes. [Certain]

    A compiler applies these rules over the whole graph. TorchInductor, the backend of torch.compile, does it greedily: it lists the legal fusion pairs, scores them (first by fusion category, then by the number of bytes saved, then by how close the two nodes are in the graph), applies them in that order, and repeats until nothing more can be fused. [Certain]

    Algorithm 1: greedy fusion pass (the idea of the TorchInductor scheduler)
    1. input: graph of operations, each with its loop shape (pointwise, reduction, template GEMM)
    2. groups \(\leftarrow\) one group per operation
    3. repeat
    4.   \(P \leftarrow\) all pairs of groups \((a, b)\) that share data and pass the tile rule (127.4) // can_fuse
    5.   sort \(P\) by score: fusion type, then bytes saved, then graph distance // score_fusion
    6.   for \((a, b)\) in \(P\): if still legal then merge \(a\) and \(b\) into one group
    7. until no pair was merged // one fusion can open new ones
    8. for each group: generate one kernel (Triton on GPU, C++ on CPU) that keeps internal tensors in registers

    How much of the speed-up comes from fusion? The PyTorch 2 paper turned off one optimization at a time (FP16 HuggingFace models on an A100, inference). With all optimizations: 1.91×. Without CUDA graphs: 1.81×. Without fusion: 1.68×. Without inlining (copying small pointwise bodies into their users): 1.58×. Without both fusion and inlining: 0.80×, a slowdown. [Certain] The reason for the slowdown: Inductor first decomposes big operators into small primitives, and only fusion puts them back together into fast kernels. [Certain]

    Fusing a whole transformer layer

    Now apply the rules to one decoder layer of Llama 2 7B (width 4,096, 32 heads with full key–value heads, MLP width 11,008). In eager PyTorch, with the usual Hugging Face style code, our count is 36 kernels per layer: 6 for each of the two RMSNorms, 3 separate Q, K, V GEMMs, 5 for rotary embedding (RoPE) on Q and 5 on K, 2 copies into the KV cache, 1 attention kernel, the output GEMM, 2 residual adds, the gate and up GEMMs, SiLU, a multiply and the down GEMM. [Likely] (The exact count depends on the code; profile your own model as in session 123.)

    We fuse in three levels, each adding one family of fusions from the table above:

    • Level B, pointwise and norm fusion: each RMSNorm becomes 1 kernel; RoPE and the cache writes become 1 kernel; SiLU × up becomes 1; the first residual add joins the second norm. 13 kernels.
    • Level C, plus horizontal and epilogue fusion: one QKV GEMM and one gate+up GEMM with SiLU × up in its epilogue; residual adds in the epilogues of the output and down GEMMs. 8 kernels.
    • Level D, “deep” fusion: the norms join the GEMM that follows them (prologue), and RoPE + cache write join the attention kernel. 5 kernels. This is close to the four fusion regions that DeepSpeed Inference uses at small batch: QKV GEMM with the input norm, transposition with attention, post-attention norm with the MLP GEMM, and bias with residual add. [Certain]
    Phase
    normGEMMRoPE, cache, attentionpointwise
    Figure 3. One Llama 2 7B decoder layer as 36 operations (the coloured cells, in execution order) and the kernels they form (the brackets below) at four fusion levels. The bars compare the activation bytes each level moves per layer (weights and KV-cache reads excluded, since fusion does not change them). The readout gives kernels for 32 layers and the time of the activation traffic at 3.35 TB/s.
    How to read Figure 3

    Step through the levels. The brackets get longer as more operations share one kernel: 36, 13, 8, then 5 kernels per layer. Switch the phase. In prefill the eager layer moves 1.32 GB of activations, more than 3 times the 405 MB of layer weights, and level D cuts it to 0.43 GB: fusion removes real bytes. In decode the activations are only 0.64 MB per layer, 0.16% of the weights, so fusion saves almost no bytes; it saves kernels (1,152 down to 160 for the model), and that matters for the CPU, as the next section shows.

    CUDA graphs: one launch for the whole step

    Fusion makes fewer kernels. But even a few hundred kernels can be too many in decode, because of the CPU. To start a kernel, the CPU runs Python, the PyTorch dispatcher (which picks the right implementation for the dtype and device), and the CUDA driver call that puts the kernel into the GPU’s queue (a stream). That costs microseconds per kernel. The launch is asynchronous: the CPU does not wait for the kernel to finish, so normally it runs ahead and the GPU always has work queued. [Certain]

    That breaks down when the GPU finishes a kernel faster than the CPU can launch the next one. Then the GPU waits, and the step time is set by the CPU. In decode at batch 1, a kernel on a 1B model takes about 1 µs of memory time, while a launch from eager PyTorch takes several microseconds. [Likely] The NVIDIA example from the CUDA graphs blog (20 small kernels, 1,000 steps, V100) measured per kernel: 9.6 µs when the CPU waits after each kernel, 3.8 µs when launches overlap with execution, 3.4 µs with a CUDA graph, against 2.9 µs of pure kernel time. [Certain]

    A CUDA graph removes the per-kernel CPU work. You run the step once in capture mode: the kernels are not executed, they are recorded, with all their arguments and memory addresses, into a graph. Later, one call (graph.replay()) submits the whole recorded sequence to the GPU at once. [Certain] A simple timing model of one decode step with \(K\) kernels:

    \[ T_{\text{eager}} \approx \max\big(K c,\ K(\bar g + \delta)\big), \qquad T_{\text{graph}} \approx G + K(\bar g + \delta) \tag{127.5} \]
    Read the symbols
    \(c\)
    CPU time to launch one kernel from eager PyTorch (Python + dispatcher + driver)
    \(\bar g\)
    average GPU time of one kernel
    \(\delta\)
    small gap between two back-to-back kernels on the GPU (we use 0.5 µs)
    \(G\)
    one-time CPU cost to launch the whole graph (we use 10 µs)

    When \(c \gt \bar g + \delta\), eager decode is CPU-bound: the GPU is idle for part of every gap. A CUDA graph makes the time depend on the GPU only. Fusion attacks the same problem from the other side: it makes \(K\) smaller.

    Figure 4 runs formula 127.5 kernel by kernel, on a short sequence of kernels, with one clock for both lanes. In the eager lane the CPU launches kernel \(k\) at time \(L_k = k\,c\), and the GPU starts it at \(s_k = \max(L_k,\ e_{k-1} + \delta)\), the later of “launch done” and “previous kernel done”. In the graph lane the CPU does one replay call, and then only the second term matters. The red parts of the GPU rows are the time the GPU sits idle.

    CPU launchingGPU kernel runningGPU idlegraph replay call
    Eager: launch one kernel at a time
    1. for \(k = 1, \dots, K\):
    2. CPU: launch kernel \(k\) // done at \(L_k = k\,c\)
    3. GPU: \(s_k = \max(L_k,\ e_{k-1} + \delta)\) // wait
    4. GPU: \(e_k = s_k + \bar g\) // run
    5. \(T_{\text{eager}} = e_K\) // (127.5)
    CUDA graph: one replay call
    1. CPU: graph.replay() // done at \(G\)
    2. for \(k = 1, \dots, K\):
    3. GPU: \(s_k = \max(G,\ e_{k-1} + \delta)\) // no CPU wait
    4. GPU: \(e_k = s_k + \bar g\) // run
    5. \(T_{\text{graph}} = e_K = G + K(\bar g + \delta) - \delta\) // (127.5)
    Figure 4. A few kernels of one decode step, simulated event by event with formula 127.5 (\(\delta = 0.5\) µs, \(G = 10\) µs). The vertical line is the clock; Step jumps to the next event in either lane. The default \(\bar g = 1.28\) µs is the Llama 3.2 1B value from the worked example below, and \(c = 8\) µs is the same assumed launch cost. Both are model values, not measurements.
    How to read Figure 4

    Press Play with the defaults. In the eager lane the GPU runs kernel 1 from 8.00 to 9.28 µs, then waits: kernel 2 is launched only at 16 µs. Every kernel starts at \(s_k = \max(8k,\ e_{k-1} + 0.5) = 8k\), so the step ends at 65.28 µs and the GPU is busy only 15.7% of the time. The graph lane starts after the 10 µs replay call and then runs the kernels back to back, 1.78 µs apart: it ends at 23.74 µs, 2.75× sooner. Now move \(c\) down to 1.5 µs: the CPU keeps up (\(s_2 = \max(3.00,\ 2.78 + 0.5) = 3.28\)), eager ends at 15.24 µs, and with only 8 kernels the 10 µs replay call makes the graph slower. When the CPU keeps up, a graph only adds its fixed call. Its gain appears when \(c \gt \bar g + \delta\), and it grows with \(K\): at \(c = 8\) µs and \(K = 24\), eager takes 193.28 µs and the graph 52.22 µs (3.70×).

    Worked example: Llama 3.2 1B decode at batch 1 on an H100

    Llama 3.2 1B has 1.24 billion parameters (16 layers, width 2,048, from its config). In BF16 its weights are 2.47 GB, so one decode step needs at least \(2.47 / 3.35 = 738\) µs of memory time. [Certain] With 36 eager kernels per layer, \(K = 576\), so \(\bar g = 738 / 576 = 1.28\) µs.

    Take \(c = 8\) µs per launch (an assumed, typical order for eager PyTorch). Eager: \(T \approx 576 \times 8 = 4{,}609\) µs, the GPU is busy only 16% of the time, and the model runs at 217 tokens/s. CUDA graph: \(T \approx 10 + 576 \times 1.78 = 1{,}036\) µs, 4.45× faster (966 tokens/s). Fused to 5 kernels per layer (\(K = 80\)): eager already takes about 785 µs, and the graph adds nothing (787 µs). [Certain] (arithmetic of 127.5; the real values of \(c\) and \(\delta\) depend on CPU, GPU and driver.)

    Check against the gpt-fast numbers: 25.5 tokens/s is 39.2 ms per token for Llama-7B, while reading 13.5 GB of FP16 weights at the A100’s 2 TB/s takes only 6.74 ms. Most of the eager step was overhead. After compile with CUDA graphs: 107 tokens/s, 9.35 ms per token. [Certain]

    CPU launchingGPU kernel runningGPU idle
    Figure 5. One decode step at batch 1 on an H100, simulated with formula 127.5: kernels are launched by the CPU one after another (eager) or all at once (CUDA graph). The GPU time of the step is the weight-read time (BF16 weights at 3.35 TB/s), spread evenly over the kernels; we use \(\delta = 0.5\) µs and \(G = 10\) µs. The two rows share one time axis.
    How to read Figure 5

    With the 1B model in eager mode, the CPU row is a solid bar and the GPU row is mostly empty: each kernel finishes long before the next one is launched. The CUDA graph row ends much earlier. Move \(c\) down to about 1.5 µs: now the CPU keeps up and both rows take about the same time. Switch to “fused”: with 80 kernels the eager version is no longer CPU-bound even at \(c = 8\) µs. The 7B model has more GPU work per kernel, so it needs a larger \(c\) to become CPU-bound; at \(c = 8\) µs eager is still 2.0× slower than the graph.

    The rules of CUDA graphs

    A graph is a recording, so everything that was fixed at capture time is fixed forever. [Certain]

    • Static memory addresses. The kernels read and write the exact buffers used during capture. New input must be copied into the captured input buffer; the output appears in the captured output buffer. PyTorch gives each graph a private memory pool so these buffers are not freed or reused. [Certain]
    • Static shapes. The kernel sizes (grid dimensions) are recorded. One graph serves exactly one shape. A KV cache that grows by concatenation breaks this; a static KV cache of maximum length, with a mask for unused positions, fixes it (this is what gpt-fast does). [Certain]
    • No CPU work inside. Python code, .item(), print of a tensor or anything that synchronizes with the CPU is not recorded and is not allowed during capture. Random numbers need special handling (PyTorch registers the generator state with the graph). [Likely]
    • Capture costs time and memory. In the NVIDIA example, creating the graph took about 400 µs, paid once. [Certain] Each captured batch size keeps its own graph and buffers.

    Serving engines have variable batch sizes, so they capture one graph per bucket of batch sizes and pad each real batch up to the nearest bucket. The padded slots are wasted work:

    \[ w = \frac{b_{\text{cap}} - b}{b_{\text{cap}}}, \qquad b_{\text{cap}} = \min\{\, s \in \mathcal{S} : s \ge b \,\} \tag{127.6} \]
    Read the symbols
    \(b\)
    real batch size of this step
    \(\mathcal{S}\)
    set of captured sizes, for example 1, 2, 4, then multiples of 8
    \(w\)
    fraction of the step spent on padding slots

    With buckets 1, 2, 4, 8, 16, 24, 32, …: a batch of 5 runs as 8 (37.5% waste), 9 as 16 (43.8%), 17 as 24 (29.2%), 33 as 40 (17.5%). In memory-bound decode the padded rows are almost free, because the weights are read once for the whole batch (session 126). So moderate padding costs little time, and finer buckets cost graph memory and capture time. vLLM captures decode graphs by request count, and a batch takes the smallest captured graph that covers it. [Certain] Its “piecewise” mode keeps attention (and other graph-incompatible operations) in eager mode and puts everything else into CUDA graphs. Attention is the part whose shapes change with each request’s context length. [Certain]

    Algorithm 2: decode with CUDA graphs and batch buckets
    1. at start-up, for each bucket \(s \in \mathcal{S}\) (largest first, to share the memory pool) do
    2.   allocate static buffers: input ids \([s]\), positions \([s]\), output logits \([s, V]\); KV cache is preallocated
    3.   run the decode step 2 to 3 times on a side stream // warm-up: lazy init, autotuning, allocator
    4.   capture one decode step into graph \(g_s\)
    5. at each step with \(b\) active sequences do
    6.   \(s \leftarrow\) smallest bucket \(\ge b\); if none, run eager // (127.6)
    7.   copy the \(b\) real inputs into the static buffers of \(g_s\); fill the other \(s - b\) slots with dummy tokens
    8.   \(g_s\).replay() // one CPU call for hundreds of kernels
    9.   read the first \(b\) rows of the static logits buffer; sample on the GPU

    torch.compile: capture, fuse, replay

    Writing fused kernels by hand for every model is a lot of work. torch.compile does it automatically. You wrap a function or module, model = torch.compile(model), and keep writing normal Python. It has three stages. [Certain]

    Pythonyour model code(bytecode) TorchDynamoFX graph+ guards AOTAutograddecompose toprimitive ops TorchInductorfuse (Algorithm 1)Triton / C++ code CUDA graphwith mode=reduce-overhead graph break: run this bytecode in Python,then trace the rest again guards fail: recompile max-autotune: also tries TritonGEMM templates with fused epilogues
    Figure 6. The torch.compile pipeline. TorchDynamo reads the Python bytecode and records the PyTorch operations into an FX graph, plus guards (checks) that must hold for the graph to be reused. AOTAutograd decomposes the operations into primitives (and, for training, also builds the backward graph). TorchInductor fuses them and generates Triton kernels for GPUs or C++ for CPUs. In mode="reduce-overhead" the result is replayed with CUDA graphs.
    1. TorchDynamo: graph capture. Dynamo installs a hook into CPython’s frame evaluation, so it sees each Python function just before it runs. It reads the bytecode instruction by instruction and records every PyTorch operation into an FX graph (a simple list of operations and their inputs). Plain Python logic (loops over a fixed list, attribute lookups, if-statements on Python values) is executed at trace time and disappears from the graph. Dynamo also records guards: conditions that the graph assumes, such as input shapes, dtypes, devices, and Python constants. The paper counts about 30 types of guards. [Certain] Next time the function is called, Dynamo checks the guards; if they hold, it runs the cached compiled code; if not, it compiles again.
    2. AOTAutograd. Runs the graph on “fake” tensors (shapes and dtypes, no data) to produce a graph of primitive operations. For training it also builds the backward graph ahead of time. [Certain]
    3. TorchInductor: fusion and code generation. Lowers the primitives to a loop-level form, fuses with Algorithm 1, and writes one Triton kernel (GPU) or C++ loop (CPU) per fused group. Matrix products go to cuBLAS by default, or to Triton GEMM templates with fused epilogues. [Certain]

    The mode argument picks a preset. "default": fusion and code generation. "reduce-overhead": also wraps the compiled code in CUDA graphs, to remove launch overhead; it is turned off automatically when it is not safe, for example with dynamic shapes or non-CUDA tensors. [Certain] "max-autotune": also benchmarks several kernel choices, including Triton GEMM templates with fused epilogues, and picks the fastest; compile time is much longer. [Certain] In recent PyTorch versions max-autotune also uses CUDA graphs on GPU, and "max-autotune-no-cudagraphs" turns that off. [Likely]

    Graph breaks

    Dynamo cannot put everything into a graph. When it meets bytecode it cannot handle, it makes a graph break: it compiles the graph so far, runs the unsupported code in normal Python, and then starts tracing again in a generated “continuation” function. [Certain] The model still runs correctly, but in pieces. Each break means: a smaller graph (fewer fusion chances across the break), a return to Python (CPU overhead), and, with reduce-overhead, several CUDA graphs with Python in between. Common causes, all checked with PyTorch 2.8 and torch._dynamo.explain:

    • Python control flow on a tensor value: if h.sum() > 0:. Python needs a real bool, but during tracing the value does not exist yet. Fix: torch.where, or compute the condition on the CPU side from Python values.
    • Tensor to Python number: .item(), .tolist(), int(t). Fix: keep it a tensor; or set torch._dynamo.config.capture_scalar_outputs = True.
    • Data-dependent output shapes: h[h > 0], torch.nonzero. The size of the result depends on the data. Fix: masks with fixed shapes, (h * m).sum() / m.sum().
    • Unsupported calls: print, logging, some C extensions, time.time(), NumPy on tensors. Fix: move them out of the compiled region.

    In the paper’s measurements, 70% of TorchBench models, 89% of HuggingFace models and 100% of TIMM models compiled with zero graph breaks (FP32 inference on an A100). [Certain] For a decode step you want exactly zero: use fullgraph=True, which raises an error at the first break instead of silently splitting. gpt-fast does this. [Certain]

    Fix or keep each line:
    compiled graphgraph break: back to Python
    Figure 7. A small function with four lines that can cause graph breaks. Each button switches one line between a break-free form and a breaking form. The counts are the real output of torch._dynamo.explain in PyTorch 2.8 for all 16 combinations: every breaking line adds exactly one break and one more graph. The break reasons come from running each breaking line on its own.
    How to read Figure 7

    All four on the left (break-free): one graph, zero breaks, and fullgraph=True succeeds. Switch any one line: the function is cut into two graphs at that line, highlighted in the listing. Switch all four: five graphs. Each small block between two graphs is a return to Python, and each graph is fused and (with reduce-overhead) replayed separately, so no fusion can cross a gap.

    Recompilation and the cost of compiling

    If a guard fails, Dynamo compiles again. The classic case is a new input shape. By default, PyTorch first compiles for the exact shape; when it sees a second shape it recompiles with a dynamic (symbolic) size, so that later shapes reuse it. In our test with PyTorch 2.8, four calls with 4, 8, 16 and 32 rows produced exactly 2 compiled graphs. [Certain] After a limit of recompilations of one function, Dynamo stops compiling it and falls back to eager. [Likely] Use TORCH_LOGS="graph_breaks,recompiles" to see both.

    Compiling also takes time, from seconds to minutes for a large model with max-autotune. It pays off after

    \[ n^{\ast} = \frac{T_{\text{compile}}}{t_{\text{eager}} - t_{\text{compiled}}} \ \text{ steps} \tag{127.7} \]

    For example, 60 s of compile time and a step that goes from 20 ms to 14 ms pay off after 10,000 steps: a few minutes of serving, but too long for a short notebook test. Servers compile once at start-up (and can cache the compiled artifacts on disk), and keep shapes in a small set of buckets so that no recompiles happen while serving.

    Algorithm 3: what happens on each call of a compiled function
    1. input: function \(f\), arguments \(x\); cache of (guards, compiled code) for \(f\)
    2. for each cache entry \(e\) do
    3.   if \(e\).guards\((x)\) hold then return \(e\).code\((x)\) // fast path: no tracing
    4. if too many entries then return \(f(x)\) in eager // recompile limit
    5. trace the bytecode of \(f\) into an FX graph until the end or an unsupported instruction
    6. compile the graph (AOTAutograd, then Inductor); at a break, emit: graph call, the unsupported bytecode, call of a continuation function
    7. store (guards, code) in the cache; run it

    How inference engines fuse whole layers

    torch.compile finds fusions automatically and safely. Inference engines go further with hand-written fused kernels for the patterns that matter most in transformers.

    • TensorRT builds an engine from a model graph ahead of time. In its first phase, layers are fused whenever possible: the builder detects supported patterns and merges several layers into one (for example adjacent pointwise layers into one pointwise layer, or the chain of small ops that computes GELU into one GELU). [Certain] It then times several kernel implementations (“tactics”) for each layer on the target GPU and keeps the fastest. [Likely] See session 131.
    • TensorRT-LLM uses its own fused operators for LLMs. In the generation (decode) phase, attention is one kernel, the masked multi-head attention kernel, which also adds the QKV bias, applies RoPE, and does quantization and dequantization. [Certain] For grouped-query attention it can use a special XQA kernel, chosen by a heuristic. [Certain] In the context (prefill) phase, a fused multi-head attention kernel does the whole attention block in one kernel. [Certain]
    • DeepSpeed Inference introduced Deep-Fusion: fuse not only pointwise ops but also reductions, data transpositions and GEMMs, as long as there are no cross-tile dependencies (127.4), and keep the data in registers or shared memory. It adds a custom small-batch GEMM that can be fused, and CUDA graphs because, once kernels get fast, “the main latency bottleneck shifts from kernel execution to the kernel launch overhead on the CPU side”. [Certain] The paper reports up to 7.3× lower latency than the state of the art for latency-oriented scenarios. [Certain]
    • ByteTransformer (for BERT-like encoders) fuses add-bias with LayerNorm into one kernel that reads memory once (61% faster for that kernel, +3.2% for the layer), and puts bias + GELU into a custom CUTLASS GEMM epilogue (+3.8% for the layer). [Certain] Its main idea is a padding-free layout: a prefix sum over the attention mask packs only the real tokens, and the pack and unpack steps are themselves fused into existing memory-bound kernels. Its fused attention is 6.13× faster than PyTorch attention on variable-length inputs (A100). [Certain] The memory-bound ops (layernorm, bias, activation) took only 11% to 17% of BERT time, which is why these fusions give single-digit percentages each. [Certain]
    • vLLM runs the model through torch.compile and uses CUDA graphs, by default in a piecewise way around attention, as described above. [Certain] See session 138.

    The pattern is the same everywhere: GEMMs and attention are separate, carefully tuned kernels; everything small is folded into their prologues and epilogues; and the remaining launches are hidden behind a CUDA graph.

    In code

    The code below runs on a CPU with PyTorch 2.x (we ran it with 2.8). It counts the bytes saved by fusion, checks that a compiled add + RMSNorm gives the same result, finds and removes graph breaks, shows a recompilation, and simulates the launch timeline. The last part captures and replays a CUDA graph and runs only when an NVIDIA GPU is present (we did not run that part).

    import torch
    torch.manual_seed(0)
    
    # 1) bytes moved by add + RMSNorm, eager (7 kernels) vs fused (1 kernel), formula (127.2)
    T, d, B = 2048, 4096, 3.35e12                      # tokens, width, H100 HBM bytes/s
    N = T * d
    eager = [(2*N, N), (N, N), (N, T), (T, T), (T, T), (N + T, N), (N, N)]  # (read, write) elements
    fused = [(2*N, 2*N)]                                # read x, res; write h, y
    for name, ks in (("eager", eager), ("fused", fused)):
        q = sum(2 * (r + w) for r, w in ks)             # BF16: 2 bytes per element
        print(f"{name}: {len(ks)} kernels, {q/1e6:.1f} MB, {q/B*1e6:.1f} us at full bandwidth")
    
    # 2) torch.compile fuses the seven ops into one generated kernel
    def add_rmsnorm(x, res, w, eps=1e-5):
        h = x + res                                     # residual add
        y = h * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + eps) * w
        return h, y
    x, res, w = torch.randn(8, d), torch.randn(8, d), torch.randn(d)
    h0, y0 = add_rmsnorm(x, res, w)
    h1, y1 = torch.compile(add_rmsnorm)(x, res, w)
    print("compiled max diff:", (y0 - y1).abs().max().item())
    
    # 3) graph breaks: find them, then remove them
    def f_bad(x):
        y = torch.sin(x) * 2
        if y.sum() > 0:                                 # Python branch on a tensor value
            y = y + 1
        n = int((y > 1).sum().item())                   # tensor -> Python int
        return y * n
    def f_good(x):
        y = torch.sin(x) * 2
        y = torch.where(y.sum() > 0, y + 1, y)          # the branch stays in the graph
        n = (y > 1).sum()                               # stays a tensor
        return y * n
    for f in (f_bad, f_good):
        torch._dynamo.reset()
        e = torch._dynamo.explain(f)(x)
        print(f.__name__, "graphs:", e.graph_count, "breaks:", e.graph_break_count)
    g = torch.compile(f_good, fullgraph=True)           # fullgraph=True: raise on any break
    print("same result:", torch.allclose(f_bad(x), g(x)))
    
    # 4) recompilation: a new input shape fails a guard
    from torch._dynamo.utils import counters
    torch._dynamo.reset(); counters.clear()
    h = torch.compile(lambda t: torch.relu(t) * 2)
    for n in (4, 8, 16, 32):
        h(torch.randn(n, 64))
    print("frames compiled for 4 shapes:", counters["stats"]["unique_graphs"])
    
    # 5) launch-overhead model of one decode step, formula (127.5)
    def step_time(K, g, c, gap=0.5, G=10.0):
        end = -gap
        for i in range(K):                              # eager: kernel i is launched at (i+1)*c
            end = max((i + 1) * c, end + gap) + g
        return end, G + K * (g + gap) - gap             # graph: one launch, then back-to-back
    for K in (576, 80):                                 # Llama 3.2 1B: eager vs fused layers
        te, tg = step_time(K, 738.0 / K, 8.0)
        print(f"K={K}: eager {te:.0f} us, CUDA graph {tg:.0f} us, speed-up {te/tg:.2f}x")
    
    # 6) CUDA graph capture and replay (needs an NVIDIA GPU)
    if torch.cuda.is_available():
        lin = torch.nn.Linear(d, d, device="cuda", dtype=torch.bfloat16)
        static_x = torch.zeros(8, d, device="cuda", dtype=torch.bfloat16)
        s = torch.cuda.Stream(); s.wait_stream(torch.cuda.current_stream())
        with torch.cuda.stream(s):                      # warm up on a side stream
            for _ in range(3): static_y = torch.nn.functional.silu(lin(static_x))
        torch.cuda.current_stream().wait_stream(s)
        graph = torch.cuda.CUDAGraph()
        with torch.cuda.graph(graph):                   # record, do not run
            static_y = torch.nn.functional.silu(lin(static_x))
        new_x = torch.randn(8, d, device="cuda", dtype=torch.bfloat16)
        static_x.copy_(new_x)                           # same address, new values
        graph.replay()                                  # one launch for all kernels
        ref = torch.nn.functional.silu(lin(new_x))
        print("graph replay matches eager:", torch.allclose(static_y, ref))

    Output (PyTorch 2.8, CPU): eager: 7 kernels, 167.8 MB, 50.1 us and fused: 1 kernels, 67.1 MB, 20.0 us; compiled max difference about \(2 \times 10^{-6}\) (float rounding); f_bad graphs: 3 breaks: 2 and f_good graphs: 1 breaks: 0; same result True; 2 frames compiled for 4 shapes (one static, then one dynamic); K=576: eager 4609 us, CUDA graph 1036 us, speed-up 4.45x and K=80: eager 786 us, CUDA graph 788 us, speed-up 1.00x. Dynamo also prints a warning for the .item() break that suggests capture_scalar_outputs.

    Where this is used

    SettingMain bottleneckWhat helps
    Chat decode at batch 1 to 8 (local assistants, coding tools)CPU launch overhead; hundreds of tiny kernelsCUDA graphs with a static KV cache; fusion to cut kernels; torch.compile(mode="reduce-overhead", fullgraph=True) as in gpt-fast
    High-throughput serving (vLLM, SGLang, TensorRT-LLM)Variable batch sizes and context lengthsOne graph per batch bucket (127.6); piecewise graphs with attention outside; fused attention and GEMM epilogues (session 135)
    Long-prompt prefill, RAGActivation bytes of norms, RoPE, activationsPointwise, norm and epilogue fusion; FlashAttention
    VLM vision encoder on video framesMany frames of a fixed size through a ViTStatic shapes make the encoder easy to compile fully and to capture as one graph; fuse LayerNorm, GELU and bias
    Video retrieval and video fingerprinting (embedding many clips)Throughput of a fixed encoderA TensorRT engine with fused layers, built once for a fixed input size and batch
    Live captioning and key-moment detection on a streamLatency of small-batch decode per segmentCUDA graphs per bucket; zero graph breaks in the decode step
    TrainingMemory-bound pointwise ops in forward and backwardtorch.compile fusion (1.41× geometric mean in the PT2 paper [Certain]); AOTAutograd recomputes cheap ops instead of storing them

    Common mistakes

    • Benchmarking the first calls of a compiled model. The first call compiles, the next ones may recompile; with reduce-overhead the first calls also capture graphs. Warm up, then time many steps with torch.cuda.synchronize().
    • Silent graph breaks. A stray print, .item() or data-dependent if cuts the decode step into pieces and brings back CPU overhead. Use fullgraph=True or TORCH_LOGS="graph_breaks".
    • Recompiling on every new length. A prompt length that changes every request triggers recompiles until the dynamic-shape path kicks in or the limit is reached. Compile prefill with dynamic shapes or buckets, and decode with static shapes.
    • Growing the KV cache with torch.cat inside a CUDA graph. Shapes and addresses change every step. Preallocate a static cache and write into it by position.
    • Feeding a graph new tensors instead of copying into its buffers. The replay reads the captured addresses, so new input passed any other way is ignored and you get stale results. Copy with static_x.copy_(x), and copy the output out if you keep it, because the next replay overwrites it.
    • Expecting fusion to save bytes in decode. At batch 1 the activations are 0.16% of the weight bytes; fusion helps decode by cutting kernel launches, not memory traffic.
    • Too many capture buckets. Every bucket costs capture time at start-up and GPU memory for its graph and buffers. Small padding waste in memory-bound decode is cheaper than hundreds of graphs.

    Interview questions

    What is kernel fusion and why does it speed up a model?basic
    Fusion merges several operations into one GPU kernel. Intermediate results stay in registers or shared memory instead of being written to HBM and read back by the next kernel, and there are fewer kernels to start. With formula 127.2, an unfused chain costs \(K t_0 + \sum Q_k / B\) and the fused one \(t_0 + Q_{\text{ext}}/B\). For RMSNorm at 2,048 × 4,096 on an H100 that is 117 MB vs 34 MB and about 53 µs vs 13 µs.
    Which operations can be fused, and what prevents fusion?core
    Fusion is legal when, after a suitable tiling, each tile of the consumer depends on exactly one tile of the producer (127.4), so no thread block needs data made by another block. That covers chains of pointwise ops, reductions over a row that one block owns (norms, softmax), GEMM epilogues (bias, activation, residual, quantization), horizontal merges of sibling GEMMs (QKV, gate+up), and attention via online softmax. It fails across a global dependency, for example GEMM → GEMM, where each output tile needs a full row produced by many blocks; that needs a global barrier, so a new kernel.
    Fusion barely changes the bytes moved in decode. Why does it still help?core
    At batch 1 each activation is a single vector (8 KB at width 4,096), so the activation traffic of a Llama 2 7B layer is about 0.64 MB against 405 MB of weights. The cost of the small kernels is their fixed per-kernel cost and the CPU launch, not bytes. Fusing 36 kernels per layer into 5 cuts the decode step from 1,152 to 160 kernels, which removes most of the launch overhead (127.3 in the small-tensor limit, 127.5 for the CPU).
    Why is decode at batch 1 often CPU-bound, and how do CUDA graphs fix it?core
    Eager PyTorch spends several microseconds of CPU time per kernel (Python, dispatcher, driver call). A decode kernel of a small model runs in about 1 µs of memory time, so the GPU finishes before the next launch and waits: \(T \approx Kc\). A CUDA graph records the kernels of one step once and replays them all with one call, so the time becomes \(G + K(\bar g + \delta)\), set by the GPU. For Llama 3.2 1B on an H100 the model gives about 4.6 ms eager vs 1.0 ms with a graph at \(c = 8\) µs. gpt-fast went from 25.5 to 107 tokens/s on Llama-7B with compile plus CUDA graphs.
    What are the constraints of CUDA graphs, and how do serving engines live with variable batch sizes?advanced
    The graph freezes shapes, memory addresses and the kernel sequence: inputs must be copied into the captured buffers, no host synchronization or Python logic can happen inside, and dynamic allocation (like a growing KV cache) is not allowed, so engines preallocate a static or paged KV cache. For variable batch sizes they capture a graph per bucket (for example 1, 2, 4, then multiples of 8) and pad the batch up to the next bucket (127.6). Padding is cheap in memory-bound decode. vLLM also uses piecewise graphs that leave attention, whose shapes depend on each request’s context, in eager mode.
    Explain how torch.compile works, from Python to kernels.core
    TorchDynamo hooks CPython frame evaluation, traces the bytecode and records the PyTorch ops into an FX graph, together with guards on shapes, dtypes and Python values. AOTAutograd traces it with fake tensors into primitive ops (and builds the backward graph for training). TorchInductor lowers to a loop-level IR, fuses greedily (pointwise, reductions, epilogues, horizontal), and generates Triton kernels for GPUs or C++ for CPUs. The compiled code is cached with its guards; mode="reduce-overhead" adds CUDA graphs, mode="max-autotune" benchmarks kernel choices such as Triton GEMM templates.
    What is a graph break? Give causes, the cost, and how to find and remove them.core
    When Dynamo meets code it cannot trace, it compiles the graph so far, runs that code in Python and resumes tracing in a continuation function. Causes: Python branches on tensor values, .item()/.tolist(), data-dependent shapes (nonzero, boolean indexing), unsupported calls (print, some libraries). Cost: smaller graphs, so fusion cannot cross the break; CPU overhead to return to Python; several CUDA graphs instead of one. Find them with torch._dynamo.explain, TORCH_LOGS="graph_breaks" or fullgraph=True; fix with torch.where, fixed-shape masks, keeping values as tensors, and moving logging outside.
    Your compiled server is fast in a benchmark but slow in production. What do you check?advanced
    First recompilations: real traffic has new prompt lengths, batch sizes or dtypes that fail guards (check TORCH_LOGS="recompiles"); past the recompile limit, functions fall back to eager. Then CUDA graph misses: batches larger than the biggest bucket or shapes that disable graphs run eager. Then graph breaks that appear only on some code paths (for example a logging branch). Fixes: buckets for batch and length, dynamic shapes for prefill, static decode shapes, warm-up of all buckets at start-up, and a cache for compiled artifacts.
    What is the difference between mode="reduce-overhead" and mode="max-autotune"?core
    reduce-overhead keeps the default code generation and wraps the compiled graph in CUDA graphs to remove launch overhead; it helps small-batch, launch-bound workloads like decode, and is disabled automatically when unsafe (dynamic shapes, CPU tensors). It uses extra memory for the graph pools. max-autotune spends much more compile time benchmarking alternative kernels (including Triton GEMM templates with fused epilogues) and keeps the fastest; it helps compute-heavy shapes. Recent versions also use CUDA graphs in max-autotune [Likely].
    How do TensorRT-LLM and DeepSpeed Inference fuse a transformer layer?advanced
    Both keep GEMMs and attention as tuned kernels and fold the small ops around them. TensorRT-LLM’s decode attention is one masked multi-head attention kernel that also adds the QKV bias, applies RoPE and does (de)quantization; prefill uses a fused MHA kernel; TensorRT fuses pointwise chains and GELU at engine build time. DeepSpeed’s Deep-Fusion fuses at tile granularity across reductions, transposes and GEMMs as long as no tile depends on another block’s tile, giving four regions per layer at small batch (input norm + QKV GEMM, transpose + attention, post-attention norm + MLP GEMM, bias + residual), plus CUDA graphs to remove launch overhead.
    GPU utilization is 30% during batch-1 decode. How do you diagnose it?core
    Take a trace (session 123). If the GPU timeline has many tiny kernels with gaps between them while the CPU thread is busy launching, the step is CPU-bound: compare \(K c\) with the weight-read time. Count kernels per layer. Fixes in order: remove graph breaks, fuse (torch.compile or fused kernels), and capture the decode step in a CUDA graph with a static KV cache. If instead kernels are back to back but slow, the problem is kernel efficiency (bandwidth), not overhead.

    Exercises

    1. A bias add followed by GELU runs on a \(4{,}096 \times 16{,}384\) BF16 tensor on an H100 (ignore the bias vector). Compute the bytes and memory time unfused (2 kernels) and fused. With \(t_0 = 3\) µs, what is the speed-up?
    2. A decode step has \(K = 800\) kernels of \(\bar g = 2\) µs each. The CPU needs \(c = 6\) µs per launch; \(\delta = 0.5\) µs and \(G = 10\) µs. Estimate the eager and graph times with formula 127.5 and the speed-up.
    3. Captured buckets are 1, 2, 4 and multiples of 8. What fraction of the step is padding for a batch of 37? For 41?
    4. Compiling takes 90 s. It cuts a decode step from 12 ms to 9 ms. After how many steps does it pay off?
    5. Which lines of this function break the graph, and how would you rewrite them (\(x\) is a 1-D tensor)? def f(x): y = x.relu(); k = int(y.argmax()); return y[:k] if y.mean() > 0 else y
    6. Llama 3.2 1B has 16 layers. How many kernels per decode step at 36 and at 5 kernels per layer? With \(c = 8\) µs, how much CPU launch time is that, compared with the 738 µs weight-read time on an H100?
    Answerscheck
    1. One tensor is \(4{,}096 \times 16{,}384 \times 2 = 134.2\) MB. Unfused: each kernel reads and writes it, so 4 passes: 536.9 MB, 160.3 µs. Fused: 2 passes, 268.4 MB, 80.1 µs. Times with \(t_0\): \(6 + 160.3 = 166.3\) µs vs \(3 + 80.1 = 83.1\) µs, a speed-up of 2.0×. 2. \(Kc = 4{,}800\) µs is more than \(K(\bar g + \delta) = 2{,}000\) µs, so eager is CPU-bound: about 4,802 µs (the last kernel ends 2 µs after the last launch). Graph: \(10 + 800 \times 2.5 - 0.5 = 2{,}009.5\) µs. Speed-up 2.39×. 3. 37 runs as 40: \(3/40 = 7.5\%\). 41 runs as 48: \(7/48 = 14.6\%\). 4. \(n^\ast = 90 / 0.003 = 30{,}000\) steps (about 4.5 minutes of steps at 9 ms). 5. int(y.argmax()) turns a tensor into a Python int (break, reported as an unsupported Tensor.item()), and if y.mean() > 0 branches on a tensor value (break): torch._dynamo.explain in PyTorch 2.8 reports 3 graphs and 2 breaks. The slice y[:k] does not break, because after the first break \(k\) is a plain Python int, but its output shape depends on the data, so new values of \(k\) can cause recompiles [Likely]. Rewrite with fixed shapes: build a mask m = torch.arange(y.numel()) < y.argmax(), return torch.where(y.mean() > 0, y * m, y) (zeros instead of a shorter tensor), or move the slicing out of the compiled region. This version gives 1 graph and 0 breaks, and compiles with fullgraph=True. 6. 576 and 80 kernels. CPU time: 4,608 µs and 640 µs. The eager model is CPU-bound by more than 6× (4,608 vs 738 µs); the fused model is just below the weight-read time, so it is GPU-bound even without a graph.

    Key takeaways

    • Small ops (bias, activation, norm, residual, RoPE) are memory-bound and each has a fixed cost. Fusion keeps intermediates on chip: in prefill it removes bytes, in decode it removes kernels (127.2, 127.3).
    • Two ops fuse if each output tile needs one input tile (127.4): pointwise chains, row reductions, GEMM epilogues and horizontal GEMM merges fuse; GEMM → GEMM does not.
    • Small-batch decode in eager PyTorch is often CPU-bound: \(T \approx Kc\). CUDA graphs replay the whole step with one launch (127.5), but need static shapes, static addresses and no host sync, so engines capture per batch bucket and pad (127.6).
    • torch.compile = TorchDynamo (bytecode to FX graph + guards) + AOTAutograd + TorchInductor (fusion, Triton). reduce-overhead adds CUDA graphs. Graph breaks and recompiles silently take back the gains; use fullgraph=True and logs.
    • Inference engines (TensorRT-LLM, DeepSpeed, ByteTransformer, vLLM) keep GEMMs and attention as tuned kernels, fold everything small into them, and hide the rest behind CUDA graphs.

    Key formulas

    \[ T_{\text{unfused}} = K t_0 + \tfrac{1}{B}\textstyle\sum_k Q_k, \quad T_{\text{fused}} = t_0 + \tfrac{Q_{\text{ext}}}{B}, \quad S_{\text{fuse}} \to K \ \text{or}\ \tfrac{\sum_k Q_k}{Q_{\text{ext}}} \]
    \[ T_{\text{eager}} \approx \max\big(Kc,\ K(\bar g + \delta)\big), \quad T_{\text{graph}} \approx G + K(\bar g + \delta), \quad w = \tfrac{b_{\text{cap}} - b}{b_{\text{cap}}}, \quad n^\ast = \tfrac{T_{\text{compile}}}{t_{\text{eager}} - t_{\text{compiled}}} \]

    References for this session

    1. Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
    2. Zhai et al. (2022). ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs
    3. Aminabadi et al. (2022). DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
    4. NVIDIA (docs). TensorRT documentation
    5. Ansel et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation
    6. Aminabadi et al. (2022). DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
    7. Zhai et al. (2022). ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs
    8. NVIDIA (docs). TensorRT documentation
    9. NVIDIA (code). TensorRT-LLM
    10. NVIDIA (docs). CUDA C++ Programming Guide
    11. PyTorch Team (2023). Accelerating Generative AI with PyTorch II: GPT, Fast
    12. Gray (NVIDIA) (2019). Getting Started with CUDA Graphs
    13. Tillet et al. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations
    14. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
    15. vLLM project (docs). vLLM documentation
    16. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models