GPU kernels 1: CUDA basics for ML engineers
Every PyTorch operation runs on the GPU as a kernel: one small program that thousands of threads run at the same time. This session teaches the CUDA model behind it: threads, warps, blocks and grids, occupancy, the three kinds of memory, coalesced reads and shared-memory bank conflicts. Then it shows why fusing kernels saves time, and how to place any kernel you measure on the roofline.
What you will learn
- How a kernel launch works: the grid of blocks, the block of threads, and the index formula every kernel starts with
- What a warp and a lane are, why threads of a warp should follow the same branch, and what occupancy means
- How to compute occupancy from threads, registers and shared memory per block, with the real H100, A100 and RTX 4090 limits
- Registers, shared memory and global memory; why a warp should read neighboring addresses (coalescing) and how to count wasted bytes
- Shared-memory banks: why reading a column of a 32 × 32 tile is 32 times slower, and how one padding column fixes it
- Why kernel fusion helps: count the HBM bytes of an unfused and a fused bias + GELU + residual chain
- How to place a kernel you measured on the roofline, and what to do when it sits far below the roof
Why this matters
When you call torch.matmul, F.gelu or F.softmax on a GPU tensor, PyTorch does not compute anything itself. It launches a kernel: a small program that runs on the GPU, copied into thousands of threads that each handle a small piece of the data. Every speed question in LLM serving ends inside such kernels.
In session 84 you saw that decoding is limited by how fast the GPU reads the weights from memory, and that batching helps because one read serves many tokens. In session 42 you met the hardware: streaming multiprocessors (SMs), warps, shared memory and HBM, and the roofline. This session is the software side of the same story. How does a kernel decide which thread does which element? Why does the same kernel run at 90% of the memory bandwidth with one memory layout and at 12% with another? Why are FlashAttention, fused RMSNorm and fused MoE kernels the heart of vLLM and TensorRT-LLM? And when your profiler says a kernel took 150 µs, is that good or bad?
You do not need to become a full-time CUDA programmer. But you should be able to read a simple kernel, count its memory traffic, and tell if it is close to the hardware limit. In session 86 you will write real kernels in the Triton language, which hides some details shown here but not the ideas.
The CUDA model: host, device, kernel
CUDA is NVIDIA’s programming model for its GPUs. It uses two words for the two processors: the host is the CPU with its memory, the device is the GPU with its memory. A kernel is a function marked __global__. The host launches it; the device runs it.
When you launch a kernel, you say how many threads to start, in two levels:
- A thread block (or just block) is a group of up to 1,024 threads. [Certain] All threads of one block run on the same SM, can share a fast on-chip memory (shared memory), and can wait for each other with a barrier (
__syncthreads()). - The grid is the set of all blocks of one launch. Blocks are independent: they may run in any order, on any SM, and they cannot wait for each other inside the kernel.
Inside the kernel, each thread reads four built-in variables to learn who it is: threadIdx.x (its number inside the block), blockIdx.x (the number of its block), blockDim.x (threads per block) and gridDim.x (blocks in the grid). From these it computes which element it owns:
To cover \(n\) elements with blocks of \(b\) threads, you launch
- Read the symbols
- \(i\)
- the global index: which element of the array this thread works on
- \(n\)
- the number of elements to process
- \(b\)
- block size =
blockDim.x, threads per block (usually 128, 256 or 512) - \(G\)
- grid size =
gridDim.x, the number of blocks; \(\lceil\cdot\rceil\) rounds up
Because \(G\,b\) is usually a bit larger than \(n\), the last block has threads with nothing to do. That is why almost every kernel starts with a bounds check: if (i < n). Here is the complete “hello world” of CUDA, a vector add:
__global__ void add(const float* x, const float* y, float* z, int n) { int i = blockIdx.x * blockDim.x + threadIdx.x; // formula (85.1) if (i < n) z[i] = x[i] + y[i]; // bounds check for the last block } // host side: n = 1000, b = 256 gives G = 4 blocks, formula (85.2) int b = 256, G = (n + b - 1) / b; add<<<G, b>>>(x, y, z, n); // launch: returns at once, runs asynchronously
Blocks and grids can also be 2-D or 3-D (threadIdx.y, blockIdx.y, …). That is only a convenience for matrices and images: the hardware sees a flat list of threads. A 2-D tile index \((r, c)\) of a row-major matrix with \(C\) columns lives at address \(r \cdot C + c\), so threads with neighboring threadIdx.x should take neighboring \(c\). You will see why in the section on coalescing.
blockIdx, threadIdx, global index (formula 85.1), warp and lane.How to read Figure 1
With \(n = 1000\) and \(b = 256\) you get 4 blocks and 1,024 threads; the last 24 threads are idle. Click a cell in the third block: the global index is \(2 \cdot 256 + \text{threadIdx.x}\). Now compare block sizes 96 and 100. 96 is a multiple of 32 (3 full warps). 100 is not: with 100, every block ends with a warp that has only 4 real threads: the hardware still schedules a full warp of 32, so 28 lanes are wasted in every block, not only at the end. That is why block sizes are multiples of 32.
Warps, lanes and divergence
The SM does not run threads one at a time. It runs them in groups of 32 consecutive threads of a block, called warps. [Certain] Thread \(t\) of a block (with \(t\) = threadIdx.x) belongs to
The lane is the thread’s position inside its warp, from 0 to 31. The 32 lanes of a warp execute the same instruction at the same time, each on its own data (NVIDIA calls this SIMT, single instruction, multiple threads). Two rules follow, and the rest of this session is built on them:
- Divergence. If lanes of one warp take different sides of an
if, the warp runs both sides one after the other, with the lanes of the other side switched off. A branch that depends onthreadIdx.x % 2halves the speed of that code; a branch that is the same for all 32 lanes (for example, it depends on the warp number or the block number) costs nothing extra. Volta and newer GPUs schedule diverged lanes more flexibly, but the inactive lanes still do no useful work. [Likely] - Memory accesses are per warp. When a warp executes a load, the 32 addresses are handled together. Whether those 32 addresses are close to each other decides how many bytes really move (the coalescing section).
Warps are also the unit that hides memory latency. A load from HBM takes hundreds of clock cycles. [Likely] While one warp waits, the warp scheduler issues instructions from another warp that is ready. So an SM needs many warps “in flight” at once. How many it can hold depends on the kernel, and that is occupancy.
Occupancy: how many warps fit on an SM
Each SM has fixed resources, and every resident block takes a share of them. On an H100 (compute capability 9.0) an SM can hold at most 64 warps (2,048 threads), at most 32 blocks, 65,536 registers of 32 bits, and 228 KB of shared memory, of which one block can use at most 227 KB; CUDA also reserves 1 KB of shared memory per block. [Certain] A block of \(T\) threads that uses \(r\) registers per thread and \(s\) KB of shared memory can be placed on the SM this many times:
Then occupancy is the fraction of the SM’s warp slots that are filled:
- Read the symbols
- \(T\)
- threads per block (the block size \(b\) from before)
- \(r\)
- registers per thread, decided by the compiler (you can see it with
nvcc -Xptxas -v); the hardware hands registers out in chunks, so we round \(r\) up to a multiple of 8 [Likely] - \(s\)
- shared memory per block in KB (the
__shared__arrays plus any dynamic shared memory) - 2048, 32, 65536, 228
- the H100 limits per SM: threads, blocks, registers, KB of shared memory. The A100 has 164 KB of shared memory per SM; the RTX 4090 has 48 warps (1,536 threads), 24 blocks and 100 KB per SM [Certain]
A kernel uses \(T = 256\) threads, \(r = 64\) registers per thread and \(s = 48\) KB of shared memory per block. On an H100, the four limits of formula (85.4) are: threads \(\lfloor 2048/256 \rfloor = 8\) blocks; blocks 32; registers \(\lfloor 65536 / (64 \cdot 256) \rfloor = 4\); shared memory \(\lfloor 228 / 49 \rfloor = 4\). The minimum is 4 blocks, so 32 warps of 64: occupancy 50%.
To reach 100% you must fix both limits. Cut the registers to 32 and the shared memory to 24 KB: now the limits are 8, 32, 8 and \(\lfloor 228/25 \rfloor = 9\), so 8 blocks and 100%. Cutting only the shared memory leaves you at 50%, because registers still allow only 4 blocks.
The same kernel (\(r = 64\), \(s = 48\)) on an A100 gets \(\lfloor 164/49 \rfloor = 3\) blocks, 37.5%; on an RTX 4090, \(\lfloor 100/49 \rfloor = 2\) blocks of 48 warp slots, 33.3%.
How to read Figure 2
Start at the worked example (256 threads, 64 registers, 48 KB): two bars tie at 4 blocks. Lower the registers to 32: the shared-memory bar now decides alone. Then lower the shared memory to 24 KB: 100%. Try 1,024 threads with 40 registers: one block needs 40,960 registers, so only one block fits and half the SM is empty. Push the registers to 255: one block of 256 threads, 12.5%. Switch GPUs: the same kernel can be fine on one GPU and starved on another.
Is high occupancy the goal?
No. Occupancy is a means to hide latency, not a score. A kernel can hide latency in another way: each thread issues several independent loads before it needs the first result, or keeps more data in registers and reuses it. Fast matrix-multiplication kernels (cuBLAS, CUTLASS, FlashAttention) often run at low occupancy, with many registers per thread and large shared-memory tiles, because reuse matters more for them than the number of warps. [Likely] The useful rule: if a memory-bound kernel has low occupancy and reaches a low share of the bandwidth, more warps will probably help. If it already moves bytes near the bandwidth limit, more occupancy cannot help.
Three kinds of memory a kernel sees
From inside a kernel, there are three memory spaces that matter. They match the hardware levels from session 42:
| Space | How you write it | Who sees it | Lives in | Lifetime |
|---|---|---|---|---|
| Registers | ordinary local variables: float v = x[i]; | one thread | the SM’s register file (256 KB per SM on H100) | the thread |
| Shared memory | __shared__ float tile[32][33]; | all threads of one block | on-chip SRAM of the SM (up to 228 KB per SM on H100) | the block |
| Global memory | pointers passed to the kernel: x[i] | all threads, all kernels, and the host (by copies) | HBM (80 GB at 3.35 TB/s on H100 SXM), cached in L2 | until freed |
The rule of thumb: read from global memory as few times as possible, and in a good pattern; keep values you reuse in registers; use shared memory when threads of a block need each other’s data (a reduction, a tile of a matrix, a transpose). Shared memory needs a __syncthreads() between the writes of some threads and the reads of others, or a thread may read a value that is not written yet.
Coalesced access: how a warp reads global memory
Global memory is not read byte by byte. On current GPUs, the loads of one warp are served in 32-byte sectors: the hardware collects the 32 addresses of the warp and fetches every 32-byte sector that contains at least one requested byte. [Certain] If the 32 lanes read 32 neighboring 4-byte floats, that is 128 bytes in exactly 4 sectors: nothing is wasted. We call this a coalesced access. If lane \(k\) reads element \(k \cdot S\) (a stride of \(S\) elements), the addresses spread out and more sectors move for the same useful bytes:
- Read the symbols
- \(w\)
- bytes per element: 4 for FP32, 2 for BF16 or FP16
- \(N_{\text{sec}}\)
- the number of different 32-byte sectors that the 32 addresses fall into
- efficiency
- the share of the moved bytes that the kernel really uses; the rest is wasted bandwidth
For FP32 and a stride of 2, the warp touches 8 sectors (256 bytes) to use 128 bytes: 50%, as the CUDA Best Practices Guide states. [Certain] At a stride of 8 or more, every lane hits its own sector: 32 sectors, 1,024 bytes moved, 12.5% efficiency. For BF16 it is even worse: a large stride gives \(2/32 = 6.25\%\). A start address that is not a multiple of 32 bytes costs one extra sector (5 instead of 4 for FP32, 80%).
How to read Figure 3
At stride 1 with FP32, the lanes fill 4 sectors exactly. Move the stride to 2: twice as many sectors for the same data. At stride 8 each lane gets its own sector, and from then on it cannot get worse: 32 sectors, 12.5%. Set the stride back to 1 and move the start offset to 1: the warp now straddles one more sector. Switch to BF16: at stride 1 the warp reads only 64 bytes (2 sectors), so with BF16 each thread should load two or more elements (for example as one 4-byte or 16-byte vector load) to keep the warp’s requests large.
Where strided access comes from in ML
The usual cause is a transposed or permuted tensor. A PyTorch tensor of shape \((M, N)\) is stored row by row. If a kernel puts neighboring threads on neighboring rows (for example to sum each column), lane \(k\) reads address \(k N\): stride \(N\), the worst case. x.t(), x.permute(...) and slicing every second element create such views without moving data. Many PyTorch kernels then call .contiguous() internally (an extra copy), or run a slow strided path. The fix in your own kernel: map threadIdx.x to the last (fastest-changing) dimension, or stage the data through shared memory (next section).
Shared memory and bank conflicts
Shared memory is fast, but it has its own access rule. It is split into 32 banks; consecutive 4-byte words go to consecutive banks, so
for a byte address \(a\) inside shared memory. [Certain] Each bank can serve one 4-byte word per cycle. If the 32 lanes of a warp access 32 words in 32 different banks, the access takes one pass. If several lanes need different words in the same bank, the accesses are serialized: a \(k\)-way bank conflict takes \(k\) passes. If several lanes read the same word, the value is broadcast to all of them and there is no conflict. [Certain]
The classic case is a \(32 \times 32\) tile of floats, __shared__ float tile[32][32]. Element \((r, c)\) is word \(32r + c\), so its bank is \(c\). When a warp reads a row (lane \(k\) reads \((0, k)\)), the banks are \(0, 1, \dots, 31\): perfect. When it reads a column (lane \(k\) reads \((k, 0)\)), every word is in bank 0: a 32-way conflict, 32 times slower. Matrix transposes, and many attention and matmul kernels, read tiles by columns.
The fix is one extra column: tile[32][33]. Now \((r, c)\) is word \(33r + c\), and its bank is \((33r + c) \bmod 32 = (r + c) \bmod 32\). A column read hits banks \(0, 1, \dots, 31\) again. We waste 32 floats (128 bytes) of shared memory and remove the conflict. Mark Harris’s transpose article shows this change bringing a shared-memory transpose to about 95% of the speed of a plain copy. [Certain]
How to read Figure 4
With the unpadded tile, “read a column” stacks all 32 words in bank 0: 32 passes. Click the padded tile: the stack spreads into 32 banks of height 1. “Every 2nd word” gives a 2-way conflict for any tile width, because even word numbers only hit even banks. “All lanes, one word” is a broadcast: 32 lanes, one word, one pass.
A real kernel: softmax over the rows of a matrix
Attention scores, logits and router outputs all need a softmax over each row. A good beginner kernel gives one block to each row. The threads of the block walk along the row with a stride of blockDim.x, so that at each step the 32 lanes of a warp read 32 neighboring numbers (coalesced). The block then needs two reductions: the maximum of the row (for numerical safety) and the sum of the exponentials.
A reduction inside a warp does not need shared memory. The instruction __shfl_down_sync(mask, v, o) lets lane \(k\) read the register v of lane \(k + o\). Five rounds with \(o = 16, 8, 4, 2, 1\) combine 32 values into lane 0, since \(2^5 = 32\). The warps then meet through a small shared array, one slot per warp.
- input: a row \(x\) of \(N\) numbers in global memory; one block of \(T\) threads (\(T\) a multiple of 32)
- each thread: \(v \leftarrow\) identity (0 for sum, \(-\infty\) for max)
- for \(j = \text{threadIdx.x};\ j \lt N;\ j \mathrel{+}= T\) do \(v \leftarrow v \oplus x_j\) // strided walk: each warp reads 32 neighbors
- for \(o = 16, 8, 4, 2, 1\) do \(v \leftarrow v \oplus \text{shfl\_down}(v, o)\) // inside the warp, registers only
- if \(\text{lane} = 0\) then \(\text{red}[\text{warp}] \leftarrow v\) // one value per warp into shared memory
__syncthreads()// wait until every warp has written- if \(\text{warp} = 0\) then \(v \leftarrow \text{red}[\text{lane}]\) if \(\text{lane} \lt T/32\), else identity; repeat step 4; lane 0 writes \(\text{red}[0]\)
__syncthreads(); every thread reads the result \(\text{red}[0]\)
__device__ float warp_max(float v) { for (int o = 16; o > 0; o >>= 1) v = fmaxf(v, __shfl_down_sync(0xffffffff, v, o)); return v; } __device__ float warp_sum(float v) { for (int o = 16; o > 0; o >>= 1) v += __shfl_down_sync(0xffffffff, v, o); return v; } // one block per row; launch with softmax_rows<<<M, 256>>>(x, y, N) __global__ void softmax_rows(const float* x, float* y, int N) { const float* xr = x + (long)blockIdx.x * N; float* yr = y + (long)blockIdx.x * N; __shared__ float red[32]; // one slot per warp (at most 1024/32) int lane = threadIdx.x % 32, wid = threadIdx.x / 32, nw = blockDim.x / 32; // formula (85.3) float m = -INFINITY; // 1) row maximum, Algorithm 1 with max for (int j = threadIdx.x; j < N; j += blockDim.x) m = fmaxf(m, xr[j]); m = warp_max(m); if (lane == 0) red[wid] = m; __syncthreads(); if (wid == 0) { m = warp_max(lane < nw ? red[lane] : -INFINITY); if (lane == 0) red[0] = m; } __syncthreads(); m = red[0]; __syncthreads(); // last barrier: red[] is reused below float s = 0.f; // 2) sum of exp(x - max), Algorithm 1 with sum for (int j = threadIdx.x; j < N; j += blockDim.x) s += __expf(xr[j] - m); s = warp_sum(s); if (lane == 0) red[wid] = s; __syncthreads(); if (wid == 0) { s = warp_sum(lane < nw ? red[lane] : 0.f); if (lane == 0) red[0] = s; } __syncthreads(); float inv = 1.f / red[0]; for (int j = threadIdx.x; j < N; j += blockDim.x) yr[j] = __expf(xr[j] - m) * inv; // 3) write }
This kernel reads each row three times. For rows of a few thousand floats, the second and third reads usually hit the L1 or L2 cache, so HBM sees close to one read and one write. [Likely] For very long rows you would keep the row in registers or shared memory, or use the online softmax trick, which computes the maximum and the sum in a single pass. That trick is the core of FlashAttention (session 124).
Why fusion helps: fewer trips to HBM
Most operations in a transformer besides the matrix multiplications are elementwise (bias add, GELU or SiLU, residual add, scaling, dropout, type casts) or small reductions (RMSNorm, softmax). They do a few FLOPs per element. From session 42, their arithmetic intensity is around 1 FLOP per byte or less, far below the H100 ridge point of 295. They are memory-bound: their time is the bytes they move divided by the bandwidth.
In eager PyTorch, each operation is its own kernel. Each kernel reads its inputs from HBM and writes its output to HBM, and the next kernel reads that output back. Kernel fusion does the whole chain in one kernel: each thread loads its element once, applies all operations in registers, and writes only the final result. The math is identical; only the memory traffic changes. For a chain of operations on \(n\) elements of \(w\) bytes:
- Read the symbols
- \(K\)
- the number of operations in the chain
- \(a_k\)
- how many full-size tensors operation \(k\) reads (1 for GELU, 2 for a residual add); the \(+1\) is its output. Small inputs such as a bias vector are ignored
- \(a_{\text{ext}}\)
- the number of full-size tensors the whole chain reads from outside (the first input, plus a residual if there is one)
- \(Q\)
- bytes moved between HBM and the chip; for a memory-bound kernel the time is about \(Q / B\), with \(B\) the bandwidth
Take the MLP output chain \(y = \text{GELU}(x + \text{bias}) + r\) on \(M = 4096\) tokens and width 16,384, in BF16: \(n = 4096 \times 16384 = 67{,}108{,}864\) elements, \(w = 2\) bytes.
Unfused, three kernels: the bias add reads \(x\) and writes \(t_1\) (2 tensors), GELU reads \(t_1\) and writes \(t_2\) (2), the residual add reads \(t_2\) and \(r\) and writes \(y\) (3). Total \(7 n w = 939.5\) MB, which takes at least \(939.5\ \text{MB} / 3.35\ \text{TB/s} = 280\) µs on an H100.
Fused, one kernel: read \(x\) and \(r\), write \(y\): \(3 n w = 402.7\) MB, at least 120 µs. The same math, \(7/3 = 2.33\) times less traffic, and two fewer kernel launches. The intermediate tensors \(t_1\) and \(t_2\) (134 MB each) are also never allocated.
How to read Figure 5
Each colored segment of the top bar is one kernel of the unfused chain: its reads and its write. The bottom bar is the fused kernel. Add operations: the unfused bar grows by \(2nw\) per unary operation, the fused bar does not grow at all. With all five operations selected, fusion cuts traffic by \(11/3 = 3.7\times\). Move \(M\) down to 1: the bytes become tiny (98 KB fused), and the time is then dominated by the few microseconds of each kernel launch, a second reason to fuse (session 50).
Fusion has limits. A matrix multiplication followed by an elementwise operation can be fused as an epilogue (the GEMM applies the bias and activation to its output tile before writing it), and cuBLASLt and CUTLASS offer this. [Certain] Two large reductions over different dimensions are hard to fuse. And fusion can raise register use and lower occupancy. In practice you get most fusions from torch.compile, which generates fused Triton kernels for elementwise and reduction chains, and from hand-written kernels for the important patterns: FlashAttention, fused RMSNorm with residual, fused rotary embedding, fused MoE routing, and fused dequantize-plus-GEMV for quantized decoding. [Likely]
- input: pointers to \(x, r, y\) (each \(n\) elements) and a bias of \(d\) elements; grid of \(G\) blocks of \(T\) threads
- \(i \leftarrow \text{blockIdx.x} \cdot T + \text{threadIdx.x}\) // formula (85.1)
- while \(i \lt n\) do
- \(v \leftarrow x_i + \text{bias}_{i \bmod d}\) // one coalesced read of x; bias stays in cache
- \(g \leftarrow \text{GELU}(v)\) // registers only, nothing written to HBM
- \(y_i \leftarrow g + r_i\) // one read of r, one write of y
- \(i \leftarrow i + G \cdot T\) // grid-stride: a fixed grid covers any n
The grid-stride loop lets you launch a fixed number of blocks (for example a few per SM) whatever the size of the tensor. Each thread handles elements \(i, i + GT, i + 2GT, \dots\), and at every step the warp still reads 32 neighbors.
Placing a kernel on the roofline
You wrote or profiled a kernel and measured its time \(t\). Is it good? The roofline answers this with three numbers you can compute on paper: the FLOPs \(F\) it must do, the bytes \(Q\) it must move to and from HBM (count them as in formula 85.8, assuming each byte crosses once), and the hardware peaks \(P\) and \(B\). Then:
- Read the symbols
- \(I\)
- arithmetic intensity in FLOP per byte: the x-position of the kernel on the roofline plot
- \(t_{\min}\)
- the fastest the kernel could possibly run on this GPU (formula 42.5 of session 42)
- \(F/t\), \(Q/t\)
- achieved FLOP/s (the y-position on the plot) and achieved bandwidth
- \(P\), \(B\)
- peak FLOP/s for the units the kernel uses (989 TFLOP/s dense BF16 on tensor cores, 67 TFLOP/s for ordinary FP32 math on H100 SXM) and peak bandwidth (3.35 TB/s) [Certain]
The fused bias + GELU + residual kernel from above moves \(Q = 402.7\) MB. Count about 10 FLOPs per element (1 for the bias, about 8 for the tanh form of GELU, 1 for the residual; the exact count does not change the answer): \(F = 6.71 \times 10^8\). So \(I = F/Q = 1.67\) FLOP/byte: far left of the ridge, memory-bound. \(t_{\min} = Q/B = 120\) µs.
Suppose you measure \(t = 150\) µs (an example value). Achieved bandwidth \(= 402.7\ \text{MB} / 150\ \mu\text{s} = 2.68\) TB/s, which is 80% of 3.35 TB/s. Achieved compute is only 4.47 TFLOP/s, but that does not matter: the kernel is limited by bytes, and it is at 80% of its roof. Memory-bound kernels usually top out somewhat below the peak bandwidth, so this is a good kernel. [Likely] There is little left to gain by tuning it; the only big win is to move fewer bytes (fuse more, or use smaller number formats).
- count \(F\) and \(Q\) from the shapes, assuming each input byte is read once and each output byte written once
- time the kernel with CUDA events after warm-up, or read its duration from Nsight Systems or the PyTorch profiler
- compute \(I\), \(t_{\min}\), \(F/t\), \(Q/t\) and the fraction of roof (formulas 85.9, 85.10)
- if \(I \lt P/B\) (memory-bound) then
- if fraction is high (say above 70%): done; only fewer bytes help (fusion, BF16 or FP8, avoid copies)
- else: look for uncoalesced access (formula 85.6), too few blocks or warps (85.4), register spills, or extra traffic from re-reads
- else (compute-bound): check that the math runs on tensor cores (right dtype, shapes that are multiples of 8 or 16), then tiling, bank conflicts (85.7) and occupancy
How to read Figure 6
Every elementwise kernel, the softmax and the batch-1 decode GEMV sit near \(I \approx 1\), far left of both ridges: they can only be judged by bandwidth. The GEMM sits far right at \(I = 1365\): it is judged by tensor-core FLOP/s. Slide the measured time up: the orange point drops straight down, because \(I\) depends only on the work, not on how well you did it. Compare “unfused chain” and “fused chain”: fusion does not move a kernel up the slope by making it run better; it moves it to the right by removing bytes, and the roof above it is higher there.
Calling your own kernel from PyTorch
The code below collects the session in one script. Part 1 implements formulas (85.2), (85.4), (85.6) and (85.7) as plain Python and prints the numbers from the figures. Part 2 counts the bytes of the fused chain (85.8). Part 3 compiles the fused kernel of Algorithm 2 with load_inline from torch.utils.cpp_extension, checks it against PyTorch, and times both with CUDA events. Parts 1 and 2 run anywhere; part 3 needs an NVIDIA GPU with the CUDA toolkit (nvcc), and its printed times depend on your GPU.
import math, torch torch.manual_seed(0) # 1) The rules of this session as small functions def grid_size(n, block): # formula (85.2) return (n + block - 1) // block def occupancy(T, regs, smem_kb, max_threads=2048, max_blocks=32, regs_sm=65536, smem_sm_kb=228, reserved_kb=1): # H100 limits, formula (85.4) r = math.ceil(regs / 8) * 8 # registers are handed out in chunks lim = {"threads": max_threads // T, "blocks": max_blocks, "registers": regs_sm // (r * T), "shared": smem_sm_kb // (smem_kb + reserved_kb) if smem_kb > 0 else max_blocks} b = min(lim.values()) return b, b * T / max_threads, min(lim, key=lim.get) def sectors(stride, elem=4, offset=0): # 32-byte sectors touched by one warp s = {(offset + lane * stride * elem + k) // 32 for lane in range(32) for k in range(elem)} return len(s), 32 * elem / (32 * len(s)) # formula (85.6): useful / moved def bank_conflict(word_idx): # formula (85.7), words of 4 bytes banks = {} for w in set(word_idx): # the same word twice is a broadcast banks.setdefault(w % 32, set()).add(w) return max(len(v) for v in banks.values()) print(grid_size(1000, 256), occupancy(256, 64, 48)) # 4 (4, 0.5, 'registers') print(sectors(1), sectors(2), sectors(8)) # (4, 1.0) (8, 0.5) (32, 0.125) print(bank_conflict([r * 32 for r in range(32)]), # column of a 32x32 tile: 32 bank_conflict([r * 33 for r in range(32)])) # padded to 33 columns: 1 # 2) HBM bytes of y = gelu(x + b) + r: unfused (3 kernels) vs fused (1 kernel), formula (85.8) n, s = 4096 * 16384, 2 # BF16 unfused = (2 + 2 + 3) * n * s # add: x->t1, gelu: t1->t2, add: t2,r->y fused = 3 * n * s # read x and r once, write y once print(unfused / 1e6, fused / 1e6, unfused / 3.35e12 * 1e6, fused / 3.35e12 * 1e6) # 939.5 MB, 402.7 MB, 280 us, 120 us on an H100 # 3) The fused kernel (Algorithm 2) in CUDA C++, compiled from Python cuda_src = r""" __global__ void bias_gelu_res(const float* __restrict__ x, const float* __restrict__ b, const float* __restrict__ r, float* __restrict__ y, long n, int d) { for (long i = blockIdx.x * (long)blockDim.x + threadIdx.x; i < n; i += (long)blockDim.x * gridDim.x) { // grid-stride loop float v = x[i] + b[i % d]; // neighbors read neighbors: coalesced float g = 0.5f * v * (1.f + tanhf(0.7978845608f * (v + 0.044715f * v * v * v))); y[i] = g + r[i]; // one write, no temporaries in HBM } } torch::Tensor fused(torch::Tensor x, torch::Tensor b, torch::Tensor r) { TORCH_CHECK(x.is_cuda() && x.is_contiguous() && r.is_contiguous()); auto y = torch::empty_like(x); long n = x.numel(); int d = b.numel(); int threads = 256; long blocks = std::min<long>((n + threads - 1) / threads, 132L * 8); bias_gelu_res<<<blocks, threads>>>(x.data_ptr<float>(), b.data_ptr<float>(), r.data_ptr<float>(), y.data_ptr<float>(), n, d); return y; } """ def reference(x, b, r): return torch.nn.functional.gelu(x + b, approximate="tanh") + r # 3 kernels in eager mode if torch.cuda.is_available(): from torch.utils.cpp_extension import load_inline ext = load_inline("fused_ext", cuda_sources=cuda_src, functions=["fused"], cpp_sources="torch::Tensor fused(torch::Tensor x, torch::Tensor b, torch::Tensor r);") M, d = 4096, 16384 x, r = torch.randn(M, d, device="cuda"), torch.randn(M, d, device="cuda") b = torch.randn(d, device="cuda") print((ext.fused(x, b, r) - reference(x, b, r)).abs().max().item()) # tiny, about 1e-6 for name, fn in [("eager", reference), ("fused", ext.fused)]: for _ in range(3): fn(x, b, r) # warm-up e0, e1 = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True) e0.record(); [fn(x, b, r) for _ in range(20)]; e1.record() torch.cuda.synchronize() # launches are async: wait before reading t = e0.elapsed_time(e1) / 20 * 1e-3 # seconds per call q = 3 * x.numel() * 4 # bytes the fused kernel must move (FP32) print(name, f"{t*1e6:.0f} us, {q/t/1e12:.2f} TB/s of useful traffic") else: x, b, r = torch.randn(8, 16), torch.randn(16), torch.randn(8, 16) v = x + b mine = 0.5 * v * (1 + torch.tanh(0.7978845608 * (v + 0.044715 * v ** 3))) + r print((mine - reference(x, b, r)).abs().max().item()) # the kernel's math matches PyTorch
Two details in this code matter in real work. The index uses long: a tensor with more than \(2^{31} - 1\) elements (for example \(32768 \times 65536\)) overflows a 32-bit int and silently writes to wrong addresses. And the timing records CUDA events and synchronizes; timing a kernel launch with Python’s time.time() alone measures only the launch, because launches are asynchronous.
Where this is used
| Where | Which idea from this session | Why it matters |
|---|---|---|
| FlashAttention in every serving engine | tiles in shared memory, online softmax, fusion of QKᵀ, softmax and PV | the \(T \times T\) score matrix never goes to HBM (session 124) |
| vLLM, TensorRT-LLM, SGLang custom kernels | fused RMSNorm + residual, fused rotary embedding, fused MoE, PagedAttention | decode is memory-bound; every saved byte is saved time [Likely] |
| Quantized decoding (INT4 or FP8 weights) | fused dequantize + GEMV; coalesced, vectorized loads of packed weights | fewer bytes per weight moves the GEMV right on the roofline (session 82) |
| torch.compile and Triton | automatic fusion of elementwise and reduction chains | you get formula (85.8) savings without writing CUDA (session 86) |
| Video preprocessing for VLMs and video classification | one fused kernel for decode output → resize → normalize → patchify, coalesced over pixels | at many frames per clip, separate kernels per step re-read every frame from HBM [Likely] |
| Video retrieval, Video-RAG and fingerprinting | fused similarity + top-k kernels over large embedding or hash tables | scoring millions of clip vectors is memory-bound; never write the full score matrix [Likely] |
| Profiling any training or serving job | placing kernels on the roofline (Algorithm 3), occupancy from Nsight Compute | tells you whether to optimize a kernel or remove it |
Common mistakes
- No bounds check. The last block reads and writes past the end of the array. Sometimes it crashes, often it silently corrupts another tensor. Always guard with
if (i < n)or a grid-stride loop. - Block size not a multiple of 32. A block of 100 threads still occupies 4 full warps; 28 lanes per block do nothing (Figure 1).
- A missing
__syncthreads()between writing shared memory and reading what other threads wrote, or before reusing a shared buffer. The result is wrong only sometimes, which makes it hard to find. A barrier inside a branch that not all threads of the block reach can hang the kernel. - Threads mapped to the wrong dimension. Neighboring
threadIdx.xon neighboring rows gives stride-\(N\) reads at 12.5% efficiency for FP32 (Figure 3). MapthreadIdx.xto the contiguous dimension, and pass contiguous tensors. - Ignoring bank conflicts in column reads of a shared tile. Pad the tile to 33 columns (Figure 4).
- Chasing 100% occupancy. More warps help only if latency is the problem. Measure the fraction of the roof first (Algorithm 3).
- 32-bit index overflow on tensors with more than about 2.1 billion elements. Use 64-bit indices for large tensors.
- Timing without synchronization, or timing the first call (which includes compilation and memory allocation). Warm up, then use CUDA events or a profiler.
Interview questions
What are threads, warps, blocks and grids, and what does each level guarantee?basic
__syncthreads(). The grid is all blocks of a launch; blocks are independent, run in any order and cannot synchronize with each other inside the kernel. Each thread finds its element with blockIdx.x * blockDim.x + threadIdx.x.What is warp divergence, and when does a branch cost nothing?basic
What limits occupancy? Compute it for 256 threads, 64 registers, 48 KB shared memory on an H100.core
Is higher occupancy always faster?core
What is coalesced global memory access? Give the efficiency of a stride-2 and a stride-32 FP32 read.core
What is a shared-memory bank conflict, and how does padding fix the transpose case?core
float tile[32][32], element \((r, c)\) is in bank \(c\), so reading a column puts all 32 lanes in one bank: 32 passes. With tile[32][33] the bank is \((r + c) \bmod 32\), so a column spreads over all 32 banks. Lanes that read the same word get a broadcast, which is not a conflict.Why does kernel fusion help if it does exactly the same math?basic
Your fused kernel takes 150 µs and moves 400 MB on an H100. Is it good? What next?core
How does a block-wide reduction work in a softmax kernel?advanced
__shfl_down_sync steps (offsets 16, 8, 4, 2, 1), with no shared memory. Lane 0 of each warp writes its result to a small shared array; after __syncthreads(), the first warp reduces those values the same way and broadcasts the final result through shared memory. Softmax does this twice (max, then sum of exponentials), with a barrier before the shared buffer is reused. Online softmax merges both into one pass.Why do GEMM kernels need tiling but elementwise kernels do not?advanced
A colleague times a kernel with time.time() around the call and reports 8 µs for a 1 GB copy. What is wrong?basic
torch.cuda.synchronize(), warm up first, average several runs, or read the kernel duration from a profiler.Exercises
- You process \(n = 10^6\) elements with blocks of 256 threads. How many blocks do you launch, and how many threads are idle? Which block and lane handle element 777,777?
- On an H100, a kernel uses 128 threads per block, 96 registers per thread and no shared memory. What is its occupancy, and which limit decides?
- A BF16 kernel reads with a stride of 4 elements. What is its coalescing efficiency? What if each thread instead reads 8 neighboring BF16 values (16 bytes) with one vector load, and lanes are 16 bytes apart?
- A shared tile is
float tile[32][64]. A warp reads column 5 (lane \(k\) reads \((k, 5)\)). What is the conflict degree? Which padding fixes it? - Use formula (85.8) for the chain: scale, add bias, SiLU, multiply by another tensor \(g\) (as in a SwiGLU gate), on \(n\) elements of 2 bytes. Count unfused and fused bytes.
- A decode GEMV with a \(4096 \times 4096\) BF16 weight takes 14 µs on an H100. Place it on the roofline: \(I\), \(t_{\min}\), the fraction of the roof.
Answerscheck
threadIdx.x \(= 777777 - 3038 \cdot 256 = 49\), so warp 1, lane 17. 2. Registers: \(65536 / (96 \cdot 128) = 5.33\), so 5 blocks; threads allow 16, blocks 32. \(5\) blocks \(\times\ 4\) warps \(= 20\) of 64 warps: 31.25%, limited by registers. 3. Lanes are \(4 \times 2 = 8\) bytes apart: 32 lanes span 256 bytes = 8 sectors for 64 useful bytes: 25%. With 16-byte vector loads per lane, the warp reads 512 contiguous bytes in 16 sectors: 100%. 4. Element \((k, 5)\) is word \(64k + 5\), bank 5 for every \(k\): 32-way. Pad to tile[32][65]: bank \((65k + 5) \bmod 32 = (k + 5) \bmod 32\), all different. 5. Unfused: scale 2, bias 2, SiLU 2, multiply 3, total \(9nw = 18n\) bytes. Fused: read \(x\) and \(g\), write \(y\): \(3nw = 6n\) bytes, 3× less. 6. \(F = 2 \cdot 4096^2 = 3.36 \times 10^7\), \(Q \approx 2 \cdot 4096^2 = 3.36 \times 10^7\) bytes (the vectors are tiny), \(I \approx 1\). \(t_{\min} = 33.6\ \text{MB}/3.35\ \text{TB/s} = 10.0\) µs. Fraction \(= 10.0/14 = 72\%\) of the memory roof (2.40 TB/s).Key takeaways
- A kernel runs as a grid of independent blocks of up to 1,024 threads; each thread finds its element with formula (85.1), and the launch needs \(\lceil n/b \rceil\) blocks plus a bounds check (85.2).
- Threads run in warps of 32 lanes (85.3). Keep branches uniform per warp and block sizes a multiple of 32.
- Occupancy (85.4, 85.5) is set by the tightest of four limits: threads, blocks, registers and shared memory. It hides latency but is not a goal in itself.
- Global memory moves 32-byte sectors per warp: neighboring lanes must read neighboring addresses (85.6). Shared memory has 32 banks: pad tiles to avoid column conflicts (85.7).
- Elementwise chains are memory-bound; fusion cuts HBM bytes (85.8), for bias + GELU + residual from \(7nw\) to \(3nw\).
- To judge any kernel: count \(F\) and \(Q\), compute \(t_{\min}\), and report the fraction of the roof (85.9, 85.10). Memory-bound and near the roof means: move fewer bytes.
Key formulas
References for this session
- NVIDIA (docs). CUDA C++ Programming Guide
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- NVIDIA (docs). CUDA C++ Best Practices Guide
- NVIDIA (docs). NVIDIA Hopper Tuning Guide (H100 occupancy and shared-memory limits)
- NVIDIA (docs). NVIDIA Ampere GPU Architecture Tuning Guide (A100 limits)
- NVIDIA (docs). NVIDIA Ada GPU Architecture Tuning Guide (RTX 4090 limits)
- Hwu, Kirk & El Hajj (2022). Programming Massively Parallel Processors: A Hands-on Approach (4th edition)
- Harris (2013). An Efficient Matrix Transpose in CUDA C/C++ (NVIDIA technical blog)