LLM & VLM courseBlog
85Track 5: LLM inference and serving · session 8 of 65

GPU kernels 1: CUDA basics for ML engineers

Every PyTorch operation runs on the GPU as a kernel: one small program that thousands of threads run at the same time. This session teaches the CUDA model behind it: threads, warps, blocks and grids, occupancy, the three kinds of memory, coalesced reads and shared-memory bank conflicts. Then it shows why fusing kernels saves time, and how to place any kernel you measure on the roofline.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 42 and 84 (GPU architecture for ML; batching basics). You can read C-like code.

What you will learn

  • How a kernel launch works: the grid of blocks, the block of threads, and the index formula every kernel starts with
  • What a warp and a lane are, why threads of a warp should follow the same branch, and what occupancy means
  • How to compute occupancy from threads, registers and shared memory per block, with the real H100, A100 and RTX 4090 limits
  • Registers, shared memory and global memory; why a warp should read neighboring addresses (coalescing) and how to count wasted bytes
  • Shared-memory banks: why reading a column of a 32 × 32 tile is 32 times slower, and how one padding column fixes it
  • Why kernel fusion helps: count the HBM bytes of an unfused and a fused bias + GELU + residual chain
  • How to place a kernel you measured on the roofline, and what to do when it sits far below the roof

Why this matters

When you call torch.matmul, F.gelu or F.softmax on a GPU tensor, PyTorch does not compute anything itself. It launches a kernel: a small program that runs on the GPU, copied into thousands of threads that each handle a small piece of the data. Every speed question in LLM serving ends inside such kernels.

In session 84 you saw that decoding is limited by how fast the GPU reads the weights from memory, and that batching helps because one read serves many tokens. In session 42 you met the hardware: streaming multiprocessors (SMs), warps, shared memory and HBM, and the roofline. This session is the software side of the same story. How does a kernel decide which thread does which element? Why does the same kernel run at 90% of the memory bandwidth with one memory layout and at 12% with another? Why are FlashAttention, fused RMSNorm and fused MoE kernels the heart of vLLM and TensorRT-LLM? And when your profiler says a kernel took 150 µs, is that good or bad?

You do not need to become a full-time CUDA programmer. But you should be able to read a simple kernel, count its memory traffic, and tell if it is close to the hardware limit. In session 86 you will write real kernels in the Triton language, which hides some details shown here but not the ideas.

The CUDA model: host, device, kernel

CUDA is NVIDIA’s programming model for its GPUs. It uses two words for the two processors: the host is the CPU with its memory, the device is the GPU with its memory. A kernel is a function marked __global__. The host launches it; the device runs it.

When you launch a kernel, you say how many threads to start, in two levels:

  • A thread block (or just block) is a group of up to 1,024 threads. [Certain] All threads of one block run on the same SM, can share a fast on-chip memory (shared memory), and can wait for each other with a barrier (__syncthreads()).
  • The grid is the set of all blocks of one launch. Blocks are independent: they may run in any order, on any SM, and they cannot wait for each other inside the kernel.

Inside the kernel, each thread reads four built-in variables to learn who it is: threadIdx.x (its number inside the block), blockIdx.x (the number of its block), blockDim.x (threads per block) and gridDim.x (blocks in the grid). From these it computes which element it owns:

\[ i = \text{blockIdx.x} \cdot \text{blockDim.x} + \text{threadIdx.x} \tag{85.1} \]

To cover \(n\) elements with blocks of \(b\) threads, you launch

\[ G = \left\lceil \frac{n}{b} \right\rceil \text{ blocks}, \qquad \text{idle threads} = G\, b - n \tag{85.2} \]
Read the symbols
\(i\)
the global index: which element of the array this thread works on
\(n\)
the number of elements to process
\(b\)
block size = blockDim.x, threads per block (usually 128, 256 or 512)
\(G\)
grid size = gridDim.x, the number of blocks; \(\lceil\cdot\rceil\) rounds up

Because \(G\,b\) is usually a bit larger than \(n\), the last block has threads with nothing to do. That is why almost every kernel starts with a bounds check: if (i < n). Here is the complete “hello world” of CUDA, a vector add:

__global__ void add(const float* x, const float* y, float* z, int n) {
  int i = blockIdx.x * blockDim.x + threadIdx.x;   // formula (85.1)
  if (i < n) z[i] = x[i] + y[i];                   // bounds check for the last block
}
// host side: n = 1000, b = 256 gives G = 4 blocks, formula (85.2)
int b = 256, G = (n + b - 1) / b;
add<<<G, b>>>(x, y, z, n);                         // launch: returns at once, runs asynchronously

Blocks and grids can also be 2-D or 3-D (threadIdx.y, blockIdx.y, …). That is only a convenience for matrices and images: the hardware sees a flat list of threads. A 2-D tile index \((r, c)\) of a row-major matrix with \(C\) columns lives at address \(r \cdot C + c\), so threads with neighboring threadIdx.x should take neighboring \(c\). You will see why in the section on coalescing.

Block size \(b\):
active thread (has an element)idle thread (fails the bounds check)the thread you clickedempty lane (block size not a multiple of 32)
Figure 1. A 1-D launch, thread by thread. Each row is one warp of 32 threads; a colored band groups the warps of one block. Click any cell to see its blockIdx, threadIdx, global index (formula 85.1), warp and lane.
How to read Figure 1

With \(n = 1000\) and \(b = 256\) you get 4 blocks and 1,024 threads; the last 24 threads are idle. Click a cell in the third block: the global index is \(2 \cdot 256 + \text{threadIdx.x}\). Now compare block sizes 96 and 100. 96 is a multiple of 32 (3 full warps). 100 is not: with 100, every block ends with a warp that has only 4 real threads: the hardware still schedules a full warp of 32, so 28 lanes are wasted in every block, not only at the end. That is why block sizes are multiples of 32.

Warps, lanes and divergence

The SM does not run threads one at a time. It runs them in groups of 32 consecutive threads of a block, called warps. [Certain] Thread \(t\) of a block (with \(t\) = threadIdx.x) belongs to

\[ \text{warp} = \left\lfloor \frac{t}{32} \right\rfloor, \qquad \text{lane} = t \bmod 32 \tag{85.3} \]

The lane is the thread’s position inside its warp, from 0 to 31. The 32 lanes of a warp execute the same instruction at the same time, each on its own data (NVIDIA calls this SIMT, single instruction, multiple threads). Two rules follow, and the rest of this session is built on them:

  1. Divergence. If lanes of one warp take different sides of an if, the warp runs both sides one after the other, with the lanes of the other side switched off. A branch that depends on threadIdx.x % 2 halves the speed of that code; a branch that is the same for all 32 lanes (for example, it depends on the warp number or the block number) costs nothing extra. Volta and newer GPUs schedule diverged lanes more flexibly, but the inactive lanes still do no useful work. [Likely]
  2. Memory accesses are per warp. When a warp executes a load, the 32 addresses are handled together. Whether those 32 addresses are close to each other decides how many bytes really move (the coalescing section).

Warps are also the unit that hides memory latency. A load from HBM takes hundreds of clock cycles. [Likely] While one warp waits, the warp scheduler issues instructions from another warp that is ready. So an SM needs many warps “in flight” at once. How many it can hold depends on the kernel, and that is occupancy.

Occupancy: how many warps fit on an SM

Each SM has fixed resources, and every resident block takes a share of them. On an H100 (compute capability 9.0) an SM can hold at most 64 warps (2,048 threads), at most 32 blocks, 65,536 registers of 32 bits, and 228 KB of shared memory, of which one block can use at most 227 KB; CUDA also reserves 1 KB of shared memory per block. [Certain] A block of \(T\) threads that uses \(r\) registers per thread and \(s\) KB of shared memory can be placed on the SM this many times:

\[ n_{\text{blocks}} = \min\left( \left\lfloor \frac{2048}{T} \right\rfloor,\ 32,\ \left\lfloor \frac{65536}{r\, T} \right\rfloor,\ \left\lfloor \frac{228}{s + 1} \right\rfloor \right) \tag{85.4} \]

Then occupancy is the fraction of the SM’s warp slots that are filled:

\[ \text{occupancy} = \frac{n_{\text{blocks}} \cdot T / 32}{64} \tag{85.5} \]
Read the symbols
\(T\)
threads per block (the block size \(b\) from before)
\(r\)
registers per thread, decided by the compiler (you can see it with nvcc -Xptxas -v); the hardware hands registers out in chunks, so we round \(r\) up to a multiple of 8 [Likely]
\(s\)
shared memory per block in KB (the __shared__ arrays plus any dynamic shared memory)
2048, 32, 65536, 228
the H100 limits per SM: threads, blocks, registers, KB of shared memory. The A100 has 164 KB of shared memory per SM; the RTX 4090 has 48 warps (1,536 threads), 24 blocks and 100 KB per SM [Certain]
Worked example: why is my kernel at 50% occupancy?

A kernel uses \(T = 256\) threads, \(r = 64\) registers per thread and \(s = 48\) KB of shared memory per block. On an H100, the four limits of formula (85.4) are: threads \(\lfloor 2048/256 \rfloor = 8\) blocks; blocks 32; registers \(\lfloor 65536 / (64 \cdot 256) \rfloor = 4\); shared memory \(\lfloor 228 / 49 \rfloor = 4\). The minimum is 4 blocks, so 32 warps of 64: occupancy 50%.

To reach 100% you must fix both limits. Cut the registers to 32 and the shared memory to 24 KB: now the limits are 8, 32, 8 and \(\lfloor 228/25 \rfloor = 9\), so 8 blocks and 100%. Cutting only the shared memory leaves you at 50%, because registers still allow only 4 blocks.

The same kernel (\(r = 64\), \(s = 48\)) on an A100 gets \(\lfloor 164/49 \rfloor = 3\) blocks, 37.5%; on an RTX 4090, \(\lfloor 100/49 \rfloor = 2\) blocks of 48 warp slots, 33.3%.

Figure 2. An occupancy calculator (formula 85.4) with the published limits of three GPUs. Each bar is the number of blocks one limit would allow; the shortest bar (orange) decides. The strip below shows the 64 (or 48) warp slots of one SM.
How to read Figure 2

Start at the worked example (256 threads, 64 registers, 48 KB): two bars tie at 4 blocks. Lower the registers to 32: the shared-memory bar now decides alone. Then lower the shared memory to 24 KB: 100%. Try 1,024 threads with 40 registers: one block needs 40,960 registers, so only one block fits and half the SM is empty. Push the registers to 255: one block of 256 threads, 12.5%. Switch GPUs: the same kernel can be fine on one GPU and starved on another.

Is high occupancy the goal?

No. Occupancy is a means to hide latency, not a score. A kernel can hide latency in another way: each thread issues several independent loads before it needs the first result, or keeps more data in registers and reuses it. Fast matrix-multiplication kernels (cuBLAS, CUTLASS, FlashAttention) often run at low occupancy, with many registers per thread and large shared-memory tiles, because reuse matters more for them than the number of warps. [Likely] The useful rule: if a memory-bound kernel has low occupancy and reaches a low share of the bandwidth, more warps will probably help. If it already moves bytes near the bandwidth limit, more occupancy cannot help.

Three kinds of memory a kernel sees

From inside a kernel, there are three memory spaces that matter. They match the hardware levels from session 42:

SpaceHow you write itWho sees itLives inLifetime
Registersordinary local variables: float v = x[i];one threadthe SM’s register file (256 KB per SM on H100)the thread
Shared memory__shared__ float tile[32][33];all threads of one blockon-chip SRAM of the SM (up to 228 KB per SM on H100)the block
Global memorypointers passed to the kernel: x[i]all threads, all kernels, and the host (by copies)HBM (80 GB at 3.35 TB/s on H100 SXM), cached in L2until freed

The rule of thumb: read from global memory as few times as possible, and in a good pattern; keep values you reuse in registers; use shared memory when threads of a block need each other’s data (a reduction, a tile of a matrix, a transpose). Shared memory needs a __syncthreads() between the writes of some threads and the reads of others, or a thread may read a value that is not written yet.

Coalesced access: how a warp reads global memory

Global memory is not read byte by byte. On current GPUs, the loads of one warp are served in 32-byte sectors: the hardware collects the 32 addresses of the warp and fetches every 32-byte sector that contains at least one requested byte. [Certain] If the 32 lanes read 32 neighboring 4-byte floats, that is 128 bytes in exactly 4 sectors: nothing is wasted. We call this a coalesced access. If lane \(k\) reads element \(k \cdot S\) (a stride of \(S\) elements), the addresses spread out and more sectors move for the same useful bytes:

\[ \text{efficiency} = \frac{\text{bytes the warp asked for}}{32 \cdot (\text{number of sectors touched})} = \frac{32\, w}{32\, N_{\text{sec}}} \tag{85.6} \]
Read the symbols
\(w\)
bytes per element: 4 for FP32, 2 for BF16 or FP16
\(N_{\text{sec}}\)
the number of different 32-byte sectors that the 32 addresses fall into
efficiency
the share of the moved bytes that the kernel really uses; the rest is wasted bandwidth

For FP32 and a stride of 2, the warp touches 8 sectors (256 bytes) to use 128 bytes: 50%, as the CUDA Best Practices Guide states. [Certain] At a stride of 8 or more, every lane hits its own sector: 32 sectors, 1,024 bytes moved, 12.5% efficiency. For BF16 it is even worse: a large stride gives \(2/32 = 6.25\%\). A start address that is not a multiple of 32 bytes costs one extra sector (5 instead of 4 for FP32, 80%).

bytes the warp usessector that is movedsector not touched
Figure 3. One warp load. Lane \(k\) reads element \(\text{offset} + k S\). Each small box at the bottom is a 32-byte sector of global memory; orange sectors are moved in full, and the blue part of each is what the kernel actually uses (formula 85.6).
How to read Figure 3

At stride 1 with FP32, the lanes fill 4 sectors exactly. Move the stride to 2: twice as many sectors for the same data. At stride 8 each lane gets its own sector, and from then on it cannot get worse: 32 sectors, 12.5%. Set the stride back to 1 and move the start offset to 1: the warp now straddles one more sector. Switch to BF16: at stride 1 the warp reads only 64 bytes (2 sectors), so with BF16 each thread should load two or more elements (for example as one 4-byte or 16-byte vector load) to keep the warp’s requests large.

Where strided access comes from in ML

The usual cause is a transposed or permuted tensor. A PyTorch tensor of shape \((M, N)\) is stored row by row. If a kernel puts neighboring threads on neighboring rows (for example to sum each column), lane \(k\) reads address \(k N\): stride \(N\), the worst case. x.t(), x.permute(...) and slicing every second element create such views without moving data. Many PyTorch kernels then call .contiguous() internally (an extra copy), or run a slow strided path. The fix in your own kernel: map threadIdx.x to the last (fastest-changing) dimension, or stage the data through shared memory (next section).

Shared memory and bank conflicts

Shared memory is fast, but it has its own access rule. It is split into 32 banks; consecutive 4-byte words go to consecutive banks, so

\[ \text{bank}(a) = \left\lfloor \frac{a}{4} \right\rfloor \bmod 32 \tag{85.7} \]

for a byte address \(a\) inside shared memory. [Certain] Each bank can serve one 4-byte word per cycle. If the 32 lanes of a warp access 32 words in 32 different banks, the access takes one pass. If several lanes need different words in the same bank, the accesses are serialized: a \(k\)-way bank conflict takes \(k\) passes. If several lanes read the same word, the value is broadcast to all of them and there is no conflict. [Certain]

The classic case is a \(32 \times 32\) tile of floats, __shared__ float tile[32][32]. Element \((r, c)\) is word \(32r + c\), so its bank is \(c\). When a warp reads a row (lane \(k\) reads \((0, k)\)), the banks are \(0, 1, \dots, 31\): perfect. When it reads a column (lane \(k\) reads \((k, 0)\)), every word is in bank 0: a 32-way conflict, 32 times slower. Matrix transposes, and many attention and matmul kernels, read tiles by columns.

The fix is one extra column: tile[32][33]. Now \((r, c)\) is word \(33r + c\), and its bank is \((33r + c) \bmod 32 = (r + c) \bmod 32\). A column read hits banks \(0, 1, \dots, 31\) again. We waste 32 floats (128 bytes) of shared memory and remove the conflict. Mark Harris’s transpose article shows this change bringing a shared-memory transpose to about 95% of the speed of a plain copy. [Certain]

Figure 4. One warp accesses a tile of floats in shared memory. Each column is one of the 32 banks; each colored box is a different word that a lane needs from that bank. The tallest stack is the number of passes the access needs (formula 85.7).
How to read Figure 4

With the unpadded tile, “read a column” stacks all 32 words in bank 0: 32 passes. Click the padded tile: the stack spreads into 32 banks of height 1. “Every 2nd word” gives a 2-way conflict for any tile width, because even word numbers only hit even banks. “All lanes, one word” is a broadcast: 32 lanes, one word, one pass.

A real kernel: softmax over the rows of a matrix

Attention scores, logits and router outputs all need a softmax over each row. A good beginner kernel gives one block to each row. The threads of the block walk along the row with a stride of blockDim.x, so that at each step the 32 lanes of a warp read 32 neighboring numbers (coalesced). The block then needs two reductions: the maximum of the row (for numerical safety) and the sum of the exponentials.

A reduction inside a warp does not need shared memory. The instruction __shfl_down_sync(mask, v, o) lets lane \(k\) read the register v of lane \(k + o\). Five rounds with \(o = 16, 8, 4, 2, 1\) combine 32 values into lane 0, since \(2^5 = 32\). The warps then meet through a small shared array, one slot per warp.

Algorithm 1: block-wide reduction (sum or max) of one row
  1. input: a row \(x\) of \(N\) numbers in global memory; one block of \(T\) threads (\(T\) a multiple of 32)
  2. each thread: \(v \leftarrow\) identity (0 for sum, \(-\infty\) for max)
  3. for \(j = \text{threadIdx.x};\ j \lt N;\ j \mathrel{+}= T\) do \(v \leftarrow v \oplus x_j\) // strided walk: each warp reads 32 neighbors
  4. for \(o = 16, 8, 4, 2, 1\) do \(v \leftarrow v \oplus \text{shfl\_down}(v, o)\) // inside the warp, registers only
  5. if \(\text{lane} = 0\) then \(\text{red}[\text{warp}] \leftarrow v\) // one value per warp into shared memory
  6. __syncthreads() // wait until every warp has written
  7. if \(\text{warp} = 0\) then \(v \leftarrow \text{red}[\text{lane}]\) if \(\text{lane} \lt T/32\), else identity; repeat step 4; lane 0 writes \(\text{red}[0]\)
  8. __syncthreads(); every thread reads the result \(\text{red}[0]\)
__device__ float warp_max(float v) { for (int o = 16; o > 0; o >>= 1) v = fmaxf(v, __shfl_down_sync(0xffffffff, v, o)); return v; }
__device__ float warp_sum(float v) { for (int o = 16; o > 0; o >>= 1) v += __shfl_down_sync(0xffffffff, v, o); return v; }

// one block per row; launch with softmax_rows<<<M, 256>>>(x, y, N)
__global__ void softmax_rows(const float* x, float* y, int N) {
  const float* xr = x + (long)blockIdx.x * N;  float* yr = y + (long)blockIdx.x * N;
  __shared__ float red[32];                      // one slot per warp (at most 1024/32)
  int lane = threadIdx.x % 32, wid = threadIdx.x / 32, nw = blockDim.x / 32;   // formula (85.3)

  float m = -INFINITY;                              // 1) row maximum, Algorithm 1 with max
  for (int j = threadIdx.x; j < N; j += blockDim.x) m = fmaxf(m, xr[j]);
  m = warp_max(m);  if (lane == 0) red[wid] = m;  __syncthreads();
  if (wid == 0) { m = warp_max(lane < nw ? red[lane] : -INFINITY); if (lane == 0) red[0] = m; }
  __syncthreads();  m = red[0];  __syncthreads();   // last barrier: red[] is reused below

  float s = 0.f;                                    // 2) sum of exp(x - max), Algorithm 1 with sum
  for (int j = threadIdx.x; j < N; j += blockDim.x) s += __expf(xr[j] - m);
  s = warp_sum(s);  if (lane == 0) red[wid] = s;  __syncthreads();
  if (wid == 0) { s = warp_sum(lane < nw ? red[lane] : 0.f); if (lane == 0) red[0] = s; }
  __syncthreads();
  float inv = 1.f / red[0];

  for (int j = threadIdx.x; j < N; j += blockDim.x) yr[j] = __expf(xr[j] - m) * inv;   // 3) write
}

This kernel reads each row three times. For rows of a few thousand floats, the second and third reads usually hit the L1 or L2 cache, so HBM sees close to one read and one write. [Likely] For very long rows you would keep the row in registers or shared memory, or use the online softmax trick, which computes the maximum and the sum in a single pass. That trick is the core of FlashAttention (session 124).

Why fusion helps: fewer trips to HBM

Most operations in a transformer besides the matrix multiplications are elementwise (bias add, GELU or SiLU, residual add, scaling, dropout, type casts) or small reductions (RMSNorm, softmax). They do a few FLOPs per element. From session 42, their arithmetic intensity is around 1 FLOP per byte or less, far below the H100 ridge point of 295. They are memory-bound: their time is the bytes they move divided by the bandwidth.

In eager PyTorch, each operation is its own kernel. Each kernel reads its inputs from HBM and writes its output to HBM, and the next kernel reads that output back. Kernel fusion does the whole chain in one kernel: each thread loads its element once, applies all operations in registers, and writes only the final result. The math is identical; only the memory traffic changes. For a chain of operations on \(n\) elements of \(w\) bytes:

\[ Q_{\text{unfused}} = n\, w \sum_{k=1}^{K} (a_k + 1), \qquad Q_{\text{fused}} = n\, w\, (a_{\text{ext}} + 1) \tag{85.8} \]
Read the symbols
\(K\)
the number of operations in the chain
\(a_k\)
how many full-size tensors operation \(k\) reads (1 for GELU, 2 for a residual add); the \(+1\) is its output. Small inputs such as a bias vector are ignored
\(a_{\text{ext}}\)
the number of full-size tensors the whole chain reads from outside (the first input, plus a residual if there is one)
\(Q\)
bytes moved between HBM and the chip; for a memory-bound kernel the time is about \(Q / B\), with \(B\) the bandwidth
Worked example: bias + GELU + residual in an MLP

Take the MLP output chain \(y = \text{GELU}(x + \text{bias}) + r\) on \(M = 4096\) tokens and width 16,384, in BF16: \(n = 4096 \times 16384 = 67{,}108{,}864\) elements, \(w = 2\) bytes.

Unfused, three kernels: the bias add reads \(x\) and writes \(t_1\) (2 tensors), GELU reads \(t_1\) and writes \(t_2\) (2), the residual add reads \(t_2\) and \(r\) and writes \(y\) (3). Total \(7 n w = 939.5\) MB, which takes at least \(939.5\ \text{MB} / 3.35\ \text{TB/s} = 280\) µs on an H100.

Fused, one kernel: read \(x\) and \(r\), write \(y\): \(3 n w = 402.7\) MB, at least 120 µs. The same math, \(7/3 = 2.33\) times less traffic, and two fewer kernel launches. The intermediate tensors \(t_1\) and \(t_2\) (134 MB each) are also never allocated.

Figure 5. HBM traffic of an elementwise chain, unfused (one kernel per operation) and fused (one kernel), from formula (85.8). Times are the memory-bound lower bound on an H100 SXM (3.35 TB/s). The dropout here makes its random mask from a seed inside the kernel, so it reads and writes one tensor.
How to read Figure 5

Each colored segment of the top bar is one kernel of the unfused chain: its reads and its write. The bottom bar is the fused kernel. Add operations: the unfused bar grows by \(2nw\) per unary operation, the fused bar does not grow at all. With all five operations selected, fusion cuts traffic by \(11/3 = 3.7\times\). Move \(M\) down to 1: the bytes become tiny (98 KB fused), and the time is then dominated by the few microseconds of each kernel launch, a second reason to fuse (session 50).

Fusion has limits. A matrix multiplication followed by an elementwise operation can be fused as an epilogue (the GEMM applies the bias and activation to its output tile before writing it), and cuBLASLt and CUTLASS offer this. [Certain] Two large reductions over different dimensions are hard to fuse. And fusion can raise register use and lower occupancy. In practice you get most fusions from torch.compile, which generates fused Triton kernels for elementwise and reduction chains, and from hand-written kernels for the important patterns: FlashAttention, fused RMSNorm with residual, fused rotary embedding, fused MoE routing, and fused dequantize-plus-GEMV for quantized decoding. [Likely]

Algorithm 2: a fused elementwise kernel with a grid-stride loop
  1. input: pointers to \(x, r, y\) (each \(n\) elements) and a bias of \(d\) elements; grid of \(G\) blocks of \(T\) threads
  2. \(i \leftarrow \text{blockIdx.x} \cdot T + \text{threadIdx.x}\) // formula (85.1)
  3. while \(i \lt n\) do
  4. \(v \leftarrow x_i + \text{bias}_{i \bmod d}\) // one coalesced read of x; bias stays in cache
  5. \(g \leftarrow \text{GELU}(v)\) // registers only, nothing written to HBM
  6. \(y_i \leftarrow g + r_i\) // one read of r, one write of y
  7. \(i \leftarrow i + G \cdot T\) // grid-stride: a fixed grid covers any n

The grid-stride loop lets you launch a fixed number of blocks (for example a few per SM) whatever the size of the tensor. Each thread handles elements \(i, i + GT, i + 2GT, \dots\), and at every step the warp still reads 32 neighbors.

Placing a kernel on the roofline

You wrote or profiled a kernel and measured its time \(t\). Is it good? The roofline answers this with three numbers you can compute on paper: the FLOPs \(F\) it must do, the bytes \(Q\) it must move to and from HBM (count them as in formula 85.8, assuming each byte crosses once), and the hardware peaks \(P\) and \(B\). Then:

\[ I = \frac{F}{Q}, \qquad t_{\min} = \max\!\left(\frac{F}{P},\ \frac{Q}{B}\right), \qquad \text{achieved} = \frac{F}{t}, \quad \frac{Q}{t} \tag{85.9} \]
\[ \text{fraction of roof} = \frac{F / t}{\min(P,\ I \cdot B)} = \frac{t_{\min}}{t} \tag{85.10} \]
Read the symbols
\(I\)
arithmetic intensity in FLOP per byte: the x-position of the kernel on the roofline plot
\(t_{\min}\)
the fastest the kernel could possibly run on this GPU (formula 42.5 of session 42)
\(F/t\), \(Q/t\)
achieved FLOP/s (the y-position on the plot) and achieved bandwidth
\(P\), \(B\)
peak FLOP/s for the units the kernel uses (989 TFLOP/s dense BF16 on tensor cores, 67 TFLOP/s for ordinary FP32 math on H100 SXM) and peak bandwidth (3.35 TB/s) [Certain]
Worked example: is 150 µs good for the fused kernel?

The fused bias + GELU + residual kernel from above moves \(Q = 402.7\) MB. Count about 10 FLOPs per element (1 for the bias, about 8 for the tanh form of GELU, 1 for the residual; the exact count does not change the answer): \(F = 6.71 \times 10^8\). So \(I = F/Q = 1.67\) FLOP/byte: far left of the ridge, memory-bound. \(t_{\min} = Q/B = 120\) µs.

Suppose you measure \(t = 150\) µs (an example value). Achieved bandwidth \(= 402.7\ \text{MB} / 150\ \mu\text{s} = 2.68\) TB/s, which is 80% of 3.35 TB/s. Achieved compute is only 4.47 TFLOP/s, but that does not matter: the kernel is limited by bytes, and it is at 80% of its roof. Memory-bound kernels usually top out somewhat below the peak bandwidth, so this is a good kernel. [Likely] There is little left to gain by tuning it; the only big win is to move fewer bytes (fuse more, or use smaller number formats).

Algorithm 3: place a kernel on the roofline and decide what to do
  1. count \(F\) and \(Q\) from the shapes, assuming each input byte is read once and each output byte written once
  2. time the kernel with CUDA events after warm-up, or read its duration from Nsight Systems or the PyTorch profiler
  3. compute \(I\), \(t_{\min}\), \(F/t\), \(Q/t\) and the fraction of roof (formulas 85.9, 85.10)
  4. if \(I \lt P/B\) (memory-bound) then
  5. if fraction is high (say above 70%): done; only fewer bytes help (fusion, BF16 or FP8, avoid copies)
  6. else: look for uncoalesced access (formula 85.6), too few blocks or warps (85.4), register spills, or extra traffic from re-reads
  7. else (compute-bound): check that the math runs on tensor cores (right dtype, shapes that are multiples of 8 or 16), then tiling, bank conflicts (85.7) and occupancy
roof: \(\min(P, I \cdot B)\)your kernel at the measured timeother kernels at 80% of their roof
Figure 6. Placing a kernel on the H100 SXM roofline (formulas 85.9 and 85.10). The upper flat roof is dense BF16 on tensor cores (989 TFLOP/s), the lower dashed one is ordinary FP32 math (67 TFLOP/s); the slope is 3.35 TB/s. Pick a kernel and set the measured time as a multiple of its lower bound \(t_{\min}\).
How to read Figure 6

Every elementwise kernel, the softmax and the batch-1 decode GEMV sit near \(I \approx 1\), far left of both ridges: they can only be judged by bandwidth. The GEMM sits far right at \(I = 1365\): it is judged by tensor-core FLOP/s. Slide the measured time up: the orange point drops straight down, because \(I\) depends only on the work, not on how well you did it. Compare “unfused chain” and “fused chain”: fusion does not move a kernel up the slope by making it run better; it moves it to the right by removing bytes, and the roof above it is higher there.

Calling your own kernel from PyTorch

The code below collects the session in one script. Part 1 implements formulas (85.2), (85.4), (85.6) and (85.7) as plain Python and prints the numbers from the figures. Part 2 counts the bytes of the fused chain (85.8). Part 3 compiles the fused kernel of Algorithm 2 with load_inline from torch.utils.cpp_extension, checks it against PyTorch, and times both with CUDA events. Parts 1 and 2 run anywhere; part 3 needs an NVIDIA GPU with the CUDA toolkit (nvcc), and its printed times depend on your GPU.

import math, torch
torch.manual_seed(0)

# 1) The rules of this session as small functions
def grid_size(n, block):                          # formula (85.2)
    return (n + block - 1) // block

def occupancy(T, regs, smem_kb, max_threads=2048, max_blocks=32,
              regs_sm=65536, smem_sm_kb=228, reserved_kb=1):   # H100 limits, formula (85.4)
    r = math.ceil(regs / 8) * 8                   # registers are handed out in chunks
    lim = {"threads": max_threads // T, "blocks": max_blocks,
           "registers": regs_sm // (r * T),
           "shared": smem_sm_kb // (smem_kb + reserved_kb) if smem_kb > 0 else max_blocks}
    b = min(lim.values())
    return b, b * T / max_threads, min(lim, key=lim.get)

def sectors(stride, elem=4, offset=0):            # 32-byte sectors touched by one warp
    s = {(offset + lane * stride * elem + k) // 32 for lane in range(32) for k in range(elem)}
    return len(s), 32 * elem / (32 * len(s))      # formula (85.6): useful / moved

def bank_conflict(word_idx):                      # formula (85.7), words of 4 bytes
    banks = {}
    for w in set(word_idx):                       # the same word twice is a broadcast
        banks.setdefault(w % 32, set()).add(w)
    return max(len(v) for v in banks.values())

print(grid_size(1000, 256), occupancy(256, 64, 48))           # 4 (4, 0.5, 'registers')
print(sectors(1), sectors(2), sectors(8))                     # (4, 1.0) (8, 0.5) (32, 0.125)
print(bank_conflict([r * 32 for r in range(32)]),             # column of a 32x32 tile: 32
      bank_conflict([r * 33 for r in range(32)]))             # padded to 33 columns: 1

# 2) HBM bytes of y = gelu(x + b) + r: unfused (3 kernels) vs fused (1 kernel), formula (85.8)
n, s = 4096 * 16384, 2                            # BF16
unfused = (2 + 2 + 3) * n * s                     # add: x->t1, gelu: t1->t2, add: t2,r->y
fused = 3 * n * s                                 # read x and r once, write y once
print(unfused / 1e6, fused / 1e6, unfused / 3.35e12 * 1e6, fused / 3.35e12 * 1e6)
# 939.5 MB, 402.7 MB, 280 us, 120 us on an H100

# 3) The fused kernel (Algorithm 2) in CUDA C++, compiled from Python
cuda_src = r"""
__global__ void bias_gelu_res(const float* __restrict__ x, const float* __restrict__ b,
                              const float* __restrict__ r, float* __restrict__ y,
                              long n, int d) {
  for (long i = blockIdx.x * (long)blockDim.x + threadIdx.x; i < n;
       i += (long)blockDim.x * gridDim.x) {               // grid-stride loop
    float v = x[i] + b[i % d];                            // neighbors read neighbors: coalesced
    float g = 0.5f * v * (1.f + tanhf(0.7978845608f * (v + 0.044715f * v * v * v)));
    y[i] = g + r[i];                                      // one write, no temporaries in HBM
  }
}
torch::Tensor fused(torch::Tensor x, torch::Tensor b, torch::Tensor r) {
  TORCH_CHECK(x.is_cuda() && x.is_contiguous() && r.is_contiguous());
  auto y = torch::empty_like(x);
  long n = x.numel(); int d = b.numel(); int threads = 256;
  long blocks = std::min<long>((n + threads - 1) / threads, 132L * 8);
  bias_gelu_res<<<blocks, threads>>>(x.data_ptr<float>(), b.data_ptr<float>(),
      r.data_ptr<float>(), y.data_ptr<float>(), n, d);
  return y;
}
"""
def reference(x, b, r):
    return torch.nn.functional.gelu(x + b, approximate="tanh") + r   # 3 kernels in eager mode

if torch.cuda.is_available():
    from torch.utils.cpp_extension import load_inline
    ext = load_inline("fused_ext", cuda_sources=cuda_src, functions=["fused"],
        cpp_sources="torch::Tensor fused(torch::Tensor x, torch::Tensor b, torch::Tensor r);")
    M, d = 4096, 16384
    x, r = torch.randn(M, d, device="cuda"), torch.randn(M, d, device="cuda")
    b = torch.randn(d, device="cuda")
    print((ext.fused(x, b, r) - reference(x, b, r)).abs().max().item())   # tiny, about 1e-6
    for name, fn in [("eager", reference), ("fused", ext.fused)]:
        for _ in range(3): fn(x, b, r)                    # warm-up
        e0, e1 = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
        e0.record(); [fn(x, b, r) for _ in range(20)]; e1.record()
        torch.cuda.synchronize()                          # launches are async: wait before reading
        t = e0.elapsed_time(e1) / 20 * 1e-3               # seconds per call
        q = 3 * x.numel() * 4                             # bytes the fused kernel must move (FP32)
        print(name, f"{t*1e6:.0f} us, {q/t/1e12:.2f} TB/s of useful traffic")
else:
    x, b, r = torch.randn(8, 16), torch.randn(16), torch.randn(8, 16)
    v = x + b
    mine = 0.5 * v * (1 + torch.tanh(0.7978845608 * (v + 0.044715 * v ** 3))) + r
    print((mine - reference(x, b, r)).abs().max().item())   # the kernel's math matches PyTorch

Two details in this code matter in real work. The index uses long: a tensor with more than \(2^{31} - 1\) elements (for example \(32768 \times 65536\)) overflows a 32-bit int and silently writes to wrong addresses. And the timing records CUDA events and synchronizes; timing a kernel launch with Python’s time.time() alone measures only the launch, because launches are asynchronous.

Where this is used

WhereWhich idea from this sessionWhy it matters
FlashAttention in every serving enginetiles in shared memory, online softmax, fusion of QKᵀ, softmax and PVthe \(T \times T\) score matrix never goes to HBM (session 124)
vLLM, TensorRT-LLM, SGLang custom kernelsfused RMSNorm + residual, fused rotary embedding, fused MoE, PagedAttentiondecode is memory-bound; every saved byte is saved time [Likely]
Quantized decoding (INT4 or FP8 weights)fused dequantize + GEMV; coalesced, vectorized loads of packed weightsfewer bytes per weight moves the GEMV right on the roofline (session 82)
torch.compile and Tritonautomatic fusion of elementwise and reduction chainsyou get formula (85.8) savings without writing CUDA (session 86)
Video preprocessing for VLMs and video classificationone fused kernel for decode output → resize → normalize → patchify, coalesced over pixelsat many frames per clip, separate kernels per step re-read every frame from HBM [Likely]
Video retrieval, Video-RAG and fingerprintingfused similarity + top-k kernels over large embedding or hash tablesscoring millions of clip vectors is memory-bound; never write the full score matrix [Likely]
Profiling any training or serving jobplacing kernels on the roofline (Algorithm 3), occupancy from Nsight Computetells you whether to optimize a kernel or remove it

Common mistakes

  • No bounds check. The last block reads and writes past the end of the array. Sometimes it crashes, often it silently corrupts another tensor. Always guard with if (i < n) or a grid-stride loop.
  • Block size not a multiple of 32. A block of 100 threads still occupies 4 full warps; 28 lanes per block do nothing (Figure 1).
  • A missing __syncthreads() between writing shared memory and reading what other threads wrote, or before reusing a shared buffer. The result is wrong only sometimes, which makes it hard to find. A barrier inside a branch that not all threads of the block reach can hang the kernel.
  • Threads mapped to the wrong dimension. Neighboring threadIdx.x on neighboring rows gives stride-\(N\) reads at 12.5% efficiency for FP32 (Figure 3). Map threadIdx.x to the contiguous dimension, and pass contiguous tensors.
  • Ignoring bank conflicts in column reads of a shared tile. Pad the tile to 33 columns (Figure 4).
  • Chasing 100% occupancy. More warps help only if latency is the problem. Measure the fraction of the roof first (Algorithm 3).
  • 32-bit index overflow on tensors with more than about 2.1 billion elements. Use 64-bit indices for large tensors.
  • Timing without synchronization, or timing the first call (which includes compilation and memory allocation). Warm up, then use CUDA events or a profiler.

Interview questions

What are threads, warps, blocks and grids, and what does each level guarantee?basic
A thread runs the kernel on its own data with its own registers. A warp is 32 consecutive threads of a block that the hardware executes together, one instruction at a time (SIMT). A block is up to 1,024 threads that run on one SM, can share shared memory and can synchronize with __syncthreads(). The grid is all blocks of a launch; blocks are independent, run in any order and cannot synchronize with each other inside the kernel. Each thread finds its element with blockIdx.x * blockDim.x + threadIdx.x.
What is warp divergence, and when does a branch cost nothing?basic
If lanes of one warp take different paths of a branch, the warp executes both paths in turn with the lanes of the other path disabled, so the cost is the sum of both paths. A branch is free when all 32 lanes of every warp take the same path, for example when the condition depends on the block index, the warp index, or a value that is the same for the whole warp. The bounds check is cheap because only the single partly filled warp at the end diverges.
What limits occupancy? Compute it for 256 threads, 64 registers, 48 KB shared memory on an H100.core
Four limits per SM: threads (2,048), blocks (32), registers (65,536) and shared memory (228 KB, plus 1 KB reserved per block). Threads allow 8 blocks, registers \(65536/(64 \cdot 256) = 4\), shared memory \(\lfloor 228/49 \rfloor = 4\). So 4 blocks, 32 of 64 warps: 50%. You must lower both registers and shared memory to raise it.
Is higher occupancy always faster?core
No. Occupancy is one way to hide memory latency: more warps to switch to while others wait. Kernels can also hide latency with instruction-level parallelism (several independent loads in flight per thread) and data reuse in registers. GEMM and attention kernels often run at low occupancy with large tiles and many registers and still reach most of the peak. Raise occupancy when a memory-bound kernel is far below the bandwidth roof and the profiler shows warps stalled on memory with few eligible warps.
What is coalesced global memory access? Give the efficiency of a stride-2 and a stride-32 FP32 read.core
The 32 addresses of a warp load are served in 32-byte sectors. When the lanes read consecutive elements, the warp’s 128 bytes (FP32) need exactly 4 sectors: coalesced, 100% efficiency. Stride 2 touches 8 sectors for the same 128 useful bytes: 50%. Stride 32 (or any stride of 8 or more) gives each lane its own sector: 32 sectors, 1,024 bytes moved, 12.5%. Strided access usually comes from mapping threads to the wrong dimension of a row-major tensor.
What is a shared-memory bank conflict, and how does padding fix the transpose case?core
Shared memory has 32 banks; word \(k\) (4 bytes) is in bank \(k \bmod 32\). If lanes of a warp access different words in the same bank, the accesses are serialized. In a float tile[32][32], element \((r, c)\) is in bank \(c\), so reading a column puts all 32 lanes in one bank: 32 passes. With tile[32][33] the bank is \((r + c) \bmod 32\), so a column spreads over all 32 banks. Lanes that read the same word get a broadcast, which is not a conflict.
Why does kernel fusion help if it does exactly the same math?basic
Elementwise and normalization operations are memory-bound: their time is bytes divided by bandwidth. Unfused, every operation writes its output to HBM and the next one reads it back. Fused, each element is loaded once, all operations run in registers, and only the final result is written. For bias + GELU + residual this is \(7nw\) versus \(3nw\) bytes, 2.33× less traffic, plus fewer kernel launches and no temporary tensors.
Your fused kernel takes 150 µs and moves 400 MB on an H100. Is it good? What next?core
Its intensity is around 1–2 FLOP per byte, far below the ridge (295), so it is memory-bound. The floor is \(0.4\ \text{GB} / 3.35\ \text{TB/s} \approx 120\) µs; 150 µs is about 80% of the memory roof (2.68 TB/s). That is close to what memory-bound kernels reach in practice. Further tuning of the same kernel will gain little; to go faster, move fewer bytes: fuse it with its neighbors, use BF16 or FP8 instead of FP32, or avoid a copy before it.
How does a block-wide reduction work in a softmax kernel?advanced
Each thread first reduces a strided slice of the row into a register (coalesced reads). Then each warp reduces its 32 values with five __shfl_down_sync steps (offsets 16, 8, 4, 2, 1), with no shared memory. Lane 0 of each warp writes its result to a small shared array; after __syncthreads(), the first warp reduces those values the same way and broadcasts the final result through shared memory. Softmax does this twice (max, then sum of exponentials), with a barrier before the shared buffer is reused. Online softmax merges both into one pass.
Why do GEMM kernels need tiling but elementwise kernels do not?advanced
An elementwise kernel uses each input once, so there is nothing to reuse; its best case is to read and write each byte once with coalesced access. A GEMM uses each input number many times (each element of A meets a whole row of B). Without tiling, each thread reloads its row and column from global memory and the intensity falls to about 0.5 FLOP per byte in BF16. Tiles in shared memory and registers let a block reuse each loaded number many times, which raises the intensity past the ridge so the tensor cores become the limit. Column reads from those tiles are also where bank conflicts appear.
A colleague times a kernel with time.time() around the call and reports 8 µs for a 1 GB copy. What is wrong?basic
Kernel launches are asynchronous: the call returns as soon as the kernel is queued, so 8 µs is the launch cost, not the run time. A 1 GB read plus write cannot take less than \(2\ \text{GB} / 3.35\ \text{TB/s} \approx 0.6\) ms on an H100. Use CUDA events with torch.cuda.synchronize(), warm up first, average several runs, or read the kernel duration from a profiler.

Exercises

  1. You process \(n = 10^6\) elements with blocks of 256 threads. How many blocks do you launch, and how many threads are idle? Which block and lane handle element 777,777?
  2. On an H100, a kernel uses 128 threads per block, 96 registers per thread and no shared memory. What is its occupancy, and which limit decides?
  3. A BF16 kernel reads with a stride of 4 elements. What is its coalescing efficiency? What if each thread instead reads 8 neighboring BF16 values (16 bytes) with one vector load, and lanes are 16 bytes apart?
  4. A shared tile is float tile[32][64]. A warp reads column 5 (lane \(k\) reads \((k, 5)\)). What is the conflict degree? Which padding fixes it?
  5. Use formula (85.8) for the chain: scale, add bias, SiLU, multiply by another tensor \(g\) (as in a SwiGLU gate), on \(n\) elements of 2 bytes. Count unfused and fused bytes.
  6. A decode GEMV with a \(4096 \times 4096\) BF16 weight takes 14 µs on an H100. Place it on the roofline: \(I\), \(t_{\min}\), the fraction of the roof.
Answerscheck
1. \(G = \lceil 10^6/256 \rceil = 3907\) blocks, \(3907 \times 256 - 10^6 = 192\) idle threads. Element 777,777: block \(\lfloor 777777/256 \rfloor = 3038\), threadIdx.x \(= 777777 - 3038 \cdot 256 = 49\), so warp 1, lane 17. 2. Registers: \(65536 / (96 \cdot 128) = 5.33\), so 5 blocks; threads allow 16, blocks 32. \(5\) blocks \(\times\ 4\) warps \(= 20\) of 64 warps: 31.25%, limited by registers. 3. Lanes are \(4 \times 2 = 8\) bytes apart: 32 lanes span 256 bytes = 8 sectors for 64 useful bytes: 25%. With 16-byte vector loads per lane, the warp reads 512 contiguous bytes in 16 sectors: 100%. 4. Element \((k, 5)\) is word \(64k + 5\), bank 5 for every \(k\): 32-way. Pad to tile[32][65]: bank \((65k + 5) \bmod 32 = (k + 5) \bmod 32\), all different. 5. Unfused: scale 2, bias 2, SiLU 2, multiply 3, total \(9nw = 18n\) bytes. Fused: read \(x\) and \(g\), write \(y\): \(3nw = 6n\) bytes, 3× less. 6. \(F = 2 \cdot 4096^2 = 3.36 \times 10^7\), \(Q \approx 2 \cdot 4096^2 = 3.36 \times 10^7\) bytes (the vectors are tiny), \(I \approx 1\). \(t_{\min} = 33.6\ \text{MB}/3.35\ \text{TB/s} = 10.0\) µs. Fraction \(= 10.0/14 = 72\%\) of the memory roof (2.40 TB/s).

Key takeaways

  • A kernel runs as a grid of independent blocks of up to 1,024 threads; each thread finds its element with formula (85.1), and the launch needs \(\lceil n/b \rceil\) blocks plus a bounds check (85.2).
  • Threads run in warps of 32 lanes (85.3). Keep branches uniform per warp and block sizes a multiple of 32.
  • Occupancy (85.4, 85.5) is set by the tightest of four limits: threads, blocks, registers and shared memory. It hides latency but is not a goal in itself.
  • Global memory moves 32-byte sectors per warp: neighboring lanes must read neighboring addresses (85.6). Shared memory has 32 banks: pad tiles to avoid column conflicts (85.7).
  • Elementwise chains are memory-bound; fusion cuts HBM bytes (85.8), for bias + GELU + residual from \(7nw\) to \(3nw\).
  • To judge any kernel: count \(F\) and \(Q\), compute \(t_{\min}\), and report the fraction of the roof (85.9, 85.10). Memory-bound and near the roof means: move fewer bytes.

Key formulas

\[ i = \text{blockIdx}\cdot\text{blockDim} + \text{threadIdx}, \quad n_{\text{blocks}} = \min\!\left(\tfrac{2048}{T}, 32, \tfrac{65536}{rT}, \tfrac{228}{s+1}\right), \quad \text{bank} = \tfrac{a}{4} \bmod 32 \]
\[ Q_{\text{fused}} = n w (a_{\text{ext}} + 1), \quad I = \frac{F}{Q}, \quad t_{\min} = \max\!\left(\frac{F}{P}, \frac{Q}{B}\right), \quad \text{fraction} = \frac{t_{\min}}{t} \]

References for this session

  1. NVIDIA (docs). CUDA C++ Programming Guide
  2. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  3. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  4. NVIDIA (docs). CUDA C++ Best Practices Guide
  5. NVIDIA (docs). NVIDIA Hopper Tuning Guide (H100 occupancy and shared-memory limits)
  6. NVIDIA (docs). NVIDIA Ampere GPU Architecture Tuning Guide (A100 limits)
  7. NVIDIA (docs). NVIDIA Ada GPU Architecture Tuning Guide (RTX 4090 limits)
  8. Hwu, Kirk & El Hajj (2022). Programming Massively Parallel Processors: A Hands-on Approach (4th edition)
  9. Harris (2013). An Efficient Matrix Transpose in CUDA C/C++ (NVIDIA technical blog)