GPU architecture for ML
A GPU is thousands of small arithmetic units fed by a memory system that is much slower than the arithmetic. This session opens the box: streaming multiprocessors, warps and tensor cores, then registers, shared memory, L2 and HBM. With the roofline model and arithmetic intensity you will predict whether an operation is limited by compute or by memory, and with MFU you will measure how much of the GPU a training run really uses.
What you will learn
- How a GPU is built: streaming multiprocessors (SMs), warps of 32 threads, CUDA cores and tensor cores
- Why GPUs run many warps at once: hiding memory latency by switching between warps
- The memory hierarchy: registers, shared memory (SRAM), L2 cache and HBM or GDDR, with real sizes and speeds for H100, A100 and RTX 4090
- Arithmetic intensity and the roofline: how to tell if an operation is compute-bound or memory-bound
- Why a matrix multiplication needs about 345 rows (tokens) on an H100 to become compute-bound, and why batch-1 decoding is memory-bound
- How tiling in shared memory raises the arithmetic intensity of a matrix multiplication
- Model FLOPs utilization (MFU): how to compute it, what real runs reach (PaLM 46.2%, Llama 3 405B 38 to 43%), and why the definition matters
Why this matters
An H100 GPU can do about 989 trillion floating-point operations (FLOPs) per second in BF16, but it can only read about 3.35 trillion bytes per second from its main memory. [Certain] That gap, about 295 FLOPs for every byte, decides almost everything about how fast your model trains and runs.
In session 25 you learned to count FLOPs, and you turned FLOPs into GPU-days with a number called MFU. In session 37 you saw why BF16 and FP8 are faster than FP32. This session explains the machine underneath. Why is a large matrix multiplication fast, but a LayerNorm or a softmax slow per FLOP? Why does generating one token at a time use less than 1% of the GPU’s arithmetic? Why are FlashAttention and kernel fusion so important? The answers all come from two facts: how the GPU is built, and how its memory works.
You do not need to write CUDA to use this knowledge. Choosing a batch size, deciding whether to fuse two operations, reading a profiler trace, and estimating the cost of a training run all need the same mental model. Writing kernels comes later, in session 85 and session 86.
A GPU is a throughput machine
A CPU has a few large cores. Each core is built to finish one stream of instructions as fast as possible: big caches, branch prediction, out-of-order execution. A GPU makes the opposite choice. It has many small, simple cores and very little control logic per core. It is slow at any single task but fast at doing the same operation on millions of numbers at once. We say a CPU is built for latency (time for one task) and a GPU for throughput (tasks finished per second).
Deep learning is almost all banquet work. A transformer layer is mostly large matrix multiplications plus simple element-by-element operations (adds, GELU, normalization). All of these apply the same rule to many numbers. That is why GPUs, not CPUs, train models.
Streaming multiprocessors, warps and threads
The GPU chip is divided into many identical units called streaming multiprocessors (SMs). The H100 SXM has 132 SMs, the A100 has 108, and the RTX 4090 has 128. [Certain] Each SM is a small processor of its own, with its own arithmetic units, its own fast memory and its own schedulers. All SMs share the large L2 cache and the main memory.
The programming model: threads, blocks, grid
When you launch work on a GPU, you launch a kernel: one function that many threads run at the same time, each on a different piece of the data. Threads are grouped into thread blocks (often 128 to 1024 threads). The whole launch is a grid of blocks. Each block runs on one SM from start to finish, and threads of the same block can share data through the SM’s fast memory. Blocks cannot talk to each other cheaply, so different blocks should do independent work (for example, different tiles of the output matrix).
Warps
The hardware does not run threads one by one. It runs them in groups of 32 called warps. [Certain] All 32 threads of a warp execute the same instruction at the same moment, each on its own data. NVIDIA calls this SIMT (single instruction, multiple threads). Two consequences matter for ML:
- Divergence. If threads in one warp take different branches of an
if, the warp runs both branches one after the other, with some threads switched off. Work that follows one path for all 32 threads is faster. - Coalescing. When the 32 threads read 32 neighboring numbers, the hardware merges this into a few wide memory transactions. Scattered reads waste bandwidth. This is why memory layout (contiguous tensors, which dimension is last) affects speed.
An H100 SM can hold up to 64 warps (2,048 threads) at the same time. [Certain] It is divided into four parts (sub-partitions), each with its own warp scheduler. [Likely] Every clock cycle, each scheduler picks one warp that is ready and issues its next instruction.
CUDA cores and tensor cores
Each SM has two kinds of arithmetic units. CUDA cores do ordinary scalar math: one FP32 add or multiply per lane. Tensor cores do a small matrix multiply-accumulate in one instruction: \(D = A B + C\) on small tiles (for example, a few 16 × 8 or 16 × 16 blocks), with low-precision inputs (FP16, BF16, FP8) and an FP32 accumulator. The H100 has 4 tensor cores per SM, 528 in total. [Certain]
The difference in speed is huge. On the H100 SXM, ordinary FP32 math peaks at 67 TFLOP/s, while dense BF16 on the tensor cores peaks at 989 TFLOP/s, about 14.8 times more. [Certain] So in practice, only matrix multiplications can reach the GPU’s headline number. Everything else (softmax, normalization, activation functions, residual adds) runs on the CUDA cores at a much lower peak, and as we will see, it is usually limited by memory anyway.
Figure 1 walks from the full chip down to one warp, and then follows the data of one matrix multiplication on its way from main memory to a tensor core.
How to read Figure 1
Step through with Next. The colors follow the data: orange for the big, slow memory (HBM), blue for the fast memory close to the arithmetic (shared memory and registers), and green for the units that compute. The last steps show a tile of a matrix moving from HBM through L2 and shared memory into registers, where the tensor core uses it many times. That reuse is the whole trick of fast GPU code.
Hiding latency with many warps
A read from HBM takes hundreds of clock cycles to come back. [Likely] A CPU would try to avoid this wait with big caches. A GPU instead hides the wait: when one warp is waiting for memory, the scheduler simply issues an instruction from another warp that is ready. If there are enough warps, the arithmetic units never sit idle.
A simple model shows how many warps are enough. Say each warp computes for \(c\) cycles, then waits \(\ell\) cycles for memory, again and again. One scheduler can keep busy only if, during one warp’s wait, other warps have enough compute to fill it:
- Read the symbols
- \(U\)
- utilization: the fraction of cycles in which the scheduler issues useful work
- \(W\)
- number of warps that the scheduler can choose from
- \(c\)
- cycles of compute a warp does before it needs new data from memory
- \(\ell\)
- memory latency: cycles until the data arrives
So you need about \(W \ge (c + \ell)/c\) warps. With \(c = 10\) and \(\ell = 90\), that is 10 warps. The fraction of the SM’s warp slots that are filled is called occupancy. Each warp needs registers and shared memory, and the SM has a fixed amount of both, so a kernel that uses many registers per thread can fit fewer warps. This is a real trade-off for kernel writers. Formula (42.1) is a teaching model, not an exact law. [Certain] Real kernels also hide latency inside one warp, by starting the next memory loads before they need them (prefetching, pipelining).
How to read Figure 2
With 1 warp, the scheduler is idle most of the time: it waits with the warp. Add warps and the gaps fill in, until the bottom row is solid and utilization reaches 100%. The readout compares the simulation with formula (42.1). Then raise the latency: you need more warps for the same utilization. Raise the compute per load (\(c\)): you need fewer. A kernel that does more math per byte loaded is easier to keep busy. The next sections turn this idea into the roofline.
The memory hierarchy
A GPU has several levels of memory. The closer a level is to the arithmetic units, the faster and the smaller it is. The table gives the numbers for an H100 SXM.
| Level | Where | Size (H100 SXM) | Who can see it | Speed |
|---|---|---|---|---|
| Registers | inside each SM sub-partition | 256 KB per SM (65,536 registers of 32 bits); about 33 MB in total | one thread | fastest; read every cycle |
| Shared memory (SRAM) / L1 | inside each SM | 256 KB per SM combined, of which up to 228 KB can be shared memory | all threads of one block | very fast, a few tens of cycles [Likely] |
| L2 cache | on the chip, shared by all SMs | 50 MB | all SMs | faster than HBM; hundreds of cycles [Likely] |
| HBM3 (main GPU memory) | stacks next to the chip | 80 GB | everything; the host copies data here | 3.35 TB/s; hundreds of cycles of latency [Likely] |
The sizes and the HBM bandwidth are from NVIDIA’s Hopper documentation and H100 product page. [Certain] Shared memory is SRAM (static RAM) on the chip. The programmer controls it directly: a kernel says “load this tile into shared memory”, and all threads of the block can then read it many times. That makes it the main tool for reuse. HBM (high-bandwidth memory) is DRAM stacked next to the GPU die; it is large but far away. Consumer GPUs use GDDR memory instead, which is cheaper and slower: the RTX 4090 has 24 GB of GDDR6X at 1,008 GB/s and a 72 MB L2 cache. [Certain]
Llama 3 8B has about 8.03 billion parameters. In BF16 (2 bytes each), the weights are \(8.03 \times 10^9 \times 2 = 16.06\) GB. Reading them once from HBM at 3.35 TB/s takes \(16.06 / 3350 = 4.79\) ms.
When a chatbot generates text for one user, every new token needs all the weights once. So even with infinitely fast arithmetic, one H100 cannot produce more than about \(1 / 4.79\ \text{ms} \approx 209\) tokens per second for one user. Memory, not compute, sets this limit. Reading the whole 80 GB of HBM takes 23.9 ms.
Arithmetic intensity
Every operation does some FLOPs and moves some bytes between HBM and the chip. Their ratio is the arithmetic intensity:
- Read the symbols
- \(F\)
- the number of floating-point operations the kernel does (a multiply-add counts as 2)
- \(Q\)
- the number of bytes it reads from and writes to HBM (main memory)
- \(I\)
- arithmetic intensity: how much math is done for each byte of memory traffic
Three examples, all in BF16 (2 bytes per number):
- Elementwise add \(z = x + y\) on \(n\) numbers: \(n\) FLOPs; read \(2n\) numbers, write \(n\), so \(6n\) bytes. \(I = 1/6 \approx 0.167\). Very low.
- Normalization, softmax, GELU: a few FLOPs per number, but each number is read and written once. \(I\) is around 1 or a bit more. [Likely]
- Matrix multiplication \((M \times K)\) times \((K \times N)\): \(2MKN\) FLOPs (from Session 25), and the bytes for the three matrices:
For square matrices with \(M = K = N = n\), this is \(n/3\): it grows with size. A big matrix multiplication reuses each loaded number many times, because each number of \(A\) meets a whole row of \(B\). That is why it is special.
A case that matters a lot for LLMs: a linear layer with a \(d \times d\) weight, applied to \(M\) tokens (so \(K = N = d\)). Formula (42.3) becomes
When \(M\) is small this is about \(M\) FLOPs per byte; when \(M\) is large it approaches \(d/2\). With \(d = 4096\): one token (\(M = 1\)) gives \(I \approx 1.0\); 64 tokens give 62; 4,096 tokens give 1,365. The number of tokens in the batch, not the size of the model, sets the intensity of the weight matrices.
The roofline model
An operation needs time for its math and time for its memory traffic. If the GPU overlaps the two perfectly, the time is set by the slower one. This is the roofline model (Williams, Waterman and Patterson, 2009):
Divide the FLOPs by this time and you get the best speed the operation can reach:
- Read the symbols
- \(P\)
- peak compute of the GPU in FLOP/s, for the number format you use (dense BF16: 989 TFLOP/s on H100 SXM)
- \(B\)
- memory bandwidth in bytes per second (3.35 TB/s on H100 SXM)
- \(I^{*}\)
- the ridge point: the intensity where the two limits meet
On a log-log plot, \(I \cdot B\) is a sloped line and \(P\) is a flat line: together they look like a roof. An operation with \(I \lt I^{*}\) sits under the slope: it is memory-bound, and a faster memory would make it faster, faster arithmetic would not. An operation with \(I \gt I^{*}\) sits under the flat part: it is compute-bound.
| GPU | Dense BF16 peak \(P\) | Memory bandwidth \(B\) | Ridge point \(I^{*}\) |
|---|---|---|---|
| H100 SXM (HBM3) | 989 TFLOP/s | 3.35 TB/s | 295 FLOP/byte |
| A100 SXM 80 GB (HBM2e) | 312 TFLOP/s | 2.039 TB/s | 153 FLOP/byte |
| RTX 4090 (GDDR6X) | 165.2 TFLOP/s (FP32 accumulate) | 1.008 TB/s | 164 FLOP/byte |
The peaks and bandwidths are NVIDIA’s published numbers (H100 and A100 product pages list them “with sparsity”, which we halve; the 4090 number is from the Ada whitepaper). [Certain] Notice the trend: compute grew faster than bandwidth from A100 to H100, so the ridge point moved up. Newer GPUs need even more reuse per byte to be compute-bound.
Add two BF16 vectors of \(2^{27}\) numbers (134 million). Bytes: \(6 \times 2^{27} = 805\) MB, so memory time is \(805 / 3.35 \times 10^{6} = 240\) µs. Even on the slow FP32 units (67 TFLOP/s), the math takes 2 µs. Memory-bound by more than 100×.
Square matrix multiplication with \(n = 8192\). FLOPs: \(2 \times 8192^3 = 1.10 \times 10^{12}\). Bytes: \(3 \times 8192^2 \times 2 = 403\) MB. \(I = 2731\), far above 295. Math takes 1.11 ms, memory 0.12 ms: compute-bound.
A \(4096 \times 4096\) linear layer. By formula (42.4), it reaches the ridge point when \(M d / (2M + d) = 295.2\), so \(M = 295.2 \times 4096 / (4096 - 590.4) \approx 345\). With fewer than about 345 tokens in the batch, this layer is memory-bound on an H100. On an A100 the same layer needs only about 165 tokens.
Figure 3 runs this estimate step by step. For each batch size it computes \(F\) and \(Q\), the intensity, the two times of formula (42.5), and the attained speed of formula (42.6). The live panel shows every number put into the formulas.
How to read Figure 3
Press Play. The algorithm runs once for each batch size \(M = 1, 2, 4, \dots, 16384\), and the purple trail shows the points already done. At \(M = 1\) (decoding one token for one user) the memory time is 10.02 µs and the math time only 0.03 µs, so the point sits far down the sloped roof: 3.3 TFLOP/s, 0.3% of the peak. At \(M = 64\) the memory time is still the larger one (10.33 µs against 2.17 µs): 207.9 TFLOP/s. The two bars become equal near \(M = 345\), the ridge on the H100, and above it the math bar is longer and the point stays on the flat roof. Switch to the A100: its ridge is lower, so the same layer becomes compute-bound earlier, but its flat roof is also much lower. Switch to “Any operation” and move \(I\): the attained speed is \(I \cdot B\) until \(I^*\), then flat. Elementwise operations never get close to the ridge. They can only get faster by moving fewer bytes, which is what kernel fusion does.
Tiling: how a matrix multiplication reaches the roof
Formula (42.3) assumes each number crosses the HBM boundary only once. A naive kernel does not achieve that. If each thread computes one output \(C_{ij}\) and reads its row of \(A\) and column of \(B\) straight from global memory, it moves \(2K\) numbers for \(2K\) FLOPs: \(I = 0.5\) FLOP per byte in BF16, as bad as an elementwise op.
The fix is tiling. A thread block computes a \(b_M \times b_N\) tile of \(C\). It walks along \(K\) in steps of \(b_K\). At each step it loads a \(b_M \times b_K\) slice of \(A\) and a \(b_K \times b_N\) slice of \(B\) into shared memory, and then every thread reuses those numbers many times from shared memory and registers. The tensor cores consume the tiles.
- input: \(A \in \R^{M \times K}\), \(B \in \R^{K \times N}\) in HBM; tile sizes \(b_M, b_N, b_K\); this block’s tile index \((i, j)\)
- \(\text{acc} \leftarrow 0\) // a \(b_M \times b_N\) FP32 accumulator, held in registers
- for \(k = 0, b_K, 2b_K, \dots \lt K\) do
- load \(A[i b_M : (i+1) b_M,\ k : k + b_K]\) into shared memory // coalesced reads
- load \(B[k : k + b_K,\ j b_N : (j+1) b_N]\) into shared memory
- synchronize the block // wait until both tiles are complete
- \(\text{acc} \leftarrow \text{acc} + A_{\text{tile}} B_{\text{tile}}\) // tensor-core MMA instructions, operands from shared memory and registers
- synchronize the block // before the tiles are overwritten
- write \(\text{acc}\) (converted to BF16) to \(C[i b_M : (i+1) b_M,\ j b_N : (j+1) b_N]\)
Count the traffic. The block does \(2 b_M b_N K\) FLOPs and loads \((b_M + b_N) K\) numbers. So the intensity against global memory is
Larger tiles mean more reuse: square tiles of 16 give 8 FLOP/byte, 128 give 64, and 128 × 256 gives 85. The limit on tile size is the fast memory: the accumulator must fit in registers and the input tiles in shared memory. Neighboring blocks read the same rows of \(A\) and columns of \(B\), so many of these loads hit the 50 MB L2 cache instead of HBM; the real HBM traffic is lower than formula (42.7) suggests. [Certain] Production kernels (cuBLAS, CUTLASS) add more tricks: several tiles in flight at once so loads overlap with math, and on Hopper, a dedicated copy engine (the Tensor Memory Accelerator) and tiles shared across neighboring SMs. [Likely]
Figure 4 runs Algorithm 1 on a real example, small enough to see every number: \(A\) and \(B\) are \(8 \times 8\) matrices of small integers, so \(M = N = K = 8\). You choose the tile sizes and the size of a toy L2 cache. Each step is one line of the algorithm. Tiles fly from HBM (orange) through L2 into shared memory (blue), the tensor core multiplies them into the accumulator in registers, and finished tiles are written back to \(C\) in HBM (green). The counters apply formula (42.2) to the bytes really moved so far.
How to read Figure 4
Press Play, or Step through one line at a time. With the defaults (\(b_M = b_N = 4\), \(b_K = 2\)) the first block computes the top-left \(4 \times 4\) tile of \(C\) in four \(k\) steps. The watched element \(C_{00}\) grows 0 → 11 → 13 → 24 → 24: for example the first step adds \(1 \cdot 2 + 3 \cdot 3 = 11\). At the end, 1,024 FLOPs were done and 512 bytes were loaded into shared memory: \(I = 2.0\), exactly \(b_M b_N/(b_M + b_N)\) from (42.7). Now set \(b_M = b_N = 2\): the same 1,024 FLOPs need 1,024 bytes, \(I = 1.0\). Small tiles reload the same numbers more often. Then change the L2 size. The loads into shared memory do not change, but the HBM traffic does: with room for 64 numbers, 8 of the 32 tile loads hit L2 and HBM reads drop from 512 to 384 bytes; with room for all 128 numbers, every number crosses HBM only once (256 bytes). That is the L2 effect described above.
How to read Figure 5
Start with tiles of 8 × 8: the kernel loads the inputs 512 times over, and the intensity is only 4 FLOP/byte. Grow the tiles and the traffic falls in proportion. Watch the shared-memory line: big tiles with a large \(b_K\) stop fitting, and a kernel that wants two or three tiles in flight (to overlap loads with math) needs two or three times that amount. Tile shape is a balance between reuse and fitting in fast memory.
The same idea, applied to attention, is FlashAttention: it computes attention in tiles held in shared memory and never writes the large \(T \times T\) score matrix to HBM. [Certain] You will see it in session 124. Kernel fusion is the same idea for elementwise chains: do GELU, bias add and dropout in one kernel, so the data crosses HBM once instead of three times (session 50).
MFU: how much of the GPU you really use
The roofline tells you the best case for one operation. For a full training run we want one number that says how close the whole system comes to the peak. That number is model FLOPs utilization (MFU), introduced in the PaLM paper. [Certain]
Here \(F_{\text{token}}\) is the number of FLOPs that the model needs per training token, from the formula, not from a hardware counter. The PaLM paper uses
- Read the symbols
- \(N\)
- number of parameters (without the embedding table is the usual choice; the difference is small for big models)
- \(6N\)
- forward (2N) plus backward (4N) for the weight matrices, from session 25
- \(12 L d T\)
- the attention score and value matmuls, forward and backward; \(L\) layers, model width \(d\), sequence length \(T\)
- \(n_{\text{GPU}}\), \(P\)
- number of GPUs and the dense peak FLOP/s of one GPU
MFU counts only useful model FLOPs. If you use activation checkpointing, the GPU recomputes parts of the forward pass, but those FLOPs do not count in MFU. A second number, hardware FLOPs utilization (HFU), counts them. For PaLM 540B on 6,144 TPU v4 chips: MFU 46.2%, HFU 57.8%. [Certain]
| Run | Hardware | MFU | Source |
|---|---|---|---|
| GPT-3 175B | V100 | 21.3% | PaLM paper, Table 3 [Certain] |
| Megatron-Turing NLG 530B | 2,240 A100 | 30.2% | PaLM paper, Table 3 [Certain] |
| PaLM 540B | 6,144 TPU v4 | 46.2% | PaLM paper [Certain] |
| Llama 3 405B | 8,192 to 16,384 H100 | 38 to 43% (380 to 430 TFLOP/s per GPU) | Llama 3 paper, Table 4 [Certain] |
Check the Llama 3 numbers yourself: \(430 / 989 = 43.5\%\) and \(380 / 989 = 38.4\%\). Even the best large runs use less than half of the peak. Where does the rest go? Memory-bound operations (normalization, softmax, elementwise), communication between GPUs that is not hidden behind compute (session 43), pipeline bubbles, small matmuls that do not fill all 132 SMs, data loading, and kernels that never reach the roof.
You train Llama 3 8B (\(N = 8.03 \times 10^9\), \(L = 32\), \(d = 4096\)) with sequences of \(T = 8192\) tokens and measure 8,000 tokens per second per H100.
Only \(6N\): \(6 \times 8.03 \times 10^9 = 4.82 \times 10^{10}\) FLOPs per token, so \(8000 \times 4.82 \times 10^{10} = 385\) TFLOP/s per GPU, and MFU \(= 385 / 989 = 39.0\%\).
With attention: \(12 \times 32 \times 4096 \times 8192 = 1.29 \times 10^{10}\) more, 27% on top of \(6N\). Now \(F_{\text{token}} = 6.11 \times 10^{10}\) and MFU \(= 49.4\%\).
Same run, same speed, two MFU numbers. Always say which formula you used. With long sequences the attention term is large; for Llama 3 405B at \(T = 8192\) it adds only 8%, because \(d = 16384\) makes \(6N\) much bigger.
How to read Figure 6
Pick Llama 3 405B, H100, \(T = 8192\), and set about 163 tokens per second per GPU: the MFU is about 43%, the Llama 3 paper’s best value. Turn off the attention term and the same throughput gives a lower MFU. Switch to the A100: the same tokens per second would mean a much higher MFU, often above 100%, which is impossible. That is a quick sanity check for any reported throughput.
- input: a model (\(N, L, d\)), sequence length \(T\), measured tokens per second, GPU peak \(P\) and bandwidth \(B\)
- compute MFU with formulas (42.8) and (42.9) // state whether attention is included
- if MFU \(\gt 1\) then the throughput or the FLOP formula is wrong stop
- profile one step; group the kernel time into matmul, attention, elementwise and normalization, communication, idle
- for each large kernel do
- estimate \(F\) and \(Q\); \(I \leftarrow F / Q\); compare with \(I^{*} = P / B\) // formula (42.6)
- if \(I \lt I^{*}\) then reduce bytes: fuse kernels, use FlashAttention, lower precision
- else raise compute efficiency: larger matmul shapes, dimensions multiple of 64, BF16 or FP8 tensor cores
- if much time is communication or idle then overlap communication with compute, change parallelism, check data loading
- re-measure tokens per second and MFU
In code
The script computes the roofline for the linear layer of Figure 3, then measures your own device: the matmul speed (compute) and the speed of an elementwise add (memory bandwidth). It runs on a CPU too (in FP32); the measured numbers depend on the machine, the analytic ones do not.
import time, torch # 1) arithmetic intensity and the roofline, formulas (42.2) to (42.6) PEAK, BW = 989e12, 3.35e12 # H100 SXM: dense BF16 FLOP/s, HBM bytes/s def roofline(flops, nbytes, peak=PEAK, bw=BW): I = flops / nbytes # FLOP per byte t = max(flops / peak, nbytes / bw) # best possible time, formula (42.5) return I, flops / t, "compute" if I > peak / bw else "memory" def gemm(M, K, N, b=2): # (M x K) @ (K x N), b bytes per number return 2 * M * K * N, b * (M * K + K * N + M * N) for M in [1, 64, 345, 4096]: I, perf, bound = roofline(*gemm(M, 4096, 4096)) print(f"M={M:5d} I={I:7.1f} FLOP/B best={perf/1e12:6.1f} TFLOP/s {bound}-bound") n = 2**27 # elementwise z = x + y in BF16 print("add:", round(roofline(n, 3 * 2 * n)[0], 3)) # 0.167 FLOP per byte # 2) measure what your own device really does (numbers depend on the machine) dev = "cuda" if torch.cuda.is_available() else "cpu" dt = torch.bfloat16 if dev == "cuda" else torch.float32 torch.manual_seed(0) def timeit(fn, reps=10): fn() # warm-up (kernel selection, caches) if dev == "cuda": torch.cuda.synchronize() t0 = time.perf_counter() for _ in range(reps): fn() if dev == "cuda": torch.cuda.synchronize() # GPU work is asynchronous return (time.perf_counter() - t0) / reps S = 4096 if dev == "cuda" else 1024 a, b = torch.randn(S, S, device=dev, dtype=dt), torch.randn(S, S, device=dev, dtype=dt) t = timeit(lambda: a @ b) print(f"{dev} matmul {S}^3: {2*S**3/t/1e12:.2f} TFLOP/s") x, y = torch.randn(n // 8, device=dev, dtype=dt), torch.randn(n // 8, device=dev, dtype=dt) t = timeit(lambda: x + y) print(f"{dev} add: {3 * x.numel() * x.element_size() / t / 1e9:.1f} GB/s") # 3) MFU, formulas (42.8) and (42.9) def mfu(tok_per_s_per_gpu, N, L, d, T, peak=PEAK, attention=True): per_tok = 6 * N + (12 * L * d * T if attention else 0) return tok_per_s_per_gpu * per_tok / peak print("8B, 8000 tok/s/GPU:", round(mfu(8000, 8.03e9, 32, 4096, 8192, attention=False), 3), round(mfu(8000, 8.03e9, 32, 4096, 8192), 3)) # 0.39 and 0.494
The first part prints: \(M = 1\) gives \(I = 1.0\) and 3.3 TFLOP/s (memory-bound); \(M = 64\) gives 62.1 and 207.9; \(M = 345\) gives 295.3 and the full 989 (compute-bound); \(M = 4096\) gives 1365.3. On a real H100, a large BF16 matmul typically measures well above half of the peak and the add measures most of the 3.35 TB/s. [Likely] Never time GPU code without torch.cuda.synchronize(): kernel launches return immediately, and you would time only the launch.
Where this is used
| Setting | What the GPU model tells you |
|---|---|
| LLM pretraining | Large matmuls are compute-bound; MFU of 35 to 50% is good at scale [Likely]. Lost time is in memory-bound ops and communication. |
| LLM decoding (chat serving) | Batch 1 is memory-bound (\(I \approx 1\)); throughput grows almost for free with batch size until the ridge (about 345 tokens per weight read on H100). This is why continuous batching and weight quantization exist (session 84). |
| Prefill of a long prompt | Thousands of tokens at once: compute-bound, like training. Prefill and decode have opposite bottlenecks, which motivates splitting them onto different GPUs (session 137). |
| Attention kernels | FlashAttention tiles attention in shared memory to avoid HBM traffic for the \(T \times T\) matrix. |
| Video classification and action recognition | A video model with many frames produces many tokens per clip, so the matmuls are large and compute-bound; decoding the video and moving frames to the GPU can become the bottleneck instead [Likely]. |
| Video retrieval and fingerprinting at scale | Embedding millions of frames is a large-batch, compute-bound job; the nearest-neighbor search over embeddings is a matmul with a small query batch, often memory-bound. |
| Choosing a GPU | Compare peak FLOP/s, bandwidth and memory size against your workload: a consumer RTX 4090 has a similar ridge point to an A100 but only 24 GB of GDDR6X. |
Common mistakes
- Using the sparse peak. MFU against 1,979 TFLOP/s instead of 989 looks half as good, and comparisons with papers become wrong.
- Timing without synchronizing. CUDA calls are asynchronous; without
torch.cuda.synchronize()(or CUDA events) you measure the launch, not the work. Also skip the first call (warm-up). - Thinking more FLOPs always means more time. A memory-bound op runs at the bandwidth limit; removing FLOPs from it changes nothing. Removing bytes does.
- Odd matrix shapes. Dimensions that are not multiples of 8, 16 or 64 can waste tensor-core tiles or fall back to slower paths. A vocabulary of 50,257 is often padded to 50,304 for this reason. [Likely]
- Comparing MFU numbers with different formulas. With or without the attention term, with or without embeddings: the same run can show 39% or 49%.
- Confusing MFU and HFU. Recomputed activations (checkpointing) raise HFU but not MFU. MFU is the honest measure of useful work.
- Too little work for 132 SMs. A small matmul makes fewer tiles than there are SMs; most of the GPU idles. Larger batches or fusing small ops into bigger ones helps.
- Ignoring the format. An FP32 matmul without TF32 runs on CUDA cores at 67 TFLOP/s, not on the tensor cores. Use BF16 autocast (or enable TF32).
Interview questions
What is a warp, and why does it matter for performance?basic
What is an SM? How many does an H100 have?basic
Explain the GPU memory hierarchy with numbers.core
What is arithmetic intensity? Give it for an elementwise add and a square matmul in BF16.core
What is the roofline model? What is the ridge point of an H100?core
Why is batch-1 LLM decoding memory-bound, and how do we fix it?core
How many tokens must a 4096 × 4096 linear layer see to be compute-bound on an H100?advanced
Why does tiling make a matrix multiplication fast?core
Define MFU. How is it different from HFU, and what values are typical?core
Your 8B run reports 60% MFU on H100. Your colleague’s reports 45% at the same speed. How?advanced
Why do GPUs need many warps per SM? What limits that number?advanced
Exercises
- What is the arithmetic intensity of \(y = \text{GELU}(x)\) in BF16 if GELU costs about 8 FLOPs per element? Is it memory-bound on an A100?
- A 70B model in BF16 is served at batch size 1 on one GPU with 3.35 TB/s (ignore that it does not fit in 80 GB). What is the best possible time per token?
- Compute the ridge point of an RTX 4090 and say whether a \(2048 \times 2048 \times 2048\) BF16 matmul is compute-bound on it.
- For a \(d = 8192\) linear layer, how many tokens are needed to reach the H100 ridge point?
- A kernel uses tiles with \(b_M = 128\), \(b_N = 64\). What is its intensity against global memory? By how much does it beat 64 × 64 tiles?
- Llama 3 70B (\(L = 80\), \(d = 8192\), \(N = 70.6 \times 10^9\)) trains at 1,000 tokens per second per H100 with \(T = 8192\). Compute the MFU with and without the attention term.
Answerscheck
Key takeaways
- A GPU is many SMs (132 on H100), each running warps of 32 threads in lockstep; tensor cores do small matrix multiply-accumulates and give about 15 times the FP32 throughput on H100.
- The scheduler hides memory latency by switching between warps (42.1); occupancy is limited by registers and shared memory.
- Memory goes from registers and shared memory (KB per SM, fast) to L2 (50 MB) to HBM (80 GB, 3.35 TB/s). Fast kernels load once and reuse on the chip.
- Arithmetic intensity (42.2) and the roofline (42.5, 42.6) tell you whether an operation is memory-bound or compute-bound; the H100 ridge is about 295 FLOP per byte.
- Elementwise ops and batch-1 decoding are memory-bound; large matmuls are compute-bound once \(M\) is above about 345 on H100 (42.4); tiling (42.7) is how matmuls reach the roof.
- MFU (42.8, 42.9) measures useful work against the dense peak. Good large runs reach about 40 to 46%; always state the FLOP formula you used.
Key formulas
References for this session
- NVIDIA (docs). CUDA C++ Programming Guide
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- NVIDIA (2022). NVIDIA Hopper Architecture In-Depth (technical blog)
- NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA A100 Tensor Core GPU: product specifications
- NVIDIA (2022). NVIDIA Ada GPU Architecture (whitepaper, RTX 4090 specifications)