LLM & VLM courseBlog
42Track 3: Pretraining and scale · session 12 of 22

GPU architecture for ML

A GPU is thousands of small arithmetic units fed by a memory system that is much slower than the arithmetic. This session opens the box: streaming multiprocessors, warps and tensor cores, then registers, shared memory, L2 and HBM. With the roofline model and arithmetic intensity you will predict whether an operation is limited by compute or by memory, and with MFU you will measure how much of the GPU a training run really uses.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 25 and 37 (counting FLOPs and memory; number formats on GPUs)

What you will learn

  • How a GPU is built: streaming multiprocessors (SMs), warps of 32 threads, CUDA cores and tensor cores
  • Why GPUs run many warps at once: hiding memory latency by switching between warps
  • The memory hierarchy: registers, shared memory (SRAM), L2 cache and HBM or GDDR, with real sizes and speeds for H100, A100 and RTX 4090
  • Arithmetic intensity and the roofline: how to tell if an operation is compute-bound or memory-bound
  • Why a matrix multiplication needs about 345 rows (tokens) on an H100 to become compute-bound, and why batch-1 decoding is memory-bound
  • How tiling in shared memory raises the arithmetic intensity of a matrix multiplication
  • Model FLOPs utilization (MFU): how to compute it, what real runs reach (PaLM 46.2%, Llama 3 405B 38 to 43%), and why the definition matters

Why this matters

An H100 GPU can do about 989 trillion floating-point operations (FLOPs) per second in BF16, but it can only read about 3.35 trillion bytes per second from its main memory. [Certain] That gap, about 295 FLOPs for every byte, decides almost everything about how fast your model trains and runs.

In session 25 you learned to count FLOPs, and you turned FLOPs into GPU-days with a number called MFU. In session 37 you saw why BF16 and FP8 are faster than FP32. This session explains the machine underneath. Why is a large matrix multiplication fast, but a LayerNorm or a softmax slow per FLOP? Why does generating one token at a time use less than 1% of the GPU’s arithmetic? Why are FlashAttention and kernel fusion so important? The answers all come from two facts: how the GPU is built, and how its memory works.

You do not need to write CUDA to use this knowledge. Choosing a batch size, deciding whether to fuse two operations, reading a profiler trace, and estimating the cost of a training run all need the same mental model. Writing kernels comes later, in session 85 and session 86.

A GPU is a throughput machine

A CPU has a few large cores. Each core is built to finish one stream of instructions as fast as possible: big caches, branch prediction, out-of-order execution. A GPU makes the opposite choice. It has many small, simple cores and very little control logic per core. It is slow at any single task but fast at doing the same operation on millions of numbers at once. We say a CPU is built for latency (time for one task) and a GPU for throughput (tasks finished per second).

Deep learning is almost all banquet work. A transformer layer is mostly large matrix multiplications plus simple element-by-element operations (adds, GELU, normalization). All of these apply the same rule to many numbers. That is why GPUs, not CPUs, train models.

Streaming multiprocessors, warps and threads

The GPU chip is divided into many identical units called streaming multiprocessors (SMs). The H100 SXM has 132 SMs, the A100 has 108, and the RTX 4090 has 128. [Certain] Each SM is a small processor of its own, with its own arithmetic units, its own fast memory and its own schedulers. All SMs share the large L2 cache and the main memory.

The programming model: threads, blocks, grid

When you launch work on a GPU, you launch a kernel: one function that many threads run at the same time, each on a different piece of the data. Threads are grouped into thread blocks (often 128 to 1024 threads). The whole launch is a grid of blocks. Each block runs on one SM from start to finish, and threads of the same block can share data through the SM’s fast memory. Blocks cannot talk to each other cheaply, so different blocks should do independent work (for example, different tiles of the output matrix).

Warps

The hardware does not run threads one by one. It runs them in groups of 32 called warps. [Certain] All 32 threads of a warp execute the same instruction at the same moment, each on its own data. NVIDIA calls this SIMT (single instruction, multiple threads). Two consequences matter for ML:

  • Divergence. If threads in one warp take different branches of an if, the warp runs both branches one after the other, with some threads switched off. Work that follows one path for all 32 threads is faster.
  • Coalescing. When the 32 threads read 32 neighboring numbers, the hardware merges this into a few wide memory transactions. Scattered reads waste bandwidth. This is why memory layout (contiguous tensors, which dimension is last) affects speed.

An H100 SM can hold up to 64 warps (2,048 threads) at the same time. [Certain] It is divided into four parts (sub-partitions), each with its own warp scheduler. [Likely] Every clock cycle, each scheduler picks one warp that is ready and issues its next instruction.

CUDA cores and tensor cores

Each SM has two kinds of arithmetic units. CUDA cores do ordinary scalar math: one FP32 add or multiply per lane. Tensor cores do a small matrix multiply-accumulate in one instruction: \(D = A B + C\) on small tiles (for example, a few 16 × 8 or 16 × 16 blocks), with low-precision inputs (FP16, BF16, FP8) and an FP32 accumulator. The H100 has 4 tensor cores per SM, 528 in total. [Certain]

The difference in speed is huge. On the H100 SXM, ordinary FP32 math peaks at 67 TFLOP/s, while dense BF16 on the tensor cores peaks at 989 TFLOP/s, about 14.8 times more. [Certain] So in practice, only matrix multiplications can reach the GPU’s headline number. Everything else (softmax, normalization, activation functions, residual adds) runs on the CUDA cores at a much lower peak, and as we will see, it is usually limited by memory anyway.

Figure 1 walks from the full chip down to one warp, and then follows the data of one matrix multiplication on its way from main memory to a tensor core.

Figure 1. Inside an H100 SXM, step by step. The chip has 132 SMs that share a 50 MB L2 cache and 80 GB of HBM3. Each SM has four sub-partitions with a warp scheduler, registers, CUDA cores and a tensor core, plus up to 228 KB of shared memory.
How to read Figure 1

Step through with Next. The colors follow the data: orange for the big, slow memory (HBM), blue for the fast memory close to the arithmetic (shared memory and registers), and green for the units that compute. The last steps show a tile of a matrix moving from HBM through L2 and shared memory into registers, where the tensor core uses it many times. That reuse is the whole trick of fast GPU code.

Hiding latency with many warps

A read from HBM takes hundreds of clock cycles to come back. [Likely] A CPU would try to avoid this wait with big caches. A GPU instead hides the wait: when one warp is waiting for memory, the scheduler simply issues an instruction from another warp that is ready. If there are enough warps, the arithmetic units never sit idle.

A simple model shows how many warps are enough. Say each warp computes for \(c\) cycles, then waits \(\ell\) cycles for memory, again and again. One scheduler can keep busy only if, during one warp’s wait, other warps have enough compute to fill it:

\[ U \approx \min\!\left(1,\ \frac{W\, c}{c + \ell}\right) \tag{42.1} \]
Read the symbols
\(U\)
utilization: the fraction of cycles in which the scheduler issues useful work
\(W\)
number of warps that the scheduler can choose from
\(c\)
cycles of compute a warp does before it needs new data from memory
\(\ell\)
memory latency: cycles until the data arrives

So you need about \(W \ge (c + \ell)/c\) warps. With \(c = 10\) and \(\ell = 90\), that is 10 warps. The fraction of the SM’s warp slots that are filled is called occupancy. Each warp needs registers and shared memory, and the SM has a fixed amount of both, so a kernel that uses many registers per thread can fit fewer warps. This is a real trade-off for kernel writers. Formula (42.1) is a teaching model, not an exact law. [Certain] Real kernels also hide latency inside one warp, by starting the next memory loads before they need them (prefetching, pipelining).

computing (issuing)waiting for memoryready but not chosen
Figure 2. One warp scheduler with \(W\) warps, simulated cycle by cycle. Each warp computes for \(c\) cycles, then waits \(\ell\) cycles for memory. The bottom row shows whether the scheduler did useful work in each cycle.
How to read Figure 2

With 1 warp, the scheduler is idle most of the time: it waits with the warp. Add warps and the gaps fill in, until the bottom row is solid and utilization reaches 100%. The readout compares the simulation with formula (42.1). Then raise the latency: you need more warps for the same utilization. Raise the compute per load (\(c\)): you need fewer. A kernel that does more math per byte loaded is easier to keep busy. The next sections turn this idea into the roofline.

The memory hierarchy

A GPU has several levels of memory. The closer a level is to the arithmetic units, the faster and the smaller it is. The table gives the numbers for an H100 SXM.

LevelWhereSize (H100 SXM)Who can see itSpeed
Registersinside each SM sub-partition256 KB per SM (65,536 registers of 32 bits); about 33 MB in totalone threadfastest; read every cycle
Shared memory (SRAM) / L1inside each SM256 KB per SM combined, of which up to 228 KB can be shared memoryall threads of one blockvery fast, a few tens of cycles [Likely]
L2 cacheon the chip, shared by all SMs50 MBall SMsfaster than HBM; hundreds of cycles [Likely]
HBM3 (main GPU memory)stacks next to the chip80 GBeverything; the host copies data here3.35 TB/s; hundreds of cycles of latency [Likely]

The sizes and the HBM bandwidth are from NVIDIA’s Hopper documentation and H100 product page. [Certain] Shared memory is SRAM (static RAM) on the chip. The programmer controls it directly: a kernel says “load this tile into shared memory”, and all threads of the block can then read it many times. That makes it the main tool for reuse. HBM (high-bandwidth memory) is DRAM stacked next to the GPU die; it is large but far away. Consumer GPUs use GDDR memory instead, which is cheaper and slower: the RTX 4090 has 24 GB of GDDR6X at 1,008 GB/s and a 72 MB L2 cache. [Certain]

Worked example: how long does it take just to read the weights?

Llama 3 8B has about 8.03 billion parameters. In BF16 (2 bytes each), the weights are \(8.03 \times 10^9 \times 2 = 16.06\) GB. Reading them once from HBM at 3.35 TB/s takes \(16.06 / 3350 = 4.79\) ms.

When a chatbot generates text for one user, every new token needs all the weights once. So even with infinitely fast arithmetic, one H100 cannot produce more than about \(1 / 4.79\ \text{ms} \approx 209\) tokens per second for one user. Memory, not compute, sets this limit. Reading the whole 80 GB of HBM takes 23.9 ms.

Arithmetic intensity

Every operation does some FLOPs and moves some bytes between HBM and the chip. Their ratio is the arithmetic intensity:

\[ I = \frac{F}{Q} \quad \left[\frac{\text{FLOP}}{\text{byte}}\right] \tag{42.2} \]
Read the symbols
\(F\)
the number of floating-point operations the kernel does (a multiply-add counts as 2)
\(Q\)
the number of bytes it reads from and writes to HBM (main memory)
\(I\)
arithmetic intensity: how much math is done for each byte of memory traffic

Three examples, all in BF16 (2 bytes per number):

  • Elementwise add \(z = x + y\) on \(n\) numbers: \(n\) FLOPs; read \(2n\) numbers, write \(n\), so \(6n\) bytes. \(I = 1/6 \approx 0.167\). Very low.
  • Normalization, softmax, GELU: a few FLOPs per number, but each number is read and written once. \(I\) is around 1 or a bit more. [Likely]
  • Matrix multiplication \((M \times K)\) times \((K \times N)\): \(2MKN\) FLOPs (from Session 25), and the bytes for the three matrices:
\[ I_{\text{GEMM}} = \frac{2MKN}{2\,(MK + KN + MN)} = \frac{MKN}{MK + KN + MN} \tag{42.3} \]

For square matrices with \(M = K = N = n\), this is \(n/3\): it grows with size. A big matrix multiplication reuses each loaded number many times, because each number of \(A\) meets a whole row of \(B\). That is why it is special.

A case that matters a lot for LLMs: a linear layer with a \(d \times d\) weight, applied to \(M\) tokens (so \(K = N = d\)). Formula (42.3) becomes

\[ I = \frac{M d}{2M + d} \tag{42.4} \]

When \(M\) is small this is about \(M\) FLOPs per byte; when \(M\) is large it approaches \(d/2\). With \(d = 4096\): one token (\(M = 1\)) gives \(I \approx 1.0\); 64 tokens give 62; 4,096 tokens give 1,365. The number of tokens in the batch, not the size of the model, sets the intensity of the weight matrices.

The roofline model

An operation needs time for its math and time for its memory traffic. If the GPU overlaps the two perfectly, the time is set by the slower one. This is the roofline model (Williams, Waterman and Patterson, 2009):

\[ t \ \ge\ \max\!\left(\frac{F}{P},\ \frac{Q}{B}\right) \tag{42.5} \]

Divide the FLOPs by this time and you get the best speed the operation can reach:

\[ \text{attainable FLOP/s} = \min\big(P,\ I \cdot B\big), \qquad I^{*} = \frac{P}{B} \tag{42.6} \]
Read the symbols
\(P\)
peak compute of the GPU in FLOP/s, for the number format you use (dense BF16: 989 TFLOP/s on H100 SXM)
\(B\)
memory bandwidth in bytes per second (3.35 TB/s on H100 SXM)
\(I^{*}\)
the ridge point: the intensity where the two limits meet

On a log-log plot, \(I \cdot B\) is a sloped line and \(P\) is a flat line: together they look like a roof. An operation with \(I \lt I^{*}\) sits under the slope: it is memory-bound, and a faster memory would make it faster, faster arithmetic would not. An operation with \(I \gt I^{*}\) sits under the flat part: it is compute-bound.

GPUDense BF16 peak \(P\)Memory bandwidth \(B\)Ridge point \(I^{*}\)
H100 SXM (HBM3)989 TFLOP/s3.35 TB/s295 FLOP/byte
A100 SXM 80 GB (HBM2e)312 TFLOP/s2.039 TB/s153 FLOP/byte
RTX 4090 (GDDR6X)165.2 TFLOP/s (FP32 accumulate)1.008 TB/s164 FLOP/byte

The peaks and bandwidths are NVIDIA’s published numbers (H100 and A100 product pages list them “with sparsity”, which we halve; the 4090 number is from the Ada whitepaper). [Certain] Notice the trend: compute grew faster than bandwidth from A100 to H100, so the ridge point moved up. Newer GPUs need even more reuse per byte to be compute-bound.

Worked example: three operations on an H100

Add two BF16 vectors of \(2^{27}\) numbers (134 million). Bytes: \(6 \times 2^{27} = 805\) MB, so memory time is \(805 / 3.35 \times 10^{6} = 240\) µs. Even on the slow FP32 units (67 TFLOP/s), the math takes 2 µs. Memory-bound by more than 100×.

Square matrix multiplication with \(n = 8192\). FLOPs: \(2 \times 8192^3 = 1.10 \times 10^{12}\). Bytes: \(3 \times 8192^2 \times 2 = 403\) MB. \(I = 2731\), far above 295. Math takes 1.11 ms, memory 0.12 ms: compute-bound.

A \(4096 \times 4096\) linear layer. By formula (42.4), it reaches the ridge point when \(M d / (2M + d) = 295.2\), so \(M = 295.2 \times 4096 / (4096 - 590.4) \approx 345\). With fewer than about 345 tokens in the batch, this layer is memory-bound on an H100. On an A100 the same layer needs only about 165 tokens.

Figure 3 runs this estimate step by step. For each batch size it computes \(F\) and \(Q\), the intensity, the two times of formula (42.5), and the attained speed of formula (42.6). The live panel shows every number put into the formulas.

memory roof \(I \cdot B\) and memory timecompute roof \(P\) and math timethe current operationpoints already visited
Algorithm: roofline estimate for one operation
    Figure 3. The roofline (formula 42.6) on log-log axes, with real peak numbers, computed in your browser. In “Linear layer” mode the blue point is a \(4096 \times 4096\) BF16 linear layer applied to \(M\) tokens (formula 42.4): \(F = 2Md^2\) and \(Q = 2(2Md + d^2)\) bytes. In “Any operation” mode you set \(I\) directly, and the operation moves \(Q = 1\) GB. The bars below the plot are the two times of formula (42.5). Gray points are fixed operations; click any of them.
    How to read Figure 3

    Press Play. The algorithm runs once for each batch size \(M = 1, 2, 4, \dots, 16384\), and the purple trail shows the points already done. At \(M = 1\) (decoding one token for one user) the memory time is 10.02 µs and the math time only 0.03 µs, so the point sits far down the sloped roof: 3.3 TFLOP/s, 0.3% of the peak. At \(M = 64\) the memory time is still the larger one (10.33 µs against 2.17 µs): 207.9 TFLOP/s. The two bars become equal near \(M = 345\), the ridge on the H100, and above it the math bar is longer and the point stays on the flat roof. Switch to the A100: its ridge is lower, so the same layer becomes compute-bound earlier, but its flat roof is also much lower. Switch to “Any operation” and move \(I\): the attained speed is \(I \cdot B\) until \(I^*\), then flat. Elementwise operations never get close to the ridge. They can only get faster by moving fewer bytes, which is what kernel fusion does.

    Tiling: how a matrix multiplication reaches the roof

    Formula (42.3) assumes each number crosses the HBM boundary only once. A naive kernel does not achieve that. If each thread computes one output \(C_{ij}\) and reads its row of \(A\) and column of \(B\) straight from global memory, it moves \(2K\) numbers for \(2K\) FLOPs: \(I = 0.5\) FLOP per byte in BF16, as bad as an elementwise op.

    The fix is tiling. A thread block computes a \(b_M \times b_N\) tile of \(C\). It walks along \(K\) in steps of \(b_K\). At each step it loads a \(b_M \times b_K\) slice of \(A\) and a \(b_K \times b_N\) slice of \(B\) into shared memory, and then every thread reuses those numbers many times from shared memory and registers. The tensor cores consume the tiles.

    Algorithm 1: tiled matrix multiplication \(C = AB\) on one thread block
    1. input: \(A \in \R^{M \times K}\), \(B \in \R^{K \times N}\) in HBM; tile sizes \(b_M, b_N, b_K\); this block’s tile index \((i, j)\)
    2. \(\text{acc} \leftarrow 0\) // a \(b_M \times b_N\) FP32 accumulator, held in registers
    3. for \(k = 0, b_K, 2b_K, \dots \lt K\) do
    4. load \(A[i b_M : (i+1) b_M,\ k : k + b_K]\) into shared memory // coalesced reads
    5. load \(B[k : k + b_K,\ j b_N : (j+1) b_N]\) into shared memory
    6. synchronize the block // wait until both tiles are complete
    7. \(\text{acc} \leftarrow \text{acc} + A_{\text{tile}} B_{\text{tile}}\) // tensor-core MMA instructions, operands from shared memory and registers
    8. synchronize the block // before the tiles are overwritten
    9. write \(\text{acc}\) (converted to BF16) to \(C[i b_M : (i+1) b_M,\ j b_N : (j+1) b_N]\)

    Count the traffic. The block does \(2 b_M b_N K\) FLOPs and loads \((b_M + b_N) K\) numbers. So the intensity against global memory is

    \[ I_{\text{tile}} = \frac{2\, b_M b_N K}{2\,(b_M + b_N) K} = \frac{b_M\, b_N}{b_M + b_N} \tag{42.7} \]

    Larger tiles mean more reuse: square tiles of 16 give 8 FLOP/byte, 128 give 64, and 128 × 256 gives 85. The limit on tile size is the fast memory: the accumulator must fit in registers and the input tiles in shared memory. Neighboring blocks read the same rows of \(A\) and columns of \(B\), so many of these loads hit the 50 MB L2 cache instead of HBM; the real HBM traffic is lower than formula (42.7) suggests. [Certain] Production kernels (cuBLAS, CUTLASS) add more tricks: several tiles in flight at once so loads overlap with math, and on Hopper, a dedicated copy engine (the Tensor Memory Accelerator) and tiles shared across neighboring SMs. [Likely]

    Figure 4 runs Algorithm 1 on a real example, small enough to see every number: \(A\) and \(B\) are \(8 \times 8\) matrices of small integers, so \(M = N = K = 8\). You choose the tile sizes and the size of a toy L2 cache. Each step is one line of the algorithm. Tiles fly from HBM (orange) through L2 into shared memory (blue), the tensor core multiplies them into the accumulator in registers, and finished tiles are written back to \(C\) in HBM (green). The counters apply formula (42.2) to the bytes really moved so far.

    \(b_M\) \(b_N\) \(b_K\) L2 holds numbers
    read from HBM (L2 miss)L2 hit / finished C tiletile in shared memoryaccumulator in registers
    Algorithm 1, running on the whole grid of blocks
      Figure 4. Algorithm 1 computed in your browser on \(A_{rc} = (3r + 2c + 1) \bmod 5\) and \(B_{rc} = (r + 3c + 2) \bmod 4\), with numbers in BF16 (2 bytes). Blocks run one after another in row order (on a real GPU they run in parallel on different SMs and share L2). The toy L2 keeps the most recently used tiles and throws out the oldest when it is full. Shaded cells in the L2 maps are the tiles it holds now. The watched number is the top-left element of the current output tile.
      How to read Figure 4

      Press Play, or Step through one line at a time. With the defaults (\(b_M = b_N = 4\), \(b_K = 2\)) the first block computes the top-left \(4 \times 4\) tile of \(C\) in four \(k\) steps. The watched element \(C_{00}\) grows 0 → 11 → 13 → 24 → 24: for example the first step adds \(1 \cdot 2 + 3 \cdot 3 = 11\). At the end, 1,024 FLOPs were done and 512 bytes were loaded into shared memory: \(I = 2.0\), exactly \(b_M b_N/(b_M + b_N)\) from (42.7). Now set \(b_M = b_N = 2\): the same 1,024 FLOPs need 1,024 bytes, \(I = 1.0\). Small tiles reload the same numbers more often. Then change the L2 size. The loads into shared memory do not change, but the HBM traffic does: with room for 64 numbers, 8 of the 32 tile loads hit L2 and HBM reads drop from 512 to 384 bytes; with room for all 128 numbers, every number crosses HBM only once (256 bytes). That is the L2 effect described above.

      Figure 5. Tiling a BF16 matrix multiplication with \(M = N = K = 4096\). Each output tile (blue) needs a strip of \(A\) and a strip of \(B\) (orange). The readout gives the global-memory traffic, the intensity from formula (42.7), and the shared memory that one pair of input tiles uses on an H100 (up to 228 KB per SM).
      How to read Figure 5

      Start with tiles of 8 × 8: the kernel loads the inputs 512 times over, and the intensity is only 4 FLOP/byte. Grow the tiles and the traffic falls in proportion. Watch the shared-memory line: big tiles with a large \(b_K\) stop fitting, and a kernel that wants two or three tiles in flight (to overlap loads with math) needs two or three times that amount. Tile shape is a balance between reuse and fitting in fast memory.

      The same idea, applied to attention, is FlashAttention: it computes attention in tiles held in shared memory and never writes the large \(T \times T\) score matrix to HBM. [Certain] You will see it in session 124. Kernel fusion is the same idea for elementwise chains: do GELU, bias add and dropout in one kernel, so the data crosses HBM once instead of three times (session 50).

      MFU: how much of the GPU you really use

      The roofline tells you the best case for one operation. For a full training run we want one number that says how close the whole system comes to the peak. That number is model FLOPs utilization (MFU), introduced in the PaLM paper. [Certain]

      \[ \text{MFU} = \frac{\text{tokens per second} \times F_{\text{token}}}{n_{\text{GPU}} \times P} \tag{42.8} \]

      Here \(F_{\text{token}}\) is the number of FLOPs that the model needs per training token, from the formula, not from a hardware counter. The PaLM paper uses

      \[ F_{\text{token}} = 6N + 12\, L\, d\, T \tag{42.9} \]
      Read the symbols
      \(N\)
      number of parameters (without the embedding table is the usual choice; the difference is small for big models)
      \(6N\)
      forward (2N) plus backward (4N) for the weight matrices, from session 25
      \(12 L d T\)
      the attention score and value matmuls, forward and backward; \(L\) layers, model width \(d\), sequence length \(T\)
      \(n_{\text{GPU}}\), \(P\)
      number of GPUs and the dense peak FLOP/s of one GPU

      MFU counts only useful model FLOPs. If you use activation checkpointing, the GPU recomputes parts of the forward pass, but those FLOPs do not count in MFU. A second number, hardware FLOPs utilization (HFU), counts them. For PaLM 540B on 6,144 TPU v4 chips: MFU 46.2%, HFU 57.8%. [Certain]

      RunHardwareMFUSource
      GPT-3 175BV10021.3%PaLM paper, Table 3 [Certain]
      Megatron-Turing NLG 530B2,240 A10030.2%PaLM paper, Table 3 [Certain]
      PaLM 540B6,144 TPU v446.2%PaLM paper [Certain]
      Llama 3 405B8,192 to 16,384 H10038 to 43% (380 to 430 TFLOP/s per GPU)Llama 3 paper, Table 4 [Certain]

      Check the Llama 3 numbers yourself: \(430 / 989 = 43.5\%\) and \(380 / 989 = 38.4\%\). Even the best large runs use less than half of the peak. Where does the rest go? Memory-bound operations (normalization, softmax, elementwise), communication between GPUs that is not hidden behind compute (session 43), pipeline bubbles, small matmuls that do not fill all 132 SMs, data loading, and kernels that never reach the roof.

      Worked example: MFU of an 8B model on H100s

      You train Llama 3 8B (\(N = 8.03 \times 10^9\), \(L = 32\), \(d = 4096\)) with sequences of \(T = 8192\) tokens and measure 8,000 tokens per second per H100.

      Only \(6N\): \(6 \times 8.03 \times 10^9 = 4.82 \times 10^{10}\) FLOPs per token, so \(8000 \times 4.82 \times 10^{10} = 385\) TFLOP/s per GPU, and MFU \(= 385 / 989 = 39.0\%\).

      With attention: \(12 \times 32 \times 4096 \times 8192 = 1.29 \times 10^{10}\) more, 27% on top of \(6N\). Now \(F_{\text{token}} = 6.11 \times 10^{10}\) and MFU \(= 49.4\%\).

      Same run, same speed, two MFU numbers. Always say which formula you used. With long sequences the attention term is large; for Llama 3 405B at \(T = 8192\) it adds only 8%, because \(d = 16384\) makes \(6N\) much bigger.

      Figure 6. An MFU calculator built on formulas (42.8) and (42.9). Model shapes are from the Llama 3 configs (8B: 32 layers, width 4096; 70B: 80 and 8192; 405B: 126 and 16384). The bar shows the achieved model TFLOP/s per GPU against the dense BF16 peak.
      How to read Figure 6

      Pick Llama 3 405B, H100, \(T = 8192\), and set about 163 tokens per second per GPU: the MFU is about 43%, the Llama 3 paper’s best value. Turn off the attention term and the same throughput gives a lower MFU. Switch to the A100: the same tokens per second would mean a much higher MFU, often above 100%, which is impossible. That is a quick sanity check for any reported throughput.

      Algorithm 2: diagnose a slow training step with the roofline and MFU
      1. input: a model (\(N, L, d\)), sequence length \(T\), measured tokens per second, GPU peak \(P\) and bandwidth \(B\)
      2. compute MFU with formulas (42.8) and (42.9) // state whether attention is included
      3. if MFU \(\gt 1\) then the throughput or the FLOP formula is wrong stop
      4. profile one step; group the kernel time into matmul, attention, elementwise and normalization, communication, idle
      5. for each large kernel do
      6. estimate \(F\) and \(Q\); \(I \leftarrow F / Q\); compare with \(I^{*} = P / B\) // formula (42.6)
      7. if \(I \lt I^{*}\) then reduce bytes: fuse kernels, use FlashAttention, lower precision
      8. else raise compute efficiency: larger matmul shapes, dimensions multiple of 64, BF16 or FP8 tensor cores
      9. if much time is communication or idle then overlap communication with compute, change parallelism, check data loading
      10. re-measure tokens per second and MFU

      In code

      The script computes the roofline for the linear layer of Figure 3, then measures your own device: the matmul speed (compute) and the speed of an elementwise add (memory bandwidth). It runs on a CPU too (in FP32); the measured numbers depend on the machine, the analytic ones do not.

      import time, torch
      
      # 1) arithmetic intensity and the roofline, formulas (42.2) to (42.6)
      PEAK, BW = 989e12, 3.35e12                  # H100 SXM: dense BF16 FLOP/s, HBM bytes/s
      def roofline(flops, nbytes, peak=PEAK, bw=BW):
          I = flops / nbytes                      # FLOP per byte
          t = max(flops / peak, nbytes / bw)      # best possible time, formula (42.5)
          return I, flops / t, "compute" if I > peak / bw else "memory"
      
      def gemm(M, K, N, b=2):                     # (M x K) @ (K x N), b bytes per number
          return 2 * M * K * N, b * (M * K + K * N + M * N)
      
      for M in [1, 64, 345, 4096]:
          I, perf, bound = roofline(*gemm(M, 4096, 4096))
          print(f"M={M:5d}  I={I:7.1f} FLOP/B  best={perf/1e12:6.1f} TFLOP/s  {bound}-bound")
      n = 2**27                                   # elementwise z = x + y in BF16
      print("add:", round(roofline(n, 3 * 2 * n)[0], 3))   # 0.167 FLOP per byte
      
      # 2) measure what your own device really does (numbers depend on the machine)
      dev = "cuda" if torch.cuda.is_available() else "cpu"
      dt = torch.bfloat16 if dev == "cuda" else torch.float32
      torch.manual_seed(0)
      def timeit(fn, reps=10):
          fn()                                    # warm-up (kernel selection, caches)
          if dev == "cuda": torch.cuda.synchronize()
          t0 = time.perf_counter()
          for _ in range(reps): fn()
          if dev == "cuda": torch.cuda.synchronize()   # GPU work is asynchronous
          return (time.perf_counter() - t0) / reps
      
      S = 4096 if dev == "cuda" else 1024
      a, b = torch.randn(S, S, device=dev, dtype=dt), torch.randn(S, S, device=dev, dtype=dt)
      t = timeit(lambda: a @ b)
      print(f"{dev} matmul {S}^3: {2*S**3/t/1e12:.2f} TFLOP/s")
      x, y = torch.randn(n // 8, device=dev, dtype=dt), torch.randn(n // 8, device=dev, dtype=dt)
      t = timeit(lambda: x + y)
      print(f"{dev} add: {3 * x.numel() * x.element_size() / t / 1e9:.1f} GB/s")
      
      # 3) MFU, formulas (42.8) and (42.9)
      def mfu(tok_per_s_per_gpu, N, L, d, T, peak=PEAK, attention=True):
          per_tok = 6 * N + (12 * L * d * T if attention else 0)
          return tok_per_s_per_gpu * per_tok / peak
      print("8B, 8000 tok/s/GPU:", round(mfu(8000, 8.03e9, 32, 4096, 8192, attention=False), 3),
            round(mfu(8000, 8.03e9, 32, 4096, 8192), 3))   # 0.39 and 0.494

      The first part prints: \(M = 1\) gives \(I = 1.0\) and 3.3 TFLOP/s (memory-bound); \(M = 64\) gives 62.1 and 207.9; \(M = 345\) gives 295.3 and the full 989 (compute-bound); \(M = 4096\) gives 1365.3. On a real H100, a large BF16 matmul typically measures well above half of the peak and the add measures most of the 3.35 TB/s. [Likely] Never time GPU code without torch.cuda.synchronize(): kernel launches return immediately, and you would time only the launch.

      Where this is used

      SettingWhat the GPU model tells you
      LLM pretrainingLarge matmuls are compute-bound; MFU of 35 to 50% is good at scale [Likely]. Lost time is in memory-bound ops and communication.
      LLM decoding (chat serving)Batch 1 is memory-bound (\(I \approx 1\)); throughput grows almost for free with batch size until the ridge (about 345 tokens per weight read on H100). This is why continuous batching and weight quantization exist (session 84).
      Prefill of a long promptThousands of tokens at once: compute-bound, like training. Prefill and decode have opposite bottlenecks, which motivates splitting them onto different GPUs (session 137).
      Attention kernelsFlashAttention tiles attention in shared memory to avoid HBM traffic for the \(T \times T\) matrix.
      Video classification and action recognitionA video model with many frames produces many tokens per clip, so the matmuls are large and compute-bound; decoding the video and moving frames to the GPU can become the bottleneck instead [Likely].
      Video retrieval and fingerprinting at scaleEmbedding millions of frames is a large-batch, compute-bound job; the nearest-neighbor search over embeddings is a matmul with a small query batch, often memory-bound.
      Choosing a GPUCompare peak FLOP/s, bandwidth and memory size against your workload: a consumer RTX 4090 has a similar ridge point to an A100 but only 24 GB of GDDR6X.

      Common mistakes

      • Using the sparse peak. MFU against 1,979 TFLOP/s instead of 989 looks half as good, and comparisons with papers become wrong.
      • Timing without synchronizing. CUDA calls are asynchronous; without torch.cuda.synchronize() (or CUDA events) you measure the launch, not the work. Also skip the first call (warm-up).
      • Thinking more FLOPs always means more time. A memory-bound op runs at the bandwidth limit; removing FLOPs from it changes nothing. Removing bytes does.
      • Odd matrix shapes. Dimensions that are not multiples of 8, 16 or 64 can waste tensor-core tiles or fall back to slower paths. A vocabulary of 50,257 is often padded to 50,304 for this reason. [Likely]
      • Comparing MFU numbers with different formulas. With or without the attention term, with or without embeddings: the same run can show 39% or 49%.
      • Confusing MFU and HFU. Recomputed activations (checkpointing) raise HFU but not MFU. MFU is the honest measure of useful work.
      • Too little work for 132 SMs. A small matmul makes fewer tiles than there are SMs; most of the GPU idles. Larger batches or fusing small ops into bigger ones helps.
      • Ignoring the format. An FP32 matmul without TF32 runs on CUDA cores at 67 TFLOP/s, not on the tensor cores. Use BF16 autocast (or enable TF32).

      Interview questions

      What is a warp, and why does it matter for performance?basic
      A warp is a group of 32 threads that the GPU schedules together; all 32 execute the same instruction at the same time on different data (SIMT). It matters in two ways: if threads in a warp take different branches, both branches run one after the other (divergence); and if the 32 threads read neighboring addresses, the reads combine into a few wide transactions (coalescing). Code that keeps warps uniform and reads memory contiguously is faster.
      What is an SM? How many does an H100 have?basic
      A streaming multiprocessor is one of the GPU’s identical processing units, with its own schedulers, registers, shared memory, CUDA cores and tensor cores. A thread block runs on one SM. The H100 SXM has 132 SMs with 4 tensor cores each (528 total); the A100 has 108. Work must be split into at least as many blocks as there are SMs, and usually many more, to keep the whole chip busy.
      Explain the GPU memory hierarchy with numbers.core
      From fast and small to slow and large, for H100 SXM: registers (256 KB per SM, private to a thread), shared memory / L1 (256 KB per SM combined, up to 228 KB shared, visible to one block and managed by the programmer), L2 cache (50 MB, shared by all SMs), and HBM3 (80 GB at 3.35 TB/s). Fast kernels load data from HBM once into shared memory or registers and reuse it many times there.
      What is arithmetic intensity? Give it for an elementwise add and a square matmul in BF16.core
      Arithmetic intensity is FLOPs divided by bytes moved to and from main memory. Elementwise add: 1 FLOP per element, 6 bytes (two 2-byte reads, one write), so 1/6 FLOP per byte. A square \(n \times n\) matmul: \(2n^3\) FLOPs and \(3 \cdot 2n^2\) bytes, so \(n/3\) FLOP per byte, which is 2,731 for \(n = 8192\). The matmul reuses each number about \(n\) times; the add uses each number once.
      What is the roofline model? What is the ridge point of an H100?core
      The roofline says attainable performance is \(\min(P, I \cdot B)\): either the peak compute or the intensity times the bandwidth, whichever is lower. The ridge point \(I^* = P/B\) separates memory-bound from compute-bound operations. For H100 SXM in dense BF16: \(989 / 3.35 \approx 295\) FLOP per byte. For the A100 it is about 153.
      Why is batch-1 LLM decoding memory-bound, and how do we fix it?core
      For one token, each weight is read from HBM once and used for one multiply-add: about 1 FLOP per byte in BF16, roughly 300 times below the H100 ridge. The speed limit is weight bytes divided by bandwidth: about 4.8 ms per token for an 8B model. Fixes raise the work per byte read: batch many requests (each weight read serves many tokens), quantize weights to 8 or 4 bits (fewer bytes), and speculative decoding (verify several tokens per pass). The KV cache adds more bytes per token, which grouped-query attention reduces.
      How many tokens must a 4096 × 4096 linear layer see to be compute-bound on an H100?advanced
      Intensity is \(Md/(2M + d)\) in BF16. Setting it equal to 295.2 with \(d = 4096\) gives \(M = 295.2 \cdot 4096/(4096 - 590.4) \approx 345\) tokens. Below that the layer is memory-bound. On the A100 (ridge 153) the threshold is about 165 tokens. This is one reason inference servers batch requests and why the best batch size grows on newer GPUs.
      Why does tiling make a matrix multiplication fast?core
      A naive kernel reads a row and a column from global memory for every output element: 0.5 FLOP per byte in BF16. With tiling, a block loads a \(b_M \times b_K\) slice of \(A\) and a \(b_K \times b_N\) slice of \(B\) into shared memory and reuses each loaded number \(b_N\) or \(b_M\) times, so the intensity against global memory becomes \(b_M b_N/(b_M + b_N)\) (64 for 128 × 128 tiles). L2 hits between neighboring blocks lower the real HBM traffic further. Tile size is limited by registers and shared memory per SM.
      Define MFU. How is it different from HFU, and what values are typical?core
      \(\text{MFU} = (\text{tokens per second} \times \text{model FLOPs per token}) / (\text{number of GPUs} \times \text{dense peak FLOP/s})\), with model FLOPs from a formula such as \(6N + 12LdT\). HFU also counts FLOPs the hardware actually does but the model does not need, such as recomputation for activation checkpointing. PaLM 540B reached 46.2% MFU and 57.8% HFU; Llama 3 405B reported 38 to 43% MFU on H100s; GPT-3 was about 21%.
      Your 8B run reports 60% MFU on H100. Your colleague’s reports 45% at the same speed. How?advanced
      The definitions differ. Typical causes: one counts the attention term \(12LdT\) and the other does not (at \(T = 8192\) this term is 27% of \(6N\) for an 8B model); one includes embedding parameters in \(N\); one uses a different peak (sparse vs dense, or a different GPU variant such as H100 PCIe); or one counts recomputed FLOPs (that is HFU, not MFU). Agree on the formula, then compare.
      Why do GPUs need many warps per SM? What limits that number?advanced
      Memory reads take hundreds of cycles. While one warp waits, the scheduler issues instructions from another, so with enough warps the arithmetic units stay busy: roughly \(W \ge (c + \ell)/c\) warps for \(c\) compute cycles per \(\ell\) cycles of latency. The number of resident warps is limited by registers per thread, shared memory per block and the hardware maximum (64 warps per SM on H100). A kernel using many registers can run fewer warps (lower occupancy), and must hide latency with prefetching instead.

      Exercises

      1. What is the arithmetic intensity of \(y = \text{GELU}(x)\) in BF16 if GELU costs about 8 FLOPs per element? Is it memory-bound on an A100?
      2. A 70B model in BF16 is served at batch size 1 on one GPU with 3.35 TB/s (ignore that it does not fit in 80 GB). What is the best possible time per token?
      3. Compute the ridge point of an RTX 4090 and say whether a \(2048 \times 2048 \times 2048\) BF16 matmul is compute-bound on it.
      4. For a \(d = 8192\) linear layer, how many tokens are needed to reach the H100 ridge point?
      5. A kernel uses tiles with \(b_M = 128\), \(b_N = 64\). What is its intensity against global memory? By how much does it beat 64 × 64 tiles?
      6. Llama 3 70B (\(L = 80\), \(d = 8192\), \(N = 70.6 \times 10^9\)) trains at 1,000 tokens per second per H100 with \(T = 8192\). Compute the MFU with and without the attention term.
      Answerscheck
      1. 8 FLOPs per 4 bytes (read 2, write 2): \(I = 2\). The A100 ridge is 153, so it is strongly memory-bound. 2. Weights: \(70 \times 10^9 \times 2 = 140\) GB; \(140 / 3350\) s \(= 41.8\) ms per token, so at most about 24 tokens per second. 3. \(165.2 / 1.008 = 164\) FLOP/byte. The matmul has \(I = 2048/3 = 683\): compute-bound. 4. \(M = 295.2 \times 8192/(8192 - 590.4) = 318\) tokens. 5. \(128 \cdot 64/192 = 42.7\) FLOP/byte versus 32: 1.33 times better. 6. \(6N = 4.24 \times 10^{11}\), \(12LdT = 6.44 \times 10^{10}\). Without attention: \(1000 \times 4.24\times10^{11} / 989\times10^{12} = 42.8\%\); with: \(4.88 \times 10^{11}\) per token, \(49.3\%\).

      Key takeaways

      • A GPU is many SMs (132 on H100), each running warps of 32 threads in lockstep; tensor cores do small matrix multiply-accumulates and give about 15 times the FP32 throughput on H100.
      • The scheduler hides memory latency by switching between warps (42.1); occupancy is limited by registers and shared memory.
      • Memory goes from registers and shared memory (KB per SM, fast) to L2 (50 MB) to HBM (80 GB, 3.35 TB/s). Fast kernels load once and reuse on the chip.
      • Arithmetic intensity (42.2) and the roofline (42.5, 42.6) tell you whether an operation is memory-bound or compute-bound; the H100 ridge is about 295 FLOP per byte.
      • Elementwise ops and batch-1 decoding are memory-bound; large matmuls are compute-bound once \(M\) is above about 345 on H100 (42.4); tiling (42.7) is how matmuls reach the roof.
      • MFU (42.8, 42.9) measures useful work against the dense peak. Good large runs reach about 40 to 46%; always state the FLOP formula you used.

      Key formulas

      \[ I = \frac{F}{Q}, \quad \text{perf} = \min(P,\ I B), \quad I^{*} = \frac{P}{B}, \quad I_{\text{GEMM}} = \frac{MKN}{MK + KN + MN} \]
      \[ I_{\text{tile}} = \frac{b_M b_N}{b_M + b_N}, \quad \text{MFU} = \frac{\text{tok/s} \cdot (6N + 12LdT)}{n_{\text{GPU}}\, P} \]

      References for this session

      1. NVIDIA (docs). CUDA C++ Programming Guide
      2. Austin et al., Google DeepMind (2025). How to Scale Your Model
      3. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
      4. Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
      5. Llama Team, Meta (2024). The Llama 3 Herd of Models
      6. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
      7. NVIDIA (2022). NVIDIA Hopper Architecture In-Depth (technical blog)
      8. NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
      9. NVIDIA (docs). NVIDIA A100 Tensor Core GPU: product specifications
      10. NVIDIA (2022). NVIDIA Ada GPU Architecture (whitepaper, RTX 4090 specifications)