Attention operator 1: FlashAttention 1, 2 and 3
Standard attention writes a T×T table of scores to the GPU's main memory and reads it back, and that traffic, not the arithmetic, is what makes it slow. FlashAttention computes attention block by block inside fast on-chip memory, using an online softmax, and recomputes instead of storing in the backward pass. This session builds the algorithm step by step, then shows what FlashAttention-2 (better work split) and FlashAttention-3 (Hopper asynchrony and FP8) changed, and why the results differ slightly from standard attention.
What you will learn
- Why standard attention is memory-bound: count its HBM bytes and apply the roofline, with real H100 and A100 numbers
- Online softmax: an exact softmax in one pass over blocks, with a running maximum, a running sum and rescaling
- The FlashAttention forward pass: tiles in SRAM, one fused kernel, O(T) extra memory, and its IO complexity Θ(T²d²/M)
- The backward pass: recompute P from the stored log-sum-exp instead of storing the T×T matrix, and the D = rowsum(dO∘O) trick
- FlashAttention-2: fewer non-matmul FLOPs, parallelism over the sequence, split-Q between warps, causal tile skipping
- FlashAttention-3 on Hopper: warp specialization, ping-pong scheduling, the exponential bottleneck, FP8 with block quantization and incoherent processing
- Numerics: why FlashAttention is exact in math but not bit-identical in floating point, and how to test it
Why this matters
Attention compares every query with every key. For a sequence of \(T\) tokens, that makes a \(T \times T\) table of scores in every head of every layer. At \(T = 8192\) one such table in BF16 takes 128 MiB; a layer with 32 heads needs 4 GiB of them. Standard attention writes these tables to the GPU’s main memory and reads them back. That traffic, not the arithmetic, is what makes it slow.
In session 123 we saw where the time of a real model goes. The efficient-inference survey of Zhou et al. reports that attention and linear operators together often take more than 75% of inference time, and it names FlashAttention as the fix for the attention part: it “fuses the entire attention operation into a single, memory-efficient operator”. [Certain] This session explains how that single operator works, why it is faster even though it does more arithmetic, and what changed in each version:
- FlashAttention (Dao et al., 2022): tiling plus online softmax, so the \(T \times T\) table never goes to main memory; recomputation in the backward pass. Memory becomes linear in \(T\), and the result is exact, not an approximation.
- FlashAttention-2 (Dao, 2023): the same math, with better work splitting on the GPU. About 2× faster than version 1, up to 73% of the A100’s peak in the forward pass. [Certain]
- FlashAttention-3 (Shah et al., 2024): rewritten for Hopper GPUs (H100). It overlaps data movement, matrix multiplications and softmax, and adds FP8. Up to 740 TFLOP/s in FP16, about 75% of the H100’s peak. [Certain]
Every serious training and serving stack uses one of these kernels. PyTorch’s scaled_dot_product_attention dispatches to a FlashAttention kernel when the inputs allow it, and serving engines such as vLLM build their attention on FlashAttention-style kernels. [Likely] Long context for text, many image tokens for VLMs, and thousands of frame tokens for video models all depend on it.
Standard attention and its trip to HBM
Recall the formula from session 17. For one head, with \(Q, K, V \in \R^{T \times d}\):
- Read the symbols
- \(T\)
- sequence length (number of tokens)
- \(d\)
- head dimension, usually 64 or 128
- \(S\)
- the scores: one number per (query, key) pair
- \(P\)
- the attention probabilities; each row sums to 1
- \(O\)
- the output: for each query, a weighted average of the value rows
A GPU has two kinds of memory that matter here (session 42). HBM (high-bandwidth memory) is the big main memory: 80 GB on an H100, read at about 3.35 TB/s. SRAM (shared memory, on the chip) is tiny, at most 228 KB per streaming multiprocessor (SM) on an H100, but much faster. [Certain] The FlashAttention paper gives the A100 numbers: 40 to 80 GB of HBM at 1.5 to 2.0 TB/s, and 192 KB of SRAM in each of 108 SMs, with an estimated SRAM bandwidth of about 19 TB/s. [Certain]
The standard implementation runs three separate GPU programs (kernels), and each kernel reads its input from HBM and writes its output to HBM:
- read \(Q, K\) from HBM; compute \(S = QK^\top/\sqrt d\); write \(S\) (\(T^2\) numbers) to HBM
- read \(S\) from HBM; compute \(P = \softmax(S)\); write \(P\) (\(T^2\) numbers) to HBM
- read \(P\) and \(V\) from HBM; compute \(O = PV\); write \(O\) to HBM
- // training also keeps \(P\) (or \(S\)) in HBM for the backward pass: \(O(T^2)\) memory
Count the numbers that cross the HBM boundary, with \(b\) bytes per number. \(Q, K, V\) are read once and \(O\) is written once (\(4Td\)), and the two tables are each written once and read once (\(4T^2\)):
When \(T\) is much larger than \(d\), the \(4T^2\) term is everything. A fused kernel that keeps \(S\) and \(P\) on the chip only needs the \(4Td\) part (we will refine this later). Real implementations also add masking and dropout kernels, each one another full pass over a \(T^2\) table, so (124.2) is the best case for standard attention.
To turn bytes into time, use the roofline rule from session 42: a kernel takes at least as long as its arithmetic and at least as long as its memory traffic.
- Read the symbols
- \(F\)
- floating-point operations (FLOPs) of the kernel: \(2T^2 d\) for each of the two matrix multiplications
- \(\pi\)
- peak compute: 989 TFLOP/s on an H100 SXM with BF16 tensor cores, 312 on an A100
- \(N\)
- bytes moved to and from HBM
- \(\beta\)
- HBM bandwidth: 3.35 TB/s on an H100 SXM, 2.039 TB/s on an A100 SXM 80 GB
Take \(d = 128\), BF16 (\(b = 2\)). The two matrix multiplications have \(4T^2d = 34.4\) GFLOP, which needs at least \(34.4 \times 10^9 / 989 \times 10^{12} = 34.7\) µs.
Standard attention moves \(2(4Td + 4T^2) = 545.3\) MB. Kernel by kernel with (124.3): \(QK^\top\) needs 41.3 µs of memory time against 17.4 µs of compute; softmax needs 80.1 µs and almost no compute; \(PV\) again 41.3 µs. All three are memory-bound. Total: 162.8 µs.
A fused kernel moves only \(2 \cdot 4Td = 8.39\) MB (2.5 µs), so it is compute-bound at 34.7 µs. That is 4.68× faster, with the same FLOPs. The real gain is smaller, because no kernel reaches peak compute. Try other lengths and GPUs in Figure 1.
How to read Figure 1
Move \(T\) up: the standard bars grow like \(T^2\) and are always orange (memory-bound), because they read and write \(T^2\) tables. The fused bar is blue (compute-bound) for all but the shortest sequences. Switch to A100: its ridge point is lower (153 FLOP/byte against 295), so the gap is smaller. Tick the checkbox to charge every re-read of \(K\) and \(V\) to HBM (explained in Section 4): the fused kernel then moves about \(2T^2 d / B_r\) numbers, and the speedup drops to 2× when \(B_r = d = 128\). In practice most of those re-reads hit the 50 MB L2 cache. [Likely] This is an ideal model: it ignores launch overhead and the fact that real kernels reach only part of the peak.
The paper measured the same effect on GPT-2 medium (sequence length 1024, forward plus backward): standard attention did 66.6 GFLOP and moved 40.3 GB in 41.7 ms; FlashAttention did more work, 75.2 GFLOP, but moved only 4.4 GB and took 7.3 ms. [Certain] The extra FLOPs come from recomputation in the backward pass (Section 5). Fewer bytes won.
Online softmax: softmax without the whole row
Fusing is easy for the matrix multiplications: they can be done block by block. The problem is the softmax in the middle. Each output row needs
\[ \mathbf{o}_i = \frac{\sum_j e^{s_{ij} - m_i}\, \mathbf{v}_j}{\sum_j e^{s_{ij} - m_i}}, \qquad m_i = \max_j s_{ij}, \]
and both the maximum \(m_i\) (needed for stability, see session 2) and the sum in the denominator depend on all keys. If we can only hold a few keys on the chip at a time, we seem to need two passes over the row.
The online softmax (Milakov and Gimelshein, 2018) removes that need. Keep three running quantities per query row: the maximum so far \(m\), the sum of exponentials so far \(\ell\) (measured against that maximum), and the un-normalized output so far \(\tilde{\mathbf{o}}\). When a new block of scores \(\mathbf{s}^{(j)}\) with values \(V^{(j)}\) arrives:
- Read the symbols
- \(m, \ell, \tilde{\mathbf{o}}\)
- running maximum, running sum of exponentials, running weighted sum of values, for one query row; start with \(m = -\infty\), \(\ell = 0\), \(\tilde{\mathbf{o}} = \mathbf{0}\)
- \(\mathbf{s}^{(j)}, \mathbf{v}^{(j)}_k\)
- the scores of this query against key block \(j\), and the value rows of that block
- \(\alpha\)
- the correction factor: if the maximum went up, all old terms were measured against a too-small maximum and must shrink by \(e^{m - m'} \le 1\)
Why is this exact? Every old term was stored as \(e^{s - m}\). Multiplying by \(e^{m - m'}\) turns it into \(e^{s - m'}\), exactly what it would have been if we had known \(m'\) from the start. After the last block, divide once:
\(L\) is the log-sum-exp of the row, one number. It is all we need to rebuild any probability later: \(p_j = e^{s_j - L}\). FlashAttention stores it for the backward pass.
Scores \(\mathbf{s} = (2, 1, 3, 0)\), scalar values \(v = (10, 20, 30, 40)\), blocks of two.
Block 1 \((2, 1)\): \(m = 2\), \(\ell = e^0 + e^{-1} = 1.3679\), \(\tilde o = 10 \cdot 1 + 20 \cdot e^{-1} = 17.3576\).
Block 2 \((3, 0)\): the maximum rises to \(m' = 3\), so \(\alpha = e^{2-3} = 0.3679\). Then \(\ell' = 1.3679 \times 0.3679 + e^0 + e^{-3} = 0.5032 + 1 + 0.0498 = 1.5530\) and \(\tilde o' = 17.3576 \times 0.3679 + 30 + 40\,e^{-3} = 6.3855 + 30 + 1.9915 = 38.3770\).
End: \(o = 38.3770 / 1.5530 = 24.7115\) and \(L = 3 + \log 1.5530 = 3.4402\). The ordinary softmax gives probabilities \((0.2369, 0.0871, 0.6439, 0.0321)\) and the same \(o = 24.7115\). Exact, with one pass.
- \(m \leftarrow -\infty\); \(\ell \leftarrow 0\); \(\tilde o \leftarrow 0\)
- for each block \((s, v)\) of the row:
- \(m' \leftarrow \max(m, \max s)\); \(\alpha \leftarrow e^{m - m'}\) // rescale old terms
- \(\ell \leftarrow \alpha\,\ell + \sum e^{s - m'}\); \(\tilde o \leftarrow \alpha\,\tilde o + \sum e^{s - m'} v\); \(m \leftarrow m'\)
- \(o \leftarrow \tilde o / \ell\); \(L \leftarrow m + \log \ell\) // once, at the end
How to read Figure 2
Press Step and follow the highlighted line of the small algorithm. Watch the green bars. They are the weights \(e^{s - m}\) of all keys seen so far, measured against the current maximum. When a block contains a new maximum, all older bars shrink at once by \(\alpha\) (line 3): that is the rescaling. When the maximum does not change, \(\alpha = 1\) and nothing shrinks. Line 4 then adds the new block’s bars to \(\ell\) and its weighted values to \(\tilde o\). Change the block size: the path is different, but the final output always equals the exact softmax result (shown in the last step) up to rounding in the last digits.
FlashAttention: tiling the whole computation
Now combine online softmax with tiles. Split \(Q\) into blocks of \(B_r\) rows, and \(K, V\) into blocks of \(B_c\) rows. One block of queries, one block of keys and values, and the small \(B_r \times B_c\) tile of scores all fit in SRAM. The kernel loops over key blocks and keeps \(m\), \(\ell\) and \(\tilde O\) for its query block on the chip. The \(T \times T\) matrices exist only as tiles, one at a time, and never reach HBM. This is kernel fusion: one kernel does what took three.
FlashAttention-2 uses this loop order (queries outside, keys inside), shown in Algorithm 2. FlashAttention-1 used the opposite order; we compare the two in Figure 4.
- input: \(Q, K, V \in \R^{T \times d}\) in HBM; block sizes \(B_r, B_c\)
- for each query block \(Q_i\) (\(B_r\) rows) in parallel // one thread block each
- load \(Q_i\) into SRAM; set \(m_i = -\infty\), \(\ell_i = 0\), \(\tilde O_i = 0\) (on chip)
- for each key block \(j\) (skip blocks above the diagonal if causal)
- load \(K_j, V_j\) into SRAM
- \(S_{ij} = Q_i K_j^\top / \sqrt d\) // \(B_r \times B_c\) tile, on chip only
- \(m' = \max(m_i, \text{rowmax}(S_{ij}))\); \(\tilde P = e^{S_{ij} - m'}\); \(\alpha = e^{m_i - m'}\)
- \(\ell_i \leftarrow \alpha\,\ell_i + \text{rowsum}(\tilde P)\); \(\tilde O_i \leftarrow \text{diag}(\alpha)\,\tilde O_i + \tilde P V_j\); \(m_i \leftarrow m'\) // formula (124.4)
- write \(O_i = \text{diag}(\ell_i)^{-1} \tilde O_i\) and \(L_i = m_i + \log \ell_i\) to HBM // formula (124.5), once
Memory: besides \(Q, K, V, O\), the only thing written is \(L\), one FP32 number per row: 32 KiB per head at \(T = 8192\), instead of 128 MiB for \(P\). The FlashAttention paper states it formally: exact attention with \(O(T^2 d)\) FLOPs and \(O(T)\) extra memory. [Certain]
Figure 3 runs Algorithm 2 for real on a toy head: \(T = 8\) tokens, \(d = 2\), with fixed numbers for \(Q\), \(K\) and \(V\). You can watch each tile travel from HBM into SRAM, see the tile of scores, the running maximum \(m\), the correction \(\alpha\), the running sum \(\ell\) and the un-normalized output \(\tilde O\) change, and count every number that crosses the HBM boundary. At the end, the kernel’s output is compared with standard attention (124.1).
- input: \(Q, K, V\) in HBM; block sizes \(B_r, B_c\)
- for each query block \(i\) // in parallel on a GPU
- load \(Q_i\) to SRAM; \(m \leftarrow -\infty\), \(\ell \leftarrow 0\), \(\tilde O \leftarrow 0\)
- for each key block \(j\) // skip if above the diagonal
- load \(K_j, V_j\) to SRAM
- \(S \leftarrow Q_i K_j^\top / \sqrt d\) // masked \(\to -\infty\)
- \(m' \leftarrow \max(m, \text{rowmax}\, S)\); \(\tilde P \leftarrow e^{S - m'}\); \(\alpha \leftarrow e^{m - m'}\)
- \(\ell \leftarrow \alpha\,\ell + \text{rowsum}\, \tilde P\); \(\tilde O \leftarrow \alpha\,\tilde O + \tilde P V_j\); \(m \leftarrow m'\)
- write \(O_i = \tilde O / \ell\) and \(L_i = m + \log \ell\) to HBM
- check: \(O\) against \(\softmax(QK^\top/\sqrt d)\,V\)
How to read Figure 3
Press Step and follow the highlighted line. A query tile flies from HBM into SRAM (line 3), then key and value tiles fly in one pair at a time (line 5). Each pair gives a small tile of scores (line 6); the cells are filled in proportion to \(\tilde p = e^{s - m'}\), so the largest score of each row is a full cell. Watch the \(\alpha\) column: it is 0 for the first tile (the old \(m\) was \(-\infty\)), below 1 when a row finds a larger score, and exactly 1 when its maximum does not change. Only line 9 writes to HBM: one row of \(O\) and one number \(L\) per query. With \(B_r = B_c = 2\) the run ends with 144 numbers read and 24 written, 168 in total, against 320 for standard attention; the largest difference from standard attention is about \(10^{-16}\), rounding in double precision. Set \(B_r = 4\): \(K\) and \(V\) are read only twice instead of four times, and traffic drops to 104. Turn on the causal mask: tiles above the diagonal are skipped, and entries above the diagonal inside a tile become \(-\infty\) (weight 0).
The toy shows the same pattern as a real kernel, only smaller. The traffic of Algorithm 2 is \(Td\) (read \(Q\)) + \(2Td\lceil T/B_r \rceil\) (read \(K, V\) once per query block) + \(Td\) (write \(O\)) + \(T\) (write \(L\)) numbers. For one head with \(T = 8192\), \(d = 128\), \(B_r = 128\), that is about 136 million numbers if every re-read of \(K\) and \(V\) went to HBM, against 273 million for standard attention: only 2× less. If the re-reads hit the L2 cache, HBM sees about 4.2 million numbers, 65× less. That is why the loop order and the cache matter, and why the analysis below is more careful than (124.2).
How many bytes, really?
Formula (124.2) said \(4Td\), but that hides the loops. In Algorithm 2 each query block reads all of \(K\) and \(V\), so \(K\) and \(V\) are read \(\lceil T/B_r \rceil\) times. The paper analyzes its own loop order with an SRAM of \(M\) numbers and proves (its Theorem 2 and Proposition 3):
and no exact attention algorithm can do asymptotically better for all \(M\) in that range. [Certain] Since \(d^2\) is much smaller than \(M\), the factor \(d^2/M\) is small. For example, 192 KB of SRAM holds \(M = 98{,}304\) BF16 numbers; with \(d = 64\), \(d^2/M = 1/24\). The paper sets the block sizes from \(M\): \(B_c = \lceil M/4d \rceil\) and \(B_r = \min(\lceil M/4d \rceil, d)\), which gives \(B_c = 192\) and \(B_r = 128\) for \(M = 98{,}304\), \(d = 128\). [Certain] In practice FlashAttention-2 uses tiles of 64 or 128 in each direction, chosen per head dimension. [Certain]
How to read Figure 4
In FlashAttention-1 order, the outer loop runs over key/value blocks: each \(K_j, V_j\) is loaded once, but for every tile the partial output \(O_i\) (and \(m_i, \ell_i\)) must be read from HBM and written back, because the next update of that row comes much later. In FlashAttention-2 order, a query block stays on the chip for its whole row, and \(O_i\) is written once at the end; \(K\) and \(V\) are re-read instead, often from the L2 cache. Turn on the causal mask: tiles fully above the diagonal are skipped, so only 21 of 36 tiles are computed, and only the 6 diagonal tiles need the mask applied element by element.
The causal case deserves a formula. With \(n = T/B\) blocks per side, a causal kernel computes the tiles on or below the diagonal:
For \(n = 6\) that is 0.583, for \(n = 64\) it is 0.508. Standard attention with a mask computes the full table and then throws half away. FlashAttention-2 reports a 1.7 to 1.8× speedup from skipping these blocks, compared with the same kernel without the causal mask. [Certain]
The backward pass: recompute instead of store
Training needs gradients. With \(dO = \partial \mathcal{L}/\partial O\) coming from above, the gradients of attention are (session 17 derives them):
- Read the symbols
- \(\circ\)
- element-wise product
- \(D_i\)
- one number per query row: the dot product of the output row and its gradient. It replaces the row sum \(\sum_j P_{ij}\, dP_{ij}\) of the softmax gradient, because the two are equal
- \(dS\)
- gradient with respect to the scores, the softmax backward
Every line needs \(P\), a \(T \times T\) matrix. Standard attention keeps it in HBM from the forward pass. FlashAttention does not have it. Instead, it recomputes each tile of \(P\) in the backward pass from \(Q\), \(K\) and the stored \(L\):
No running maximum is needed here: \(L_i\) already is the exact normalizer. This costs one extra \(QK^\top\) multiplication per tile, but it avoids reading a \(T^2\) matrix from HBM. It is the same trade as activation checkpointing (session 39), applied inside one kernel, and it is why FlashAttention did 75.2 GFLOP against 66.6 and was still 5.7× faster in the GPT-2 measurement above. [Certain] FlashAttention-1 stored both \(m\) and \(\ell\); FlashAttention-2 stores only \(L = m + \log \ell\). [Certain]
- input: \(Q, K, V, O, dO\) and \(L\) in HBM
- \(D = \text{rowsum}(dO \circ O)\) // small pre-pass, \(T\) numbers
- for each key/value block \(j\) in parallel
- load \(K_j, V_j\); set \(dK_j = 0\), \(dV_j = 0\) on chip
- for each query block \(i\)
- load \(Q_i, dO_i, L_i, D_i\); \(S_{ij} = Q_i K_j^\top / \sqrt d\); \(P_{ij} = e^{S_{ij} - L_i}\) // recompute, (124.9)
- \(dV_j \mathrel{+}= P_{ij}^\top dO_i\); \(dP_{ij} = dO_i V_j^\top\); \(dS_{ij} = P_{ij} \circ (dP_{ij} - D_i)\)
- \(dK_j \mathrel{+}= dS_{ij}^\top Q_i / \sqrt d\); add \(dS_{ij} K_j / \sqrt d\) into \(dQ_i\) in HBM // atomic add
- write \(dK_j, dV_j\) to HBM
FlashAttention-2: the same math, better work split
FlashAttention-1 reached only 25 to 40% of the A100’s peak FLOP/s. The FlashAttention-2 paper traces this to how the work is split across thread blocks and warps, and makes three changes. [Certain]
1. Fewer non-matmul FLOPs
An A100 does 312 TFLOP/s of FP16/BF16 matrix multiplication on tensor cores, but only 19.5 TFLOP/s of other FP32 arithmetic. So each non-matmul FLOP (exp, max, rescaling) costs about 16× more time than a matmul FLOP. [Certain] FlashAttention-2 therefore keeps \(\tilde O\) un-normalized and divides by \(\ell\) only once at the end (Algorithm 2, line 9), instead of rescaling the normalized output at every step, and it stores just \(L\) instead of \(m\) and \(\ell\).
2. Parallelism over the sequence
FlashAttention-1 used one thread block per (batch, head) pair. A thread block runs on one SM. With batch 1 and 16 heads, only 16 of the A100’s 108 SMs have work. Long sequences usually come with small batches, so this case is common. FlashAttention-2 also splits the sequence: one thread block per query block, so a single head at \(T = 16{,}384\) with \(B_r = 128\) gives 128 thread blocks. This works because query blocks are independent in the forward pass; only key/value blocks interact through the softmax, and that interaction is inside the loop. [Certain]
How to read Figure 5
“Waves”: if there are 300 thread blocks and 108 SMs, the GPU runs them in \(\lceil 300/108 \rceil = 3\) waves, and the last wave is partly empty. Utilization in this model is blocks / (waves × SMs). Start with batch 1 and 16 heads: FlashAttention-1 fills 16 of 108 SMs (15%). FlashAttention-2 fills almost all of them. Now raise the batch to 32: both versions are fine, because there are enough (batch, head) pairs. The change matters most for long sequences with small batches, exactly the long-context case. Real GPUs can run more than one thread block per SM, so the real numbers differ; the direction does not.
3. Splitting work between warps
Inside a thread block, 4 or 8 warps (groups of 32 threads) share the work. FlashAttention-1 split \(K\) and \(V\) across warps (“split-K”): each warp got the full query block and part of the keys, so at the end the warps had to write partial results to shared memory, synchronize and add them up. FlashAttention-2 splits \(Q\) across warps (“split-Q”): every warp sees all of \(K\) and \(V\) but owns its own query rows, so each warp finishes its rows alone, with no exchange. [Certain] The backward pass also avoids split-K, though it still needs some synchronization. [Certain]
The result: 1.7 to 3.0× faster than FlashAttention-1, up to 73% of peak in the forward pass and 63% in the backward pass, and up to 225 TFLOP/s per A100 when training GPT-style models end to end (72% model FLOPs utilization). [Certain] FlashAttention-2 also supports multi-query and grouped-query attention (session 97) by indexing into the shared key/value heads instead of copying them. [Certain]
| Model (8× A100, TFLOP/s per GPU) | Context | No FlashAttention | FlashAttention | FlashAttention-2 |
|---|---|---|---|---|
| GPT3-1.3B | 2k | 72 to 149 (by model and context) | 189 | 196 |
| GPT3-1.3B | 8k | 170 | 220 | |
| GPT3-2.7B | 2k | 189 | 205 | |
| GPT3-2.7B | 8k | 175 | 225 |
Notice the pattern in the table, from the FlashAttention-2 paper: the gain over version 1 is small at 2k and large at 8k, where attention takes a bigger share of the time. [Certain]
FlashAttention-3: asynchrony and FP8 on Hopper
Run on an H100, FlashAttention-2 reaches only about 35% utilization. [Certain] The H100 tensor cores are so fast that everything else becomes the bottleneck. Hopper adds hardware that FlashAttention-3 uses:
- TMA (Tensor Memory Accelerator): a unit that copies tiles between HBM and shared memory by itself, so threads do not have to compute addresses and issue loads.
- WGMMA: asynchronous matrix multiplication issued by a warpgroup (4 warps, 128 threads). The warpgroup starts it and can do other work while the tensor cores run.
The exponential is the new bottleneck
Per score, the two matrix multiplications cost \(4d\) FLOPs (\(2d\) for \(QK^\top\), \(2d\) for \(PV\)), while the softmax needs one exponential. But the H100 has 989 TFLOP/s of FP16 matmul and only 3.9 TFLOP/s for special functions such as exp. [Certain] With \(d = 128\):
So the exponentials alone can take about half as long as all the matrix multiplications. If they run one after the other, the tensor cores sit idle a third of the time. FlashAttention-3 overlaps them in two ways. [Certain]
- Warp specialization. Some warps are producers: they only issue TMA loads of \(K\) and \(V\). The others are consumers: they issue WGMMAs and compute the softmax. Registers are moved from producers to consumers (the
setmaxnreginstruction), since consumers need many more. - Ping-pong scheduling. Two consumer warpgroups take turns: while warpgroup 1 computes the softmax of its tile, warpgroup 2 runs its matrix multiplications, and then they swap. Inside one warpgroup, a 2-stage pipeline also overlaps the softmax of one tile with the WGMMA of the next.
How to read Figure 6
The bottom row is the tensor cores. Without overlap, they are busy \(1/(1+r)\) of the time: 67% at \(r = 0.5\), the ratio from (124.10). With ping-pong, as long as \(r \le 1\) the softmax of one warpgroup fits inside the other’s GEMM and the tensor cores never wait (100% in this toy). Above \(r = 1\) even ping-pong cannot hide it: utilization is \(2/(1+r)\). Real kernels have other costs; the paper measured 570 → 620 to 640 TFLOP/s from ping-pong (FP16, head dimension 128, length 8192). [Certain]
FP8 attention
The H100 doubles its tensor-core throughput in FP8 (see session 105). FP8 attention raises two problems. First, a layout rule: FP8 WGMMA needs its operands in “k-major” order, so FlashAttention-3 transposes the \(V\) tile inside the kernel. Second, accuracy: LLM activations have outliers, a few very large values, and with one scale for a whole tensor they force every other value into the few lowest FP8 levels. FlashAttention-3 uses two fixes. [Certain]
- Block quantization: one scale per block of \(Q\), \(K\) or \(V\), not per tensor. The tiles are already there, so this is cheap.
- Incoherent processing: multiply \(Q\) and \(K\) by the same random orthogonal matrix \(M\) (random \(\pm 1\) signs times a Hadamard matrix) before quantizing. Because \(M M^\top = I\), the scores do not change, but each outlier is spread over many coordinates:
This is the same idea as the rotations of session 109 (QuIP, QuaRot). The paper’s test data had about 0.1% outlier entries; the FP8 root-mean-square error was \(9.1 \times 10^{-3}\) for FlashAttention-3 against \(2.4 \times 10^{-2}\) for a baseline FP8 attention, 2.6× lower. [Certain] Overall: 1.5 to 2.0× faster than FlashAttention-2 in FP16, up to 740 TFLOP/s, and close to 1.2 PFLOP/s in FP8. [Certain]
| FlashAttention-1 (2022) | FlashAttention-2 (2023) | FlashAttention-3 (2024) | |
|---|---|---|---|
| Main idea | tiling + online softmax + recompute in backward | fewer non-matmul FLOPs, parallel over sequence, split-Q between warps | warp specialization, ping-pong, GEMM–softmax overlap, FP8 |
| Target GPU | A100 (and older) | A100; runs on H100 without Hopper features | H100 / Hopper only |
| Kept for backward | \(O\), \(m\), \(\ell\) | \(O\), \(L\) | \(O\), \(L\) |
| Reported speed | 25–40% of A100 peak | 50–73% of A100 peak (forward); 335 TFLOP/s on H100 | up to 740 TFLOP/s FP16 (75% of H100), about 1.2 PFLOP/s FP8 |
Numerics: exact, but not bit-identical
FlashAttention is “exact”: in exact arithmetic it gives the same output as (124.1). In floating point the results differ a little, for four reasons:
- Different order of additions. The row sums and \(PV\) products are accumulated block by block. Floating-point addition is not associative, so a different order gives different last digits.
- Rescaling. Each \(\alpha = e^{m - m'}\) multiplication adds one more rounding.
- Rounding of \(P\). The tile of probabilities is rounded to BF16 or FP16 before the \(\tilde P V\) multiplication on the tensor cores. A standard implementation that keeps \(P\) in FP32 avoids this step.
- Atomic adds in backward (Algorithm 3): the order changes from run to run.
How big is this? Golden et al. (2024) measured FlashAttention in BF16 against a baseline attention in an isolated forward pass and found roughly an order of magnitude more numeric deviation; their analysis estimates its effect on model weights during training as 2 to 5 times smaller than the effect of low-precision training itself. [Certain] The efficient-inference survey also warns in a footnote that FlashAttention “might cause the numeric deviation”. [Certain] In FP16 tests, the FlashAttention-3 paper reports the opposite direction: an RMSE of \(1.9 \times 10^{-4}\) for FlashAttention-2 and -3 against \(3.2 \times 10^{-4}\) for a standard FP16 implementation, because the kernels keep the running sums in FP32. [Certain] The practical rule: compare against a high-precision reference with a tolerance, never with ==.
The algorithm in PyTorch
The code below is a reference implementation, not a fast one: Python loops over tiles. It shows exactly what the kernel does. It checks the tiled forward against naive attention, checks the recomputing backward (formulas 124.8 and 124.9, left untiled for clarity) against autograd, and measures low-precision errors. It runs on a CPU in a few seconds.
import math, torch
torch.manual_seed(0)
def naive_attention(q, k, v, causal=False):
# q, k, v: (B, h, T, d). Builds the full T x T matrices, like standard attention.
S = q @ k.transpose(-2, -1) / math.sqrt(q.shape[-1])
if causal:
T = q.shape[-2]
S = S.masked_fill(torch.ones(T, T, dtype=torch.bool).triu(1), float("-inf"))
return torch.softmax(S, dim=-1) @ v
def flash_forward(q, k, v, Br=64, Bc=64, causal=False):
# FlashAttention-2 order: outer loop over query blocks, inner loop over key/value blocks.
B, h, T, d = q.shape
scale = 1.0 / math.sqrt(d)
O = torch.empty_like(q)
L = torch.empty(B, h, T, dtype=torch.float32)
for i in range(0, T, Br): # one thread block per query block on a GPU
qi = q[:, :, i:i+Br].float()
m = torch.full(qi.shape[:-1], float("-inf")) # running max, per row
l = torch.zeros(qi.shape[:-1]) # running sum of exp, per row
acc = torch.zeros_like(qi) # un-normalized output
for j in range(0, T, Bc):
if causal and j > i + Br - 1:
break # block is fully above the diagonal: skip it
kj, vj = k[:, :, j:j+Bc].float(), v[:, :, j:j+Bc].float()
s = qi @ kj.transpose(-2, -1) * scale # (Br, Bc) tile, lives in SRAM only
if causal:
rows = torch.arange(i, i + qi.shape[-2])[:, None]
cols = torch.arange(j, j + kj.shape[-2])[None, :]
s = s.masked_fill(cols > rows, float("-inf"))
m_new = torch.maximum(m, s.amax(-1))
p = torch.exp(s - m_new[..., None])
alpha = torch.exp(m - m_new) # rescale factor for the old partial results
l = alpha * l + p.sum(-1)
acc = alpha[..., None] * acc + p.to(q.dtype).float() @ vj # P is rounded to the input dtype, as on a GPU
m = m_new
O[:, :, i:i+Br] = (acc / l[..., None]).to(q.dtype) # divide once, at the end (FA2)
L[:, :, i:i+Br] = m + torch.log(l) # one number per row is kept for backward
return O, L
def flash_backward(q, k, v, O, L, dO, causal=False):
# Recompute P from Q, K and the stored log-sum-exp L instead of reading a stored T x T matrix.
d = q.shape[-1]
S = q.float() @ k.float().transpose(-2, -1) / math.sqrt(d)
if causal:
T = q.shape[-2]
S = S.masked_fill(torch.ones(T, T, dtype=torch.bool).triu(1), float("-inf"))
P = torch.exp(S - L[..., None]) # exact softmax, no max or sum needed
D = (dO.float() * O.float()).sum(-1, keepdim=True) # D_i = rowsum(dO * O)
dV = P.transpose(-2, -1) @ dO.float()
dP = dO.float() @ v.float().transpose(-2, -1)
dS = P * (dP - D)
dQ = dS @ k.float() / math.sqrt(d)
dK = dS.transpose(-2, -1) @ q.float() / math.sqrt(d)
return dQ, dK, dV
B, h, T, d = 1, 2, 256, 64
q, k, v = (torch.randn(B, h, T, d, dtype=torch.float64) for _ in range(3))
# 1) the tiled forward matches naive attention (the tiles compute in fp32, like a GPU kernel)
for causal in (False, True):
O, L = flash_forward(q, k, v, causal=causal)
print("forward max error, causal =", causal, (O - naive_attention(q, k, v, causal)).abs().max().item())
# 2) the recomputing backward matches autograd through the naive version
qr, kr, vr = (t.clone().requires_grad_(True) for t in (q, k, v))
out = naive_attention(qr, kr, vr, causal=True)
dO = torch.randn_like(out)
out.backward(dO)
O, L = flash_forward(q, k, v, causal=True)
dQ, dK, dV = flash_backward(q, k, v, O, L, dO, causal=True)
print("backward max error:", max((a - b).abs().max().item() for a, b in ((dQ, qr.grad), (dK, kr.grad), (dV, vr.grad))))
# 3) numerics in low precision: both methods round, in different places
ref = naive_attention(q, k, v) # float64 reference
for dt in (torch.float16, torch.bfloat16):
qh, kh, vh = q.to(dt), k.to(dt), v.to(dt)
std = naive_attention(qh.float(), kh.float(), vh.float()).to(dt) # standard math in fp32, output rounded
fl, _ = flash_forward(qh, kh, vh)
e_std = (std.double() - ref).abs().max().item()
e_fl = (fl.double() - ref).abs().max().item()
e_between = (std.double() - fl.double()).abs().max().item()
print(dt, "max error vs fp64: standard %.2e, tiled %.2e, between them %.2e" % (e_std, e_fl, e_between))
With PyTorch 2.8 on a CPU this prints forward errors of about \(3 \times 10^{-7}\) (non-causal) and \(6 \times 10^{-7}\) (causal), which is FP32 rounding, and a backward error of about \(3 \times 10^{-6}\). In FP16 both methods have a maximum error of \(4.1 \times 10^{-4}\) against the FP64 reference and differ from each other by \(2.4 \times 10^{-4}\). In BF16, which has fewer mantissa bits, the tiled version is a little worse (\(3.8 \times 10^{-3}\) against \(2.7 \times 10^{-3}\)), mostly because \(\tilde P\) is rounded to BF16 before the \(PV\) product. On a GPU you would call the real kernel: F.scaled_dot_product_attention(q, k, v, is_causal=True) (with F = torch.nn.functional), or flash_attn_func from the flash-attn package.
Where this is used
| Setting | What FlashAttention gives | Notes |
|---|---|---|
| LLM pretraining and fine-tuning | Attention memory linear in \(T\); no stored \(P\); 2–4× faster attention | The 8k rows of the FlashAttention-2 table; long-context continued pretraining depends on it |
| Prefill in LLM serving | Fast compute-bound attention over the whole prompt | Decode has one query row per step, a different problem: session 125 |
| Packed or variable-length batches | “varlen” kernels take concatenated sequences plus their start offsets, with no padding | Pairs with sequence packing (session 18) [Likely] |
| Vision transformers and VLM encoders | High-resolution images give thousands of patch tokens per image | Bidirectional attention: no causal skipping, so the full tile grid is computed |
| Video models (action recognition, key-moment detection, video retrieval, Video-RAG) | Frames × patches easily reach tens of thousands of tokens; without fused attention the \(T^2\) table alone would not fit | Many video LLMs also reduce tokens first; FlashAttention makes the remaining long sequences affordable [Likely] |
| Long-document models | Exact full attention to 16K–64K tokens on one GPU | FlashAttention made the first transformers better than chance on Path-X (16K tokens, 61.4%) and, with block-sparse FlashAttention, Path-256 (64K, 63.1%) [Certain] |
| Sparse and sliding-window attention | The same tiling with whole tiles skipped | Block-sparse FlashAttention in the first paper; local windows in later kernels (session 116) |
Common mistakes
- “FlashAttention reduces the FLOPs.” It does not. Forward FLOPs are the same \(O(T^2 d)\), and training does more because of recomputation. It reduces HBM traffic and memory.
- “FlashAttention is an approximation.” It is exact up to floating-point rounding. Linformer, Performer and sparse attention are approximations; FlashAttention is not.
- Expecting bit-identical outputs after switching kernels, or comparing with
==in tests. Use a tolerance against an FP32 or FP64 reference. - Silently falling back to the slow path.
scaled_dot_product_attentionpicks a backend from dtype, head dimension, mask type and hardware. An arbitrary dense mask tensor, FP32 inputs or an unsupported head size can send you to the math backend with its \(T^2\) memory. Check which backend runs. [Likely] - Building a full \(T \times T\) boolean mask for causal attention and passing it in. That allocates \(T^2\) memory and blocks causal tile skipping; use
is_causal=True. - Thinking FlashAttention solves decoding. At decode time there is one query per sequence, so splitting over query blocks gives no parallel work; decode needs a split over keys (session 125).
- Using FlashAttention-3 features on the wrong GPU. Its speedups come from Hopper hardware (TMA, WGMMA); on an A100 use FlashAttention-2.
Interview questions
Why is FlashAttention faster if it does the same number of FLOPs?basic
Explain online softmax. Why is it exact?core
What does FlashAttention store for the backward pass, and why is recomputation a good trade?core
Write the backward formulas of attention. What is the D term?advanced
What is the IO complexity of FlashAttention, and is it optimal?advanced
What are the three main changes in FlashAttention-2?core
Why does FlashAttention-3 need Hopper, and what does it overlap?advanced
How does FlashAttention-3 keep FP8 attention accurate?advanced
Is FlashAttention exact? Why might outputs differ from standard attention?basic
How does the causal mask make FlashAttention faster?core
Your long-context training run with batch size 1 is slow in attention. What do you check?core
is_causal instead of a mask tensor so masked tiles are skipped, and use variable-length kernels instead of padding. On H100, FlashAttention-3 gives a further 1.5–2×.Exercises
- For \(T = 32{,}768\) and 32 heads in BF16, how much memory does one layer’s probability matrix \(P\) need? How much does FlashAttention’s \(L\) need (FP32)?
- Run online softmax by hand on scores \((1, 3)\) then \((2)\), with values \((0, 10)\) then \((5)\). Give \(m\), \(\ell\), \(\tilde o\) after each block and the final output.
- Show that \(\sum_j P_{ij}\, dP_{ij} = dO_i \cdot O_i\) using \(dP = dO\,V^\top\) and \(O = PV\).
- With 8 query blocks per side, what fraction of tiles does a causal kernel compute? How many need element-wise masking?
- Batch 2, 8 heads, \(T = 4096\), \(B_r = 128\), on an A100 with 108 SMs. How many thread blocks do FlashAttention-1 and FlashAttention-2 launch? What is the toy utilization of Figure 5 for each?
- Using (124.10), what is the exp-to-matmul time ratio for \(d = 64\)? Is overlap more or less important than for \(d = 128\)?
Answerscheck
Key takeaways
- Standard attention is slow because it writes and reads the \(T \times T\) matrices in HBM (124.2); by the roofline (124.3) its kernels are memory-bound.
- Online softmax (124.4, 124.5) computes an exact softmax in one pass over blocks by rescaling old partial sums when the maximum grows.
- FlashAttention tiles \(Q\), \(K\), \(V\) into SRAM, fuses everything into one kernel, uses \(O(T)\) extra memory, and is IO-optimal: \(\Theta(T^2 d^2/M)\) HBM accesses (124.6). Causal tiles above the diagonal are skipped (124.7).
- The backward pass recomputes \(P = e^{S - L}\) from the stored log-sum-exp (124.8, 124.9): more FLOPs, far fewer bytes.
- FlashAttention-2: fewer non-matmul FLOPs, parallel over the sequence, split-Q between warps; about 2× over version 1. FlashAttention-3: Hopper asynchrony (warp specialization, ping-pong) hides the slow exponentials (124.10), and FP8 with block quantization and incoherent processing (124.11).
- Results are exact in math, slightly different in floating point; test with tolerances.
Key formulas
References for this session
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Shah et al. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- Milakov & Gimelshein (2018). Online normalizer calculation for softmax
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Shah et al. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- Milakov & Gimelshein (2018). Online normalizer calculation for softmax
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Golden, Hsia, Sun et al. (2024). Is Flash Attention Stable?