LLM & VLM courseBlog
124Track 5: LLM inference and serving · session 47 of 65

Attention operator 1: FlashAttention 1, 2 and 3

Standard attention writes a T×T table of scores to the GPU's main memory and reads it back, and that traffic, not the arithmetic, is what makes it slow. FlashAttention computes attention block by block inside fast on-chip memory, using an online softmax, and recomputes instead of storing in the backward pass. This session builds the algorithm step by step, then shows what FlashAttention-2 (better work split) and FlashAttention-3 (Hopper asynchrony and FP8) changed, and why the results differ slightly from standard attention.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 17 (attention math), 42 (GPU architecture and the roofline) and 123 (runtime profiling)

What you will learn

  • Why standard attention is memory-bound: count its HBM bytes and apply the roofline, with real H100 and A100 numbers
  • Online softmax: an exact softmax in one pass over blocks, with a running maximum, a running sum and rescaling
  • The FlashAttention forward pass: tiles in SRAM, one fused kernel, O(T) extra memory, and its IO complexity Θ(T²d²/M)
  • The backward pass: recompute P from the stored log-sum-exp instead of storing the T×T matrix, and the D = rowsum(dO∘O) trick
  • FlashAttention-2: fewer non-matmul FLOPs, parallelism over the sequence, split-Q between warps, causal tile skipping
  • FlashAttention-3 on Hopper: warp specialization, ping-pong scheduling, the exponential bottleneck, FP8 with block quantization and incoherent processing
  • Numerics: why FlashAttention is exact in math but not bit-identical in floating point, and how to test it

Why this matters

Attention compares every query with every key. For a sequence of \(T\) tokens, that makes a \(T \times T\) table of scores in every head of every layer. At \(T = 8192\) one such table in BF16 takes 128 MiB; a layer with 32 heads needs 4 GiB of them. Standard attention writes these tables to the GPU’s main memory and reads them back. That traffic, not the arithmetic, is what makes it slow.

In session 123 we saw where the time of a real model goes. The efficient-inference survey of Zhou et al. reports that attention and linear operators together often take more than 75% of inference time, and it names FlashAttention as the fix for the attention part: it “fuses the entire attention operation into a single, memory-efficient operator”. [Certain] This session explains how that single operator works, why it is faster even though it does more arithmetic, and what changed in each version:

  • FlashAttention (Dao et al., 2022): tiling plus online softmax, so the \(T \times T\) table never goes to main memory; recomputation in the backward pass. Memory becomes linear in \(T\), and the result is exact, not an approximation.
  • FlashAttention-2 (Dao, 2023): the same math, with better work splitting on the GPU. About 2× faster than version 1, up to 73% of the A100’s peak in the forward pass. [Certain]
  • FlashAttention-3 (Shah et al., 2024): rewritten for Hopper GPUs (H100). It overlaps data movement, matrix multiplications and softmax, and adds FP8. Up to 740 TFLOP/s in FP16, about 75% of the H100’s peak. [Certain]

Every serious training and serving stack uses one of these kernels. PyTorch’s scaled_dot_product_attention dispatches to a FlashAttention kernel when the inputs allow it, and serving engines such as vLLM build their attention on FlashAttention-style kernels. [Likely] Long context for text, many image tokens for VLMs, and thousands of frame tokens for video models all depend on it.

Standard attention and its trip to HBM

Recall the formula from session 17. For one head, with \(Q, K, V \in \R^{T \times d}\):

\[ S = \frac{Q K^\top}{\sqrt{d}} \in \R^{T \times T}, \qquad P = \softmax(S) \ \text{(row by row)}, \qquad O = P\, V \in \R^{T \times d} \tag{124.1} \]
Read the symbols
\(T\)
sequence length (number of tokens)
\(d\)
head dimension, usually 64 or 128
\(S\)
the scores: one number per (query, key) pair
\(P\)
the attention probabilities; each row sums to 1
\(O\)
the output: for each query, a weighted average of the value rows

A GPU has two kinds of memory that matter here (session 42). HBM (high-bandwidth memory) is the big main memory: 80 GB on an H100, read at about 3.35 TB/s. SRAM (shared memory, on the chip) is tiny, at most 228 KB per streaming multiprocessor (SM) on an H100, but much faster. [Certain] The FlashAttention paper gives the A100 numbers: 40 to 80 GB of HBM at 1.5 to 2.0 TB/s, and 192 KB of SRAM in each of 108 SMs, with an estimated SRAM bandwidth of about 19 TB/s. [Certain]

The standard implementation runs three separate GPU programs (kernels), and each kernel reads its input from HBM and writes its output to HBM:

Algorithm 1: standard attention (three kernels)
  1. read \(Q, K\) from HBM; compute \(S = QK^\top/\sqrt d\); write \(S\) (\(T^2\) numbers) to HBM
  2. read \(S\) from HBM; compute \(P = \softmax(S)\); write \(P\) (\(T^2\) numbers) to HBM
  3. read \(P\) and \(V\) from HBM; compute \(O = PV\); write \(O\) to HBM
  4. // training also keeps \(P\) (or \(S\)) in HBM for the backward pass: \(O(T^2)\) memory

Count the numbers that cross the HBM boundary, with \(b\) bytes per number. \(Q, K, V\) are read once and \(O\) is written once (\(4Td\)), and the two tables are each written once and read once (\(4T^2\)):

\[ N_{\text{std}} = b\,\big(4Td + 4T^2\big) \ \text{bytes per head}, \qquad N_{\text{flash}} \approx b \cdot 4Td \tag{124.2} \]

When \(T\) is much larger than \(d\), the \(4T^2\) term is everything. A fused kernel that keeps \(S\) and \(P\) on the chip only needs the \(4Td\) part (we will refine this later). Real implementations also add masking and dropout kernels, each one another full pass over a \(T^2\) table, so (124.2) is the best case for standard attention.

To turn bytes into time, use the roofline rule from session 42: a kernel takes at least as long as its arithmetic and at least as long as its memory traffic.

\[ t_{\text{kernel}} \ge \max\!\left( \frac{F}{\pi},\ \frac{N}{\beta} \right) \tag{124.3} \]
Read the symbols
\(F\)
floating-point operations (FLOPs) of the kernel: \(2T^2 d\) for each of the two matrix multiplications
\(\pi\)
peak compute: 989 TFLOP/s on an H100 SXM with BF16 tensor cores, 312 on an A100
\(N\)
bytes moved to and from HBM
\(\beta\)
HBM bandwidth: 3.35 TB/s on an H100 SXM, 2.039 TB/s on an A100 SXM 80 GB
Worked example: one head at \(T = 8192\) on an H100

Take \(d = 128\), BF16 (\(b = 2\)). The two matrix multiplications have \(4T^2d = 34.4\) GFLOP, which needs at least \(34.4 \times 10^9 / 989 \times 10^{12} = 34.7\) µs.

Standard attention moves \(2(4Td + 4T^2) = 545.3\) MB. Kernel by kernel with (124.3): \(QK^\top\) needs 41.3 µs of memory time against 17.4 µs of compute; softmax needs 80.1 µs and almost no compute; \(PV\) again 41.3 µs. All three are memory-bound. Total: 162.8 µs.

A fused kernel moves only \(2 \cdot 4Td = 8.39\) MB (2.5 µs), so it is compute-bound at 34.7 µs. That is 4.68× faster, with the same FLOPs. The real gain is smaller, because no kernel reaches peak compute. Try other lengths and GPUs in Figure 1.

memory-bound partcompute-bound part
Figure 1. Roofline estimate (124.3) for one attention head in BF16: three standard kernels against one fused kernel. Each bar is the lower bound \(\max(F/\pi, N/\beta)\); its color shows which limit wins.
How to read Figure 1

Move \(T\) up: the standard bars grow like \(T^2\) and are always orange (memory-bound), because they read and write \(T^2\) tables. The fused bar is blue (compute-bound) for all but the shortest sequences. Switch to A100: its ridge point is lower (153 FLOP/byte against 295), so the gap is smaller. Tick the checkbox to charge every re-read of \(K\) and \(V\) to HBM (explained in Section 4): the fused kernel then moves about \(2T^2 d / B_r\) numbers, and the speedup drops to 2× when \(B_r = d = 128\). In practice most of those re-reads hit the 50 MB L2 cache. [Likely] This is an ideal model: it ignores launch overhead and the fact that real kernels reach only part of the peak.

The paper measured the same effect on GPT-2 medium (sequence length 1024, forward plus backward): standard attention did 66.6 GFLOP and moved 40.3 GB in 41.7 ms; FlashAttention did more work, 75.2 GFLOP, but moved only 4.4 GB and took 7.3 ms. [Certain] The extra FLOPs come from recomputation in the backward pass (Section 5). Fewer bytes won.

Online softmax: softmax without the whole row

Fusing is easy for the matrix multiplications: they can be done block by block. The problem is the softmax in the middle. Each output row needs

\[ \mathbf{o}_i = \frac{\sum_j e^{s_{ij} - m_i}\, \mathbf{v}_j}{\sum_j e^{s_{ij} - m_i}}, \qquad m_i = \max_j s_{ij}, \]

and both the maximum \(m_i\) (needed for stability, see session 2) and the sum in the denominator depend on all keys. If we can only hold a few keys on the chip at a time, we seem to need two passes over the row.

The online softmax (Milakov and Gimelshein, 2018) removes that need. Keep three running quantities per query row: the maximum so far \(m\), the sum of exponentials so far \(\ell\) (measured against that maximum), and the un-normalized output so far \(\tilde{\mathbf{o}}\). When a new block of scores \(\mathbf{s}^{(j)}\) with values \(V^{(j)}\) arrives:

\[ \begin{aligned} m' &= \max\!\big(m,\ \max_k s^{(j)}_k\big), \qquad \alpha = e^{\,m - m'} \\ \ell' &= \alpha\,\ell + \textstyle\sum_k e^{\,s^{(j)}_k - m'} \\ \tilde{\mathbf{o}}' &= \alpha\,\tilde{\mathbf{o}} + \textstyle\sum_k e^{\,s^{(j)}_k - m'}\, \mathbf{v}^{(j)}_k \end{aligned} \tag{124.4} \]
Read the symbols
\(m, \ell, \tilde{\mathbf{o}}\)
running maximum, running sum of exponentials, running weighted sum of values, for one query row; start with \(m = -\infty\), \(\ell = 0\), \(\tilde{\mathbf{o}} = \mathbf{0}\)
\(\mathbf{s}^{(j)}, \mathbf{v}^{(j)}_k\)
the scores of this query against key block \(j\), and the value rows of that block
\(\alpha\)
the correction factor: if the maximum went up, all old terms were measured against a too-small maximum and must shrink by \(e^{m - m'} \le 1\)

Why is this exact? Every old term was stored as \(e^{s - m}\). Multiplying by \(e^{m - m'}\) turns it into \(e^{s - m'}\), exactly what it would have been if we had known \(m'\) from the start. After the last block, divide once:

\[ \mathbf{o} = \frac{\tilde{\mathbf{o}}}{\ell}, \qquad L = m + \log \ell = \log \textstyle\sum_j e^{s_j} \tag{124.5} \]

\(L\) is the log-sum-exp of the row, one number. It is all we need to rebuild any probability later: \(p_j = e^{s_j - L}\). FlashAttention stores it for the backward pass.

Worked example: four scores in two blocks

Scores \(\mathbf{s} = (2, 1, 3, 0)\), scalar values \(v = (10, 20, 30, 40)\), blocks of two.

Block 1 \((2, 1)\): \(m = 2\), \(\ell = e^0 + e^{-1} = 1.3679\), \(\tilde o = 10 \cdot 1 + 20 \cdot e^{-1} = 17.3576\).

Block 2 \((3, 0)\): the maximum rises to \(m' = 3\), so \(\alpha = e^{2-3} = 0.3679\). Then \(\ell' = 1.3679 \times 0.3679 + e^0 + e^{-3} = 0.5032 + 1 + 0.0498 = 1.5530\) and \(\tilde o' = 17.3576 \times 0.3679 + 30 + 40\,e^{-3} = 6.3855 + 30 + 1.9915 = 38.3770\).

End: \(o = 38.3770 / 1.5530 = 24.7115\) and \(L = 3 + \log 1.5530 = 3.4402\). The ordinary softmax gives probabilities \((0.2369, 0.0871, 0.6439, 0.0321)\) and the same \(o = 24.7115\). Exact, with one pass.

block in SRAM nowalready processedweight \(e^{s - m}\) against the current maximum
  1. \(m \leftarrow -\infty\); \(\ell \leftarrow 0\); \(\tilde o \leftarrow 0\)
  2. for each block \((s, v)\) of the row:
  3. \(m' \leftarrow \max(m, \max s)\); \(\alpha \leftarrow e^{m - m'}\) // rescale old terms
  4. \(\ell \leftarrow \alpha\,\ell + \sum e^{s - m'}\); \(\tilde o \leftarrow \alpha\,\tilde o + \sum e^{s - m'} v\); \(m \leftarrow m'\)
  5. \(o \leftarrow \tilde o / \ell\); \(L \leftarrow m + \log \ell\) // once, at the end
Figure 2. Online softmax (124.4) on one query row with 12 keys, computed live. Each block takes two steps: first the new maximum \(m'\) and the correction \(\alpha\) (old bars shrink), then the new terms are added to \(\ell\) and \(\tilde o\) (new bars appear) and the block is forgotten. The last step divides by \(\ell\) (124.5). The live panel beside the algorithm puts the current numbers into the formulas.
How to read Figure 2

Press Step and follow the highlighted line of the small algorithm. Watch the green bars. They are the weights \(e^{s - m}\) of all keys seen so far, measured against the current maximum. When a block contains a new maximum, all older bars shrink at once by \(\alpha\) (line 3): that is the rescaling. When the maximum does not change, \(\alpha = 1\) and nothing shrinks. Line 4 then adds the new block’s bars to \(\ell\) and its weighted values to \(\tilde o\). Change the block size: the path is different, but the final output always equals the exact softmax result (shown in the last step) up to rounding in the last digits.

FlashAttention: tiling the whole computation

Now combine online softmax with tiles. Split \(Q\) into blocks of \(B_r\) rows, and \(K, V\) into blocks of \(B_c\) rows. One block of queries, one block of keys and values, and the small \(B_r \times B_c\) tile of scores all fit in SRAM. The kernel loops over key blocks and keeps \(m\), \(\ell\) and \(\tilde O\) for its query block on the chip. The \(T \times T\) matrices exist only as tiles, one at a time, and never reach HBM. This is kernel fusion: one kernel does what took three.

FlashAttention-2 uses this loop order (queries outside, keys inside), shown in Algorithm 2. FlashAttention-1 used the opposite order; we compare the two in Figure 4.

Algorithm 2: FlashAttention forward (FlashAttention-2 loop order)
  1. input: \(Q, K, V \in \R^{T \times d}\) in HBM; block sizes \(B_r, B_c\)
  2. for each query block \(Q_i\) (\(B_r\) rows) in parallel // one thread block each
  3. load \(Q_i\) into SRAM; set \(m_i = -\infty\), \(\ell_i = 0\), \(\tilde O_i = 0\) (on chip)
  4. for each key block \(j\) (skip blocks above the diagonal if causal)
  5. load \(K_j, V_j\) into SRAM
  6. \(S_{ij} = Q_i K_j^\top / \sqrt d\) // \(B_r \times B_c\) tile, on chip only
  7. \(m' = \max(m_i, \text{rowmax}(S_{ij}))\); \(\tilde P = e^{S_{ij} - m'}\); \(\alpha = e^{m_i - m'}\)
  8. \(\ell_i \leftarrow \alpha\,\ell_i + \text{rowsum}(\tilde P)\); \(\tilde O_i \leftarrow \text{diag}(\alpha)\,\tilde O_i + \tilde P V_j\); \(m_i \leftarrow m'\) // formula (124.4)
  9. write \(O_i = \text{diag}(\ell_i)^{-1} \tilde O_i\) and \(L_i = m_i + \log \ell_i\) to HBM // formula (124.5), once

Memory: besides \(Q, K, V, O\), the only thing written is \(L\), one FP32 number per row: 32 KiB per head at \(T = 8192\), instead of 128 MiB for \(P\). The FlashAttention paper states it formally: exact attention with \(O(T^2 d)\) FLOPs and \(O(T)\) extra memory. [Certain]

Figure 3 runs Algorithm 2 for real on a toy head: \(T = 8\) tokens, \(d = 2\), with fixed numbers for \(Q\), \(K\) and \(V\). You can watch each tile travel from HBM into SRAM, see the tile of scores, the running maximum \(m\), the correction \(\alpha\), the running sum \(\ell\) and the un-normalized output \(\tilde O\) change, and count every number that crosses the HBM boundary. At the end, the kernel’s output is compared with standard attention (124.1).

query tile \(Q_i\)key tile \(K_j\)value tile \(V_j\)score tile (fill \(= \tilde p = e^{s - m'}\))running \(m, \alpha, \ell, \tilde O\) and the output
  1. input: \(Q, K, V\) in HBM; block sizes \(B_r, B_c\)
  2. for each query block \(i\) // in parallel on a GPU
  3. load \(Q_i\) to SRAM; \(m \leftarrow -\infty\), \(\ell \leftarrow 0\), \(\tilde O \leftarrow 0\)
  4. for each key block \(j\) // skip if above the diagonal
  5. load \(K_j, V_j\) to SRAM
  6. \(S \leftarrow Q_i K_j^\top / \sqrt d\) // masked \(\to -\infty\)
  7. \(m' \leftarrow \max(m, \text{rowmax}\, S)\); \(\tilde P \leftarrow e^{S - m'}\); \(\alpha \leftarrow e^{m - m'}\)
  8. \(\ell \leftarrow \alpha\,\ell + \text{rowsum}\, \tilde P\); \(\tilde O \leftarrow \alpha\,\tilde O + \tilde P V_j\); \(m \leftarrow m'\)
  9. write \(O_i = \tilde O / \ell\) and \(L_i = m + \log \ell\) to HBM
  10. check: \(O\) against \(\softmax(QK^\top/\sqrt d)\,V\)
Figure 3. Algorithm 2 computed live on \(T = 8\), \(d = 2\) (scale \(1/\sqrt 2\)). Top: HBM holds \(Q\), \(K\), \(V\) and receives \(O\) and \(L\) only when a query block is finished. Middle: the SRAM of one SM, with the current query tile, key and value tiles, the \(B_r \times B_c\) score tile, and the running \(m\), \(\alpha\), \(\ell\), \(\tilde O\) of every row of the block. Right: numbers read from and written to HBM, against standard attention, which moves \(4Td + 4T^2 = 320\) numbers. The panel beside the algorithm puts the live numbers of one row (choose it with “row”) into formulas (124.4) and (124.5).
How to read Figure 3

Press Step and follow the highlighted line. A query tile flies from HBM into SRAM (line 3), then key and value tiles fly in one pair at a time (line 5). Each pair gives a small tile of scores (line 6); the cells are filled in proportion to \(\tilde p = e^{s - m'}\), so the largest score of each row is a full cell. Watch the \(\alpha\) column: it is 0 for the first tile (the old \(m\) was \(-\infty\)), below 1 when a row finds a larger score, and exactly 1 when its maximum does not change. Only line 9 writes to HBM: one row of \(O\) and one number \(L\) per query. With \(B_r = B_c = 2\) the run ends with 144 numbers read and 24 written, 168 in total, against 320 for standard attention; the largest difference from standard attention is about \(10^{-16}\), rounding in double precision. Set \(B_r = 4\): \(K\) and \(V\) are read only twice instead of four times, and traffic drops to 104. Turn on the causal mask: tiles above the diagonal are skipped, and entries above the diagonal inside a tile become \(-\infty\) (weight 0).

The toy shows the same pattern as a real kernel, only smaller. The traffic of Algorithm 2 is \(Td\) (read \(Q\)) + \(2Td\lceil T/B_r \rceil\) (read \(K, V\) once per query block) + \(Td\) (write \(O\)) + \(T\) (write \(L\)) numbers. For one head with \(T = 8192\), \(d = 128\), \(B_r = 128\), that is about 136 million numbers if every re-read of \(K\) and \(V\) went to HBM, against 273 million for standard attention: only 2× less. If the re-reads hit the L2 cache, HBM sees about 4.2 million numbers, 65× less. That is why the loop order and the cache matter, and why the analysis below is more careful than (124.2).

How many bytes, really?

Formula (124.2) said \(4Td\), but that hides the loops. In Algorithm 2 each query block reads all of \(K\) and \(V\), so \(K\) and \(V\) are read \(\lceil T/B_r \rceil\) times. The paper analyzes its own loop order with an SRAM of \(M\) numbers and proves (its Theorem 2 and Proposition 3):

\[ \text{HBM accesses: } \ \Theta\!\left(Td + T^2\right) \ \text{(standard)}, \qquad \Theta\!\left(\frac{T^2 d^2}{M}\right) \ \text{(FlashAttention)}, \qquad d \le M \le Td \tag{124.6} \]

and no exact attention algorithm can do asymptotically better for all \(M\) in that range. [Certain] Since \(d^2\) is much smaller than \(M\), the factor \(d^2/M\) is small. For example, 192 KB of SRAM holds \(M = 98{,}304\) BF16 numbers; with \(d = 64\), \(d^2/M = 1/24\). The paper sets the block sizes from \(M\): \(B_c = \lceil M/4d \rceil\) and \(B_r = \min(\lceil M/4d \rceil, d)\), which gives \(B_c = 192\) and \(B_r = 128\) for \(M = 98{,}304\), \(d = 128\). [Certain] In practice FlashAttention-2 uses tiles of 64 or 128 in each direction, chosen per head dimension. [Certain]

tile being computeddoneskipped (above the diagonal)
Figure 4. The score matrix split into 6 × 6 tiles (rows: query blocks; columns: key/value blocks). The counters on the right count tile-sized transfers between HBM and SRAM.
How to read Figure 4

In FlashAttention-1 order, the outer loop runs over key/value blocks: each \(K_j, V_j\) is loaded once, but for every tile the partial output \(O_i\) (and \(m_i, \ell_i\)) must be read from HBM and written back, because the next update of that row comes much later. In FlashAttention-2 order, a query block stays on the chip for its whole row, and \(O_i\) is written once at the end; \(K\) and \(V\) are re-read instead, often from the L2 cache. Turn on the causal mask: tiles fully above the diagonal are skipped, so only 21 of 36 tiles are computed, and only the 6 diagonal tiles need the mask applied element by element.

The causal case deserves a formula. With \(n = T/B\) blocks per side, a causal kernel computes the tiles on or below the diagonal:

\[ \frac{\text{tiles computed}}{\text{all tiles}} = \frac{n(n+1)/2}{n^2} = \frac{n+1}{2n} \ \longrightarrow \ \tfrac12 \tag{124.7} \]

For \(n = 6\) that is 0.583, for \(n = 64\) it is 0.508. Standard attention with a mask computes the full table and then throws half away. FlashAttention-2 reports a 1.7 to 1.8× speedup from skipping these blocks, compared with the same kernel without the causal mask. [Certain]

The backward pass: recompute instead of store

Training needs gradients. With \(dO = \partial \mathcal{L}/\partial O\) coming from above, the gradients of attention are (session 17 derives them):

\[ \begin{aligned} dV &= P^\top dO, \qquad dP = dO\, V^\top, \qquad D_i = \textstyle\sum_c dO_{ic}\, O_{ic} \\ dS &= P \circ \big(dP - D\,\mathbf{1}^\top\big), \qquad dQ = \frac{dS\, K}{\sqrt d}, \qquad dK = \frac{dS^\top Q}{\sqrt d} \end{aligned} \tag{124.8} \]
Read the symbols
\(\circ\)
element-wise product
\(D_i\)
one number per query row: the dot product of the output row and its gradient. It replaces the row sum \(\sum_j P_{ij}\, dP_{ij}\) of the softmax gradient, because the two are equal
\(dS\)
gradient with respect to the scores, the softmax backward

Every line needs \(P\), a \(T \times T\) matrix. Standard attention keeps it in HBM from the forward pass. FlashAttention does not have it. Instead, it recomputes each tile of \(P\) in the backward pass from \(Q\), \(K\) and the stored \(L\):

\[ P_{ij} = \exp\!\big( S_{ij} - L_i \big), \qquad S_{ij} = \frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt d} \tag{124.9} \]

No running maximum is needed here: \(L_i\) already is the exact normalizer. This costs one extra \(QK^\top\) multiplication per tile, but it avoids reading a \(T^2\) matrix from HBM. It is the same trade as activation checkpointing (session 39), applied inside one kernel, and it is why FlashAttention did 75.2 GFLOP against 66.6 and was still 5.7× faster in the GPT-2 measurement above. [Certain] FlashAttention-1 stored both \(m\) and \(\ell\); FlashAttention-2 stores only \(L = m + \log \ell\). [Certain]

Algorithm 3: FlashAttention backward (one key/value block per thread block)
  1. input: \(Q, K, V, O, dO\) and \(L\) in HBM
  2. \(D = \text{rowsum}(dO \circ O)\) // small pre-pass, \(T\) numbers
  3. for each key/value block \(j\) in parallel
  4. load \(K_j, V_j\); set \(dK_j = 0\), \(dV_j = 0\) on chip
  5. for each query block \(i\)
  6. load \(Q_i, dO_i, L_i, D_i\); \(S_{ij} = Q_i K_j^\top / \sqrt d\); \(P_{ij} = e^{S_{ij} - L_i}\) // recompute, (124.9)
  7. \(dV_j \mathrel{+}= P_{ij}^\top dO_i\); \(dP_{ij} = dO_i V_j^\top\); \(dS_{ij} = P_{ij} \circ (dP_{ij} - D_i)\)
  8. \(dK_j \mathrel{+}= dS_{ij}^\top Q_i / \sqrt d\); add \(dS_{ij} K_j / \sqrt d\) into \(dQ_i\) in HBM // atomic add
  9. write \(dK_j, dV_j\) to HBM

FlashAttention-2: the same math, better work split

FlashAttention-1 reached only 25 to 40% of the A100’s peak FLOP/s. The FlashAttention-2 paper traces this to how the work is split across thread blocks and warps, and makes three changes. [Certain]

1. Fewer non-matmul FLOPs

An A100 does 312 TFLOP/s of FP16/BF16 matrix multiplication on tensor cores, but only 19.5 TFLOP/s of other FP32 arithmetic. So each non-matmul FLOP (exp, max, rescaling) costs about 16× more time than a matmul FLOP. [Certain] FlashAttention-2 therefore keeps \(\tilde O\) un-normalized and divides by \(\ell\) only once at the end (Algorithm 2, line 9), instead of rescaling the normalized output at every step, and it stores just \(L\) instead of \(m\) and \(\ell\).

2. Parallelism over the sequence

FlashAttention-1 used one thread block per (batch, head) pair. A thread block runs on one SM. With batch 1 and 16 heads, only 16 of the A100’s 108 SMs have work. Long sequences usually come with small batches, so this case is common. FlashAttention-2 also splits the sequence: one thread block per query block, so a single head at \(T = 16{,}384\) with \(B_r = 128\) gives 128 thread blocks. This works because query blocks are independent in the forward pass; only key/value blocks interact through the softmax, and that interaction is inside the loop. [Certain]

SM busy in the first waveSM idle
Figure 5. How many SMs get work. FlashAttention-1 launches one thread block per (batch, head); FlashAttention-2 launches one per (batch, head, query block of 128 rows). Toy model: one thread block per SM at a time, all blocks equally long.
How to read Figure 5

“Waves”: if there are 300 thread blocks and 108 SMs, the GPU runs them in \(\lceil 300/108 \rceil = 3\) waves, and the last wave is partly empty. Utilization in this model is blocks / (waves × SMs). Start with batch 1 and 16 heads: FlashAttention-1 fills 16 of 108 SMs (15%). FlashAttention-2 fills almost all of them. Now raise the batch to 32: both versions are fine, because there are enough (batch, head) pairs. The change matters most for long sequences with small batches, exactly the long-context case. Real GPUs can run more than one thread block per SM, so the real numbers differ; the direction does not.

3. Splitting work between warps

Inside a thread block, 4 or 8 warps (groups of 32 threads) share the work. FlashAttention-1 split \(K\) and \(V\) across warps (“split-K”): each warp got the full query block and part of the keys, so at the end the warps had to write partial results to shared memory, synchronize and add them up. FlashAttention-2 splits \(Q\) across warps (“split-Q”): every warp sees all of \(K\) and \(V\) but owns its own query rows, so each warp finishes its rows alone, with no exchange. [Certain] The backward pass also avoids split-K, though it still needs some synchronization. [Certain]

The result: 1.7 to 3.0× faster than FlashAttention-1, up to 73% of peak in the forward pass and 63% in the backward pass, and up to 225 TFLOP/s per A100 when training GPT-style models end to end (72% model FLOPs utilization). [Certain] FlashAttention-2 also supports multi-query and grouped-query attention (session 97) by indexing into the shared key/value heads instead of copying them. [Certain]

Model (8× A100, TFLOP/s per GPU)ContextNo FlashAttentionFlashAttentionFlashAttention-2
GPT3-1.3B2k72 to 149 (by model and context)189196
GPT3-1.3B8k170220
GPT3-2.7B2k189205
GPT3-2.7B8k175225

Notice the pattern in the table, from the FlashAttention-2 paper: the gain over version 1 is small at 2k and large at 8k, where attention takes a bigger share of the time. [Certain]

FlashAttention-3: asynchrony and FP8 on Hopper

Run on an H100, FlashAttention-2 reaches only about 35% utilization. [Certain] The H100 tensor cores are so fast that everything else becomes the bottleneck. Hopper adds hardware that FlashAttention-3 uses:

  • TMA (Tensor Memory Accelerator): a unit that copies tiles between HBM and shared memory by itself, so threads do not have to compute addresses and issue loads.
  • WGMMA: asynchronous matrix multiplication issued by a warpgroup (4 warps, 128 threads). The warpgroup starts it and can do other work while the tensor cores run.

The exponential is the new bottleneck

Per score, the two matrix multiplications cost \(4d\) FLOPs (\(2d\) for \(QK^\top\), \(2d\) for \(PV\)), while the softmax needs one exponential. But the H100 has 989 TFLOP/s of FP16 matmul and only 3.9 TFLOP/s for special functions such as exp. [Certain] With \(d = 128\):

\[ \frac{t_{\exp}}{t_{\text{matmul}}} = \frac{1 / 3.9}{4d / 989} = \frac{989 / 3.9}{512} \approx \frac{254}{512} \approx 0.50 \tag{124.10} \]

So the exponentials alone can take about half as long as all the matrix multiplications. If they run one after the other, the tensor cores sit idle a third of the time. FlashAttention-3 overlaps them in two ways. [Certain]

  • Warp specialization. Some warps are producers: they only issue TMA loads of \(K\) and \(V\). The others are consumers: they issue WGMMAs and compute the softmax. Registers are moved from producers to consumers (the setmaxnreg instruction), since consumers need many more.
  • Ping-pong scheduling. Two consumer warpgroups take turns: while warpgroup 1 computes the softmax of its tile, warpgroup 2 runs its matrix multiplications, and then they swap. Inside one warpgroup, a 2-stage pipeline also overlaps the softmax of one tile with the WGMMA of the next.
GEMM on tensor coressoftmax (exp units)
Figure 6. Toy timeline of the tensor cores. Each tile needs a GEMM (length 1) and then a softmax (length \(r\)). With one warpgroup and no overlap they alternate; with ping-pong two warpgroups interleave so one’s softmax hides behind the other’s GEMM.
How to read Figure 6

The bottom row is the tensor cores. Without overlap, they are busy \(1/(1+r)\) of the time: 67% at \(r = 0.5\), the ratio from (124.10). With ping-pong, as long as \(r \le 1\) the softmax of one warpgroup fits inside the other’s GEMM and the tensor cores never wait (100% in this toy). Above \(r = 1\) even ping-pong cannot hide it: utilization is \(2/(1+r)\). Real kernels have other costs; the paper measured 570 → 620 to 640 TFLOP/s from ping-pong (FP16, head dimension 128, length 8192). [Certain]

FP8 attention

The H100 doubles its tensor-core throughput in FP8 (see session 105). FP8 attention raises two problems. First, a layout rule: FP8 WGMMA needs its operands in “k-major” order, so FlashAttention-3 transposes the \(V\) tile inside the kernel. Second, accuracy: LLM activations have outliers, a few very large values, and with one scale for a whole tensor they force every other value into the few lowest FP8 levels. FlashAttention-3 uses two fixes. [Certain]

  • Block quantization: one scale per block of \(Q\), \(K\) or \(V\), not per tensor. The tiles are already there, so this is cheap.
  • Incoherent processing: multiply \(Q\) and \(K\) by the same random orthogonal matrix \(M\) (random \(\pm 1\) signs times a Hadamard matrix) before quantizing. Because \(M M^\top = I\), the scores do not change, but each outlier is spread over many coordinates:
\[ (Q M)(K M)^\top = Q\, M M^\top K^\top = Q K^\top \tag{124.11} \]

This is the same idea as the rotations of session 109 (QuIP, QuaRot). The paper’s test data had about 0.1% outlier entries; the FP8 root-mean-square error was \(9.1 \times 10^{-3}\) for FlashAttention-3 against \(2.4 \times 10^{-2}\) for a baseline FP8 attention, 2.6× lower. [Certain] Overall: 1.5 to 2.0× faster than FlashAttention-2 in FP16, up to 740 TFLOP/s, and close to 1.2 PFLOP/s in FP8. [Certain]

FlashAttention-1 (2022)FlashAttention-2 (2023)FlashAttention-3 (2024)
Main ideatiling + online softmax + recompute in backwardfewer non-matmul FLOPs, parallel over sequence, split-Q between warpswarp specialization, ping-pong, GEMM–softmax overlap, FP8
Target GPUA100 (and older)A100; runs on H100 without Hopper featuresH100 / Hopper only
Kept for backward\(O\), \(m\), \(\ell\)\(O\), \(L\)\(O\), \(L\)
Reported speed25–40% of A100 peak50–73% of A100 peak (forward); 335 TFLOP/s on H100up to 740 TFLOP/s FP16 (75% of H100), about 1.2 PFLOP/s FP8

Numerics: exact, but not bit-identical

FlashAttention is “exact”: in exact arithmetic it gives the same output as (124.1). In floating point the results differ a little, for four reasons:

  1. Different order of additions. The row sums and \(PV\) products are accumulated block by block. Floating-point addition is not associative, so a different order gives different last digits.
  2. Rescaling. Each \(\alpha = e^{m - m'}\) multiplication adds one more rounding.
  3. Rounding of \(P\). The tile of probabilities is rounded to BF16 or FP16 before the \(\tilde P V\) multiplication on the tensor cores. A standard implementation that keeps \(P\) in FP32 avoids this step.
  4. Atomic adds in backward (Algorithm 3): the order changes from run to run.

How big is this? Golden et al. (2024) measured FlashAttention in BF16 against a baseline attention in an isolated forward pass and found roughly an order of magnitude more numeric deviation; their analysis estimates its effect on model weights during training as 2 to 5 times smaller than the effect of low-precision training itself. [Certain] The efficient-inference survey also warns in a footnote that FlashAttention “might cause the numeric deviation”. [Certain] In FP16 tests, the FlashAttention-3 paper reports the opposite direction: an RMSE of \(1.9 \times 10^{-4}\) for FlashAttention-2 and -3 against \(3.2 \times 10^{-4}\) for a standard FP16 implementation, because the kernels keep the running sums in FP32. [Certain] The practical rule: compare against a high-precision reference with a tolerance, never with ==.

The algorithm in PyTorch

The code below is a reference implementation, not a fast one: Python loops over tiles. It shows exactly what the kernel does. It checks the tiled forward against naive attention, checks the recomputing backward (formulas 124.8 and 124.9, left untiled for clarity) against autograd, and measures low-precision errors. It runs on a CPU in a few seconds.

import math, torch
torch.manual_seed(0)

def naive_attention(q, k, v, causal=False):
    # q, k, v: (B, h, T, d). Builds the full T x T matrices, like standard attention.
    S = q @ k.transpose(-2, -1) / math.sqrt(q.shape[-1])
    if causal:
        T = q.shape[-2]
        S = S.masked_fill(torch.ones(T, T, dtype=torch.bool).triu(1), float("-inf"))
    return torch.softmax(S, dim=-1) @ v

def flash_forward(q, k, v, Br=64, Bc=64, causal=False):
    # FlashAttention-2 order: outer loop over query blocks, inner loop over key/value blocks.
    B, h, T, d = q.shape
    scale = 1.0 / math.sqrt(d)
    O = torch.empty_like(q)
    L = torch.empty(B, h, T, dtype=torch.float32)
    for i in range(0, T, Br):                      # one thread block per query block on a GPU
        qi = q[:, :, i:i+Br].float()
        m = torch.full(qi.shape[:-1], float("-inf"))   # running max, per row
        l = torch.zeros(qi.shape[:-1])                 # running sum of exp, per row
        acc = torch.zeros_like(qi)                     # un-normalized output
        for j in range(0, T, Bc):
            if causal and j > i + Br - 1:
                break                                  # block is fully above the diagonal: skip it
            kj, vj = k[:, :, j:j+Bc].float(), v[:, :, j:j+Bc].float()
            s = qi @ kj.transpose(-2, -1) * scale      # (Br, Bc) tile, lives in SRAM only
            if causal:
                rows = torch.arange(i, i + qi.shape[-2])[:, None]
                cols = torch.arange(j, j + kj.shape[-2])[None, :]
                s = s.masked_fill(cols > rows, float("-inf"))
            m_new = torch.maximum(m, s.amax(-1))
            p = torch.exp(s - m_new[..., None])
            alpha = torch.exp(m - m_new)               # rescale factor for the old partial results
            l = alpha * l + p.sum(-1)
            acc = alpha[..., None] * acc + p.to(q.dtype).float() @ vj   # P is rounded to the input dtype, as on a GPU
            m = m_new
        O[:, :, i:i+Br] = (acc / l[..., None]).to(q.dtype)   # divide once, at the end (FA2)
        L[:, :, i:i+Br] = m + torch.log(l)                    # one number per row is kept for backward
    return O, L

def flash_backward(q, k, v, O, L, dO, causal=False):
    # Recompute P from Q, K and the stored log-sum-exp L instead of reading a stored T x T matrix.
    d = q.shape[-1]
    S = q.float() @ k.float().transpose(-2, -1) / math.sqrt(d)
    if causal:
        T = q.shape[-2]
        S = S.masked_fill(torch.ones(T, T, dtype=torch.bool).triu(1), float("-inf"))
    P = torch.exp(S - L[..., None])                    # exact softmax, no max or sum needed
    D = (dO.float() * O.float()).sum(-1, keepdim=True) # D_i = rowsum(dO * O)
    dV = P.transpose(-2, -1) @ dO.float()
    dP = dO.float() @ v.float().transpose(-2, -1)
    dS = P * (dP - D)
    dQ = dS @ k.float() / math.sqrt(d)
    dK = dS.transpose(-2, -1) @ q.float() / math.sqrt(d)
    return dQ, dK, dV

B, h, T, d = 1, 2, 256, 64
q, k, v = (torch.randn(B, h, T, d, dtype=torch.float64) for _ in range(3))

# 1) the tiled forward matches naive attention (the tiles compute in fp32, like a GPU kernel)
for causal in (False, True):
    O, L = flash_forward(q, k, v, causal=causal)
    print("forward max error, causal =", causal, (O - naive_attention(q, k, v, causal)).abs().max().item())

# 2) the recomputing backward matches autograd through the naive version
qr, kr, vr = (t.clone().requires_grad_(True) for t in (q, k, v))
out = naive_attention(qr, kr, vr, causal=True)
dO = torch.randn_like(out)
out.backward(dO)
O, L = flash_forward(q, k, v, causal=True)
dQ, dK, dV = flash_backward(q, k, v, O, L, dO, causal=True)
print("backward max error:", max((a - b).abs().max().item() for a, b in ((dQ, qr.grad), (dK, kr.grad), (dV, vr.grad))))

# 3) numerics in low precision: both methods round, in different places
ref = naive_attention(q, k, v)                         # float64 reference
for dt in (torch.float16, torch.bfloat16):
    qh, kh, vh = q.to(dt), k.to(dt), v.to(dt)
    std = naive_attention(qh.float(), kh.float(), vh.float()).to(dt)   # standard math in fp32, output rounded
    fl, _ = flash_forward(qh, kh, vh)
    e_std = (std.double() - ref).abs().max().item()
    e_fl = (fl.double() - ref).abs().max().item()
    e_between = (std.double() - fl.double()).abs().max().item()
    print(dt, "max error vs fp64: standard %.2e, tiled %.2e, between them %.2e" % (e_std, e_fl, e_between))

With PyTorch 2.8 on a CPU this prints forward errors of about \(3 \times 10^{-7}\) (non-causal) and \(6 \times 10^{-7}\) (causal), which is FP32 rounding, and a backward error of about \(3 \times 10^{-6}\). In FP16 both methods have a maximum error of \(4.1 \times 10^{-4}\) against the FP64 reference and differ from each other by \(2.4 \times 10^{-4}\). In BF16, which has fewer mantissa bits, the tiled version is a little worse (\(3.8 \times 10^{-3}\) against \(2.7 \times 10^{-3}\)), mostly because \(\tilde P\) is rounded to BF16 before the \(PV\) product. On a GPU you would call the real kernel: F.scaled_dot_product_attention(q, k, v, is_causal=True) (with F = torch.nn.functional), or flash_attn_func from the flash-attn package.

Where this is used

SettingWhat FlashAttention givesNotes
LLM pretraining and fine-tuningAttention memory linear in \(T\); no stored \(P\); 2–4× faster attentionThe 8k rows of the FlashAttention-2 table; long-context continued pretraining depends on it
Prefill in LLM servingFast compute-bound attention over the whole promptDecode has one query row per step, a different problem: session 125
Packed or variable-length batches“varlen” kernels take concatenated sequences plus their start offsets, with no paddingPairs with sequence packing (session 18) [Likely]
Vision transformers and VLM encodersHigh-resolution images give thousands of patch tokens per imageBidirectional attention: no causal skipping, so the full tile grid is computed
Video models (action recognition, key-moment detection, video retrieval, Video-RAG)Frames × patches easily reach tens of thousands of tokens; without fused attention the \(T^2\) table alone would not fitMany video LLMs also reduce tokens first; FlashAttention makes the remaining long sequences affordable [Likely]
Long-document modelsExact full attention to 16K–64K tokens on one GPUFlashAttention made the first transformers better than chance on Path-X (16K tokens, 61.4%) and, with block-sparse FlashAttention, Path-256 (64K, 63.1%) [Certain]
Sparse and sliding-window attentionThe same tiling with whole tiles skippedBlock-sparse FlashAttention in the first paper; local windows in later kernels (session 116)

Common mistakes

  • “FlashAttention reduces the FLOPs.” It does not. Forward FLOPs are the same \(O(T^2 d)\), and training does more because of recomputation. It reduces HBM traffic and memory.
  • “FlashAttention is an approximation.” It is exact up to floating-point rounding. Linformer, Performer and sparse attention are approximations; FlashAttention is not.
  • Expecting bit-identical outputs after switching kernels, or comparing with == in tests. Use a tolerance against an FP32 or FP64 reference.
  • Silently falling back to the slow path. scaled_dot_product_attention picks a backend from dtype, head dimension, mask type and hardware. An arbitrary dense mask tensor, FP32 inputs or an unsupported head size can send you to the math backend with its \(T^2\) memory. Check which backend runs. [Likely]
  • Building a full \(T \times T\) boolean mask for causal attention and passing it in. That allocates \(T^2\) memory and blocks causal tile skipping; use is_causal=True.
  • Thinking FlashAttention solves decoding. At decode time there is one query per sequence, so splitting over query blocks gives no parallel work; decode needs a split over keys (session 125).
  • Using FlashAttention-3 features on the wrong GPU. Its speedups come from Hopper hardware (TMA, WGMMA); on an A100 use FlashAttention-2.

Interview questions

Why is FlashAttention faster if it does the same number of FLOPs?basic
Standard attention is memory-bound: it writes the \(T \times T\) score and probability matrices to HBM and reads them back, \(4T^2\) numbers per head, while the matrix multiplications themselves are cheap on tensor cores. FlashAttention computes attention in tiles inside SRAM and never writes those matrices, so HBM traffic drops to about \(4Td\) numbers (plus re-reads of \(K, V\)). On GPT-2 medium the paper measured 40.3 GB vs 4.4 GB of HBM traffic and 41.7 ms vs 7.3 ms, even though FlashAttention did more FLOPs (75.2 vs 66.6 GFLOP).
Explain online softmax. Why is it exact?core
Keep a running maximum \(m\), running sum \(\ell = \sum e^{s - m}\) and running output \(\tilde o = \sum e^{s - m} v\). When a new block raises the maximum to \(m'\), multiply the old \(\ell\) and \(\tilde o\) by \(\alpha = e^{m - m'}\), then add the new block’s terms measured against \(m'\). Every old term \(e^{s - m}\) becomes exactly \(e^{s - m'}\), so at the end \(\tilde o / \ell\) is the exact softmax-weighted average. Only rounding differs.
What does FlashAttention store for the backward pass, and why is recomputation a good trade?core
It stores the output \(O\) and one number per row: the log-sum-exp \(L = m + \log \ell\) (FlashAttention-1 stored \(m\) and \(\ell\)). In the backward pass it recomputes each tile \(S = QK^\top/\sqrt d\) and \(P = e^{S - L}\) in SRAM. This adds one matrix multiplication per tile, but tensor-core FLOPs are cheap while reading a stored \(T^2\) matrix from HBM is expensive, and the memory saving is \(O(T^2) \to O(T)\). It is activation checkpointing inside one kernel.
Write the backward formulas of attention. What is the D term?advanced
\(dV = P^\top dO\), \(dP = dO V^\top\), \(dS = P \circ (dP - D\mathbf 1^\top)\), \(dQ = dS K/\sqrt d\), \(dK = dS^\top Q/\sqrt d\). The softmax backward needs \(\sum_j P_{ij} dP_{ij}\) per row, which would need a full row of \(P\). But \(\sum_j P_{ij} (dO_i \cdot v_j) = dO_i \cdot \sum_j P_{ij} v_j = dO_i \cdot O_i = D_i\), computable from \(O\) and \(dO\) alone, \(T\) numbers in a cheap pre-pass.
What is the IO complexity of FlashAttention, and is it optimal?advanced
With SRAM of \(M\) numbers and \(d \le M \le Td\), FlashAttention needs \(\Theta(T^2 d^2 / M)\) HBM accesses, against \(\Theta(Td + T^2)\) for standard attention. Since \(d^2\) is much smaller than \(M\) (for example \(64^2\) against about 98K), that is many times fewer. The paper also proves a lower bound: no exact attention algorithm can do \(o(T^2 d^2/M)\) for all \(M\) in that range, so it is asymptotically optimal.
What are the three main changes in FlashAttention-2?core
(1) Fewer non-matmul FLOPs: keep the output un-normalized and divide once at the end; store only the log-sum-exp. Non-matmul FLOPs are about 16× more expensive on an A100 (19.5 vs 312 TFLOP/s). (2) Parallelize over the sequence dimension too (one thread block per query block), so a long sequence with batch 1 still fills the 108 SMs. (3) Split \(Q\) across warps instead of \(K\) (split-Q vs split-K), so warps do not need to exchange and sum partial results through shared memory. Result: about 2× over FlashAttention-1, 50–73% of peak forward.
Why does FlashAttention-3 need Hopper, and what does it overlap?advanced
It uses Hopper-only hardware: TMA for asynchronous tile copies and WGMMA for asynchronous warpgroup matrix multiplications. With these, producer warps load data while consumer warps compute (warp specialization), two consumer warpgroups alternate GEMM and softmax (ping-pong), and inside a warpgroup the softmax of one tile overlaps the GEMM of the next. This matters because exp throughput (3.9 TFLOP/s) is so low next to matmul (989 TFLOP/s) that at \(d = 128\) the exponentials take about half as long as the GEMMs.
How does FlashAttention-3 keep FP8 attention accurate?advanced
Two techniques. Block quantization: one scale per tile of \(Q\), \(K\), \(V\) instead of per tensor, so an outlier only hurts its own block. Incoherent processing: multiply \(Q\) and \(K\) by the same random orthogonal matrix (random signs times a Hadamard matrix); \(QM(KM)^\top = QK^\top\), but outliers are spread over many coordinates, so they quantize better. With 0.1% outliers in the test data, RMSE fell from \(2.4 \times 10^{-2}\) (baseline FP8) to \(9.1 \times 10^{-3}\).
Is FlashAttention exact? Why might outputs differ from standard attention?basic
Mathematically exact. Numerically different: blockwise accumulation changes the order of floating-point additions, rescaling by \(e^{m - m'}\) adds roundings, the probability tile is rounded to BF16/FP16 before the \(PV\) matmul, and atomic adds in the backward pass make the order vary between runs. Golden et al. measured about 10× more deviation than baseline in BF16 forward, but estimate its training effect at 2–5× smaller than that of low-precision training itself. Test with tolerances.
How does the causal mask make FlashAttention faster?core
Tiles completely above the diagonal contribute nothing, so the kernel skips them: with \(n\) blocks per side it computes \(n(n+1)/2\) of \(n^2\) tiles, approaching half. Only the diagonal tiles need element-wise masking. FlashAttention-2 reports 1.7–1.8× speedup over the same kernel without the mask. Standard attention computes the full table and then masks it, so it gets no saving.
Your long-context training run with batch size 1 is slow in attention. What do you check?core
First, that a fused kernel actually runs (profile it: session 123); a dense mask, FP32 inputs or an unsupported head size may cause a fallback. Second, that the kernel parallelizes over the sequence (FlashAttention-2 or later), because with batch 1 and few heads a per-head kernel leaves most SMs idle. Third, use is_causal instead of a mask tensor so masked tiles are skipped, and use variable-length kernels instead of padding. On H100, FlashAttention-3 gives a further 1.5–2×.

Exercises

  1. For \(T = 32{,}768\) and 32 heads in BF16, how much memory does one layer’s probability matrix \(P\) need? How much does FlashAttention’s \(L\) need (FP32)?
  2. Run online softmax by hand on scores \((1, 3)\) then \((2)\), with values \((0, 10)\) then \((5)\). Give \(m\), \(\ell\), \(\tilde o\) after each block and the final output.
  3. Show that \(\sum_j P_{ij}\, dP_{ij} = dO_i \cdot O_i\) using \(dP = dO\,V^\top\) and \(O = PV\).
  4. With 8 query blocks per side, what fraction of tiles does a causal kernel compute? How many need element-wise masking?
  5. Batch 2, 8 heads, \(T = 4096\), \(B_r = 128\), on an A100 with 108 SMs. How many thread blocks do FlashAttention-1 and FlashAttention-2 launch? What is the toy utilization of Figure 5 for each?
  6. Using (124.10), what is the exp-to-matmul time ratio for \(d = 64\)? Is overlap more or less important than for \(d = 128\)?
Answerscheck
1. \(32 \times 32768^2 \times 2\) bytes = 64 GiB per layer. \(L\): \(32 \times 32768 \times 4\) bytes = 4 MiB. 2. Block 1: \(m = 3\), \(\ell = e^{-2} + 1 = 1.1353\), \(\tilde o = 0 + 10 = 10\). Block 2: \(m' = 3\) (no change, \(\alpha = 1\)), \(\ell = 1.1353 + e^{-1} = 1.5032\), \(\tilde o = 10 + 5e^{-1} = 11.8394\). Output \(11.8394/1.5032 = 7.876\). 3. \(\sum_j P_{ij} dP_{ij} = \sum_j P_{ij} \sum_c dO_{ic} V_{jc} = \sum_c dO_{ic} \sum_j P_{ij} V_{jc} = \sum_c dO_{ic} O_{ic}\). 4. \(36/64 = 0.5625\); the 8 diagonal tiles. 5. FlashAttention-1: \(2 \times 8 = 16\) blocks, one partial wave, utilization \(16/108 = 14.8\%\). FlashAttention-2: \(16 \times 32 = 512\) blocks, \(\lceil 512/108 \rceil = 5\) waves, \(512/540 = 94.8\%\). 6. \((989/3.9)/256 \approx 0.99\): the exponentials take about as long as the GEMMs, so overlap matters even more for small heads.

Key takeaways

  • Standard attention is slow because it writes and reads the \(T \times T\) matrices in HBM (124.2); by the roofline (124.3) its kernels are memory-bound.
  • Online softmax (124.4, 124.5) computes an exact softmax in one pass over blocks by rescaling old partial sums when the maximum grows.
  • FlashAttention tiles \(Q\), \(K\), \(V\) into SRAM, fuses everything into one kernel, uses \(O(T)\) extra memory, and is IO-optimal: \(\Theta(T^2 d^2/M)\) HBM accesses (124.6). Causal tiles above the diagonal are skipped (124.7).
  • The backward pass recomputes \(P = e^{S - L}\) from the stored log-sum-exp (124.8, 124.9): more FLOPs, far fewer bytes.
  • FlashAttention-2: fewer non-matmul FLOPs, parallel over the sequence, split-Q between warps; about 2× over version 1. FlashAttention-3: Hopper asynchrony (warp specialization, ping-pong) hides the slow exponentials (124.10), and FP8 with block quantization and incoherent processing (124.11).
  • Results are exact in math, slightly different in floating point; test with tolerances.

Key formulas

\[ m' = \max(m, \max_k s_k), \quad \ell' = e^{m - m'}\ell + \textstyle\sum_k e^{s_k - m'}, \quad \tilde{\mathbf{o}}' = e^{m - m'}\tilde{\mathbf{o}} + \textstyle\sum_k e^{s_k - m'}\mathbf{v}_k \]
\[ \mathbf{o} = \tilde{\mathbf{o}}/\ell, \quad L = m + \log \ell, \quad P_{ij} = e^{S_{ij} - L_i}, \quad dS = P \circ (dP - D\mathbf{1}^\top), \quad \Theta\!\left(\frac{T^2 d^2}{M}\right) \]

References for this session

  1. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  2. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  3. Shah et al. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
  4. Milakov & Gimelshein (2018). Online normalizer calculation for softmax
  5. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  6. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  7. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  8. Shah et al. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
  9. Milakov & Gimelshein (2018). Online normalizer calculation for softmax
  10. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  11. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  12. Golden, Hsia, Sun et al. (2024). Is Flash Attention Stable?