LLM & VLM courseBlog
37Track 3: Pretraining and scale · session 7 of 22

Numbers on GPUs: FP32, TF32, FP16, BF16, FP8 and mixed precision

Every weight, activation and gradient is stored with a fixed number of bits. This session shows how those bits are split between range (the exponent) and precision (the mantissa), why FP16 training needs loss scaling while BF16 usually does not, how mixed precision keeps FP32 where it matters, and how FP8 training works with E4M3, E5M2 and scaling factors.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Session 4 (optimization and AdamW); Session 36 (training recipe and stability) helps

What you will learn

  • How sign, exponent and mantissa bits encode a number, including subnormals, infinity and NaN
  • The range and precision of FP32, TF32, FP16, BF16, E4M3 and E5M2, and how to compute them from the bit counts
  • The three ways low precision breaks training: overflow, underflow and swamping (small updates and long sums that disappear)
  • Why FP16 needs loss scaling, how dynamic loss scaling works, and why BF16 usually does not need it
  • The mixed-precision recipe: low-precision matmuls, FP32 accumulation, FP32 master weights and optimizer, and which ops stay in FP32
  • FP8 training: E4M3 for the forward pass, E5M2 for gradients, per-tensor, per-block and MXFP8 scaling, and the DeepSeek-V3 recipe

Why this matters

A GPU does not compute with “real numbers”. It computes with a small, fixed set of bit patterns. How many bits you use, and how you split them, decides three things at once: how fast the matmuls run, how much memory the model needs, and whether training converges or ends in NaN.

On an H100 SXM, plain FP32 math peaks at about 67 TFLOP/s. The same chip does about 989 TFLOP/s in BF16 and about 1,979 TFLOP/s in FP8 on its tensor cores (dense numbers; the spec sheet shows twice these values “with sparsity”). [Certain] So moving from FP32 to BF16 gives up to about 15 times more compute, and every tensor takes half the memory and half the bandwidth. Almost every large model today is trained in mixed precision: most math in 16 or 8 bits, a few critical parts in 32 bits.

The price is that low-precision numbers can break in three ways. Big values overflow to infinity. Tiny values underflow to zero. And a small number added to a big one can simply disappear. This session explains each failure with real numbers, then shows the tricks that prevent them: loss scaling, FP32 master weights, FP32 accumulation, and scaling factors for FP8.

In session 36 you saw that BF16 logits of size 1,000 can only be multiples of 4. That was one symptom. Here we build the full picture, and in session 38 we turn it into a memory budget (the famous 16 bytes per parameter).

How a computer stores a real number

A floating-point number is scientific notation in base 2. Like \(6.02 \times 10^{23}\), it has three parts: a sign, a number of significant digits, and a power. The bits are split into three fields:

  • Sign (1 bit): 0 for positive, 1 for negative.
  • Exponent (\(E\) bits): which power of 2. More exponent bits give more range (how big and how small a number can be).
  • Mantissa (\(M\) bits, also called the fraction or significand): the significant digits after the leading 1. More mantissa bits give more precision (how close two neighbouring numbers are).

The exponent field stores an unsigned integer \(e\). To allow negative powers, a fixed bias \(b = 2^{E-1} - 1\) is subtracted (127 for 8 exponent bits, 15 for 5 bits, 7 for 4 bits). For a normal number, there is an invisible leading 1 in front of the mantissa:

\[ x = (-1)^{s} \cdot 2^{\,e - b} \cdot \left(1 + \frac{m}{2^{M}}\right), \qquad 1 \le e \le e_{\max} \tag{37.1} \]
Read the symbols
\(s\)
the sign bit, 0 or 1
\(e\)
the exponent field read as an unsigned integer
\(b\)
the exponent bias, \(2^{E-1} - 1\)
\(m\)
the mantissa field read as an unsigned integer, from 0 to \(2^M - 1\)
\(M\)
the number of mantissa bits; \(1 + m/2^M\) is a number in \([1, 2)\)

When the exponent field is all zeros, the number is subnormal (also called denormal). The invisible 1 becomes a 0, and the power is fixed at the smallest normal one:

\[ x = (-1)^{s} \cdot 2^{\,1 - b} \cdot \frac{m}{2^{M}} \qquad (e = 0) \tag{37.2} \]

Subnormals fill the gap between the smallest normal number and zero with evenly spaced values. They let numbers fade to zero gradually instead of jumping, but they carry fewer significant bits. When the exponent field is all ones, IEEE formats store \(\pm\infty\) (mantissa 0) or NaN, “not a number” (mantissa not 0). NaN is what you get from \(0/0\) or \(\infty - \infty\), and it spreads: any operation with a NaN gives NaN.

Worked example: π in BF16

BF16 has \(E = 8\), \(M = 7\), bias 127. The number 3.14159 lies in \([2, 4)\), so the power is \(2^1\) and the exponent field is \(1 + 127 = 128 = \texttt{10000000}_2\).

The mantissa must satisfy \(1 + m/128 \approx 3.14159/2 = 1.5708\), so \(m \approx 73.06\), which rounds to \(m = 73 = \texttt{1001001}_2\).

The bits are 0 10000000 1001001, and the stored value is \(2 \times (1 + 73/128) = 3.140625\). The relative error is \(-0.031\%\). In FP16 the same number also becomes 3.140625, but in FP32 it is 3.1415901184, with an error of about \(4 \times 10^{-8}\).

Rounding a real number into a format is a short algorithm. You find the power of two, scale the rest into an integer range, round, check the range, and write the bits. Figure 1 runs it step by step for any number and any of the six formats. The highlighted line is the step being run, and the panel next to it shows the formula with the real values put in.

signexponent (range)mantissa (precision)flushed to zero or overflowsubnormal
  1. input: real \(x\); format with \(E\) exponent bits, \(M\) mantissa bits, bias \(b = 2^{E-1}-1\)
  2. \(s \leftarrow 1\) if \(x \lt 0\) else \(0\)
  3. \(p \leftarrow \lfloor \log_2 |x| \rfloor\); if \(p \lt 1-b\): \(p \leftarrow 1-b\) // subnormal
  4. \(t \leftarrow (|x|/2^p - 1)\cdot 2^M\) // subnormal: \(t \leftarrow |x|/2^p \cdot 2^M\)
  5. \(m \leftarrow\) round \(t\) to the nearest integer, ties to even; if \(m = 2^M\): \(m \leftarrow 0\), \(p \leftarrow p+1\)
  6. if the value is above \(x_{\max}\): overflow; if subnormal and \(m = 0\): underflow to 0
  7. \(e \leftarrow p + b\) (subnormal: \(e \leftarrow 0\)); write \(s\), \(e\), \(m\) in binary
  8. decode with (37.1) or (37.2); relative error \((\hat{x}-x)/x\)
Figure 1. Converting one number into six formats, one line of the algorithm at a time. Top: the bits, filled in as they are decided. Middle: the range of the format on a log scale (zero, subnormal, normal, overflow) with \(|x|\) marked. Bottom: the rounding step, with \(t\) between its two neighbouring integers. After the last step, click any bit to flip it and see the number it encodes.
How to read Figure 1

Press Play with 3.14159 in BF16. Line 3 finds \(p = 1\), line 4 gives \(t = 73.06\), and line 5 rounds it to \(m = 73\), exactly the worked example above. Now keep 0.1 and switch formats. FP32 stores 0.10000000149 (error about \(10^{-8}\)), FP16 stores 0.0999755859 (error 0.024%), BF16 0.10009765625 (error 0.098%), E4M3 0.1015625 (1.6%) and E5M2 0.09375 (6.3%). The number 1.00390625 is \(1 + 2^{-8}\): in BF16, \(t = 0.5\) is a tie, so line 5 picks the even neighbour \(m = 0\) and the stored value is 1. With 65519 in FP16, \(t = 1023.47\) rounds down to 1023, giving 65,504, the largest FP16 number. With 70000, FP16 gets \(p = 16\), above its largest power \(2^{15}\), so line 6 reports overflow (\(\infty\)), while BF16 keeps it as 70,144. With 1e-8, FP16 and both FP8 formats are subnormal and \(t\) rounds to 0: underflow. In E4M3, 500 is out of range: there is no infinity in that format, so a raw cast gives NaN, and real FP8 code saturates (clamps) to 448 instead. The “gap” in the readout is the distance to the next representable number; it grows with the size of the number.

Range and precision of the formats used in deep learning

From formula (37.1) we can read off the range and the precision of any format from its bit counts alone. The largest normal number uses the largest allowed exponent and an all-ones mantissa; the smallest normal number uses \(e = 1\) and \(m = 0\); the smallest subnormal uses \(e = 0\) and \(m = 1\):

\[ x_{\max} = \left(2 - 2^{-M}\right) 2^{\,e_{\max} - b}, \qquad x_{\min}^{\text{normal}} = 2^{\,1-b}, \qquad x_{\min}^{\text{sub}} = 2^{\,1-b-M} \tag{37.3} \]

For IEEE-style formats \(e_{\max} = 2^E - 2\), because the all-ones exponent is kept for infinity and NaN. For FP16: \(b = 15\), \(e_{\max} = 30\), so \(x_{\max} = (2 - 2^{-10}) \cdot 2^{15} = 65{,}504\), \(x_{\min}^{\text{normal}} = 2^{-14} \approx 6.1 \times 10^{-5}\), \(x_{\min}^{\text{sub}} = 2^{-24} \approx 6.0 \times 10^{-8}\).

Precision is about relative error. Between two powers of two, the representable numbers are evenly spaced. The gap, called one ulp (unit in the last place), and the worst rounding error are:

\[ \text{ulp}(x) = 2^{\lfloor \log_2 |x| \rfloor - M}, \qquad \frac{|\operatorname{fl}(x) - x|}{|x|} \le u = 2^{-(M+1)} \tag{37.4} \]
Read the symbols
\(\operatorname{fl}(x)\)
\(x\) rounded to the nearest number the format can store
\(\text{ulp}(x)\)
the distance from \(x\) to the next representable number of the same size
\(u\)
the unit roundoff: the largest relative rounding error for normal numbers, half of \(2^{-M}\)
\(2^{-M}\)
“machine epsilon”: the gap between 1 and the next number above 1 (torch.finfo(dtype).eps)

So the absolute gap doubles at each power of two, but the relative error stays below \(u\). That is why BF16 can store 0.001 and 1,000,000 equally well in relative terms, but cannot store \(1 + 0.001\).

FormatSign / exp. / mantissaBytesLargestSmallest normalSmallest subnormalGap above 1 (\(2^{-M}\))About this many decimal digits
FP321 / 8 / 234\(3.40 \times 10^{38}\)\(1.18 \times 10^{-38}\)\(1.40 \times 10^{-45}\)\(1.19 \times 10^{-7}\)7.2
TF32 (math mode)1 / 8 / 10(4 in memory)\(\approx 3.40 \times 10^{38}\)\(1.18 \times 10^{-38}\)(not used)\(9.77 \times 10^{-4}\)3.3
FP161 / 5 / 10265,504\(6.10 \times 10^{-5}\)\(5.96 \times 10^{-8}\)\(9.77 \times 10^{-4}\)3.3
BF161 / 8 / 72\(3.39 \times 10^{38}\)\(1.18 \times 10^{-38}\)\(9.18 \times 10^{-41}\)\(7.81 \times 10^{-3}\)2.4
FP8 E4M31 / 4 / 31448\(2^{-6} = 0.0156\)\(2^{-9} \approx 0.00195\)0.1251.2
FP8 E5M21 / 5 / 2157,344\(2^{-14} \approx 6.10 \times 10^{-5}\)\(2^{-16} \approx 1.53 \times 10^{-5}\)0.250.9

“Decimal digits” is \(\log_{10} 2^{M+1}\). All values were computed from formula (37.3) and checked against torch.finfo. [Certain] Four short stories explain the table:

  • FP16 (IEEE half precision) spends 10 bits on the mantissa and only 5 on the exponent. It is fairly precise, but its whole range covers only 40 powers of two, from \(2^{-24}\) to just under \(2^{16}\).
  • BF16 (“brain float 16”, from Google Brain, first used on TPUs) is simply the top 16 bits of an FP32 number: same 8 exponent bits, so the same range as FP32, but only 7 mantissa bits. Converting FP32 to BF16 is just rounding away the lower 16 bits.
  • TF32 (TensorFloat-32, NVIDIA Ampere and later) is not a storage format. Tensors stay FP32 in memory; the tensor cores round the matmul inputs to 10 mantissa bits, multiply, and accumulate in FP32. You get FP32 range with FP16 precision on the inputs, without changing any code.
  • FP8 comes in two flavours. E4M3 has more precision and a range up to 448; E5M2 has more range, up to 57,344. E4M3 gives up infinity and keeps a single NaN pattern per sign, which buys one extra power of two of range (without that change its maximum would be 240). [Certain]

Speed: why anyone accepts fewer bits

Tensor cores process low-precision inputs much faster, and fewer bytes means less memory traffic (session 42 explains why bandwidth often matters even more than FLOPs). Dense peak numbers for one H100 SXM (half of the datasheet values, which are quoted “with sparsity”) [Certain]:

MathWhere it runsDense peak TFLOP/sBytes per input element
FP32CUDA cores674
TF32tensor coresabout 4954 (rounded inside)
BF16 or FP16, FP32 accumulatetensor coresabout 9892
FP8 (E4M3 or E5M2)tensor coresabout 1,9791

PyTorch’s documentation gives a concrete TF32 example on an A100: a large FP32 matmul took 0.11 s in full FP32 and 0.016 s with TF32 (about 7 times faster), while the relative error grew from \(3.9 \times 10^{-5}\) to \(2.2 \times 10^{-3}\). [Certain] Since PyTorch 1.12, TF32 is off by default for matmuls (but on for cuDNN convolutions). You switch it on with torch.set_float32_matmul_precision("high").

Three ways low precision breaks

1. Overflow

A value larger than \(x_{\max}\) becomes \(\infty\) (or NaN in E4M3). In FP16 this happens at 65,520 already (65,519 still rounds down to 65,504). Activations after a big matmul, attention logits before softmax, or the sum inside a variance can easily pass that. Once one \(\infty\) appears, \(\infty - \infty\) or \(0 \cdot \infty\) soon gives NaN, and the NaN spreads to the loss and to every weight in the next update.

2. Underflow

A value smaller than half of \(x_{\min}^{\text{sub}}\) rounds to exactly zero. For FP16 that is anything below \(2^{-25} \approx 3 \times 10^{-8}\). Gradients are often this small. In the mixed-precision paper, about 5% of the weight-gradient values of a speech model had exponents below \(-24\), so they would be zero in FP16; and in an SSD object detector, many activation gradients were below the FP16 range and the model diverged unless the gradients were scaled. [Certain]

3. Swamping: small numbers that vanish when added

This one is the most important for training, because it hurts BF16 too. To add \(a + b\), the hardware first aligns the binary points, which shifts the smaller number to the right. Bits that fall off the end of the mantissa are lost. If \(b\) is smaller than half a gap of \(a\), the sum is just \(a\):

\[ \operatorname{fl}(a + b) = a \quad \text{whenever} \quad |b| \lt \tfrac12\, \text{ulp}(a) \approx 2^{-(M+1)} |a| \tag{37.5} \]
Worked example: a weight update that disappears

A weight \(w = 1.0\) receives an update \(\Delta = 0.001\) (learning rate times gradient).

In FP32, \(1 + 0.001 = 1.0010000467\). Fine.

In FP16, \(\text{ulp}(1) = 2^{-10} \approx 0.000977\). The update is larger than half of that, so the result rounds to \(1 + 2^{-10} = 1.0009765625\): kept, but with a 2.3% error on the update.

In BF16, \(\text{ulp}(1) = 2^{-7} = 0.0078125\), and half of it is 0.0039. The update is smaller, so \(1 + 0.001 = 1\). The update is lost completely. Even \(\Delta = 0.003\) is lost; \(\Delta = 0.005\) becomes 0.0078.

The mixed-precision paper states the FP16 version of this rule: once a weight is at least 2,048 times (\(2^{11}\)) larger than its update, the update can become zero. [Certain] For BF16 the ratio is only \(2^8 = 256\). With learning rates around \(10^{-4}\) and Adam steps of about one learning rate, nearly every update is far below \(1/256\) of the weight. So a pure-BF16 optimizer step would silently stop learning. This is the main reason for FP32 master weights.

The same effect hits long sums. A dot product of length \(n\), or a reduction over a tensor, adds \(n\) numbers one after another. The classic bound for this recursive summation (from numerical analysis) grows with \(n\):

\[ \left| \hat{s}_n - s_n \right| \lesssim (n - 1)\, u \sum_{i=1}^{n} |x_i| \tag{37.6} \]

For a dot product of length 4,096 with a BF16 accumulator, \((n-1)u = 4095 \times 2^{-8} \approx 16\): the bound is useless, and in practice the sum can stall completely. With an FP32 accumulator, \((n-1)u = 4095 \times 2^{-24} \approx 2.4 \times 10^{-4}\). That is why tensor cores multiply BF16 or FP16 inputs but accumulate in FP32, and why large reductions (softmax sums, norm statistics, the loss) are done in FP32. [Certain] Figure 2 shows the stall.

Each term
exactFP32 accumulatorBF16 inputs, FP32 accumulatorFP16 accumulatorBF16 accumulator
  1. \(\hat{x} \leftarrow \operatorname{fl}(x)\); \(s \leftarrow 0\) // the term and the accumulator, both in the format
  2. for \(i = 1, \dots, n\):
  3. \(s \leftarrow \operatorname{fl}(s + \hat{x})\) // lost when \(\hat{x} \lt \tfrac12 \text{ulp}(s)\), formula (37.5)
Figure 2. Adding the same small number \(n\) times, one addition at a time, with the running sum stored in different formats. Every addition is rounded to the accumulator’s format with round-to-nearest-even. Play draws the curves up to addition \(i\); the frames include the exact additions where the 16-bit accumulators stop. The panel shows formula (37.5) for the last addition of each 16-bit accumulator.
How to read Figure 2

With terms of 0.01, the BF16 accumulator stops growing at 4.0 after 350 additions: at 4, the gap between BF16 numbers is \(2^{2-7} = 1/32\), and 0.01 is less than half of that, so by formula (37.5) every further addition is lost. The FP16 accumulator stops at 32.0 after 2,798 additions for the same reason (gap \(2^{5-10} = 1/32\)). The green line uses BF16 inputs but an FP32 accumulator: its only error is that 0.01 itself was rounded to 0.010009765625 once, so the sum ends at 100.098 instead of 100. That is the tensor-core recipe: low-precision inputs, high-precision sum. Press Reset and step to addition 350 and 351: just below 4 each addition adds a full gap of \(1/64 = 0.015625\) (the term is more than half a gap, so it rounds up), and at 4 the half gap becomes \(0.015625\), larger than the term, so the sum stays at 4.

Why FP16 needs loss scaling

FP16 has enough precision for most of training, but its range is narrow and sits in the wrong place for gradients. Gradients of a large model are mostly tiny numbers, with exponents far below zero, while FP16’s top range (up to 65,504) is almost never used. The idea of loss scaling is to move the gradients up into the range FP16 can store, by multiplying the loss by a constant \(S\) before the backward pass. Because backpropagation is linear in the upstream gradient, every gradient in the network is multiplied by the same \(S\):

\[ \nabla_\theta \big( S \cdot \mathcal{L} \big) = S \cdot \nabla_\theta \mathcal{L}, \qquad \mathbf{g} = \frac{1}{S}\, \nabla_\theta \big( S \cdot \mathcal{L} \big) \ \text{ (computed in FP32)} \tag{37.7} \]
Read the symbols
\(S\)
the loss scale, usually a power of two, so multiplying and dividing by it is exact
\(\mathcal{L}\)
the loss, computed in FP32
\(\mathbf{g}\)
the true gradient, recovered by unscaling before clipping and the optimizer step

A power of two only changes the exponent, so scaling adds no rounding error. The only risk is that a large \(S\) pushes the largest gradients past 65,504. Figure 3 lets you see both sides.

Worked example: rescuing a gradient of \(10^{-8}\)

Without scaling, \(10^{-8}\) is below \(2^{-25} \approx 2.98 \times 10^{-8}\), so FP16 stores 0.

With \(S = 1024 = 2^{10}\), the value is \(1.024 \times 10^{-5}\). It is still subnormal in FP16 (below \(6.1 \times 10^{-5}\)) but representable: it is stored as \(1.0252 \times 10^{-5}\). After unscaling in FP32, we get \(1.00117 \times 10^{-8}\), an error of 0.12%.

With \(S = 65536 = 2^{16}\), the value is \(6.5536 \times 10^{-4}\), a normal FP16 number, stored as \(6.5517 \times 10^{-4}\): error 0.028%.

Gradient format
flushed to zero, or overflowsubnormal (fewer bits)normal
Figure 3. A made-up but typical gradient distribution: \(\log_2 |g|\) is normal with mean \(-20\) and standard deviation 4 (so the typical gradient is about \(10^{-6}\)). The curve shows the scaled gradients \(S g\). The fractions are computed exactly from the normal distribution.
How to read Figure 3

At \(k = 0\) (no scaling), FP16 flushes 10.6% of the gradients to zero and stores 82.8% as subnormals with reduced precision. Move \(k\) to 12: almost nothing is lost, and the expected number of infinities among \(10^9\) gradient values is about 1. At \(k = 16\) it is about 287, so nearly every step overflows. A real step has billions of gradient values, so even a tiny overflow probability matters: one \(\infty\) is enough to ruin the step. Switch to BF16: the whole curve fits inside the normal range for every \(k\), so no scaling is needed. Switch to E5M2: its range is like FP16’s but its subnormals end at \(2^{-16}\), so at \(k = 0\) it flushes 77% of these gradients; FP8 needs scaling even more.

Dynamic loss scaling

The best \(S\) changes during training, as gradients grow and shrink. Dynamic loss scaling finds it automatically by trial and error: start high, cut the scale in half whenever an overflow appears, and try doubling it after a long run of good steps:

\[ S \leftarrow \begin{cases} S / 2 & \text{if any gradient is } \infty \text{ or NaN (and skip this update)} \\ 2S & \text{after } N \text{ steps in a row without overflow} \\ S & \text{otherwise} \end{cases} \tag{37.8} \]

PyTorch’s GradScaler uses exactly this rule with defaults \(S_0 = 65536\), growth factor 2, backoff factor 0.5 and \(N = 2000\). [Certain] A skipped step costs one batch of compute, which is cheap; an applied \(\infty\) would destroy the weights.

Algorithm 1: one FP16 training step with dynamic loss scaling
  1. input: FP32 master weights \(\theta\), loss scale \(S\), good-step counter \(c\), growth interval \(N\)
  2. \(\theta_{16} \leftarrow \text{fp16}(\theta)\) // or let autocast cast on the fly
  3. \(\mathcal{L} \leftarrow\) forward pass with \(\theta_{16}\); loss computed in FP32
  4. backward pass on \(S \cdot \mathcal{L}\) // all gradients are S times larger, formula (37.7)
  5. if any gradient is \(\infty\) or NaN then \(S \leftarrow S/2\), \(c \leftarrow 0\), skip to the next batch
  6. \(\mathbf{g} \leftarrow \text{fp32}(\text{grad}) / S\) // unscale first ...
  7. clip \(\mathbf{g}\) to global norm \(1.0\) // ... then clip, so the threshold means what it says
  8. optimizer step on the FP32 \(\theta\) with FP32 states
  9. \(c \leftarrow c + 1\); if \(c = N\) then \(S \leftarrow 2S\), \(c \leftarrow 0\)

Figure 4 runs Algorithm 1 for 12 training steps on 8 concrete gradient values, from \(3 \times 10^{-10}\) up to \(1.2\). Each step multiplies all 8 by a fixed factor (between 0.8 and 3), as if the gradients drift a little. The gradients are real FP32 numbers; the scaling, the FP16 rounding, the overflow check, the unscaling and the update of \(S\) are all computed in your browser. To keep the movie short, the growth interval \(N\) is 3 instead of PyTorch’s 2,000.

Growth interval \(N\)
flushed to zero, or overflowsubnormalnormal FP16hollow: the true gradient \(g\)
  1. input: FP32 master weights \(\theta\), loss scale \(S\), counter \(c\), growth interval \(N\)
  2. \(\mathcal{L} \leftarrow\) forward pass (FP16 matmuls, loss in FP32)
  3. backward on \(S \cdot \mathcal{L}\): the gradients are \(\text{fp16}(S \cdot g)\) // (37.7)
  4. if any gradient is \(\infty\) or NaN: \(S \leftarrow S/2\), \(c \leftarrow 0\), skip the step // (37.8)
  5. \(\hat{g} \leftarrow \text{fp32}(\text{grad}) / S\) // unscale
  6. clip \(\hat{g}\); optimizer step on the FP32 \(\theta\)
  7. \(c \leftarrow c+1\); if \(c = N\): \(S \leftarrow 2S\), \(c \leftarrow 0\) // (37.8)
Figure 4. A dynamic loss scaler (Algorithm 1) running on 8 gradients for 12 steps. Top: each row is one gradient on a \(\log_2\) axis; the hollow dot is the true \(|g|\), the filled dot is \(|S g|\) after the FP16 cast, colored by the zone it lands in. Bottom: the loss scale at each step; a red cross is a skipped step. The table gives the numbers, and the last column shows what FP16 would store with no scaling at all.
How to read Figure 4

With the defaults (\(S_0 = 2^{16} = 65{,}536\), \(N = 3\)), step 1 overflows: \(65{,}536 \times 1.2 = 78{,}643\) is above 65,504, so the gradient becomes \(\infty\), the step is skipped and \(S\) drops to 32,768. Steps 2 to 4 are clean, so after step 4 the scaler doubles \(S\) back to 65,536, and step 5 overflows again. This saw-tooth is normal: the scaler keeps testing whether a larger scale is safe. At step 8 the gradients jump by a factor of 3, \(32{,}768 \times 3.6 = 117{,}965\) overflows, and \(S\) falls to 16,384. In every applied step all 8 gradients survive, with relative errors below 0.6%, while without scaling (last column) the three smallest would be stored as 0 in 11 of the 12 steps. Now set \(k = 0\): nothing ever overflows, but 1 to 3 gradients are lost in every step (the smallest one in all 12), and the scale grows so slowly that it is still only 8 at step 12. With \(k = 24\) the first 9 steps are all skipped while the scale halves its way down to 32,768.

Why BF16 usually does not need loss scaling

BF16 has the same 8 exponent bits as FP32. Its smallest normal number is \(1.18 \times 10^{-38}\) and its largest is \(3.39 \times 10^{38}\): 261 powers of two including subnormals, against 40 for FP16. A gradient of \(10^{-8}\), or even \(10^{-30}\), is stored without trouble, and activations of \(10^{5}\) do not overflow. There is nothing to scale into range, so there is no loss scaler, no skipped steps and one less hyper-parameter. A 2019 study by Intel and Facebook found that BF16 training matched FP32 “in the same number of iterations and with no changes to hyper-parameters” across many models. [Certain] Since the A100 (2020), BF16 has been the default for LLM training. [Likely]

BF16 is not free, though. It has only 8 significant bits, so the third failure, swamping, is worse than in FP16. BF16 needs the other two parts of the mixed-precision recipe even more: FP32 accumulation inside matmuls and reductions, and FP32 master weights for the update. A model that is trained in BF16 “end to end”, including the weights and the optimizer states, is a different and riskier setup; it usually needs extra tricks such as stochastic rounding or Kahan-compensated updates. [Likely]

Mixed precision: the full recipe

The recipe from the mixed-precision paper (Micikevicius et al., 2017) has three parts, and all three are still used today. [Certain]

  1. Low-precision math for the heavy work. Matmuls and convolutions take FP16 or BF16 inputs (activations and a low-precision copy of the weights). This is where nearly all FLOPs are, so this is where the speed comes from. Activations saved for the backward pass are also stored in 16 bits, which roughly halves activation memory.
  2. FP32 accumulation. Inside each matmul, the products are summed in FP32 and rounded to 16 bits only when the result is written to memory. Large reductions (softmax sums, LayerNorm and RMSNorm statistics, the loss) are computed in FP32. Elementwise ops are memory-bound, so their arithmetic precision barely affects speed.
  3. FP32 master weights. The optimizer keeps an FP32 copy of every weight, plus FP32 optimizer states (Adam’s \(\mathbf{m}\) and \(\mathbf{v}\)). Gradients are converted to FP32 and the update \(\theta \leftarrow \theta - \eta\,\Delta\) is done in FP32, so tiny updates are not swamped (formula 37.5). In the paper’s speech model, updating FP16 weights directly instead of an FP32 master copy caused an 80% relative accuracy loss. [Certain]

With FP16 you add the fourth part, loss scaling. With BF16 you do not.

FP3216-bit (FP16 or BF16)FP8
Figure 5. One mixed-precision training step. Each box is colored by the number format of the data it holds. The optimizer side (left column) is FP32 in every mode; only the compute side (middle and right) changes.
How to read Figure 5

Follow the arrows: the FP32 master weights are cast down, the forward and backward passes run in low precision with FP32 accumulation inside the matmuls, the gradients come back, and the update is applied in FP32. In FP16 mode two extra boxes appear (multiply the loss by \(S\), then unscale and check for \(\infty\)). In FP8 mode the matmul inputs become FP8 with a scale per tensor or per block; everything outside the matmuls stays in BF16 or FP32.

In PyTorch, autocast applies this recipe per operation. Inside torch.autocast("cuda", dtype=torch.bfloat16), ops like matmul, linear and mm run in the low-precision type, while ops like softmax, log_softmax, layer_norm, exp, sum, pow and cross_entropy run in FP32. [Certain] The parameters themselves stay FP32; autocast casts copies on the fly. This is the “FP32 master weights” part for free.

Algorithm 2: one BF16 mixed-precision training step
  1. input: FP32 weights \(\theta\), FP32 Adam states \(\mathbf{m}, \mathbf{v}\), a batch
  2. for each linear layer: \(Y_{16} \leftarrow \text{bf16}\big(\text{bf16}(X)\,\text{bf16}(W)\big)\) // products summed in FP32 inside the tensor core
  3. norms, softmax, loss: read BF16, compute in FP32 // reductions in FP32
  4. backward pass: matmuls in BF16 with FP32 accumulation; weight gradients arrive in FP32 (or BF16)
  5. gradient accumulation over micro-batches and the data-parallel all-reduce: in FP32 where affordable // Llama 3 did both in FP32
  6. clip global gradient norm; AdamW update of the FP32 \(\theta\) with FP32 \(\mathbf{m}, \mathbf{v}\)
  7. (no loss scaling, no overflow check)

Llama 3 is a good real example: its paper reports FP32 gradient accumulation across micro-batches and FP32 reduce-scatter of gradients across data-parallel workers in FSDP, “to ensure training convergence”. [Certain] Memory cost of the recipe for Adam: 2 bytes (16-bit weights) + 2 (gradients) + 4 (FP32 master) + 4 + 4 (\(\mathbf{m}, \mathbf{v}\)) = 16 bytes per parameter, before activations (session 38). Mixed precision makes training faster, not smaller in weight memory; the saving is in activations and bandwidth.

FP8 training

FP8 doubles the tensor-core throughput again and halves the bytes of every matmul input. But its range and precision are tiny: E4M3 covers about 18 powers of two and has 3 mantissa bits. One fixed range cannot fit every tensor in a network. The solution is to give each tensor (or each block of a tensor) its own scaling factor, chosen so that its largest value lands near the top of the FP8 range. This is loss scaling done per tensor, in both directions.

Two formats, two jobs

The FP8 paper (NVIDIA, Arm and Intel, 2022) recommends E4M3 for weights and activations (forward pass: precision matters more) and E5M2 for gradients (backward pass: gradients have a wider spread, so range matters more). [Certain] NVIDIA’s Transformer Engine calls this the HYBRID recipe. The paper trained models up to 175B parameters this way, with FP8 inputs to all matmuls, and matched the 16-bit baselines without changing any hyper-parameter. [Certain]

Scaling factors

Before a tensor \(X\) is cast to FP8, it is multiplied by a scale \(s\). The matmul then runs on FP8 inputs with FP32 accumulation, and the result is divided by the product of the two scales:

\[ s_X = \frac{q_{\max}}{\operatorname{amax}(X)}, \qquad Q_X = \operatorname{fp8}\big(\operatorname{clip}(s_X X,\, -q_{\max},\, q_{\max})\big), \qquad \hat{X} = \frac{Q_X}{s_X} \tag{37.9} \]
\[ Y = X W \approx \frac{1}{s_X\, s_W}\, \big(Q_X\, Q_W\big)_{\text{FP32 accumulate}} \tag{37.10} \]
Read the symbols
\(\operatorname{amax}(X)\)
the largest absolute value in the tensor (or in one block of it)
\(q_{\max}\)
the largest FP8 value: 448 for E4M3, 57,344 for E5M2
\(\operatorname{clip}\)
saturation: values beyond \(q_{\max}\) are clamped instead of becoming NaN or \(\infty\)
\(Q_X\)
the FP8 tensor actually stored and fed to the tensor cores
\(s_X s_W\)
one scalar per output (or per block pair) that undoes both scalings after the FP32 sum

The unscaling costs one multiply per output element, spread over thousands of multiply-adds, so it is almost free. [Certain] The FP8 paper also notes that the FP16 trick of “skip the step on overflow” is a bad fit here: with so little range, overflows are common, so FP8 saturates instead. [Certain]

Worked example: why the scale matters

A tensor has \(\operatorname{amax} = 3.2\) and contains the value 0.001.

Without scaling, 0.001 is below E4M3’s smallest normal number (\(2^{-6} = 0.0156\)) and lands among the subnormals, whose gap is \(2^{-9} = 0.00195\). It rounds to 0.00195: a 95% error.

With scaling, \(s = 448 / 3.2 = 140\). The scaled value is 0.14, which E4M3 stores as 0.140625. Dividing by 140 gives 0.0010045: an error of 0.45%.

When to measure amax: delayed or current scaling

Computing \(\operatorname{amax}(X)\) needs a full pass over the tensor before the cast. Delayed scaling avoids that wait: it uses the largest amax seen over a window of recent steps (Transformer Engine keeps an amax history; its primer example uses 16 steps) and records the current amax for the future. [Certain] It is fast but can be wrong when a value suddenly jumps. Current (just-in-time) scaling measures amax of the tensor right now: more robust, slightly more work. Fine-grained recipes use current scaling. [Likely]

Granularity: one scale per tensor, or per block

One scale per tensor fails when the tensor has outliers. LLM activations often have a few channels with values hundreds of times larger than the rest. One outlier sets \(\operatorname{amax}\), all other values are scaled down into E4M3’s subnormal range, and they lose most of their bits. The fix is to use one scale per small block, so an outlier only hurts its own block. Figure 6 shows the effect.

error under 3%3% to 10%over 10%flushed to zero
Figure 6. 512 values drawn from a standard normal distribution (fixed seed), plus one outlier at position 37, quantized to FP8 E4M3 with formula (37.9). Each thin column is one value, colored by its relative error. Row 1 uses one scale for the whole tensor; rows 2 and 3 use one scale per block of 128 and 32 values (thin lines show the blocks).
How to read Figure 6

At \(a = 0\) (no outlier) all three rows have a mean relative error of about 2%, which is what 3 mantissa bits give. Raise the outlier: with one scale per tensor, at \(a = 4\) (outlier 10,000) the mean error on the ordinary values rises to 5.6%, 43 values have more than 10% error and 5 become zero. With blocks of 128, only the first block suffers (3.3%, 16 values, 2 zeros). With blocks of 32 the damage stays inside one small block (2.4%, 5 values, no zeros). The outlier itself is always stored well, because it sets the scale.

Real recipes

DeepSeek-V3 (671B parameters) was trained with FP8 matmuls and is the best-documented large example. [Certain] Its choices, from the technical report:

  • E4M3 for all tensors, also gradients, instead of the hybrid recipe, made possible by fine-grained scaling.
  • Tile and block scales: activations are scaled per \(1 \times 128\) tile (per token, per 128 channels), weights per \(128 \times 128\) block.
  • Better accumulation: they found that FP8 matmul accumulation on H800 tensor cores keeps only about 14 bits, with a maximum relative error of nearly 2% for an inner dimension of 4,096. So every 128 elements the partial sums are moved to CUDA cores and added in full FP32.
  • What stays in higher precision: the embedding, the output head, the MoE gating, the normalization operators and the attention operators stay in BF16 or FP32. Master weights and gradients are kept in FP32; Adam’s moments are stored in BF16.
  • Result: the training loss stayed within 0.25% (relative) of a BF16 baseline.

NVIDIA Blackwell GPUs add MXFP8: one scale per block of 32 consecutive values, stored as a power of two in 8 bits (E8M0), so E4M3 can be used everywhere. [Certain] This is the same idea as Figure 6, row 3, built into the hardware.

Algorithm 3: an FP8 linear layer (forward and backward) with per-tensor current scaling
  1. input: activation \(X\) (BF16), FP32 master weight \(W\)
  2. \(s_X \leftarrow 448 / \operatorname{amax}(X)\), \(\;Q_X \leftarrow \text{e4m3}(\operatorname{clip}(s_X X))\) // formula (37.9)
  3. \(s_W \leftarrow 448 / \operatorname{amax}(W)\), \(\;Q_W \leftarrow \text{e4m3}(\operatorname{clip}(s_W W))\)
  4. \(Y \leftarrow \text{bf16}\big((Q_X Q_W)_{\text{FP32}} / (s_X s_W)\big)\) // formula (37.10); save \(Q_X, s_X\) for backward
  5. backward: \(G = \partial \mathcal{L}/\partial Y\); \(s_G \leftarrow 57344 / \operatorname{amax}(G)\), \(\;Q_G \leftarrow \text{e5m2}(\operatorname{clip}(s_G G))\) // E5M2 for gradients
  6. \(\partial \mathcal{L}/\partial X \leftarrow (Q_G Q_W^\top)_{\text{FP32}} / (s_G s_W)\), \(\;\partial \mathcal{L}/\partial W \leftarrow (Q_X^\top Q_G)_{\text{FP32}} / (s_X s_G)\)
  7. the optimizer updates the FP32 master \(W\) as in Algorithm 2

The code

This runs on a CPU and is deterministic. It prints the format limits, shows swamping and underflow, runs a small dynamic loss scaler (formula 37.8), shows that autocast keeps FP32 parameters, and emulates an FP8 matmul with per-tensor scales (formulas 37.9 and 37.10). The printed values are from a run with PyTorch 2.8.

import torch
torch.manual_seed(0)

# 1) the formats: range and precision
for dt in [torch.float32, torch.float16, torch.bfloat16, torch.float8_e4m3fn, torch.float8_e5m2]:
    fi = torch.finfo(dt)
    print(f"{str(dt):22s} max={fi.max:.4g}  min_normal={fi.tiny:.3g}  eps={fi.eps:.3g}")
# float16: max=6.55e+04 min_normal=6.1e-05 eps=0.000977;  bfloat16: max=3.39e+38 eps=0.00781

# 2) swamping, formula (37.5): a small update added to a big weight
w = torch.tensor(1.0)
for dt in [torch.float32, torch.float16, torch.bfloat16]:
    print(dt, (w.to(dt) + torch.tensor(1e-3).to(dt)).item())   # 1.0010000467, 1.0009765625, 1.0

# 3) underflow and loss scaling in FP16, formula (37.7)
g = torch.tensor(1e-8)
print(g.half().item(), (g * 1024).half().item(), ((g * 1024).half().float() / 1024).item())
# 0.0   1.0252e-05   1.00117e-08

# 4) a tiny dynamic loss scaler, formula (37.8); the model stays in FP32
model = torch.nn.Linear(8, 1)
opt = torch.optim.SGD(model.parameters(), lr=0.1)
scale, good, interval = 2.0 ** 16, 0, 2000                    # GradScaler's defaults
x, y = torch.randn(32, 8), torch.randn(32, 1)
for step in range(3):
    loss = torch.nn.functional.mse_loss(model(x), y)
    opt.zero_grad(set_to_none=True)
    (loss * scale).backward()                                 # gradients are scale times too big
    grads = [p.grad for p in model.parameters()]
    finite = all(torch.isfinite(gr).all() for gr in grads)
    if finite:
        for gr in grads: gr.div_(scale)                       # unscale BEFORE clipping
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step(); good += 1
        if good == interval: scale, good = scale * 2, 0
    else:
        scale, good = scale / 2, 0                            # skip this step
    print(step, round(loss.item(), 4), scale, finite)         # 0 1.2498 65536.0 True, ...

# 5) BF16 autocast: matmuls in BF16, parameters (master weights) stay FP32
with torch.autocast(device_type="cpu", dtype=torch.bfloat16):
    out = model(x)
print(out.dtype, model.weight.dtype)                          # torch.bfloat16 torch.float32

# 6) FP8 E4M3 with a per-tensor scale, formulas (37.9) and (37.10)
def to_fp8(t, fmax=448.0):
    s = fmax / t.abs().max().clamp(min=1e-12)                 # map amax to the top of the range
    q = (t * s).clamp(-fmax, fmax).to(torch.float8_e4m3fn)    # clamp = saturate, then cast
    return q, s
a = torch.randn(4, 256); b = torch.randn(256, 3)
qa, sa = to_fp8(a); qb, sb = to_fp8(b)
y8 = (qa.float() @ qb.float()) / (sa * sb)                   # FP32 accumulate, then unscale
print(((y8 - a @ b).norm() / (a @ b).norm()).item())          # 0.0473: about 5% error

On a CUDA GPU, the production version of steps 4 and 5 is three lines: scaler = torch.amp.GradScaler("cuda"), the forward pass under torch.autocast("cuda", dtype=torch.float16), then scaler.scale(loss).backward(), scaler.unscale_(opt), clipping, scaler.step(opt) and scaler.update(). With BF16 you drop the scaler. FP8 training is usually done with a library such as Transformer Engine or torchao rather than by hand. [Likely]

Where this is used

SettingTypical number formatsWhy
LLM pretraining on A100, H100, TPUBF16 matmuls, FP32 accumulation, FP32 master weights and Adam statesNo loss scaling; FP32 range; the standard since about 2020 [Likely]
Frontier pretraining with FP8 (DeepSeek-V3)E4M3 matmul inputs with \(1\times128\) and \(128\times128\) scales; BF16 or FP32 elsewhereAbout twice the BF16 matmul throughput; fine-grained scales handle outliers
Training on V100 or older GPUsFP16 with dynamic loss scaling, FP32 master weightsNo BF16 tensor cores before Ampere [Certain]
Fine-tuning and LoRABF16 base weights (often frozen), FP32 or BF16 adapters and optimizer states; QLoRA stores the base in 4-bitMemory; the frozen base needs no master copy (session 58)
LLM and VLM inferenceBF16 or FP16 weights; FP8 or INT8 or INT4 quantized weights and KV cacheDecoding is memory-bound, so fewer bytes per weight means faster tokens (session 103)
Video models: action recognition, video retrieval, key-moment detectionBF16 autocast for the video encoder; FP32 for the contrastive softmax and lossLong clips give huge activations; BF16 halves activation memory, and the softmax over a big batch needs FP32 sums
Video fingerprinting and Video-RAG indexesEmbeddings stored in FP16 or 8-bit, similarity accumulated in FP32Billions of vectors; half or quarter the storage with little loss in ranking quality [Likely]
Scientific or fp32-sensitive code on Ampere+FP32 with TF32 matmuls switched on or off on purposeTF32 is about 7 times faster on A100 matmuls but has a relative error near \(10^{-3}\)

Common mistakes

  • Calling .half() or .bfloat16() on the whole model and training it. The optimizer then updates 16-bit weights and tiny updates vanish (formula 37.5). Use autocast, which keeps FP32 parameters, or a framework that keeps FP32 master weights.
  • Clipping before unscaling. With a GradScaler, call scaler.unscale_(opt) before clip_grad_norm_, or you clip gradients that are \(S\) times too large.
  • Using loss scaling with BF16 “to be safe”. It does no harm by itself, but it hides bugs and adds skipped steps; BF16 problems are precision problems, not range problems.
  • Computing softmax, norms or the loss in 16 bits by hand. Custom kernels or manual casts can bypass autocast’s FP32 list. Long sums stall (Figure 2) and exp overflows in FP16 for inputs above about 11.1.
  • Expecting FP32 accuracy from “FP32” code on Ampere+. If TF32 is on, matmuls round their inputs to 10 mantissa bits. Check torch.get_float32_matmul_precision() when you compare results across machines.
  • An epsilon that is zero in the format. \(10^{-8}\) is zero in FP16. Adam’s \(\varepsilon\) and the \(\varepsilon\) in normalization must be added in FP32, or be large enough to exist in the format.
  • One FP8 scale for a tensor with outliers. Ordinary values fall into subnormals and lose their bits (Figure 6). Use per-block scales, or keep that layer in BF16.
  • Comparing runs in different precisions bit for bit. Low-precision runs differ in the last bits and the paths diverge slowly. Compare loss curves and evaluation scores, not exact weights.

Interview questions

What do the sign, exponent and mantissa bits control?basic
The sign bit gives the sign. The exponent bits give the power of two, so they control the range: the largest and smallest magnitudes, \(x_{\max} = (2 - 2^{-M})\,2^{e_{\max} - b}\) and \(x_{\min} = 2^{1-b}\). The mantissa bits give the significant digits, so they control precision: the relative rounding error is at most \(2^{-(M+1)}\). With a fixed number of bits, every extra exponent bit doubles the number of powers of two you can reach but halves the precision.
Compare FP16 and BF16.basic
Both are 16 bits. FP16 is 1/5/10: max 65,504, smallest subnormal \(6 \times 10^{-8}\), gap above 1 of \(2^{-10}\). BF16 is 1/8/7: the same exponent range as FP32 (max \(3.4 \times 10^{38}\)), but gap above 1 of \(2^{-7}\), 8 times coarser. FP16 is more precise but overflows and underflows easily, so training needs loss scaling. BF16 is safe in range, so it needs no loss scaling, and is the default for LLM training on A100, H100 and TPUs. Conversion FP32 → BF16 is a simple truncation-with-rounding of the low 16 bits.
What is TF32 and when is it used?core
TF32 is a tensor-core math mode on Ampere and later GPUs: FP32 tensors are read from memory, their mantissas are rounded to 10 bits inside the tensor core, products are accumulated in FP32. It has FP32 range and FP16-like input precision. It speeds up FP32 matmuls a lot (PyTorch’s docs show about 7 times on an A100) without code changes, at a relative error around \(10^{-3}\). In PyTorch it is off by default for matmuls since 1.12; enable it with torch.set_float32_matmul_precision("high").
Why does FP16 training need loss scaling? How does dynamic loss scaling work?core
Many gradients are smaller than FP16’s smallest subnormal (\(2^{-24}\)) and become zero, while the top of FP16’s range is unused. Multiplying the loss by \(S\) multiplies every gradient by \(S\) (backprop is linear), moving them into range; we divide by \(S\) in FP32 before clipping and the optimizer. Dynamic scaling starts with a large \(S\) (65,536 in PyTorch), halves it and skips the step when any gradient is \(\infty\) or NaN, and doubles it after 2,000 clean steps. Powers of two make scaling exact.
Why keep an FP32 master copy of the weights?core
Because of swamping: adding an update \(\Delta\) to a weight \(w\) gives back \(w\) when \(|\Delta|\) is below half an ulp of \(w\). In FP16 that happens once \(w/\Delta\) is about \(2^{11} = 2048\); in BF16 already at about \(2^8 = 256\). Typical updates are \(10^{-4}\) to \(10^{-5}\) of the weight, so 16-bit weights would stop learning. Keeping FP32 weights (and FP32 optimizer states) makes the update exact enough; the 16-bit copy is only used for compute. It costs 4 extra bytes per parameter.
Why must matmuls and reductions accumulate in FP32?core
A long sum in low precision loses every term that is below half an ulp of the running sum, and the error bound grows like \((n-1)u\sum|x_i|\). For \(n = 4096\) and BF16, \((n-1)u \approx 16\), so the result can be meaningless; summing 0.01 repeatedly in BF16 stalls at 4.0. Tensor cores therefore take 16-bit or 8-bit inputs but keep the partial sums in FP32, and softmax, norm statistics and the loss are computed in FP32. Elementwise ops are memory-bound, so their precision barely matters for speed.
What are E4M3 and E5M2, and which tensors use which?core
Two 8-bit formats. E4M3 (bias 7) has 3 mantissa bits, max 448, smallest subnormal \(2^{-9}\), no infinity and one NaN pattern. E5M2 (bias 15) has 2 mantissa bits, max 57,344, IEEE-style infinities and NaNs; it is like FP16 with fewer mantissa bits. The standard recipe uses E4M3 for weights and activations (forward) and E5M2 for gradients (backward), because gradients need more range. DeepSeek-V3 used E4M3 everywhere, thanks to fine-grained scaling; MXFP8 on Blackwell also uses E4M3 everywhere.
How do FP8 scaling factors work, and why per block rather than per tensor?advanced
Each tensor is multiplied by \(s = q_{\max}/\operatorname{amax}\) before the cast, so its largest value maps to the top of the FP8 range; values above are saturated. The matmul accumulates \(Q_X Q_W\) in FP32 and divides by \(s_X s_W\). With one scale per tensor, a single outlier sets amax and pushes ordinary values into the subnormal range or to zero. Per-block scales (DeepSeek-V3: \(1 \times 128\) for activations, \(128 \times 128\) for weights; MXFP8: 32 values with a power-of-two E8M0 scale) confine the damage to one block. Delayed scaling takes amax from a history of past steps (cheaper); current scaling measures it now (safer).
A BF16 run converges more slowly than the FP32 reference. What do you check?advanced
First, that the optimizer works on FP32 master weights with FP32 states (no model.bfloat16() plus a plain optimizer). Second, that reductions run in FP32: norms, softmax, loss, and gradient accumulation across micro-batches and the data-parallel all-reduce (Llama 3 used FP32 for both). Third, custom kernels that bypass autocast. Fourth, large logits or activations where BF16’s coarse spacing matters (z-loss or QK-norm help). If updates are very small relative to weights, also consider stochastic rounding or compensated summation for any BF16 state you keep.
Does mixed-precision training reduce the memory for weights and optimizer states?advanced
Not by itself. With Adam it needs 2 (16-bit weights) + 2 (16-bit gradients) + 4 (FP32 master) + 8 (FP32 m and v) = 16 bytes per parameter. Plain FP32 training with Adam also needs 4 (weights) + 4 (gradients) + 8 (m and v) = 16 bytes. So the weight-side memory is the same. The real savings are activations stored in 16 bits (often the biggest term), half the memory traffic, and up to about 15 times more FLOP/s than plain FP32 on an H100. To shrink weight and optimizer memory you need sharding (ZeRO, FSDP), 8-bit or BF16 optimizer states, or PEFT.

Exercises

  1. Compute the largest normal number and the smallest subnormal of E5M2 from formula (37.3). Check them against the table.
  2. What is the spacing between neighbouring BF16 numbers in \([256, 512)\) and in \([1024, 2048)\)? Which BF16 number is 1,029 stored as?
  3. A BF16 weight equals 0.5. What is the smallest update size that changes it? Explain with formula (37.5).
  4. The largest absolute gradient in a model is 0.02. What is the largest power-of-two loss scale that cannot overflow FP16?
  5. A tensor has amax 20. Quantize the value 0.05 to E4M3 with and without the per-tensor scale, and give both relative errors.
  6. Why does a BF16 running sum of 0.01 stall at exactly 4.0, and an FP16 one at exactly 32.0? Use Figure 2 to check.
  7. In Figure 3 with FP16, find the range of \(k\) for which fewer than 0.01% of gradients are flushed to zero and fewer than 1 in \(10^9\) overflow.
Answerscheck
1. \(b = 15\), \(e_{\max} = 30\), \(M = 2\): \(x_{\max} = (2 - 0.25) \cdot 2^{15} = 57{,}344\); \(x_{\min}^{\text{sub}} = 2^{1 - 15 - 2} = 2^{-16} \approx 1.53 \times 10^{-5}\). 2. \(\text{ulp} = 2^{8-7} = 2\) in \([256, 512)\) and \(2^{10-7} = 8\) in \([1024, 2048)\). 1,029 lies between 1,024 and 1,032 and is closer to 1,032, so it is stored as 1,032. 3. \(\text{ulp}(0.5) = 2^{-1-7} = 2^{-8}\), half of it is \(2^{-9} = 0.001953125\). An update of exactly that size is a tie and rounds back to 0.5 (even mantissa), so the update must be larger than 0.001953125; for example, 0.00196 gives 0.50390625. 4. We need \(0.02\, S \lt 65504\), so \(S \lt 3{,}275{,}200\); \(2^{21} = 2{,}097{,}152\) works (max scaled gradient 41,943), \(2^{22}\) would give 83,886, which overflows. In practice you leave a margin. 5. \(s = 448/20 = 22.4\); \(0.05 \times 22.4 = 1.12\), stored as 1.125, unscaled 0.050223: error 0.45%. Without scaling, 0.05 lies in \([2^{-5}, 2^{-4})\) where the gap is \(2^{-8}\); it is stored as 0.05078: error 1.56%. 6. At 4, BF16’s gap is \(2^{2-7} = 1/32 = 0.03125\); half is 0.0156, bigger than 0.01, so \(4 + 0.01\) rounds to 4. Just below 4 the gap is \(1/64\), half is 0.0078, smaller than 0.01, so the sum still grows (in steps of 1/64). FP16 reaches the same gap \(1/32\) at \(2^5 = 32\) because it has 3 more mantissa bits. 7. Zero fraction below 0.01% needs \(k \ge 10\) (0.009%); fewer than 1 overflow per \(10^9\) needs \(k \le 12\) (0.99). So \(k\) from 10 to 12: a narrow window, which is why the scale is adjusted dynamically.

Key takeaways

  • A float is sign × power of two × significant bits (37.1, 37.2). Exponent bits buy range, mantissa bits buy precision (37.3, 37.4).
  • FP16 is precise but narrow (max 65,504); BF16 has FP32’s range but only 8 significant bits; TF32 is FP32 storage with 10-bit matmul inputs; E4M3 and E5M2 are FP8 for forward and backward.
  • Low precision breaks by overflow, underflow and swamping (37.5, 37.6). Swamping is why we keep FP32 master weights and FP32 accumulation.
  • FP16 needs loss scaling (37.7), usually dynamic (37.8). BF16 does not, because of its range.
  • FP8 needs a scale per tensor or per block (37.9, 37.10); fine-grained scales handle outliers, as in DeepSeek-V3 and MXFP8.

Key formulas

\[ x = (-1)^s\, 2^{e-b} \left(1 + \tfrac{m}{2^M}\right), \quad u = 2^{-(M+1)}, \quad \operatorname{fl}(a+b) = a \text{ if } |b| \lt \tfrac12 \text{ulp}(a) \]
\[ \mathbf{g} = \tfrac{1}{S} \nabla_\theta (S\mathcal{L}), \qquad s_X = \frac{q_{\max}}{\operatorname{amax}(X)}, \qquad Y \approx \frac{(Q_X Q_W)_{\text{FP32}}}{s_X\, s_W} \]

References for this session

  1. Micikevicius et al. (2017). Mixed Precision Training
  2. Micikevicius et al. (2022). FP8 Formats for Deep Learning
  3. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  4. Micikevicius et al. (2017). Mixed Precision Training
  5. Micikevicius et al. (2022). FP8 Formats for Deep Learning
  6. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  7. Kalamkar et al. (2019). A Study of BFLOAT16 for Deep Learning Training
  8. DeepSeek-AI (2024). DeepSeek-V3 Technical Report
  9. Llama Team, Meta (2024). The Llama 3 Herd of Models
  10. NVIDIA (2025). Using FP8 with Transformer Engine (FP8 primer)
  11. PyTorch (2025). Automatic Mixed Precision package: torch.amp
  12. PyTorch (2025). CUDA semantics: TensorFloat-32 (TF32) on Ampere and later devices
  13. NVIDIA (2023). NVIDIA H100 Tensor Core GPU (specifications)