Number formats: INT8, INT4, FP8, FP4, NF4 and microscaling
Quantization stores each number with fewer bits, plus a scale that says what the bits mean. This session explains the formats used to serve LLMs: integers (INT8, INT4), small floats (FP8 E4M3 and E5M2, FP4 E2M1), block-scaled microscaling formats (MXFP4, NVFP4) and the NF4 type from QLoRA. You will see where each format puts its grid points, why outliers decide which format wins, and which GPU generation can multiply which format on its tensor cores.
What you will learn
- The quantization map: scale, zero point, clamping and rounding, and how the granularity of the scale (tensor, channel, group, block) changes the error
- Why integer grids are uniform and float grids are logarithmic, and how to estimate the error of each in decibels (6 dB per bit)
- FP8 E4M3 versus E5M2 for inference, and per-tensor scaling with a calibrated or a live amax
- FP4 E2M1 and block-scaled microscaling formats: MXFP4 (32 values, power-of-two scale) and NVFP4 (16 values, E4M3 scale plus a tensor scale)
- NF4: a 4-bit grid built from the quantiles of a normal distribution, block size 64 and double quantization
- Bits per value including scales, and the weight memory of 8B, 70B and 405B models in each format
- Which formats each GPU generation runs on tensor cores (A100, L4, H100, B200), and which formats are storage-only
Why this matters
A 70B model in BF16 needs 140 GB just for its weights. In a 4-bit format with small block scales it needs about 36 to 39 GB. The model is the same; only the number format changed.
In session 104 we saw why fewer bits make inference faster: decoding is limited by memory traffic, so every byte you do not read is time saved, and on newer GPUs low-precision tensor cores also do more multiplies per second. That session asked where to quantize (weights only, or weights and activations). This session asks how: what exactly is stored in those 8 or 4 bits, and what a stored code means.
There are many formats, and their names are confusing: INT8, INT4, FP8 E4M3, FP8 E5M2, FP4 E2M1, MXFP4, NVFP4, NF4. They differ in three things only:
- Where the grid points are. A format with \(b\) bits has at most \(2^b\) values. Integers space them evenly. Floats space them on a logarithmic scale. NF4 puts them where a bell curve has most of its mass.
- How the scale is stored. One scale per tensor, per row, per group of 128, per block of 32 or 16. The scale itself can be FP32, FP16, FP8 or a pure power of two.
- Whether the hardware can multiply it. Some formats are compute formats on a given GPU (its tensor cores take them as inputs). Others are only storage formats: the kernel unpacks them to BF16 before the multiply.
Real systems use all of them. DeepSeek-V3 was trained with FP8 matmuls and fine-grained block scales. [Certain] OpenAI’s gpt-oss-120b checkpoint stores its mixture-of-experts weights in MXFP4 and keeps attention, the router, the embedding and the output head unquantized (see its quantization_config). [Certain] QLoRA fine-tunes models whose frozen weights are stored in NF4. [Certain] NVIDIA reports that DeepSeek-R1-0528 in NVFP4 stays within about one point of FP8 on most benchmarks they show (for example MMLU-Pro 84 versus 85). [Certain: vendor claim]
The quantization map: scale, zero point, round, clamp
Every format in this session uses the same idea. We have real numbers \(x\) (weights or activations in BF16). We pick a scale \(s\) (a real number, “how much one step is worth”) and sometimes a zero point \(z\) (an integer offset). Then we divide, round to the nearest code, and clip to the allowed codes:
- \(x\)
- the original value (for example a BF16 weight).
- \(s\)
- the scale: the real value of one integer step. Stored next to the codes, usually in FP16, FP32, FP8 or as a power of two.
- \(z\)
- the zero point: the code that means the real value 0. It is 0 for symmetric quantization.
- \(q\)
- the stored code, for example an integer in \([-127, 127]\) for INT8.
- \(\hat{x}\)
- the dequantized value: what the model actually computes with. The error is \(\hat{x} - x\).
- \(\operatorname{round}\)
- round to nearest. For float formats, “round” means “round to the nearest value of the float grid” instead of the nearest integer.
Two ways to choose \(s\) and \(z\) from the data:
- \(\operatorname{amax}(x)\)
- the largest absolute value, \(\max_i |x_i|\), over the values that share this scale.
- \(b\)
- the number of bits per code. Symmetric INT8 uses codes \(-127 \dots 127\); symmetric INT4 uses \(-7 \dots 7\).
Symmetric is the default for weights and activations of LLMs, because they are centered near zero and because \(z = 0\) keeps the matmul simple. Asymmetric is useful for one-sided data, for example the output of a ReLU or softmax probabilities, and it is common in weight-only INT4 schemes (GPTQ and AWQ checkpoints usually store a scale and a zero point per group of 128). [Likely]
Granularity: who shares a scale?
The scale is computed from an amax, and the amax is taken over some group of numbers. This choice, the granularity, matters as much as the number of bits:
- Per tensor: one scale for the whole matrix. Cheapest, but one outlier sets the scale for millions of values.
- Per channel / per row: one scale per output channel of a weight matrix, or per token (row) of an activation matrix.
- Per group / per block: one scale per \(k\) neighbouring values along the reduction dimension: \(k = 128\) in typical INT4 weight-only schemes, 64 in NF4, 32 in MX, 16 in NVFP4.
Smaller groups give smaller errors but cost more scale storage. If \(k\) values share one scale of \(b_s\) bits, the true cost per value is
INT4 with an FP16 scale per 128 values costs \(4 + 16/128 = 4.125\) bits; with an FP16 zero point too, 4.25 bits. MXFP4 costs \(4 + 8/32 = 4.25\) bits. NVFP4 costs \(4 + 8/16 = 4.5\) bits. NF4 with an FP32 scale per 64 values costs 4.5 bits. These numbers matter when you compare “4-bit” methods.
Which scales survive a matrix multiply?
Quantization is only useful for speed if the multiply can run on the codes. Take \(Y = XW\) with \(X\) of shape \(T \times K\) (tokens by input features) and \(W\) of shape \(K \times N\). If \(X\) has one scale per row \(i\) and \(W\) one scale per column \(j\), the scales move outside the sum:
- \(s^{X}_{i}\)
- the scale of token \(i\) (per-token scaling of activations).
- \(s^{W}_{j}\)
- the scale of output channel \(j\) (per-channel scaling of weights).
- \(\sum q^{X} q^{W}\)
- an integer (or FP8) dot product, accumulated in INT32 (or FP32) inside the tensor core.
A scale per input channel \(k\) does not factor out: it sits inside the sum and would have to be applied to every product. That is a problem, because activation outliers in LLMs live in a few fixed input channels (LLM.int8() found them in a small set of hidden dimensions). [Certain] The classic fixes are to keep those channels in 16 bits (LLM.int8()), or to move the outlier from the activation into the weight (SmoothQuant), covered in session 108. Block formats like MX take a third way: the hardware applies a new scale every 32 values along \(k\), so the sum is split into short pieces that each have their own scale.
Integer formats: INT8 and INT4
An integer format has evenly spaced codes. With symmetric scaling the grid is \(\{-Q s, \dots, -s, 0, s, \dots, Q s\}\) with \(Q = 2^{b-1} - 1\): 127 for INT8, 7 for INT4. Two INT4 codes are packed into one byte. The rounding error is at most \(s/2\) and, for values that are spread out, roughly uniform on \([-s/2, s/2]\), so its mean square is \(s^2/12\).
A standard way to measure the quality of a format is the signal-to-quantization-noise ratio (SQNR): the energy of the signal over the energy of the error, in decibels. 10 dB means the error is 10 times smaller in energy (about 3.2 times smaller in size); every 6 dB halves the size of the error. For Gaussian data with standard deviation \(\sigma\) and a scale set by \(\operatorname{amax} = c\,\sigma\):
- \(Q\)
- the largest code, \(2^{b-1} - 1\).
- \(c\)
- how many standard deviations the amax is. For 1,024 Gaussian values it is around 3 to 4; with an outlier it can be 50 or more.
- \(6.02\, b\)
- the famous “6 dB per bit” rule: each extra bit halves the step, so the error halves. The approximation uses \(Q \approx 2^{b-1}\); for 4 bits it is about 1 dB too optimistic, so use the exact form.
With \(c = 4\): INT8 gives 40.8 dB and INT4 15.7 dB. The rule also shows the main weakness of integers: the term \(-20 \log_{10} c\). If one outlier makes \(c\) ten times larger, you lose 20 dB, the same as losing more than 3 bits.
- Input: weight matrix \(W\) of shape \(K \times N\), group size \(g = 128\) along \(K\).
- For each output column \(j\) and each group \(G\) of \(g\) rows: \(s \leftarrow \max_{k \in G} |W_{kj}| / 7\). Store \(s\) in FP16.
- For each \(k \in G\): \(q_{kj} \leftarrow \operatorname{clamp}(\operatorname{round}(W_{kj}/s), -7, 7)\).
- Pack two 4-bit codes per byte (add 8 to get unsigned nibbles 1 to 15).
- At inference: load the packed bytes and scales, unpack to BF16 as \(s \cdot q\) in registers, and run a BF16 matmul. Only the memory read was cheaper.
Floating-point formats: FP8 E4M3 and E5M2
Session 37 built the FP8 formats bit by bit; here we only need the facts that matter for inference. A float stores a sign, an exponent and a mantissa:
Inside one power of two (one binade, for example \([1, 2)\)) the \(2^M\) values are evenly spaced. Each next binade has the same number of values, twice as far apart. So the step grows with the value, and the relative error stays about the same, from \(2^{-(M+1)}\) at worst down to half of that. This is why a float grid is logarithmic.
| Format | Sign / exp. / mantissa | Largest | Smallest normal | Smallest subnormal | Distinct finite values | Range in binades |
|---|---|---|---|---|---|---|
| FP8 E4M3 (“FN”) | 1 / 4 / 3 | 448 | \(2^{-6}\) | \(2^{-9}\) | 253 | 17.8 |
| FP8 E5M2 | 1 / 5 / 2 | 57,344 | \(2^{-14}\) | \(2^{-16}\) | 247 | 31.8 |
| FP4 E2M1 | 1 / 2 / 1 | 6 | 1 | 0.5 | 15 | 3.6 |
| INT8 (for comparison) | codes −127…127 | 127 s | step s everywhere | 255 | 6.99 | |
The counts come from enumerating all bit patterns in PyTorch (torch.float8_e4m3fn, torch.float8_e5m2); “range in binades” is \(\log_2\) of largest over smallest positive value. E4M3 has no infinity and only one NaN pattern per sign, which buys it one more binade (max 448 instead of 240). [Certain] AMD’s MI300 GPUs use slightly different variants called E4M3FNUZ and E5M2FNUZ (max 240 for the 4-bit-exponent one, a different bias and no negative zero), so a checkpoint must be converted between the two families. [Likely]
The relative error of a float does not depend on the scale, only on \(M\). If values are spread over many binades, averaging the squared relative error over one binade gives
- \(M\)
- the number of mantissa bits: 3 for E4M3, 2 for E5M2, 1 for E2M1.
- \(32 \ln 2\)
- a constant from averaging: the step is \(2^{-M}\) times the binade start, and \(\E[1/x^2]\) over a binade with log-uniform values is \(3/(8 \ln 2)\).
This gives 31.5 dB for E4M3, 25.5 dB for E5M2 and 19.5 dB for E2M1, whatever the scale, as long as no value overflows or falls far into the subnormals. Compare with (105.5): INT8 per channel on clean Gaussian weights reaches about 42 dB, better than E4M3, because it spends its 8 bits on precision, not range. But INT8 loses 6 dB every time the amax doubles, and E4M3 does not.
How to read Figure 1
Each row is one format; each tick is a value it can store. Move the slider to choose a value; the readout gives the nearest grid point in each format and the relative error. At \(2^{-6.5}\) (a value 90 times smaller than the amax), INT8 is off by 28.7% while both FP8 formats are within 1.02%. Below \(2^{-8}\) of the amax INT8 has nothing left except zero, while E4M3 continues down to \(2^{-17.8}\) and E5M2 to \(2^{-31.8}\) (off the left edge). Switch to 4-bit: INT4 has only 7 positive values, all within a factor of 7 of the top; E2M1 spans 3.6 binades; NF4 packs its 8 positive values between 0.08 and 1, densest near zero where Gaussian weights are.
Take four activations \(x = (0.02,\ -0.5,\ 1.3,\ 40)\). The amax is 40.
INT8 symmetric: \(s = 40/127 = 0.31496\). Codes \(\operatorname{round}(x/s) = (0, -2, 4, 127)\). Dequantized: \((0,\ -0.630,\ 1.260,\ 40)\). Relative errors: 100% (0.02 became zero), 26.0%, 3.1%, 0%.
FP8 E4M3: \(s = 40/448 = 0.08929\). Scaled values \(x/s = (0.224, -5.6, 14.56, 448)\) round to the E4M3 grid as \((0.21875,\ -5.5,\ 15,\ 448)\): in \([0.125, 0.25)\) the step is \(2^{-6}\), in \([4, 8)\) it is 0.5, in \([8, 16)\) it is 1. Dequantized: \((0.01953,\ -0.4911,\ 1.3393,\ 40)\). Relative errors: 2.3%, 1.8%, 3.0%, 0%.
Same 8 bits, same scale rule. The integer grid spent its codes evenly up to 40 and had almost nothing left for 0.02 and 0.5. The float grid kept about 3% relative accuracy everywhere.
FP8 for inference: E4M3, and how the scale is chosen
Training recipes often use E4M3 for the forward pass and E5M2 for gradients, which need more range. Inference has no gradients, and a scale already moves the tensor into range, so the extra 14 binades of E5M2 are wasted while its missing mantissa bit costs 6 dB. FP8 inference therefore uses E4M3 for weights and activations. [Certain] The per-tensor scale is
- 448
- the largest finite E4M3 value; values above it are saturated (clipped to ±448), never cast to NaN.
- \(\operatorname{fp8}(\cdot)\)
- round to the nearest E4M3 value (round half to even).
- \((\cdot)_{\text{FP32}}\)
- the tensor core multiplies FP8 inputs and accumulates in higher precision; the two scales are applied once at the end.
For weights the amax is known offline. For activations there are two choices. Static scaling measures the amax on a few hundred calibration prompts and freezes it: no runtime cost, but a larger activation in production is clipped. Dynamic scaling computes the amax of each tensor (or each token) at run time: safer, but it costs a reduction over the tensor before the matmul. Serving engines such as vLLM and TensorRT-LLM support both. [Likely]
How to read Figure 2
With no real outlier (\(M = 1\)), INT8 wins: 42.4 dB against 31.4 dB for E4M3. As \(M\) grows, the INT8 step grows with it and the curve falls by 6 dB for every doubling. It drops below E4M3 at about \(M \approx 11\) and below E5M2 at about \(M \approx 22\). At \(M = 100\), 31% of the ordinary values round to zero and INT8 is at 12.8 dB. The float curves stay flat (E4M3 about 31.5 dB, E5M2 about 25.5 dB, as formula 105.7 predicts) until the ordinary values fall into the subnormal range. The grey line is the LLM.int8() idea: take the outliers out, and INT8 is good again.
Four bits: FP4 E2M1 and why one scale per tensor is not enough
FP4 E2M1 has 1 sign bit, 2 exponent bits and 1 mantissa bit. Its 15 distinct values are
It covers only \(\log_2(6/0.5) = 3.6\) binades. A per-tensor scale cannot work: in a weight matrix, the amax is often 10 to 20 times the typical value, and anything below \(1/24\) of the amax becomes zero. INT4 has the same problem in a different shape. The answer is to share a scale between only a few neighbouring values, so that each small block gets its own amax. This is block scaling, and when the block is tiny (16 or 32 values) and the scale is stored in hardware-friendly 8 bits, it is called microscaling.
The MX standard: 32 values and a power-of-two scale
The Open Compute Project MX specification (by AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm [Likely]) defines block formats where \(k = 32\) elements share one scale \(X\) stored as E8M0: 8 exponent bits, no mantissa, so \(X\) is always a power of two. [Certain] The shared exponent is chosen so that the block’s amax lands in the top binade of the element format:
- \(V_i\)
- the 32 original values of the block.
- \(e^{\text{elem}}_{\max}\)
- the exponent of the largest power of two in the element format: 2 for E2M1 (\(4 = 2^2\)), 8 for E4M3 (\(256 = 2^8\)).
- \(\operatorname{clamp}\)
- values above the element maximum are clipped to it, keeping the sign (the specification says so). [Certain]
- \(P_i\)
- the stored element codes (4 bits each for MXFP4).
The concrete MX formats are MXFP8 (E4M3 or E5M2 elements), MXFP6 (E2M3 or E3M2), MXFP4 (E2M1) and MXINT8, all with 32-element blocks and an E8M0 scale. [Certain] A power-of-two scale is cheap in hardware: applying it is an exponent add, not a multiply. The price is that it can be off by up to a factor of 2 from the ideal scale. Because of the floor, the block amax can land anywhere from 4 to almost 8 in units of \(X\), and E2M1 stops at 6, so the largest values of a block may be clipped. Worked example 2 shows this.
- Reshape the tensor so that each row of the reduction dimension is cut into blocks of 32 values.
- For each block: \(a \leftarrow \max_i |V_i|\). If \(a = 0\), store \(X = 2^{-127}\) and zero codes.
- \(e \leftarrow \lfloor \log_2 a \rfloor - 2\) (2 is \(e_{\max}\) of E2M1); store \(e + 127\) as the 8-bit E8M0 scale; \(X \leftarrow 2^{e}\).
- For each \(i\): \(P_i \leftarrow\) nearest E2M1 value to \(\operatorname{clamp}(V_i / X, -6, 6)\); store 4 bits.
- In the block-scaled matmul (Blackwell tensor cores), each 32-long piece of every dot product is computed on the E2M1 codes, multiplied by \(X^{A} X^{B}\), and accumulated in FP32.
NVFP4: 16 values, an FP8 scale, and a tensor scale
NVIDIA’s NVFP4 (introduced with Blackwell) keeps E2M1 elements but changes the scaling in three ways: blocks of 16 instead of 32; a block scale stored as FP8 E4M3 instead of E8M0, so it can be any value with 3 mantissa bits, not only a power of two; and a second, per-tensor FP32 scale that moves the block scales into the range of E4M3. [Certain]
- \(s_t\)
- the FP32 per-tensor scale. It makes the largest block scale equal to 448, the top of E4M3.
- \(s_b\)
- the block scale, stored in 8 bits (E4M3). The block amax maps to about 6, the top of E2M1, with an error of at most about 6% from rounding \(s_b\).
- 6
- the largest E2M1 value.
The cost is \(4 + 8/16 = 4.5\) bits per value (the FP32 tensor scale is negligible). NVIDIA reports this as about 3.5 times smaller than FP16 and 1.8 times smaller than FP8. [Certain] Our own checks: \(16/4.5 = 3.56\) and \(8/4.5 = 1.78\).
Take a block \(V = (0.31,\ -0.07,\ 0.12,\ 0.95,\ -0.44,\ 0.02,\ 0.6,\ -0.18)\) (8 values to keep it short; real blocks have 32 or 16). The amax is 0.95. For NVFP4 we take \(s_t = 1\) so only the block scale matters.
MXFP4: \(\lfloor \log_2 0.95 \rfloor = -1\), so \(X = 2^{-1-2} = 0.125\). Then \(V/X = (2.48, -0.56, 0.96, 7.6, -3.52, 0.16, 4.8, -1.44)\). The value 7.6 is above 6 and is clipped. Codes \(P = (2, -0.5, 1, 6, -4, 0, 4, -1.5)\). Dequantized: \((0.25, -0.0625, 0.125, 0.75, -0.5, 0, 0.5, -0.1875)\). The largest value 0.95 became 0.75: an error of 21%. Mean squared error: 0.00722.
NVFP4: \(0.95/6 = 0.15833\), which E4M3 stores as \(s_b = 0.15625\) (step \(2^{-6}\) in \([0.125, 0.25)\)). Then \(V/s_b = (1.98, -0.45, 0.77, 6.08, -2.82, 0.13, 3.84, -1.15)\), codes \(P = (2, -0.5, 1, 6, -3, 0, 4, -1)\). Dequantized: \((0.3125, -0.078, 0.156, 0.9375, -0.469, 0, 0.625, -0.156)\). The amax error is 1.3%. Mean squared error: 0.000495, about 14.6 times smaller.
The power-of-two scale was almost a factor of 2 too small here (ideal \(0.95/6 = 0.158\), MX chose 0.125). That is the worst case for MX; when the block amax lands near 6 in units of \(X\), MX and NVFP4 behave about the same.
NF4: a 4-bit grid shaped like a bell curve
Integer and float grids are generic. QLoRA asked a different question: if the values are known to be roughly normal, where should 16 grid points go? Its answer is quantile quantization: put the points so that each one is used about equally often. For a normal distribution that means dense points near zero and sparse points in the tails. The construction uses the quantile function \(Q_X\) of \(\mathcal{N}(0, 1)\) (the inverse of its CDF):
- \(k\)
- the number of bits (4).
- \(Q_X(p)\)
- the value below which a fraction \(p\) of a standard normal lies.
- \(q_i\)
- a grid point: the midpoint between two neighbouring quantiles.
A symmetric version of (105.12) has no exact zero, and zero matters (padding, and many weights near 0). So NF4 builds the negative and positive halves separately, with \(2^{k-1}\) quantiles for one side and \(2^{k-1} + 1\) for the other, merges them and removes the duplicate zero. [Certain] The result has 7 negative values, zero, and 8 positive values:
A block of 64 weights shares one absmax scale \(c\): \(\hat{w} = c \cdot \text{NF4}[q]\). [Certain] With \(c\) in FP32 that is \(4 + 32/64 = 4.5\) bits per weight. Double quantization quantizes the scales themselves: the FP32 scales are stored as 8-bit floats with one FP32 scale per 256 of them. The overhead falls from 0.5 to \(8/64 + 32/(64 \cdot 256) = 0.127\) bits per weight, so NF4 with double quantization costs about 4.127 bits. [Certain] QLoRA calls NF4 “information-theoretically optimal” for zero-centered normal data, in the sense that each code is used equally often. [Certain] Session 58 uses it to fine-tune.
How to read Figure 3
On plain Gaussian data, FP4 with one scale per tensor (18.4 dB), INT4 per 128 (18.5 dB) and MXFP4 (18.8 dB) are close; NVFP4 (20.4 dB) and NF4 per 64 (20.6 dB) are about 2 dB better, at 4.5 bits. Now add the outliers: every block of 128 contains one value 10 times larger. Per-tensor FP4 collapses to 8.9 dB (54% of values become zero), INT4-g128 falls to 11.3 dB because every group contains an outlier, MXFP4 keeps 16.2 dB because only one block in four is hit, NF4 falls to 17.4 dB, and NVFP4 keeps 20.7 dB: with 16-value blocks and a fine-grained scale, the outlier only spoils its own small block. In the bottom panel, look for blocks where most values sit on zero.
Bits per value and real memory
Formula (105.3) turns into gigabytes quickly. The table counts only the weights that are quantized, with all of them in the given format. Real checkpoints keep some tensors in BF16 (embeddings, the output head, norms, often the router), so real files are a little larger.
| Format | Bits per value | 8B model | 70B model | 405B model |
|---|---|---|---|---|
| BF16 | 16 | 16.0 GB | 140 GB | 810 GB |
| FP8 E4M3 / INT8, per tensor or per channel | ≈ 8 | 8.0 GB | 70 GB | 405 GB |
| INT4, FP16 scale per 128 | 4.125 | 4.1 GB | 36.1 GB | 208.8 GB |
| MXFP4 (E8M0 per 32) | 4.25 | 4.3 GB | 37.2 GB | 215.2 GB |
| NVFP4 (E4M3 per 16) | 4.5 | 4.5 GB | 39.4 GB | 227.8 GB |
| NF4 with double quantization | 4.127 | 4.1 GB | 36.1 GB | 208.9 GB |
How to read Figure 4
Pick a model size. At 70B, BF16 needs two H100s just for weights; FP8 fits on one H100 with only 10 GB left for the KV cache; the 4-bit formats leave more than 40 GB free. The difference between the 4-bit formats (36.1 to 39.4 GB) is the scale overhead. The readout also gives the minimum number of 80 GB GPUs for the weights alone.
Hardware: which formats run on tensor cores
A format helps speed in two ways. Fewer bytes help every GPU, because decode is memory-bound. More multiplies per second only come if the tensor cores accept that format as input. The table lists the data-center GPUs you are most likely to meet. Peak numbers are from NVIDIA’s product pages; “dense” means without 2:4 structured sparsity, which is half of the sparse number the pages often show. [Certain for the rows with numbers]
| GPU (architecture) | Low-precision tensor-core inputs | Dense peak, per GPU | Notes |
|---|---|---|---|
| T4 (Turing) | FP16, INT8, INT4 | — | No BF16, no FP8. [Likely] |
| A100 (Ampere) | TF32, BF16, FP16, INT8, INT4 | INT8 624 TOPS (SXM) | No FP8. FP8 checkpoints run as weight-only (upcast to BF16). |
| L4 (Ada Lovelace) | BF16, FP16, FP8 (E4M3, E5M2), INT8 | FP8 242.5 TFLOPS, BF16 121 | The cheapest data-center GPU with FP8 tensor cores; 24 GB at 300 GB/s. |
| H100 SXM (Hopper) | BF16, FP16, FP8, INT8 | FP8 1,979 TFLOPS, BF16 989.5 | No INT4 or FP4 is listed. DeepSeek-V3 notes limited FP8 accumulation precision in Hopper tensor cores and promotes partial sums to FP32 every 128 elements. [Likely] |
| B200 (Blackwell) | adds FP4 (NVFP4, MXFP4), FP6, MXFP8 | FP4 9 PFLOPS, FP8/FP6 4.5 PFLOPS | From HGX B200 (8 GPUs): FP4 72 PFLOPS dense, FP8/FP6 72 PFLOPS sparse. |
| B300 (Blackwell Ultra) | FP4, FP6, FP8 | FP4 13.5 PFLOPS | INT8 is cut back: HGX B300 lists 3 POPS sparse for 8 GPUs, against 72 for HGX B200. |
| AMD MI300X / MI355X | FP8 (FNUZ variants) / adds MXFP6, MXFP4 | — | [Likely]; check the vendor page for numbers. |
Three patterns follow. First, NVIDIA moved from integers to floats: INT4 tensor cores were on Turing and Ampere, FP8 came with Ada and Hopper, FP4 with Blackwell, and Blackwell Ultra even cuts INT8. Second, every step down in bits roughly doubled the peak: on B200, FP4 is twice FP8. Third, weight-only formats (INT4-g128, NF4, and FP8 on an A100) run on any GPU, but they only reduce memory traffic; that is exactly what decode needs, which is why they are so popular. [Certain for the first two from the spec pages; the third is the argument of session 104]
The code
This runs on a CPU and is deterministic. It implements every scheme of this session in a few lines (the MXFP4 and NVFP4 functions follow formulas 105.10 and 105.11; FP8 uses PyTorch’s own float8_e4m3fn cast) and measures the SQNR on Gaussian weights and on activations with 4 outlier channels. The printed values are from a run with PyTorch 2.8.
import torch torch.manual_seed(0) # the 16 NF4 code values (QLoRA appendix; same table as bitsandbytes) NF4 = torch.tensor([-1.0, -0.6961928, -0.5250731, -0.3949175, -0.2844414, -0.1847734, -0.0910500, 0.0, 0.0795803, 0.1609302, 0.2461123, 0.3379152, 0.4407098, 0.5626170, 0.7229568, 1.0]) E2M1 = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0]) FP4 = torch.cat([-E2M1.flip(0)[:-1], E2M1]) # 15 distinct values, max 6 def to_grid(x, grid): """Round every element of x to the nearest value in a sorted 1-D grid.""" mids = (grid[1:] + grid[:-1]) / 2 return grid[torch.bucketize(x, mids)] def int_sym(x, bits): # symmetric, one scale per row Q = 2 ** (bits - 1) - 1 s = x.abs().amax(-1, keepdim=True) / Q return torch.clamp(torch.round(x / s), -Q, Q) * s def fp8(x, dtype=torch.float8_e4m3fn): # one scale per tensor (105.8) s = x.abs().amax() / torch.finfo(dtype).max return (x / s).to(dtype).float() * s def mxfp4(x, k=32): # OCP MX: power-of-two scale per 32 (105.10) b = x.reshape(-1, k) e = torch.floor(torch.log2(b.abs().amax(1, keepdim=True))) - 2 # emax(E2M1) = 2 X = 2.0 ** e return (to_grid((b / X).clamp(-6, 6), FP4) * X).reshape(x.shape) def nvfp4(x, k=16): # E4M3 scale per 16 + FP32 per tensor (105.11) b = x.reshape(-1, k) st = x.abs().amax() / (6 * 448) sb = (b.abs().amax(1, keepdim=True) / (6 * st)).to(torch.float8_e4m3fn).float() s = st * sb return (to_grid((b / s).clamp(-6, 6), FP4) * s).reshape(x.shape) def nf4(x, k=64): # QLoRA: absmax per 64, NF4 codes b = x.reshape(-1, k) c = b.abs().amax(1, keepdim=True) return (to_grid(b / c, NF4) * c).reshape(x.shape) def sqnr(x, xq): # signal-to-quantization-noise, dB return (10 * torch.log10(x.pow(2).sum() / (x - xq).pow(2).sum())).item() W = torch.randn(1024, 1024) * 0.02 # Gaussian weights A = torch.randn(256, 1024) # activations ... A[:, :4] *= 50 # ... with 4 outlier channels schemes = { "INT8 per-row": (lambda x: int_sym(x, 8), 8.0), "FP8 E4M3": (fp8, 8.0), "INT4 per-row": (lambda x: int_sym(x, 4), 4.0), "INT4 g128": (lambda x: int_sym(x.reshape(-1, 128), 4).reshape(x.shape), 4 + 16 / 128), "MXFP4": (mxfp4, 4 + 8 / 32), "NVFP4": (nvfp4, 4 + 8 / 16), "NF4 b64": (nf4, 4 + 32 / 64), } print(f"{'format':14s} {'bits':>5s} {'W dB':>6s} {'A dB':>6s}") for name, (f, bits) in schemes.items(): print(f"{name:14s} {bits:5.3f} {sqnr(W, f(W)):6.1f} {sqnr(A, f(A)):6.1f}") # format bits W dB A dB # INT8 per-row 8.000 42.1 25.3 # FP8 E4M3 8.000 31.6 31.4 # INT4 per-row 4.000 16.9 10.1 # INT4 g128 4.125 18.6 18.0 # MXFP4 4.250 18.9 17.1 # NVFP4 4.500 20.4 22.2 # NF4 b64 4.500 20.7 20.5
Read the table as a summary of the session. On clean weights, INT8 per row (42.1 dB) beats FP8 (31.6 dB), exactly as formulas 105.5 and 105.7 predict. On activations with outlier channels, per-row INT8 drops to 25.3 dB because every row contains the outliers, while FP8 does not care (31.4 dB). At 4 bits, a scale per row is far too coarse (16.9 and 10.1 dB); small blocks fix it, and the finer the block and its scale, the better: NVFP4 is the best of the 4-bit schemes on the outlier activations (22.2 dB). Remember that 3 dB is a factor of 2 in error energy.
Where this is used
| Setting | Typical format | Why |
|---|---|---|
| Chat serving of a dense LLM on H100 | FP8 E4M3 weights and activations, per-tensor or per-channel scales | Tensor cores run FP8 at 2× BF16; accuracy is close to BF16 for most models. [Likely] |
| Local and single-GPU serving (laptops, L4, consumer GPUs) | INT4 weight-only with groups of 128 (GPTQ, AWQ), or similar 4-bit schemes | Decode is memory-bound; 4× fewer bytes than BF16 is the main gain. |
| Fine-tuning a large model on one GPU | NF4 frozen weights + BF16 LoRA adapters (QLoRA) | Weights take about a quarter of the memory; compute stays in BF16. |
| Large MoE models on Blackwell | MXFP4 or NVFP4 expert weights | Experts hold most of the parameters; gpt-oss-120b ships its experts in MXFP4. [Certain] |
| FP8 pretraining (DeepSeek-V3) | E4M3 with 1×128 activation tiles and 128×128 weight blocks | Fine-grained scales handle outliers without per-tensor tricks. [Certain] |
| Video-language models (video QA, key-moment detection) | FP8 or INT4 for the LLM; vision encoder often kept in BF16 | Long videos give many visual tokens; the KV cache and LLM weights dominate memory. [Likely] |
| Video retrieval, video fingerprinting and Video-RAG indexes | INT8 (or binary) embedding vectors with a per-vector scale | Millions of clip embeddings: 4× smaller index than FP32, integer dot products for search. [Likely] |
Common mistakes
- Casting to FP8 without a scale.
x.to(torch.float8_e4m3fn)on raw activations sends everything below \(2^{-9}\) to zero and everything above 448 out of range. Always divide by \(s = \operatorname{amax}/448\) first and keep \(s\). - Counting 4 bits per value. Scales are part of the format. NVFP4 is 4.5 bits, MXFP4 4.25, INT4-g128 with a zero point 4.25. Compare methods at equal effective bits (105.3).
- Expecting a speed-up from NF4 or INT4-g128 compute. On Hopper these are dequantized to BF16 before the multiply. They speed up memory-bound decode, not compute-bound prefill of long prompts.
- Scaling activations per input channel in an integer matmul. The scale must factor out of the sum (105.4): per token for \(X\), per output channel for \(W\), or per block along \(K\) only if the hardware supports block-scaled MMA (MX, NVFP4).
- Using E5M2 for inference. After scaling, range is not the problem; E5M2 throws away a mantissa bit (6 dB) for 14 extra binades you never use.
- Calibrating static activation scales on too few or unrepresentative prompts. A larger activation in production is clipped at 448 times \(s\). Use more varied calibration data, a margin, or dynamic scales.
- Mixing FP8 variants. E4M3FN (max 448, NVIDIA and OCP) and E4M3FNUZ (max 240, used on AMD MI300) have different biases; copying raw bytes between them changes every value. [Likely]
Interview questions
What are a scale and a zero point, and when do you use asymmetric quantization?basic
Which is more accurate, INT8 or FP8 E4M3?basic
E4M3 or E5M2 for inference, and why?core
Explain granularity. Which scales can be used in an INT8 matmul?core
What is microscaling (MX)? Why blocks of 32 and an E8M0 scale?core
How does NVFP4 differ from MXFP4?core
What is NF4, why is it better than INT4 for weights, and can a GPU multiply it?core
How many bits per weight does NF4 with double quantization use? What is the weight memory of a 70B model?advanced
Which formats does each GPU generation run on tensor cores? What does an INT4 weight-only model gain on an H100?advanced
gpt-oss-120b ships in MXFP4 but keeps attention, the router, embeddings and the head unquantized. Why?advanced
Why does the “6 dB per bit” rule hold, and why does a float’s SQNR not depend on the scale?advanced
Exercises
- Quantize values in \([-0.3, 1.0]\) to unsigned 8-bit codes 0 to 255 with asymmetric scaling (105.2). Give \(s\), \(z\), the code of 0.5 and its dequantized value. Check that −0.3 and 1.0 map to 0 and 255.
- A tensor has amax 3.0. How is the value 0.001 stored in E4M3 with the per-tensor scale (105.8), and what is the relative error?
- A block has amax 0.3. Compute the MXFP4 scale (105.10) and the stored value of 0.3. Then do the same with an NVFP4 block scale (\(s_t = 1\)).
- INT4 with an FP16 scale and an FP16 zero point per group of 128: how many bits per value? What is the weight memory of a 70B model?
- In Figure 2, find the outlier size where per-tensor INT8 falls below E4M3, and where it falls below E5M2. Explain both with formulas 105.5 and 105.7.
- Why does NF4 have 8 positive values but only 7 negative ones? What would be lost with a symmetric 16-value grid?
- Which of these matmuls run natively on H100 tensor cores: FP8 E4M3 × FP8 E4M3, NVFP4 × NVFP4, INT8 × INT8, NF4 × BF16?
Answerscheck
Key takeaways
- Every format is codes plus scales: \(\hat{x} = s(q - z)\) (105.1). What differs is where the grid points are, how fine the scale is, and how it is stored.
- Integer grids are uniform: about \(6.02b + 4.8 - 20\log_{10}c\) dB (105.5), excellent on clean data, ruined by outliers. Float grids are logarithmic: about \(6.02M + 13.5\) dB at any scale (105.7).
- For inference, FP8 means E4M3 with a per-tensor (or finer) scale \(\operatorname{amax}/448\); static or dynamic activation scales.
- At 4 bits you need small blocks: MXFP4 (32 values, power-of-two E8M0 scale, 4.25 bits) and NVFP4 (16 values, E4M3 scale plus FP32 tensor scale, 4.5 bits). NVFP4’s finer scale avoids MX’s clipping and 2× scale error.
- NF4 places 16 values at normal quantiles with an exact zero; blocks of 64; 4.127 bits with double quantization. It is a storage format only.
- Tensor cores: INT8/INT4 on Ampere, FP8 on Ada and Hopper (L4, H100), FP4/FP6 and MX on Blackwell. Weight-only formats work everywhere but only save memory traffic.
Key formulas
References for this session
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- Rouhani et al. (2023). Microscaling Data Formats for Deep Learning
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- Rouhani et al. (2023). Microscaling Data Formats for Deep Learning
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- NVIDIA (2025). Introducing NVFP4 for Efficient and Accurate Low-Precision Inference (NVIDIA Technical Blog)
- Open Compute Project (2023). OCP Microscaling Formats (MX) Specification, version 1.0
- OpenAI (2025). gpt-oss-120b: config.json (quantization_config with quant_method mxfp4)
- NVIDIA (docs). NVIDIA L4 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA A100 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA HGX platform: HGX B300 and HGX B200 specifications