Quantization-aware training: fake quant, STE, LLM-QAT, QLoRA, LoftQ
Post-training quantization rounds a finished model and hopes for the best. Quantization-aware training lets the model see the rounding while it trains, so the weights can move to places where rounding hurts less. This session builds fake quantization and the straight-through estimator from scratch, then shows how LLM-QAT, QLoRA, QA-LoRA and LoftQ make the idea affordable for large models, and when it is worth the cost.
What you will learn
- Fake quantization: quantize then dequantize in the forward pass, while FP32 latent weights receive the updates
- Why rounding has no useful gradient, and how the straight-through estimator (STE) and learned step sizes (LSQ) get around it
- Why QAT beats round-to-nearest: weights learn to cancel each other's rounding errors, and why lr decay matters against oscillation
- LLM-QAT: data-free QAT that distills the FP16 model on text it generates itself, with quantized weights, activations and KV cache
- QLoRA, QA-LoRA and LoftQ: a frozen low-bit base plus trainable low-rank adapters, and how each handles merging and initialization
- The cost of QAT in memory, data and GPU time, and a decision rule for when PTQ is enough
Why this matters
The sessions before this one were about post-training quantization (PTQ): take a finished model, round its numbers to a few bits, and fix what you can with a little calibration data. GPTQ and AWQ make 4-bit weights work well. But PTQ has a limit. At 2 or 3 bits, or when the activations and the KV cache must also be 4 or 8 bits, the rounding damage becomes too large to repair after the fact.
Quantization-aware training (QAT) takes a different road. It puts the rounding inside training. The model sees, in every forward pass, the same rounded numbers it will see at deployment, and gradient descent moves the weights to places where rounding hurts less. The cost is real training: gradients, optimizer states, data and GPU time. This session is about how QAT works, why it is hard for LLMs, and four ways to make it affordable:
- Fake quantization and the straight-through estimator (STE): the two tricks that make rounding trainable at all.
- LLM-QAT: full QAT of LLaMA models without the original training data, by distilling the FP16 model on text it writes itself.
- QLoRA and QA-LoRA: freeze a low-bit base and train only small adapters; QA-LoRA arranges things so the adapters can be merged back into a low-bit model.
- LoftQ: choose the adapters' starting values so that “low-bit base + adapter” starts close to the original model.
This is not only research. Google released Gemma 3 QAT checkpoints in 2025: about 5,000 QAT steps, using the probabilities of the unquantized checkpoint as targets, which cut the perplexity drop of 4-bit (Q4_0) quantization by 54%. The 27B model then needs 14.1 GB for its weights instead of 54 GB in BF16. [Certain] Meta’s quantized Llama 3.2 1B and 3B models for phones were made with QAT followed by LoRA adapters. [Certain] The efficient-inference survey (Zhou et al., Section 5.2.1) divides model quantization into PTQ and QAT in the same way. [Certain]
PTQ versus QAT: the idea
A quantized weight can take only a few values: the points of a grid. PTQ takes each trained weight and moves it to a nearby grid point. Each weight’s error may be small, but the errors were never trained against. They add up through the layers, and the model has no chance to adapt.
In QAT, the network is trained (or fine-tuned) while its weights are rounded. If rounding one weight up hurts, the loss goes up, and the gradient pushes other weights to compensate. The model ends up at a set of grid points that work well together, not just a set of points that are each near the original.
Two problems must be solved for this to work. First, the weights must stay in high precision during training, or small gradient steps would vanish below the grid spacing. Second, the rounding function has a gradient of zero almost everywhere, so backpropagation through it gives nothing. Fake quantization solves the first problem; the straight-through estimator solves the second.
Fake quantization
Keep a full-precision copy of each weight: the latent weight \(w\), usually in FP32. In every forward pass, quantize it and immediately dequantize it back to a floating-point number. The result \(\hat w\) is a float, but it can only take grid values. The rest of the network runs in normal floating point on \(\hat w\). Nothing is stored as integers yet; that is why it is called fake (or simulated) quantization. For a symmetric signed \(b\)-bit grid with step size (scale) \(s\):
- Read the symbols
- \(x\)
- a latent weight or an activation, in full precision
- \(s\)
- the step size (scale): the distance between two neighbouring grid points
- \(\lfloor \cdot \rceil\)
- round to the nearest integer
- \(\operatorname{clamp}(k, -Q, Q)\)
- cut \(k\) to the range \([-Q, Q]\); values outside are clipped
- \(Q\)
- the largest integer code: 7 for 4 bits, 127 for 8 bits (one code, \(-2^{b-1}\), is left unused to keep the grid symmetric)
- \(\hat x\)
- the fake-quantized value: a float that lies exactly on the grid
At deployment, the integer code \(k = \operatorname{clamp}(\lfloor x/s \rceil, -Q, Q)\) and the scale \(s\) are stored, and the kernel computes with them. Because the training forward pass used exactly \(s \cdot k\), the deployed model computes the same numbers the trained model saw (up to the order of floating-point additions). Asymmetric grids add a zero point; session 103 covers the general form.
Take \(x = (0.42, -1.30, 0.07, 2.10, -0.55)\) and the MinMax scale \(s = \max|x| / Q = 2.10 / 7 = 0.3\). Then \(x/s = (1.40, -4.33, 0.23, 7.00, -1.83)\), the codes are \(k = (1, -4, 0, 7, -2)\), and \(\hat x = (0.3, -1.2, 0.0, 2.1, -0.6)\). The errors are \((-0.12, 0.10, -0.07, 0, -0.05)\), with mean squared error 0.00636.
Now use a smaller step, \(s = 0.25\). The range shrinks to \(\pm 7 \times 0.25 = \pm 1.75\), so \(2.10\) is clipped to \(1.75\), while the other values get a finer grid: \(\hat x = (0.5, -1.25, 0.0, 1.75, -0.5)\). Choosing \(s\) is a trade-off between rounding error (fine grid) and clipping error (wide range). Figure 1 measures it.
How to read Figure 1
Start at 4 bits and ratio 1.00 (MinMax): no value is clipped, and the MSE is 0.0246, all from rounding. Move the ratio down: the steps get finer and the MSE falls until about 0.70 (MSE 0.0144), then clipping error takes over. At 2 bits the grid has only three values, \(\{-s, 0, s\}\), and the best ratio is 0.33: clipping 22% of the values is better than a grid that is too coarse. The readout also shows the LSQ gradient of the MSE with respect to \(s\) (formula 111.4): it is positive when \(s\) is too large, so a learned scale moves toward the best ratio by itself. Real LLM activations are not Gaussian: rare outliers carry important information, and LLM-QAT found that clipping them hurts (below).
The gradient problem and the straight-through estimator
The staircase in Figure 1 is flat between its jumps. Its true derivative is 0 almost everywhere and undefined at the jumps. If we backpropagate honestly, every gradient that reaches a fake-quantized weight becomes 0, and nothing trains.
The straight-through estimator (STE), named and studied by Bengio, Léonard and Courville (2013), simply pretends that the rounding step is the identity in the backward pass. The forward pass uses the rounded value; the backward pass passes the gradient straight through. With clipping, the usual version passes the gradient only where the value is inside the range:
Why is this not crazy? Over a range of many steps, the staircase follows the line \(\hat x = x\) closely; the identity is its average slope. The STE gives the gradient of that smooth line. The gradient tells the latent weight which way to move; once it has moved far enough, the rounded value jumps to the next grid point. The latent weights then take normal optimizer steps:
This is why the latent weights must be kept in high precision. With \(s = 0.01\) and a learning rate of \(10^{-5}\), one step moves a weight by about \(10^{-5}\), a thousandth of a grid step. If we stored only the rounded weight, the change would be rounded away every time. The latent weight accumulates many small steps until it crosses a rounding boundary.
Learning the step size (LSQ)
The scale \(s\) can be learned too. Learned step size quantization (LSQ), by Esser et al. (2019), applies the same STE idea to \(s\). Write \(u = x/s\) and \(k = \operatorname{clamp}(\lfloor u \rceil, -Q, Q)\). Treating rounding as the identity gives:
Inside the range, the gradient is the signed rounding error in units of \(s\): values near a grid point barely care about \(s\); values halfway between two points care most. Clipped values push \(s\) up with weight \(Q\). In the second worked example (\(s = 0.25\)), formula (111.4) gives \((0.32, 0.20, -0.28, 7, 0.20)\): the clipped value \(2.10\) dominates and asks for a larger range. LSQ also scales this gradient down by a factor that depends on the number of weights and on \(Q\), so that \(s\) learns at a speed similar to the weights. [Likely]
- input: latent FP32 weights \(W\) for every linear layer, scales \(s\) (fixed or learned), a batch, optimizer state
- for each linear layer: \(\hat W \leftarrow \operatorname{fq}(W; s_W)\) // formula (111.1), per output channel or per group
- \(\quad\) \(\hat X \leftarrow \operatorname{fq}(X; s_X)\) // only if activations will also be quantized at deployment
- \(\quad\) \(Y \leftarrow \hat X \hat W^\top\) // normal floating-point matmul
- \(\mathcal{L} \leftarrow\) loss on the batch (task loss, or distillation loss from an FP teacher)
- backward pass; at every \(\operatorname{fq}\), pass the gradient straight through inside the range, 0 outside // formula (111.2)
- gradient for each scale \(s\) by formula (111.4), if scales are learned
- optimizer step on the latent \(W\) (and \(s\)) // formula (111.3); \(W\) stays FP32
- after training: store \(k = \operatorname{clamp}(\lfloor W/s \rceil, -Q, Q)\) and \(s\); throw away the latent \(W\)
Jacob et al. (2017) made this recipe standard for integer-only inference on phones. Their version keeps running averages (exponential moving averages) of each activation’s min and max during training to set the activation scales, and folds batch normalization into the weights before quantizing them, so the trained model matches the integer model exactly. [Certain]
Why QAT beats rounding to nearest
Round-to-nearest (RTN) minimizes the error of each weight separately. But the loss does not care about each weight separately; it cares about the layer’s output. If two inputs are strongly correlated, an error of \(+0.1\) on one weight can be cancelled by \(-0.1\) on the other. RTN cannot do this. QAT can, because the gradient sees the output error. (GPTQ, from session 106, does the same compensation in closed form, one column at a time.)
Figure 2 shows this on the smallest possible model: one output that is a weighted sum of 16 inputs. The FP weights \(\mathbf{w}^*\) are given. The inputs have correlation \(\rho\) with each other, so the covariance is \(\Sigma = (1-\rho) I + \rho \mathbf{1}\mathbf{1}^\top\), and the expected output error of quantized weights \(\hat{\mathbf{w}}\) is \((\hat{\mathbf{w}} - \mathbf{w}^*)^\top \Sigma\, (\hat{\mathbf{w}} - \mathbf{w}^*)\). PTQ rounds \(\mathbf{w}^*\) once. QAT starts from \(\mathbf{w}^*\) and runs Algorithm 1 (full-batch gradient descent with the STE, fixed MinMax scale) for 300 steps.
How to read Figure 2
With the defaults (4 bits, \(\rho = 0.9\)), PTQ leaves 1.267% error and QAT ends at 0.090%, about 14 times less. Now set \(\rho = 0\): the inputs are independent, \(\Sigma = I\), the error is a sum over weights, and rounding each weight to its nearest grid point is already the best possible answer. QAT cannot win; it ends slightly worse (0.659% against 0.643%) because weights jump around their grid points. Turn off the learning-rate decay at \(\rho = 0.9\): the lowest error seen is still 0.090%, but the final error is back at 1.267%, and the count of grid flips rises from 184 to 430. Weights near a rounding boundary flip back and forth every step. Nagel et al. (2022) call this oscillation and show it is a real problem in QAT; decaying the learning rate, or freezing weights that oscillate, helps. [Certain]
Two lessons carry over to real models. QAT helps most when errors can cancel, which is the normal case in a network, because inputs and channels are correlated. And QAT needs a learning-rate schedule that ends small, or the model ends in the middle of an oscillation.
Why QAT is hard for LLMs
For a CNN on a phone, QAT means fine-tuning a few million weights on ImageNet for a few epochs. For an LLM, all three ingredients become expensive:
| Problem | Why it bites for LLMs | What helps |
|---|---|---|
| Memory | QAT is training: BF16 weights, gradients, FP32 latent weights and two Adam moments are about 16 bytes per parameter. LLaMA 7B (6.74B parameters) needs about 108 GB before activations. | Sharding (FSDP, ZeRO); or freeze the base and train adapters (QLoRA, QA-LoRA, LoftQ) |
| Data | The pretraining data is usually not available, or is trillions of tokens. A small fine-tuning set covers a narrow domain, and QAT on it can damage general skills. | Distill from the FP16 model on its own generated text (LLM-QAT); use the FP model’s probabilities as targets (Gemma 3 QAT) |
| Outliers | Some activation channels are 10 to 100 times larger than the rest (session 108). Clipping them, which works for CNNs, hurts LLMs. | Per-token activation scales, MinMax without clipping, rotations (session 109) |
| Time | Each QAT step is a full forward and backward pass of the model, plus quantize–dequantize work. | Short QAT runs (thousands of steps, not a new pretraining run) |
LLM-QAT: data-free QAT by self-distillation
LLM-QAT (Liu et al., Meta, 2023) was one of the first works to run full QAT on LLaMA 7B, 13B and 30B. [Certain] It solves the data problem with a simple idea: let the FP16 model write its own training data, then train the quantized student to match the FP16 model’s output distribution on that data.
Step 1: generate data from the model itself
Start from a random first token from the vocabulary and let the FP16 model generate the rest. The paper tried pure greedy (top-1) decoding and found it produces repetitive text; pure sampling is more diverse. Its best choice is a hybrid: take the top-1 token for the first 3 to 5 tokens, then sample the rest from the model’s distribution. It generates about 100,000 sequences, up to 1,024 tokens long (or until an end-of-sentence token). [Certain] Because the text comes from the model, it covers what the model knows, across all topics, with no need for the pretraining set.
Step 2: distill with the teacher’s full distribution
The student is the same network with fake-quantized weights, activations and KV cache. It is trained to match the teacher’s next-token distribution at every position, with a cross-entropy against the teacher’s soft predictions (session 59):
- Read the symbols
- \(X_i\)
- the context at position \(i\) of a generated sequence; \(n\) positions in the batch
- \(p^{\mathcal{T}}_c\)
- the FP16 teacher’s probability for token \(c\) (fixed; no gradient)
- \(p^{\mathcal{S}}_c\)
- the quantized student’s probability for token \(c\)
- \(V\)
- the vocabulary size: the loss uses all \(V\) probabilities, not only the token that was generated
Minus the teacher’s entropy (a constant), this is the KL divergence from teacher to student. The paper found that also matching attention maps or hidden states made results worse, so it uses the logits only. [Certain]
How to read Figure 3
Choose Greedy: from the start word, the sequence falls into the loop “a sat a sat …”, and 300 greedy sequences contain only 8 distinct 3-grams (one per start word). Training data like this would teach the student only a few contexts. Sampling gives 341 distinct 3-grams. The hybrid gives 291 here; in a bigram toy it cannot show its real advantage, which is that a confident start makes long LLM generations more coherent. The second point: at every position, the loss compares the whole blue and orange distributions. The student learns about every possible next token, not only the one that was written, which is why a modest amount of generated text is enough.
- input: FP16 model \(\mathcal{T}\), bit widths for weights, activations and KV cache
- repeat about 100,000 times: pick a random first token; for each next position, take the top-1 token of \(\mathcal{T}\) for the first 3–5 tokens, then sample from \(p^{\mathcal{T}}\); stop at end-of-sentence or 1,024 tokens
- student \(\mathcal{S} \leftarrow\) copy of \(\mathcal{T}\) with fake quantization on every linear layer’s weights (per output channel), its input activations (per token), and the keys and values (per token)
- for each batch of generated text: run \(\mathcal{T}\) (no gradient) and \(\mathcal{S}\); loss (111.5) on all positions
- \(\quad\) backward with the STE (111.2); AdamW step on the latent weights (paper: lr \(2 \times 10^{-5}\), cosine decay, no weight decay, batch size 1 per GPU on 8 GPUs)
- output: integer weights and scales; activation and KV quantizers fixed by the same rules
Quantizers: MinMax, no clipping
LLM-QAT uses symmetric MinMax quantization: the scale of each group is set by its largest absolute value, so nothing is clipped:
This is the opposite of Figure 1’s lesson for Gaussian data. In LLMs, the large values are rare but carry information, and the paper found that clipping them hurt. A learned-scale method (StatsQ for weights) scored 56.8 against 64.4 for MinMax at 4-bit weights and 8-bit activations on LLaMA 7B. [Certain]
Results
Zero-shot common-sense accuracy (average over the paper’s tasks, higher is better), written as bits for weights–activations–KV cache: [Certain]
| Model | Setting (W-A-KV) | RTN | GPTQ | SmoothQuant | LLM-QAT | FP16 |
|---|---|---|---|---|---|---|
| LLaMA 7B | 4-16-16 (weights only) | 63.0 | 60.9 | – | 64.4 | 66.2 |
| LLaMA 7B | 4-8-4 | 41.2 | – | 43.2 | 60.7 | 66.2 |
| LLaMA 30B | 4-8-4 | 42.5 | – | 40.7 | 69.9 | 71.4 |
Read the table carefully. With 4-bit weights only, LLM-QAT is better than PTQ, but the gap is small (1.4 points over RTN). The big win is in the hard setting with 4-bit KV cache and 8-bit activations: PTQ collapses to about 41–43, while QAT keeps 60.7 on 7B and 69.9 on 30B (1.5 points below FP16). QAT pays off most exactly where PTQ fails.
A low-bit base plus trainable adapters
Full QAT updates every weight, so it costs as much memory as full fine-tuning. A cheaper family freezes a low-bit base and trains a small low-rank adapter \(BA\) on top (session 57). The base never changes; only the adapter learns to work around the base’s rounding errors. This is a weaker form of quantization awareness: the base was rounded without training, but the trainable part is trained through the quantized base.
QLoRA, in one formula
Session 58 built QLoRA in detail. In short: store the frozen base in 4-bit NF4 with double-quantized scales, dequantize it to BF16 on the fly, and train a BF16 LoRA adapter:
Only \(A\) and \(B\) get gradients. For LLaMA 7B, the NF4 base takes about \(6.74 \times 10^9 \times 4.127 / 8 \approx 3.5\) GB instead of 13.5 GB in BF16. QLoRA has two weak spots, and the next two methods fix one each:
- Merging. After training, the serving model is \(\hat W + sBA\). The update \(sBA\) changes every weight by a different amount, so the sum is no longer on the 4-bit grid. Either you serve the 4-bit base and a 16-bit adapter side by side (slower, two code paths), or you quantize \(\hat W + sBA\) again and lose part of what the adapter learned. QA-LoRA fixes this.
- Initialization. LoRA starts with \(B = 0\), so training starts from the quantized model \(\hat W\), not from the original \(W\). At 2 or 3 bits, \(\hat W\) is far from \(W\), and training starts from a damaged model. LoftQ fixes this.
QA-LoRA: adapters that merge into a low-bit model
QA-LoRA (Xu et al., 2023) starts from one observation. Use group-wise asymmetric quantization: each group of \(G\) consecutive input weights in a row has its own scale \(\alpha\) and zero point \(\beta\), and \(\tilde w = \alpha k + \beta\) with integer \(k\). If an update adds the same number \(c\) to every weight of a group, then \(\tilde w + c = \alpha k + (\beta + c)\): the codes \(k\) and the scale stay; only the zero point moves. The merged weight is still exactly on a (shifted) low-bit grid.
So QA-LoRA constrains the adapter to give one value per (output row, group). The easy way to do that: sum the input inside each group before the adapter. The adapter’s \(A\) then sees \(d_{\text{in}}/G\) numbers instead of \(d_{\text{in}}\):
Each input in group \(g\) is multiplied by the same adapter weight \((sBA)_{o,g}\) for output \(o\). That is exactly a shift of the group’s zero point, so after training the adapter disappears into the zero points:
- Read the symbols
- \(G\)
- the group size along the input dimension (32 in the paper’s main experiments)
- \(\alpha_{o,g}, \beta_{o,g}\)
- scale and zero point of group \(g\) in output row \(o\); \(\tilde w = \alpha k + \beta\) with \(k \in \{0, \dots, 2^b - 1\}\)
- \(\mathcal{P}(X)\)
- the input summed over each group: a parameter-free pooling from \(d_{\text{in}}\) to \(d_{\text{in}}/G\) numbers
- \((BA)_{o,g}\)
- one adapter value per output row and group; \(BA\) has shape \(d_{\text{out}} \times d_{\text{in}}/G\)
The paper’s Algorithm 1 writes the zero point in a different convention, so its update looks different; the idea is the same. [Likely] A side effect: \(A\) is \(G\) times smaller. For a 4096 × 4096 layer with \(r = 16\) and \(G = 32\), the adapter has \(16 \times 128 + 4096 \times 16 = 67{,}584\) parameters instead of \(16 \times 8192 = 131{,}072\). The constraint costs flexibility, but group-wise quantization gives the base more freedom (more scales) in return.
Take one group \(w = (0.10, -0.20, 0.04, 0.30)\) with 2 bits (codes 0 to 3). Then \(\beta = -0.20\), \(\alpha = 0.50 / 3 = 0.1667\), the codes are \(k = (2, 0, 1, 3)\), and \(\tilde w = (0.1333, -0.2000, -0.0333, 0.3000)\).
QA-LoRA: the adapter adds \(c = 0.012\) to the whole group. Merged: \((0.1453, -0.1880, -0.0213, 0.3120)\), which is exactly \(\alpha k + \beta'\) with \(\beta' = -0.188\). Same codes, same scale, no error.
A LoRA-style adapter adds a different change to each weight: \((+0.012, -0.006, +0.009, +0.003)\). Merged: \((0.1453, -0.2060, -0.0243, 0.3030)\), off the grid. Re-quantize with MinMax: \(\beta = -0.206\), \(\alpha = 0.1697\), codes \((2, 0, 1, 3)\), giving \((0.1333, -0.2060, -0.0363, 0.3030)\). The first weight’s change of \(+0.012\) is completely lost, and the third weight’s \(+0.009\) became \(-0.003\): the wrong sign.
How to read Figure 4
With the LoRA adapter, every orange circle sits between grid lines: all 16 merged weights are off the grid. At 3 bits and groups of 8, re-quantizing moves them by 29% of the adapter’s own change; at 2 bits it is 87%, so the adapter is mostly erased. Switch to QA-LoRA: the whole grid of each group slides (dashed orange lines), and every merged weight sits exactly on it. Smaller groups give QA-LoRA more freedom (more independent shifts) and the base more scales, at the cost of storing more zero points.
Does it work? On LLaMA 7B fine-tuned on FLAN v2, 5-shot MMLU accuracy (%) from the QA-LoRA paper, Table 1: [Certain]
| Method | Model at inference | Alpaca | FLAN v2 |
|---|---|---|---|
| LLaMA 7B, no fine-tuning | 16-bit | 34.6 | |
| QLoRA | NF4 base + 16-bit adapter (“4+16”) | 38.4 | 44.3 |
| QLoRA, then GPTQ on the merged model | 4-bit | 36.0 | 41.4 |
| QA-LoRA | 4-bit, merged | 39.4 | 47.0 |
| QLoRA, then GPTQ | 3-bit | 34.0 | 36.9 |
| QA-LoRA | 3-bit, merged | 37.4 | 44.7 |
| QLoRA, then GPTQ | 2-bit | 25.8 | 25.0 |
| QA-LoRA | 2-bit, merged | 27.5 | 33.2 |
Re-quantizing a QLoRA model after training (row 3) loses 2.4 to 2.9 points at 4 bits; QA-LoRA’s merged 4-bit model is even slightly better than QLoRA’s unmerged one in this table. The QA-LoRA base uses group size 32 and GPTQ for the initial quantization. [Certain] At 2 bits, both are poor on Alpaca: an adapter cannot rescue a base that is too damaged.
LoftQ: start the adapter where the error is
LoftQ (Li et al., 2023) attacks the second weak spot. With \(B = 0\), training starts from \(\hat W\), whose error \(W - \hat W\) is large at low bits. But the adapter \(BA\) is a rank-\(r\) matrix we are free to initialize. Why not choose it to cancel as much of the quantization error as possible? LoftQ chooses the quantized base \(Q\) and the adapter together:
This is hard to solve jointly, but each half is easy when the other is fixed. With \(BA\) fixed, the best \(Q\) (approximately) quantizes what the adapter does not cover. With \(Q\) fixed, the best rank-\(r\) \(BA\) is the truncated SVD of the residual (the Eckart–Young theorem). LoftQ alternates:
- Read the symbols
- \(q(\cdot)\)
- the quantize–dequantize function of the base (NF4, NF2 or uniform in the paper)
- \(\sigma_i, \mathbf{u}_i, \mathbf{v}_i\)
- the \(i\)-th singular value and singular vectors of the residual \(W - Q_t\)
- \(B_t, A_t\)
- the split of the rank-\(r\) part: LoftQ uses \(B = [\sqrt{\sigma_i}\,\mathbf{u}_i]\) and \(A = [\sqrt{\sigma_i}\,\mathbf{v}_i]^{\top}\)
- \(t = 1, \dots, T\)
- alternating steps; the paper uses \(T = 5\) for its GLUE experiments and \(T = 1\) for BART
Note that \(t = 1\) already gives most of the gain: \(Q_1\) is the plain quantization of \(W\) (as in QLoRA), and \(B_1 A_1\) is the best rank-\(r\) fix of its error. After this, \(Q_T\) is frozen and \(A, B\) are trained exactly as in QLoRA. Only the starting point changes; training cost is the same.
- input: pretrained \(W\), bit width \(b\), rank \(r\), steps \(T\)
- \(BA \leftarrow 0\)
- for \(t = 1, \dots, T\):
- \(\quad\) \(Q \leftarrow q(W - BA)\) // quantize what the adapter does not cover
- \(\quad\) \(U, \Sigma, V^\top \leftarrow \operatorname{SVD}(W - Q)\)
- \(\quad\) \(B \leftarrow U_{:, :r} \Sigma_{:r}^{1/2}\), \(\ A \leftarrow \Sigma_{:r}^{1/2} V^\top_{:r, :}\) // formula (111.11)
- output: frozen low-bit \(Q\) and trainable \(A, B\); then fine-tune as in QLoRA
How to read Figure 5
At 2 bits, the QLoRA start is 50.6% away from \(W\) (relative Frobenius error). One LoftQ step with rank 16 brings it to 30.4%. At 4 bits the start is 10.2% and LoftQ reaches 6.0%: a similar share closed, but a much smaller absolute damage, which is why LoftQ matters most at low bits. Set the rank to 0: no adapter, no change. Look at the later steps: here the error at 2 bits rises slightly after step 2, because re-quantizing \(W - BA\) does not always lower the joint error. More steps are not always better; this toy matrix has no low-rank structure, so treat its numbers as an illustration, not a prediction for real weights.
The paper’s results on LLaMA 2 with rank 64 show the same pattern: at 4 bits, LoftQ and QLoRA are close (WikiText-2 perplexity 5.24 against 5.70 on 7B; GSM8K 35.0 against 35.1); at 2 bits, QLoRA fine-tuning did not converge, while LoftQ reached perplexity 7.85 (7B) and 7.69 (13B). On DeBERTaV3 with 2-bit NF2 and rank 16, MNLI accuracy was 84.7 for LoftQ against 75.4 for QLoRA. [Certain]
When PTQ is enough, and when QAT is worth the cost
QAT costs training; PTQ costs minutes to hours on one GPU. Use the cheapest method that meets your quality target. A rough rule, from the evidence in this and earlier sessions: [Likely]
| Target | Usually enough | Consider QAT when |
|---|---|---|
| 8-bit weights (W8A16), or FP8 weights and activations | PTQ (often plain RTN or per-channel scales) | almost never |
| 4-bit weights, 16-bit activations, 7B+ model | GPTQ or AWQ (session 107) | you need the last 1–2 points, or the model is small (1–3B) where errors matter more |
| W8A8 with outlier handling | SmoothQuant or rotations (QuaRot, SpinQuant) | accuracy drop is still too large for the task |
| 4-bit activations or 4-bit KV cache | rotations + PTQ may work (session 110) | PTQ collapses, as in LLM-QAT’s W4A8KV4 row |
| 2–3-bit weights | PTQ is fragile | almost always: QAT, or LoftQ/QA-LoRA if you fine-tune anyway |
| You are fine-tuning anyway | QLoRA, then serve NF4 + adapter | QA-LoRA if you need one merged low-bit model; LoftQ at low bits |
| A model you will ship to millions of devices | PTQ as a baseline | the one-time training cost is small next to the serving cost (Gemma 3 QAT, Llama 3.2 quantized) |
At the extreme end, QAT is not a repair step at all. BitNet b1.58 (Ma et al., 2024) trains a model from scratch with ternary weights \(\{-1, 0, +1\}\), using fake quantization and the STE throughout pretraining, and reports matching an FP16 model of the same size and training tokens. [Certain] for the method; how well this holds at the largest scales is still being studied. [Speculation]
Runnable code: fake quant, STE, QAT, LoftQ and QA-LoRA
This code runs on a CPU in under 10 seconds (most of it in the 1024 × 1024 SVDs) and prints the numbers shown in the comments. Part 1 checks the worked example and the LSQ gradient. Part 2 compares PTQ and QAT (by distillation) for a 3-bit MLP. Part 3 is LoftQ. Part 4 trains nothing but checks that a QA-LoRA adapter merges into the zero points exactly.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) fake quantization with a straight-through estimator, formulas (111.1) and (111.2) def round_ste(u): return u + (torch.round(u) - u).detach() # forward: round(u); backward: identity def fake_quant(x, s, bits): Q = 2 ** (bits - 1) - 1 # symmetric signed range [-Q, Q] return s * torch.clamp(round_ste(x / s), -Q, Q) # clamp passes gradient only inside the range x = torch.tensor([0.42, -1.30, 0.07, 2.10, -0.55], requires_grad=True) s = torch.tensor(0.25, requires_grad=True) # small scale, so 2.10 is clipped y = fake_quant(x, s, bits=4) y.sum().backward() print(y.tolist()) # [0.5, -1.25, 0.0, 1.75, -0.5] print(x.grad.tolist()) # [1.0, 1.0, 1.0, 0.0, 1.0]: STE, zero where clipped # LSQ-style gradient for the scale, formula (111.4): one value per element print([round(v, 2) for v in torch.autograd.functional.jacobian( lambda s_: fake_quant(x.detach(), s_, 4), s.detach()).tolist()]) # [0.32, 0.2, -0.28, 7.0, 0.2] # 2) a QAT linear layer: FP32 latent weights, fake-quantized per output channel in every forward class QATLinear(nn.Linear): def __init__(self, n_in, n_out, bits=3): super().__init__(n_in, n_out); self.bits = bits def forward(self, x): Q = 2 ** (self.bits - 1) - 1 s = self.weight.detach().abs().amax(dim=1, keepdim=True) / Q # MinMax scale, formula (111.6) return F.linear(x, fake_quant(self.weight, s, self.bits), self.bias) def quantize_weights_(model, bits): # PTQ baseline: round once, no training Q = 2 ** (bits - 1) - 1 with torch.no_grad(): for m in model.modules(): if isinstance(m, nn.Linear): s = m.weight.abs().amax(dim=1, keepdim=True) / Q m.weight.copy_(s * torch.clamp(torch.round(m.weight / s), -Q, Q)) teacher = nn.Sequential(nn.Linear(32, 64), nn.GELU(), nn.Linear(64, 32)) X = torch.randn(4096, 32) @ (torch.eye(32) + 0.5 * torch.randn(32, 32)) # correlated inputs with torch.no_grad(): Y = teacher(X) ptq = nn.Sequential(nn.Linear(32, 64), nn.GELU(), nn.Linear(64, 32)); ptq.load_state_dict(teacher.state_dict()) quantize_weights_(ptq, bits=3) with torch.no_grad(): print("PTQ 3-bit MSE", round(F.mse_loss(ptq(X), Y).item(), 4)) # 0.025 student = nn.Sequential(QATLinear(32, 64), nn.GELU(), QATLinear(64, 32)) student.load_state_dict(teacher.state_dict()) # start QAT from the FP weights opt = torch.optim.AdamW(student.parameters(), lr=1e-3, weight_decay=0.0) sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=600) # decay: fewer oscillations for step in range(600): # distill: match the FP teacher's outputs idx = torch.randint(0, 4096, (256,)) loss = F.mse_loss(student(X[idx]), Y[idx]) opt.zero_grad(); loss.backward(); opt.step(); sched.step() with torch.no_grad(): print("QAT 3-bit MSE", round(F.mse_loss(student(X), Y).item(), 4)) # 0.0132 # 3) LoftQ initialization, formulas (111.10) and (111.11) def quant_uniform(W, bits): # 2^bits levels in [-c, c], c = absmax per row K = 2 ** bits - 1 c = W.abs().amax(dim=1, keepdim=True) k = torch.clamp(torch.round((W / c + 1) / 2 * K), 0, K) return c * (-1 + 2 * k / K) def loftq_init(W, bits, r, T): BA = torch.zeros_like(W) for _ in range(T): Qw = quant_uniform(W - BA, bits) # quantize what the adapter does not cover U, S, Vh = torch.linalg.svd(W - Qw) # best rank-r fit of the residual B, A = U[:, :r] * S[:r].sqrt(), S[:r].sqrt()[:, None] * Vh[:r] BA = B @ A return Qw, B, A W = torch.randn(1024, 1024, dtype=torch.float64) * 0.02 for bits in (2, 4): e_qlora = (W - quant_uniform(W, bits)).norm() / W.norm() Qw, B, A = loftq_init(W, bits, r=64, T=5) print(bits, "bit QLoRA init", round(e_qlora.item(), 4), " LoftQ init", round(((W - Qw - B @ A).norm() / W.norm()).item(), 4)) # 2 bit QLoRA init 0.6762 LoftQ init 0.6208 # 4 bit QLoRA init 0.1327 LoftQ init 0.1059 # 4) QA-LoRA: pool inputs per group, then merge the adapter into the zero points, (111.8) and (111.9) d_in, d_out, G, r, bits, sc = 256, 128, 32, 8, 4, 2.0 Wf = torch.randn(d_out, d_in, dtype=torch.float64) * 0.02 Wg = Wf.view(d_out, d_in // G, G) # groups along the input dimension K = 2 ** bits - 1 beta = Wg.amin(-1, keepdim=True); alpha = (Wg.amax(-1, keepdim=True) - beta) / K Wint = torch.clamp(torch.round((Wg - beta) / alpha), 0, K) # integer codes, never change A = torch.randn(r, d_in // G, dtype=torch.float64) * 0.1 # input dimension is d_in / G, not d_in Bm = torch.randn(d_out, r, dtype=torch.float64) * 0.1 # pretend these were trained xin = torch.randn(4, d_in, dtype=torch.float64) pooled = xin.view(4, d_in // G, G).sum(-1) # sum inside each group y_train = xin @ (alpha * Wint + beta).view(d_out, d_in).T + sc * pooled @ A.T @ Bm.T beta_new = beta + sc * (Bm @ A).unsqueeze(-1) # one shift per (row, group) y_merged = xin @ (alpha * Wint + beta_new).view(d_out, d_in).T print("QA-LoRA merge max error", (y_train - y_merged).abs().max().item()) # about 3e-15
A 1024 × 1024 Gaussian matrix has no low-rank structure, so rank 64 closes only a small part of the gap in part 3 (from 0.676 to 0.621 at 2 bits). In production you would use a library: PyTorch’s torchao has QAT “quantizers” that insert fake quantization into a model and later convert it to real low-bit kernels, and the PEFT library supports LoftQ initialization. [Likely]
Where this is used
| Application | Method | Why |
|---|---|---|
| Gemma 3 QAT checkpoints (1B to 27B) | about 5,000 QAT steps with the unquantized model’s probabilities as targets; int4 and Q4_0 | 54% smaller perplexity drop than plain 4-bit; 27B fits a 24 GB GPU [Certain] |
| Llama 3.2 1B and 3B on phones | QAT during SFT, then LoRA adapters in BF16, then DPO; 4-bit group-wise weights (group 32), 8-bit per-token activations | Meta reports 2–4× speed-ups on device [Certain] |
| Fine-tuning a 7B–70B model on one GPU | QLoRA; LoftQ initialization at 2–3 bits | Memory; LoftQ gives a better start at low bits |
| Shipping one merged low-bit fine-tuned model | QA-LoRA | No 16-bit adapter path at inference; one INT4 kernel |
| Very low-bit LLMs trained from scratch | BitNet b1.58 (ternary weights, STE in pretraining) | Matmuls become additions [Certain]; the hardware to exploit this is still emerging [Likely] |
| On-device video classification and action recognition | INT8 QAT of the CNN or ViT backbone (Jacob et al. recipe) | Integer-only kernels on phone DSPs and NPUs; frames per second and battery matter |
| Video-RAG and video retrieval at scale | QAT or QAT-style fine-tuning of the embedding model to INT8 or binary codes | Billions of stored frame or clip embeddings; smaller codes make search cheaper [Likely] |
| Video LLMs on edge devices | QAT plus LoRA on the language model; PTQ often enough for the vision encoder | Long visual-token contexts make 4-bit KV cache attractive, which is where PTQ is weakest [Speculation] |
Common mistakes
- Storing only the rounded weights during QAT. Small updates are rounded away and nothing learns. Keep FP32 (or at least BF16) latent weights and fake-quantize them in the forward pass.
- A training quantizer that differs from the deployment kernel. Different granularity (per channel versus per group), a different rounding rule, or a symmetric grid in training and asymmetric in deployment: the model was trained for numbers it will never see. Match the kernel exactly.
- Forgetting to fake-quantize activations or the KV cache when they will be quantized at deployment. Weight-only QAT does not prepare the model for 4-bit KV cache.
- No learning-rate decay. Weights oscillate around rounding boundaries (Figure 2); the last checkpoint can be worse than PTQ. Decay to a small learning rate, and evaluate the fake-quantized model, not the latent one.
- QAT on a narrow task set only. The model adapts to rounding on that data and may forget the rest. Distill from the FP model on broad data, or on its own generations (LLM-QAT).
- Clipping LLM activation outliers because it helped for CNNs. For LLMs, MinMax per token was better in LLM-QAT; clip only if you measured that it helps.
- Merging a QLoRA adapter and re-quantizing without measuring. As the worked example showed, re-quantization can erase or even flip the adapter’s changes. Measure the merged low-bit model, or use QA-LoRA.
- Skipping the PTQ baseline. At 4-bit weights, GPTQ or AWQ is often within a point of QAT for a fraction of the cost. Always know what PTQ gives first.
Interview questions
What is fake quantization, and why is it called “fake”?basic
How do you backpropagate through round()? What is the straight-through estimator?basic
u + (u.round() - u).detach(). It is a biased estimate: the gradient of the smooth line the staircase follows on average.Why must QAT keep full-precision latent weights?core
Why can QAT beat round-to-nearest even with the same grid?core
What does LSQ learn, and what is its gradient?advanced
Explain LLM-QAT. Why does it generate its own data?core
Why is greedy decoding a bad way to generate QAT data?core
Why can you not simply merge a QLoRA adapter into a 4-bit model? How does QA-LoRA fix it?advanced
What does LoftQ change compared with QLoRA, and when does it matter?core
Your team wants 4-bit weights for a 70B chat model. PTQ or QAT?advanced
What is QAT oscillation and how do you reduce it?advanced
Exercises
- 3-bit symmetric fake quantization with \(s = 0.5\). Compute \(\hat x\) and the STE gradient \(\partial \hat x/\partial x\) for \(x = 1.3\) and \(x = 2.2\).
- For the same two values, compute the LSQ gradient \(\partial \hat x / \partial s\) (formula 111.4). Which one asks for a larger scale?
- A LLaMA MLP up-projection has \(d_{\text{in}} = 4096\), \(d_{\text{out}} = 11008\). Count the adapter parameters for LoRA and for QA-LoRA with \(r = 64\) and group size 32.
- Estimate the memory for full QAT of a 13B model with AdamW and mixed precision, before activations. Why is QA-LoRA or LoftQ attractive here?
- In Figure 2, set \(\rho = 0\). Explain, without running anything, why QAT cannot end below the PTQ line.
- In Figure 5, what does LoftQ give with rank 0? With \(T = 1\), which part of the LoftQ result is identical to QLoRA?
Answerscheck
Key takeaways
- QAT puts rounding inside training: fake quantization (111.1) in the forward pass on FP32 latent weights, and the straight-through estimator (111.2) in the backward pass.
- Scales can be learned (LSQ, 111.4); the rounding–clipping trade-off depends on the data, and LLM outliers favour MinMax without clipping (111.6).
- QAT beats round-to-nearest because weights learn to cancel each other’s errors; decay the learning rate to avoid oscillation.
- LLM-QAT makes full QAT data-free: the FP16 model generates its own text, and the student matches its full distribution (111.5). It shines where PTQ collapses (W4A8KV4).
- Adapters on a frozen low-bit base are the cheap form: QLoRA (111.7) trains through the quantized base; QA-LoRA (111.8–111.9) merges into zero points; LoftQ (111.10–111.11) starts \(Q + BA\) close to \(W\).
- Try PTQ first. Use QAT for 2–3 bits, low-bit activations or KV cache, small models, or models served at very large scale.
Key formulas
References for this session
- Bengio et al. (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Liu et al. (2023). LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Xu et al. (2023). QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
- Li et al. (2023). LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
- Jacob et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
- Esser et al. (2019). Learned Step Size Quantization
- Nagel et al. (2022). Overcoming Oscillations in Quantization-Aware Training
- Ma et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
- Google (Gemma team) (2025). Gemma 3 QAT Models: Bringing state-of-the-Art AI to consumer GPUs
- Meta AI (2024). Introducing quantized Llama models with increased speed and a reduced memory footprint
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Hinton et al. (2015). Distilling the Knowledge in a Neural Network
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models