LLM & VLM courseBlog
111Track 5: LLM inference and serving · session 34 of 65

Quantization-aware training: fake quant, STE, LLM-QAT, QLoRA, LoftQ

Post-training quantization rounds a finished model and hopes for the best. Quantization-aware training lets the model see the rounding while it trains, so the weights can move to places where rounding hurts less. This session builds fake quantization and the straight-through estimator from scratch, then shows how LLM-QAT, QLoRA, QA-LoRA and LoftQ make the idea affordable for large models, and when it is worth the cost.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 58 (QLoRA), 59 (knowledge distillation), 103 (quantization math) and 106 (GPTQ)

What you will learn

  • Fake quantization: quantize then dequantize in the forward pass, while FP32 latent weights receive the updates
  • Why rounding has no useful gradient, and how the straight-through estimator (STE) and learned step sizes (LSQ) get around it
  • Why QAT beats round-to-nearest: weights learn to cancel each other's rounding errors, and why lr decay matters against oscillation
  • LLM-QAT: data-free QAT that distills the FP16 model on text it generates itself, with quantized weights, activations and KV cache
  • QLoRA, QA-LoRA and LoftQ: a frozen low-bit base plus trainable low-rank adapters, and how each handles merging and initialization
  • The cost of QAT in memory, data and GPU time, and a decision rule for when PTQ is enough

Why this matters

The sessions before this one were about post-training quantization (PTQ): take a finished model, round its numbers to a few bits, and fix what you can with a little calibration data. GPTQ and AWQ make 4-bit weights work well. But PTQ has a limit. At 2 or 3 bits, or when the activations and the KV cache must also be 4 or 8 bits, the rounding damage becomes too large to repair after the fact.

Quantization-aware training (QAT) takes a different road. It puts the rounding inside training. The model sees, in every forward pass, the same rounded numbers it will see at deployment, and gradient descent moves the weights to places where rounding hurts less. The cost is real training: gradients, optimizer states, data and GPU time. This session is about how QAT works, why it is hard for LLMs, and four ways to make it affordable:

  • Fake quantization and the straight-through estimator (STE): the two tricks that make rounding trainable at all.
  • LLM-QAT: full QAT of LLaMA models without the original training data, by distilling the FP16 model on text it writes itself.
  • QLoRA and QA-LoRA: freeze a low-bit base and train only small adapters; QA-LoRA arranges things so the adapters can be merged back into a low-bit model.
  • LoftQ: choose the adapters' starting values so that “low-bit base + adapter” starts close to the original model.

This is not only research. Google released Gemma 3 QAT checkpoints in 2025: about 5,000 QAT steps, using the probabilities of the unquantized checkpoint as targets, which cut the perplexity drop of 4-bit (Q4_0) quantization by 54%. The 27B model then needs 14.1 GB for its weights instead of 54 GB in BF16. [Certain] Meta’s quantized Llama 3.2 1B and 3B models for phones were made with QAT followed by LoRA adapters. [Certain] The efficient-inference survey (Zhou et al., Section 5.2.1) divides model quantization into PTQ and QAT in the same way. [Certain]

PTQ versus QAT: the idea

A quantized weight can take only a few values: the points of a grid. PTQ takes each trained weight and moves it to a nearby grid point. Each weight’s error may be small, but the errors were never trained against. They add up through the layers, and the model has no chance to adapt.

In QAT, the network is trained (or fine-tuned) while its weights are rounded. If rounding one weight up hurts, the loss goes up, and the gradient pushes other weights to compensate. The model ends up at a set of grid points that work well together, not just a set of points that are each near the original.

Two problems must be solved for this to work. First, the weights must stay in high precision during training, or small gradient steps would vanish below the grid spacing. Second, the rounding function has a gradient of zero almost everywhere, so backpropagation through it gives nothing. Fake quantization solves the first problem; the straight-through estimator solves the second.

Fake quantization

Keep a full-precision copy of each weight: the latent weight \(w\), usually in FP32. In every forward pass, quantize it and immediately dequantize it back to a floating-point number. The result \(\hat w\) is a float, but it can only take grid values. The rest of the network runs in normal floating point on \(\hat w\). Nothing is stored as integers yet; that is why it is called fake (or simulated) quantization. For a symmetric signed \(b\)-bit grid with step size (scale) \(s\):

\[ \hat{x} = \operatorname{fq}(x; s) = s \cdot \operatorname{clamp}\!\left( \left\lfloor \frac{x}{s} \right\rceil,\ -Q,\ Q \right), \qquad Q = 2^{b-1} - 1 \tag{111.1} \]
Read the symbols
\(x\)
a latent weight or an activation, in full precision
\(s\)
the step size (scale): the distance between two neighbouring grid points
\(\lfloor \cdot \rceil\)
round to the nearest integer
\(\operatorname{clamp}(k, -Q, Q)\)
cut \(k\) to the range \([-Q, Q]\); values outside are clipped
\(Q\)
the largest integer code: 7 for 4 bits, 127 for 8 bits (one code, \(-2^{b-1}\), is left unused to keep the grid symmetric)
\(\hat x\)
the fake-quantized value: a float that lies exactly on the grid

At deployment, the integer code \(k = \operatorname{clamp}(\lfloor x/s \rceil, -Q, Q)\) and the scale \(s\) are stored, and the kernel computes with them. Because the training forward pass used exactly \(s \cdot k\), the deployed model computes the same numbers the trained model saw (up to the order of floating-point additions). Asymmetric grids add a zero point; session 103 covers the general form.

Worked example: fake quantization of five numbers to 4 bits

Take \(x = (0.42, -1.30, 0.07, 2.10, -0.55)\) and the MinMax scale \(s = \max|x| / Q = 2.10 / 7 = 0.3\). Then \(x/s = (1.40, -4.33, 0.23, 7.00, -1.83)\), the codes are \(k = (1, -4, 0, 7, -2)\), and \(\hat x = (0.3, -1.2, 0.0, 2.1, -0.6)\). The errors are \((-0.12, 0.10, -0.07, 0, -0.05)\), with mean squared error 0.00636.

Now use a smaller step, \(s = 0.25\). The range shrinks to \(\pm 7 \times 0.25 = \pm 1.75\), so \(2.10\) is clipped to \(1.75\), while the other values get a finer grid: \(\hat x = (0.5, -1.25, 0.0, 1.75, -0.5)\). Choosing \(s\) is a trade-off between rounding error (fine grid) and clipping error (wide range). Figure 1 measures it.

fake-quantized output \(\hat x\)STE gradient and clipping rangehistogram of 4,096 Gaussian values
Figure 1. Formula (111.1) on 4,096 seeded standard-normal values. Top: the staircase \(\hat x\) against \(x\). Bottom: the gradient the straight-through estimator uses (1 inside the range, 0 outside). The clip ratio sets \(s = \text{ratio} \cdot \max|x| / Q\); ratio 1 is the MinMax scale.
How to read Figure 1

Start at 4 bits and ratio 1.00 (MinMax): no value is clipped, and the MSE is 0.0246, all from rounding. Move the ratio down: the steps get finer and the MSE falls until about 0.70 (MSE 0.0144), then clipping error takes over. At 2 bits the grid has only three values, \(\{-s, 0, s\}\), and the best ratio is 0.33: clipping 22% of the values is better than a grid that is too coarse. The readout also shows the LSQ gradient of the MSE with respect to \(s\) (formula 111.4): it is positive when \(s\) is too large, so a learned scale moves toward the best ratio by itself. Real LLM activations are not Gaussian: rare outliers carry important information, and LLM-QAT found that clipping them hurts (below).

The gradient problem and the straight-through estimator

The staircase in Figure 1 is flat between its jumps. Its true derivative is 0 almost everywhere and undefined at the jumps. If we backpropagate honestly, every gradient that reaches a fake-quantized weight becomes 0, and nothing trains.

The straight-through estimator (STE), named and studied by Bengio, Léonard and Courville (2013), simply pretends that the rounding step is the identity in the backward pass. The forward pass uses the rounded value; the backward pass passes the gradient straight through. With clipping, the usual version passes the gradient only where the value is inside the range:

\[ \frac{\partial \mathcal{L}}{\partial x} \approx \frac{\partial \mathcal{L}}{\partial \hat x} \cdot \mathbb{1}\!\left[ -Q \le \frac{x}{s} \le Q \right] \tag{111.2} \]

Why is this not crazy? Over a range of many steps, the staircase follows the line \(\hat x = x\) closely; the identity is its average slope. The STE gives the gradient of that smooth line. The gradient tells the latent weight which way to move; once it has moved far enough, the rounded value jumps to the next grid point. The latent weights then take normal optimizer steps:

\[ w \leftarrow w - \eta \, \frac{\partial \mathcal{L}(\operatorname{fq}(w; s))}{\partial \hat w} \cdot \mathbb{1}\!\left[\,|w / s| \le Q\,\right] \tag{111.3} \]

This is why the latent weights must be kept in high precision. With \(s = 0.01\) and a learning rate of \(10^{-5}\), one step moves a weight by about \(10^{-5}\), a thousandth of a grid step. If we stored only the rounded weight, the change would be rounded away every time. The latent weight accumulates many small steps until it crosses a rounding boundary.

Learning the step size (LSQ)

The scale \(s\) can be learned too. Learned step size quantization (LSQ), by Esser et al. (2019), applies the same STE idea to \(s\). Write \(u = x/s\) and \(k = \operatorname{clamp}(\lfloor u \rceil, -Q, Q)\). Treating rounding as the identity gives:

\[ \frac{\partial \hat x}{\partial s} = \begin{cases} \lfloor u \rceil - u & \text{if } -Q \le u \le Q \\ k = \pm Q & \text{if the value is clipped} \end{cases} \tag{111.4} \]

Inside the range, the gradient is the signed rounding error in units of \(s\): values near a grid point barely care about \(s\); values halfway between two points care most. Clipped values push \(s\) up with weight \(Q\). In the second worked example (\(s = 0.25\)), formula (111.4) gives \((0.32, 0.20, -0.28, 7, 0.20)\): the clipped value \(2.10\) dominates and asks for a larger range. LSQ also scales this gradient down by a factor that depends on the number of weights and on \(Q\), so that \(s\) learns at a speed similar to the weights. [Likely]

Algorithm 1: one QAT training step with fake quantization and the STE
  1. input: latent FP32 weights \(W\) for every linear layer, scales \(s\) (fixed or learned), a batch, optimizer state
  2. for each linear layer: \(\hat W \leftarrow \operatorname{fq}(W; s_W)\) // formula (111.1), per output channel or per group
  3. \(\quad\) \(\hat X \leftarrow \operatorname{fq}(X; s_X)\) // only if activations will also be quantized at deployment
  4. \(\quad\) \(Y \leftarrow \hat X \hat W^\top\) // normal floating-point matmul
  5. \(\mathcal{L} \leftarrow\) loss on the batch (task loss, or distillation loss from an FP teacher)
  6. backward pass; at every \(\operatorname{fq}\), pass the gradient straight through inside the range, 0 outside // formula (111.2)
  7. gradient for each scale \(s\) by formula (111.4), if scales are learned
  8. optimizer step on the latent \(W\) (and \(s\)) // formula (111.3); \(W\) stays FP32
  9. after training: store \(k = \operatorname{clamp}(\lfloor W/s \rceil, -Q, Q)\) and \(s\); throw away the latent \(W\)

Jacob et al. (2017) made this recipe standard for integer-only inference on phones. Their version keeps running averages (exponential moving averages) of each activation’s min and max during training to set the activation scales, and folds batch normalization into the weights before quantizing them, so the trained model matches the integer model exactly. [Certain]

Why QAT beats rounding to nearest

Round-to-nearest (RTN) minimizes the error of each weight separately. But the loss does not care about each weight separately; it cares about the layer’s output. If two inputs are strongly correlated, an error of \(+0.1\) on one weight can be cancelled by \(-0.1\) on the other. RTN cannot do this. QAT can, because the gradient sees the output error. (GPTQ, from session 106, does the same compensation in closed form, one column at a time.)

Figure 2 shows this on the smallest possible model: one output that is a weighted sum of 16 inputs. The FP weights \(\mathbf{w}^*\) are given. The inputs have correlation \(\rho\) with each other, so the covariance is \(\Sigma = (1-\rho) I + \rho \mathbf{1}\mathbf{1}^\top\), and the expected output error of quantized weights \(\hat{\mathbf{w}}\) is \((\hat{\mathbf{w}} - \mathbf{w}^*)^\top \Sigma\, (\hat{\mathbf{w}} - \mathbf{w}^*)\). PTQ rounds \(\mathbf{w}^*\) once. QAT starts from \(\mathbf{w}^*\) and runs Algorithm 1 (full-batch gradient descent with the STE, fixed MinMax scale) for 300 steps.

Bits
QAT (fake quant + STE)PTQ: round to nearest, no training
Figure 2. A 16-weight linear layer with seeded FP weights (standard deviation 0.5). The y-axis is the output error after quantization as a share of the output variance, on a log scale. The learning rate is given in units of \(1/\lambda_{\max}\), where \(\lambda_{\max} = 1 + 15\rho\) is the largest eigenvalue of \(\Sigma\).
How to read Figure 2

With the defaults (4 bits, \(\rho = 0.9\)), PTQ leaves 1.267% error and QAT ends at 0.090%, about 14 times less. Now set \(\rho = 0\): the inputs are independent, \(\Sigma = I\), the error is a sum over weights, and rounding each weight to its nearest grid point is already the best possible answer. QAT cannot win; it ends slightly worse (0.659% against 0.643%) because weights jump around their grid points. Turn off the learning-rate decay at \(\rho = 0.9\): the lowest error seen is still 0.090%, but the final error is back at 1.267%, and the count of grid flips rises from 184 to 430. Weights near a rounding boundary flip back and forth every step. Nagel et al. (2022) call this oscillation and show it is a real problem in QAT; decaying the learning rate, or freezing weights that oscillate, helps. [Certain]

Two lessons carry over to real models. QAT helps most when errors can cancel, which is the normal case in a network, because inputs and channels are correlated. And QAT needs a learning-rate schedule that ends small, or the model ends in the middle of an oscillation.

Why QAT is hard for LLMs

For a CNN on a phone, QAT means fine-tuning a few million weights on ImageNet for a few epochs. For an LLM, all three ingredients become expensive:

ProblemWhy it bites for LLMsWhat helps
MemoryQAT is training: BF16 weights, gradients, FP32 latent weights and two Adam moments are about 16 bytes per parameter. LLaMA 7B (6.74B parameters) needs about 108 GB before activations.Sharding (FSDP, ZeRO); or freeze the base and train adapters (QLoRA, QA-LoRA, LoftQ)
DataThe pretraining data is usually not available, or is trillions of tokens. A small fine-tuning set covers a narrow domain, and QAT on it can damage general skills.Distill from the FP16 model on its own generated text (LLM-QAT); use the FP model’s probabilities as targets (Gemma 3 QAT)
OutliersSome activation channels are 10 to 100 times larger than the rest (session 108). Clipping them, which works for CNNs, hurts LLMs.Per-token activation scales, MinMax without clipping, rotations (session 109)
TimeEach QAT step is a full forward and backward pass of the model, plus quantize–dequantize work.Short QAT runs (thousands of steps, not a new pretraining run)

LLM-QAT: data-free QAT by self-distillation

LLM-QAT (Liu et al., Meta, 2023) was one of the first works to run full QAT on LLaMA 7B, 13B and 30B. [Certain] It solves the data problem with a simple idea: let the FP16 model write its own training data, then train the quantized student to match the FP16 model’s output distribution on that data.

Step 1: generate data from the model itself

Start from a random first token from the vocabulary and let the FP16 model generate the rest. The paper tried pure greedy (top-1) decoding and found it produces repetitive text; pure sampling is more diverse. Its best choice is a hybrid: take the top-1 token for the first 3 to 5 tokens, then sample the rest from the model’s distribution. It generates about 100,000 sequences, up to 1,024 tokens long (or until an end-of-sentence token). [Certain] Because the text comes from the model, it covers what the model knows, across all topics, with no need for the pretraining set.

Step 2: distill with the teacher’s full distribution

The student is the same network with fake-quantized weights, activations and KV cache. It is trained to match the teacher’s next-token distribution at every position, with a cross-entropy against the teacher’s soft predictions (session 59):

\[ \mathcal{L}_{\text{CE}} = -\frac{1}{n} \sum_{i=1}^{n} \sum_{c=1}^{V} p^{\mathcal{T}}_c(X_i) \, \log p^{\mathcal{S}}_c(X_i) \tag{111.5} \]
Read the symbols
\(X_i\)
the context at position \(i\) of a generated sequence; \(n\) positions in the batch
\(p^{\mathcal{T}}_c\)
the FP16 teacher’s probability for token \(c\) (fixed; no gradient)
\(p^{\mathcal{S}}_c\)
the quantized student’s probability for token \(c\)
\(V\)
the vocabulary size: the loss uses all \(V\) probabilities, not only the token that was generated

Minus the teacher’s entropy (a constant), this is the KL divergence from teacher to student. The paper found that also matching attention maps or hidden states made results worse, so it uses the logits only. [Certain]

Decoding
Figure 3. A toy version of LLM-QAT’s data generation. The “teacher” is a seeded bigram model over 8 words (each row of its logit table is a next-token distribution). The “student” is the same table with each row fake-quantized to 3 bits. Each step adds one token and shows the distillation loss (111.5) at that position.
How to read Figure 3

Choose Greedy: from the start word, the sequence falls into the loop “a sat a sat …”, and 300 greedy sequences contain only 8 distinct 3-grams (one per start word). Training data like this would teach the student only a few contexts. Sampling gives 341 distinct 3-grams. The hybrid gives 291 here; in a bigram toy it cannot show its real advantage, which is that a confident start makes long LLM generations more coherent. The second point: at every position, the loss compares the whole blue and orange distributions. The student learns about every possible next token, not only the one that was written, which is why a modest amount of generated text is enough.

Algorithm 2: LLM-QAT
  1. input: FP16 model \(\mathcal{T}\), bit widths for weights, activations and KV cache
  2. repeat about 100,000 times: pick a random first token; for each next position, take the top-1 token of \(\mathcal{T}\) for the first 3–5 tokens, then sample from \(p^{\mathcal{T}}\); stop at end-of-sentence or 1,024 tokens
  3. student \(\mathcal{S} \leftarrow\) copy of \(\mathcal{T}\) with fake quantization on every linear layer’s weights (per output channel), its input activations (per token), and the keys and values (per token)
  4. for each batch of generated text: run \(\mathcal{T}\) (no gradient) and \(\mathcal{S}\); loss (111.5) on all positions
  5. \(\quad\) backward with the STE (111.2); AdamW step on the latent weights (paper: lr \(2 \times 10^{-5}\), cosine decay, no weight decay, batch size 1 per GPU on 8 GPUs)
  6. output: integer weights and scales; activation and KV quantizers fixed by the same rules

Quantizers: MinMax, no clipping

LLM-QAT uses symmetric MinMax quantization: the scale of each group is set by its largest absolute value, so nothing is clipped:

\[ s = \frac{\max |X_{\text{group}}|}{2^{b-1} - 1}, \qquad \text{group} = \text{one output channel (weights) or one token (activations, keys, values)} \tag{111.6} \]

This is the opposite of Figure 1’s lesson for Gaussian data. In LLMs, the large values are rare but carry information, and the paper found that clipping them hurt. A learned-scale method (StatsQ for weights) scored 56.8 against 64.4 for MinMax at 4-bit weights and 8-bit activations on LLaMA 7B. [Certain]

Results

Zero-shot common-sense accuracy (average over the paper’s tasks, higher is better), written as bits for weights–activations–KV cache: [Certain]

ModelSetting (W-A-KV)RTNGPTQSmoothQuantLLM-QATFP16
LLaMA 7B4-16-16 (weights only)63.060.9–64.466.2
LLaMA 7B4-8-441.2–43.260.766.2
LLaMA 30B4-8-442.5–40.769.971.4

Read the table carefully. With 4-bit weights only, LLM-QAT is better than PTQ, but the gap is small (1.4 points over RTN). The big win is in the hard setting with 4-bit KV cache and 8-bit activations: PTQ collapses to about 41–43, while QAT keeps 60.7 on 7B and 69.9 on 30B (1.5 points below FP16). QAT pays off most exactly where PTQ fails.

A low-bit base plus trainable adapters

Full QAT updates every weight, so it costs as much memory as full fine-tuning. A cheaper family freezes a low-bit base and trains a small low-rank adapter \(BA\) on top (session 57). The base never changes; only the adapter learns to work around the base’s rounding errors. This is a weaker form of quantization awareness: the base was rounded without training, but the trainable part is trained through the quantized base.

QLoRA, in one formula

Session 58 built QLoRA in detail. In short: store the frozen base in 4-bit NF4 with double-quantized scales, dequantize it to BF16 on the fly, and train a BF16 LoRA adapter:

\[ Y = X \hat{W}^{\top} + s \, X A^{\top} B^{\top}, \qquad \hat{W} = \operatorname{dequant}(W_{\text{NF4}}), \quad A \in \R^{r \times d_{\text{in}}},\ B \in \R^{d_{\text{out}} \times r} \tag{111.7} \]

Only \(A\) and \(B\) get gradients. For LLaMA 7B, the NF4 base takes about \(6.74 \times 10^9 \times 4.127 / 8 \approx 3.5\) GB instead of 13.5 GB in BF16. QLoRA has two weak spots, and the next two methods fix one each:

  • Merging. After training, the serving model is \(\hat W + sBA\). The update \(sBA\) changes every weight by a different amount, so the sum is no longer on the 4-bit grid. Either you serve the 4-bit base and a 16-bit adapter side by side (slower, two code paths), or you quantize \(\hat W + sBA\) again and lose part of what the adapter learned. QA-LoRA fixes this.
  • Initialization. LoRA starts with \(B = 0\), so training starts from the quantized model \(\hat W\), not from the original \(W\). At 2 or 3 bits, \(\hat W\) is far from \(W\), and training starts from a damaged model. LoftQ fixes this.

QA-LoRA: adapters that merge into a low-bit model

QA-LoRA (Xu et al., 2023) starts from one observation. Use group-wise asymmetric quantization: each group of \(G\) consecutive input weights in a row has its own scale \(\alpha\) and zero point \(\beta\), and \(\tilde w = \alpha k + \beta\) with integer \(k\). If an update adds the same number \(c\) to every weight of a group, then \(\tilde w + c = \alpha k + (\beta + c)\): the codes \(k\) and the scale stay; only the zero point moves. The merged weight is still exactly on a (shifted) low-bit grid.

So QA-LoRA constrains the adapter to give one value per (output row, group). The easy way to do that: sum the input inside each group before the adapter. The adapter’s \(A\) then sees \(d_{\text{in}}/G\) numbers instead of \(d_{\text{in}}\):

\[ Y = X \tilde{W}^{\top} + s\, \mathcal{P}(X)\, A^{\top} B^{\top}, \qquad \mathcal{P}(X)_{:,g} = \sum_{j \in \text{group } g} X_{:,j}, \qquad A \in \R^{r \times d_{\text{in}}/G} \tag{111.8} \]

Each input in group \(g\) is multiplied by the same adapter weight \((sBA)_{o,g}\) for output \(o\). That is exactly a shift of the group’s zero point, so after training the adapter disappears into the zero points:

\[ \beta'_{o,g} = \beta_{o,g} + s\,(BA)_{o,g}, \qquad \tilde{W}' = \alpha \, k + \beta' \quad \text{(same integer codes } k\text{, same scales } \alpha) \tag{111.9} \]
Read the symbols
\(G\)
the group size along the input dimension (32 in the paper’s main experiments)
\(\alpha_{o,g}, \beta_{o,g}\)
scale and zero point of group \(g\) in output row \(o\); \(\tilde w = \alpha k + \beta\) with \(k \in \{0, \dots, 2^b - 1\}\)
\(\mathcal{P}(X)\)
the input summed over each group: a parameter-free pooling from \(d_{\text{in}}\) to \(d_{\text{in}}/G\) numbers
\((BA)_{o,g}\)
one adapter value per output row and group; \(BA\) has shape \(d_{\text{out}} \times d_{\text{in}}/G\)

The paper’s Algorithm 1 writes the zero point in a different convention, so its update looks different; the idea is the same. [Likely] A side effect: \(A\) is \(G\) times smaller. For a 4096 × 4096 layer with \(r = 16\) and \(G = 32\), the adapter has \(16 \times 128 + 4096 \times 16 = 67{,}584\) parameters instead of \(16 \times 8192 = 131{,}072\). The constraint costs flexibility, but group-wise quantization gives the base more freedom (more scales) in return.

Worked example: merging into a 2-bit group of four weights

Take one group \(w = (0.10, -0.20, 0.04, 0.30)\) with 2 bits (codes 0 to 3). Then \(\beta = -0.20\), \(\alpha = 0.50 / 3 = 0.1667\), the codes are \(k = (2, 0, 1, 3)\), and \(\tilde w = (0.1333, -0.2000, -0.0333, 0.3000)\).

QA-LoRA: the adapter adds \(c = 0.012\) to the whole group. Merged: \((0.1453, -0.1880, -0.0213, 0.3120)\), which is exactly \(\alpha k + \beta'\) with \(\beta' = -0.188\). Same codes, same scale, no error.

A LoRA-style adapter adds a different change to each weight: \((+0.012, -0.006, +0.009, +0.003)\). Merged: \((0.1453, -0.2060, -0.0243, 0.3030)\), off the grid. Re-quantize with MinMax: \(\beta = -0.206\), \(\alpha = 0.1697\), codes \((2, 0, 1, 3)\), giving \((0.1333, -0.2060, -0.0363, 0.3030)\). The first weight’s change of \(+0.012\) is completely lost, and the third weight’s \(+0.009\) became \(-0.003\): the wrong sign.

Adapter
Group size Bits
base weight on its gridafter merging the adaptergrid lines of each group
Figure 4. Sixteen seeded weights of one output row (standard deviation 0.02), quantized with min–max per group. The adapter’s change per weight is seeded noise (standard deviation 0.006); the QA-LoRA adapter can only give one shared change per group, here the group mean of the same changes. Re-quantization uses min–max again.
How to read Figure 4

With the LoRA adapter, every orange circle sits between grid lines: all 16 merged weights are off the grid. At 3 bits and groups of 8, re-quantizing moves them by 29% of the adapter’s own change; at 2 bits it is 87%, so the adapter is mostly erased. Switch to QA-LoRA: the whole grid of each group slides (dashed orange lines), and every merged weight sits exactly on it. Smaller groups give QA-LoRA more freedom (more independent shifts) and the base more scales, at the cost of storing more zero points.

Does it work? On LLaMA 7B fine-tuned on FLAN v2, 5-shot MMLU accuracy (%) from the QA-LoRA paper, Table 1: [Certain]

MethodModel at inferenceAlpacaFLAN v2
LLaMA 7B, no fine-tuning16-bit34.6
QLoRANF4 base + 16-bit adapter (“4+16”)38.444.3
QLoRA, then GPTQ on the merged model4-bit36.041.4
QA-LoRA4-bit, merged39.447.0
QLoRA, then GPTQ3-bit34.036.9
QA-LoRA3-bit, merged37.444.7
QLoRA, then GPTQ2-bit25.825.0
QA-LoRA2-bit, merged27.533.2

Re-quantizing a QLoRA model after training (row 3) loses 2.4 to 2.9 points at 4 bits; QA-LoRA’s merged 4-bit model is even slightly better than QLoRA’s unmerged one in this table. The QA-LoRA base uses group size 32 and GPTQ for the initial quantization. [Certain] At 2 bits, both are poor on Alpaca: an adapter cannot rescue a base that is too damaged.

LoftQ: start the adapter where the error is

LoftQ (Li et al., 2023) attacks the second weak spot. With \(B = 0\), training starts from \(\hat W\), whose error \(W - \hat W\) is large at low bits. But the adapter \(BA\) is a rank-\(r\) matrix we are free to initialize. Why not choose it to cancel as much of the quantization error as possible? LoftQ chooses the quantized base \(Q\) and the adapter together:

\[ \min_{Q,\, B,\, A} \ \big\lVert W - Q - BA \big\rVert_F, \qquad Q \text{ on the low-bit grid}, \quad \operatorname{rank}(BA) \le r \tag{111.10} \]

This is hard to solve jointly, but each half is easy when the other is fixed. With \(BA\) fixed, the best \(Q\) (approximately) quantizes what the adapter does not cover. With \(Q\) fixed, the best rank-\(r\) \(BA\) is the truncated SVD of the residual (the Eckart–Young theorem). LoftQ alternates:

\[ Q_t = q\big(W - B_{t-1} A_{t-1}\big), \qquad B_t A_t = \operatorname{SVD}_r\big(W - Q_t\big) = \sum_{i=1}^{r} \sigma_i \mathbf{u}_i \mathbf{v}_i^{\top}, \qquad B_0 A_0 = 0 \tag{111.11} \]
Read the symbols
\(q(\cdot)\)
the quantize–dequantize function of the base (NF4, NF2 or uniform in the paper)
\(\sigma_i, \mathbf{u}_i, \mathbf{v}_i\)
the \(i\)-th singular value and singular vectors of the residual \(W - Q_t\)
\(B_t, A_t\)
the split of the rank-\(r\) part: LoftQ uses \(B = [\sqrt{\sigma_i}\,\mathbf{u}_i]\) and \(A = [\sqrt{\sigma_i}\,\mathbf{v}_i]^{\top}\)
\(t = 1, \dots, T\)
alternating steps; the paper uses \(T = 5\) for its GLUE experiments and \(T = 1\) for BART

Note that \(t = 1\) already gives most of the gain: \(Q_1\) is the plain quantization of \(W\) (as in QLoRA), and \(B_1 A_1\) is the best rank-\(r\) fix of its error. After this, \(Q_T\) is frozen and \(A, B\) are trained exactly as in QLoRA. Only the starting point changes; training cost is the same.

Algorithm 3: LoftQ initialization (once per weight matrix, before fine-tuning)
  1. input: pretrained \(W\), bit width \(b\), rank \(r\), steps \(T\)
  2. \(BA \leftarrow 0\)
  3. for \(t = 1, \dots, T\):
  4. \(\quad\) \(Q \leftarrow q(W - BA)\) // quantize what the adapter does not cover
  5. \(\quad\) \(U, \Sigma, V^\top \leftarrow \operatorname{SVD}(W - Q)\)
  6. \(\quad\) \(B \leftarrow U_{:, :r} \Sigma_{:r}^{1/2}\), \(\ A \leftarrow \Sigma_{:r}^{1/2} V^\top_{:r, :}\) // formula (111.11)
  7. output: frozen low-bit \(Q\) and trainable \(A, B\); then fine-tune as in QLoRA
Bits of the base
Figure 5. LoftQ, formula (111.11), on a seeded 64 × 64 Gaussian matrix. The base uses uniform quantization with \(2^b\) levels in \([-c, c]\), where \(c\) is each row’s largest absolute value. The SVD is computed in your browser (Jacobi eigenvalue method). Grey bar: the QLoRA start (\(BA = 0\)). Orange bars: LoftQ after each step.
How to read Figure 5

At 2 bits, the QLoRA start is 50.6% away from \(W\) (relative Frobenius error). One LoftQ step with rank 16 brings it to 30.4%. At 4 bits the start is 10.2% and LoftQ reaches 6.0%: a similar share closed, but a much smaller absolute damage, which is why LoftQ matters most at low bits. Set the rank to 0: no adapter, no change. Look at the later steps: here the error at 2 bits rises slightly after step 2, because re-quantizing \(W - BA\) does not always lower the joint error. More steps are not always better; this toy matrix has no low-rank structure, so treat its numbers as an illustration, not a prediction for real weights.

The paper’s results on LLaMA 2 with rank 64 show the same pattern: at 4 bits, LoftQ and QLoRA are close (WikiText-2 perplexity 5.24 against 5.70 on 7B; GSM8K 35.0 against 35.1); at 2 bits, QLoRA fine-tuning did not converge, while LoftQ reached perplexity 7.85 (7B) and 7.69 (13B). On DeBERTaV3 with 2-bit NF2 and rank 16, MNLI accuracy was 84.7 for LoftQ against 75.4 for QLoRA. [Certain]

When PTQ is enough, and when QAT is worth the cost

QAT costs training; PTQ costs minutes to hours on one GPU. Use the cheapest method that meets your quality target. A rough rule, from the evidence in this and earlier sessions: [Likely]

TargetUsually enoughConsider QAT when
8-bit weights (W8A16), or FP8 weights and activationsPTQ (often plain RTN or per-channel scales)almost never
4-bit weights, 16-bit activations, 7B+ modelGPTQ or AWQ (session 107)you need the last 1–2 points, or the model is small (1–3B) where errors matter more
W8A8 with outlier handlingSmoothQuant or rotations (QuaRot, SpinQuant)accuracy drop is still too large for the task
4-bit activations or 4-bit KV cacherotations + PTQ may work (session 110)PTQ collapses, as in LLM-QAT’s W4A8KV4 row
2–3-bit weightsPTQ is fragilealmost always: QAT, or LoftQ/QA-LoRA if you fine-tune anyway
You are fine-tuning anywayQLoRA, then serve NF4 + adapterQA-LoRA if you need one merged low-bit model; LoftQ at low bits
A model you will ship to millions of devicesPTQ as a baselinethe one-time training cost is small next to the serving cost (Gemma 3 QAT, Llama 3.2 quantized)

At the extreme end, QAT is not a repair step at all. BitNet b1.58 (Ma et al., 2024) trains a model from scratch with ternary weights \(\{-1, 0, +1\}\), using fake quantization and the STE throughout pretraining, and reports matching an FP16 model of the same size and training tokens. [Certain] for the method; how well this holds at the largest scales is still being studied. [Speculation]

Runnable code: fake quant, STE, QAT, LoftQ and QA-LoRA

This code runs on a CPU in under 10 seconds (most of it in the 1024 × 1024 SVDs) and prints the numbers shown in the comments. Part 1 checks the worked example and the LSQ gradient. Part 2 compares PTQ and QAT (by distillation) for a 3-bit MLP. Part 3 is LoftQ. Part 4 trains nothing but checks that a QA-LoRA adapter merges into the zero points exactly.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) fake quantization with a straight-through estimator, formulas (111.1) and (111.2)
def round_ste(u):
    return u + (torch.round(u) - u).detach()      # forward: round(u); backward: identity

def fake_quant(x, s, bits):
    Q = 2 ** (bits - 1) - 1                        # symmetric signed range [-Q, Q]
    return s * torch.clamp(round_ste(x / s), -Q, Q)  # clamp passes gradient only inside the range

x = torch.tensor([0.42, -1.30, 0.07, 2.10, -0.55], requires_grad=True)
s = torch.tensor(0.25, requires_grad=True)         # small scale, so 2.10 is clipped
y = fake_quant(x, s, bits=4)
y.sum().backward()
print(y.tolist())          # [0.5, -1.25, 0.0, 1.75, -0.5]
print(x.grad.tolist())     # [1.0, 1.0, 1.0, 0.0, 1.0]: STE, zero where clipped
# LSQ-style gradient for the scale, formula (111.4): one value per element
print([round(v, 2) for v in torch.autograd.functional.jacobian(
    lambda s_: fake_quant(x.detach(), s_, 4), s.detach()).tolist()])   # [0.32, 0.2, -0.28, 7.0, 0.2]

# 2) a QAT linear layer: FP32 latent weights, fake-quantized per output channel in every forward
class QATLinear(nn.Linear):
    def __init__(self, n_in, n_out, bits=3):
        super().__init__(n_in, n_out); self.bits = bits
    def forward(self, x):
        Q = 2 ** (self.bits - 1) - 1
        s = self.weight.detach().abs().amax(dim=1, keepdim=True) / Q   # MinMax scale, formula (111.6)
        return F.linear(x, fake_quant(self.weight, s, self.bits), self.bias)

def quantize_weights_(model, bits):                # PTQ baseline: round once, no training
    Q = 2 ** (bits - 1) - 1
    with torch.no_grad():
        for m in model.modules():
            if isinstance(m, nn.Linear):
                s = m.weight.abs().amax(dim=1, keepdim=True) / Q
                m.weight.copy_(s * torch.clamp(torch.round(m.weight / s), -Q, Q))

teacher = nn.Sequential(nn.Linear(32, 64), nn.GELU(), nn.Linear(64, 32))
X = torch.randn(4096, 32) @ (torch.eye(32) + 0.5 * torch.randn(32, 32))  # correlated inputs
with torch.no_grad(): Y = teacher(X)

ptq = nn.Sequential(nn.Linear(32, 64), nn.GELU(), nn.Linear(64, 32)); ptq.load_state_dict(teacher.state_dict())
quantize_weights_(ptq, bits=3)
with torch.no_grad(): print("PTQ  3-bit MSE", round(F.mse_loss(ptq(X), Y).item(), 4))    # 0.025

student = nn.Sequential(QATLinear(32, 64), nn.GELU(), QATLinear(64, 32))
student.load_state_dict(teacher.state_dict())      # start QAT from the FP weights
opt = torch.optim.AdamW(student.parameters(), lr=1e-3, weight_decay=0.0)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=600)   # decay: fewer oscillations
for step in range(600):                            # distill: match the FP teacher's outputs
    idx = torch.randint(0, 4096, (256,))
    loss = F.mse_loss(student(X[idx]), Y[idx])
    opt.zero_grad(); loss.backward(); opt.step(); sched.step()
with torch.no_grad(): print("QAT  3-bit MSE", round(F.mse_loss(student(X), Y).item(), 4))  # 0.0132

# 3) LoftQ initialization, formulas (111.10) and (111.11)
def quant_uniform(W, bits):                        # 2^bits levels in [-c, c], c = absmax per row
    K = 2 ** bits - 1
    c = W.abs().amax(dim=1, keepdim=True)
    k = torch.clamp(torch.round((W / c + 1) / 2 * K), 0, K)
    return c * (-1 + 2 * k / K)

def loftq_init(W, bits, r, T):
    BA = torch.zeros_like(W)
    for _ in range(T):
        Qw = quant_uniform(W - BA, bits)               # quantize what the adapter does not cover
        U, S, Vh = torch.linalg.svd(W - Qw)            # best rank-r fit of the residual
        B, A = U[:, :r] * S[:r].sqrt(), S[:r].sqrt()[:, None] * Vh[:r]
        BA = B @ A
    return Qw, B, A

W = torch.randn(1024, 1024, dtype=torch.float64) * 0.02
for bits in (2, 4):
    e_qlora = (W - quant_uniform(W, bits)).norm() / W.norm()
    Qw, B, A = loftq_init(W, bits, r=64, T=5)
    print(bits, "bit  QLoRA init", round(e_qlora.item(), 4),
          " LoftQ init", round(((W - Qw - B @ A).norm() / W.norm()).item(), 4))
# 2 bit  QLoRA init 0.6762  LoftQ init 0.6208
# 4 bit  QLoRA init 0.1327  LoftQ init 0.1059

# 4) QA-LoRA: pool inputs per group, then merge the adapter into the zero points, (111.8) and (111.9)
d_in, d_out, G, r, bits, sc = 256, 128, 32, 8, 4, 2.0
Wf = torch.randn(d_out, d_in, dtype=torch.float64) * 0.02
Wg = Wf.view(d_out, d_in // G, G)                  # groups along the input dimension
K = 2 ** bits - 1
beta = Wg.amin(-1, keepdim=True); alpha = (Wg.amax(-1, keepdim=True) - beta) / K
Wint = torch.clamp(torch.round((Wg - beta) / alpha), 0, K)    # integer codes, never change
A = torch.randn(r, d_in // G, dtype=torch.float64) * 0.1     # input dimension is d_in / G, not d_in
Bm = torch.randn(d_out, r, dtype=torch.float64) * 0.1        # pretend these were trained
xin = torch.randn(4, d_in, dtype=torch.float64)
pooled = xin.view(4, d_in // G, G).sum(-1)                   # sum inside each group
y_train = xin @ (alpha * Wint + beta).view(d_out, d_in).T + sc * pooled @ A.T @ Bm.T
beta_new = beta + sc * (Bm @ A).unsqueeze(-1)                # one shift per (row, group)
y_merged = xin @ (alpha * Wint + beta_new).view(d_out, d_in).T
print("QA-LoRA merge max error", (y_train - y_merged).abs().max().item())   # about 3e-15

A 1024 × 1024 Gaussian matrix has no low-rank structure, so rank 64 closes only a small part of the gap in part 3 (from 0.676 to 0.621 at 2 bits). In production you would use a library: PyTorch’s torchao has QAT “quantizers” that insert fake quantization into a model and later convert it to real low-bit kernels, and the PEFT library supports LoftQ initialization. [Likely]

Where this is used

ApplicationMethodWhy
Gemma 3 QAT checkpoints (1B to 27B)about 5,000 QAT steps with the unquantized model’s probabilities as targets; int4 and Q4_054% smaller perplexity drop than plain 4-bit; 27B fits a 24 GB GPU [Certain]
Llama 3.2 1B and 3B on phonesQAT during SFT, then LoRA adapters in BF16, then DPO; 4-bit group-wise weights (group 32), 8-bit per-token activationsMeta reports 2–4× speed-ups on device [Certain]
Fine-tuning a 7B–70B model on one GPUQLoRA; LoftQ initialization at 2–3 bitsMemory; LoftQ gives a better start at low bits
Shipping one merged low-bit fine-tuned modelQA-LoRANo 16-bit adapter path at inference; one INT4 kernel
Very low-bit LLMs trained from scratchBitNet b1.58 (ternary weights, STE in pretraining)Matmuls become additions [Certain]; the hardware to exploit this is still emerging [Likely]
On-device video classification and action recognitionINT8 QAT of the CNN or ViT backbone (Jacob et al. recipe)Integer-only kernels on phone DSPs and NPUs; frames per second and battery matter
Video-RAG and video retrieval at scaleQAT or QAT-style fine-tuning of the embedding model to INT8 or binary codesBillions of stored frame or clip embeddings; smaller codes make search cheaper [Likely]
Video LLMs on edge devicesQAT plus LoRA on the language model; PTQ often enough for the vision encoderLong visual-token contexts make 4-bit KV cache attractive, which is where PTQ is weakest [Speculation]

Common mistakes

  • Storing only the rounded weights during QAT. Small updates are rounded away and nothing learns. Keep FP32 (or at least BF16) latent weights and fake-quantize them in the forward pass.
  • A training quantizer that differs from the deployment kernel. Different granularity (per channel versus per group), a different rounding rule, or a symmetric grid in training and asymmetric in deployment: the model was trained for numbers it will never see. Match the kernel exactly.
  • Forgetting to fake-quantize activations or the KV cache when they will be quantized at deployment. Weight-only QAT does not prepare the model for 4-bit KV cache.
  • No learning-rate decay. Weights oscillate around rounding boundaries (Figure 2); the last checkpoint can be worse than PTQ. Decay to a small learning rate, and evaluate the fake-quantized model, not the latent one.
  • QAT on a narrow task set only. The model adapts to rounding on that data and may forget the rest. Distill from the FP model on broad data, or on its own generations (LLM-QAT).
  • Clipping LLM activation outliers because it helped for CNNs. For LLMs, MinMax per token was better in LLM-QAT; clip only if you measured that it helps.
  • Merging a QLoRA adapter and re-quantizing without measuring. As the worked example showed, re-quantization can erase or even flip the adapter’s changes. Measure the merged low-bit model, or use QA-LoRA.
  • Skipping the PTQ baseline. At 4-bit weights, GPTQ or AWQ is often within a point of QAT for a fraction of the cost. Always know what PTQ gives first.

Interview questions

What is fake quantization, and why is it called “fake”?basic
In the forward pass, each weight (and optionally activation) is quantized to the low-bit grid and immediately dequantized back to a float: \(\hat x = s \cdot \operatorname{clamp}(\lfloor x/s \rceil, -Q, Q)\). The network computes in normal floating point, but only with values that the integer model could represent. Nothing is stored as integers during training, so the quantization is simulated (“fake”). After training, the codes and scales are exported for real integer kernels.
How do you backpropagate through round()? What is the straight-through estimator?basic
The derivative of round is 0 almost everywhere, so honest backprop gives no signal. The STE treats rounding as the identity in the backward pass: \(\partial \mathcal{L}/\partial x \approx \partial \mathcal{L}/\partial \hat x\), usually masked to 0 where the value was clipped. In PyTorch: u + (u.round() - u).detach(). It is a biased estimate: the gradient of the smooth line the staircase follows on average.
Why must QAT keep full-precision latent weights?core
Optimizer steps are tiny compared with the grid spacing (for example \(10^{-5}\) against a step size of 0.01). If you stored only grid values, every update would be rounded back to the same point. The latent weight accumulates many small steps until it crosses a rounding boundary, and then the quantized value changes. This also means QAT needs the full training memory: about 16 bytes per parameter with AdamW and mixed precision.
Why can QAT beat round-to-nearest even with the same grid?core
RTN minimizes each weight’s error separately, but the loss depends on the layer output. When inputs are correlated, errors on different weights can cancel, and the best set of grid points is not the set of nearest points. QAT’s gradient sees the output error and moves weights to a combination that cancels well. In the 16-weight toy with correlation 0.9 at 4 bits, the error falls from 1.27% (RTN) to 0.09%. With independent inputs, RTN is already optimal and QAT cannot help. GPTQ exploits the same effect in closed form.
What does LSQ learn, and what is its gradient?advanced
LSQ learns the step size \(s\) of each quantizer. With the STE, \(\partial \hat x/\partial s = \lfloor x/s \rceil - x/s\) for values inside the range and \(\pm Q\) for clipped values. So clipped values push the range wider and in-range values push it toward a finer grid; the balance finds the rounding–clipping trade-off from the loss, not from a fixed rule. LSQ also scales the gradient down so \(s\) does not move too fast. For LLMs, LLM-QAT found that MinMax scales worked better than a learned-scale method (StatsQ), because outliers matter.
Explain LLM-QAT. Why does it generate its own data?core
LLM-QAT runs QAT on LLaMA with quantized weights, activations and KV cache. The original pretraining data is not available, and a small task set would narrow the model. So the FP16 model generates about 100K sequences from random first tokens (top-1 for the first 3–5 tokens, then sampling), and the quantized student is trained to match the teacher’s full next-token distribution (logit distillation, formula 111.5). Because the text comes from the model, it covers the model’s own knowledge. At W4A8KV4 on LLaMA 7B it keeps 60.7 average zero-shot accuracy against 41.2 for RTN and 43.2 for SmoothQuant (FP16: 66.2).
Why is greedy decoding a bad way to generate QAT data?core
Greedy decoding is deterministic and falls into repetitive loops, so it covers few contexts; for a given first token it always produces the same text. The student then learns to match the teacher only on that narrow set. Sampling gives diverse contexts. LLM-QAT uses top-1 for the first few tokens (a confident, coherent start) and sampling after that. The distillation loss uses the full teacher distribution at every position, so each position teaches the student about every possible next token.
Why can you not simply merge a QLoRA adapter into a 4-bit model? How does QA-LoRA fix it?advanced
\(sBA\) changes every weight by a different amount, so \(\hat W + sBA\) is off the 4-bit grid; re-quantizing it erases part of the adapter and can flip small changes (in QA-LoRA’s table, QLoRA plus GPTQ loses about 2.4–2.9 MMLU points at 4 bits). QA-LoRA uses group-wise quantization and sums the inputs inside each group before the adapter, so the adapter adds one value per (row, group). Adding a constant to a group only shifts its zero point: \(\beta' = \beta + s(BA)_{o,g}\), with the same codes and scales. The merged model is exactly low-bit, and \(A\) is \(G\) times smaller.
What does LoftQ change compared with QLoRA, and when does it matter?core
Only the initialization. QLoRA starts with \(B = 0\), so the starting model is the quantized \(\hat W\). LoftQ alternates \(Q_t = q(W - B_{t-1}A_{t-1})\) and \(B_tA_t = \operatorname{SVD}_r(W - Q_t)\), so \(Q + BA\) starts close to the original \(W\). Training then runs exactly as in QLoRA. It matters most at low bits: on LLaMA 2 at 2 bits, QLoRA did not converge while LoftQ did (WikiText-2 perplexity 7.85 on 7B); at 4 bits the two are close.
Your team wants 4-bit weights for a 70B chat model. PTQ or QAT?advanced
Start with PTQ: GPTQ or AWQ with per-group scales is usually close to the BF16 model at 4-bit weights for large models, and costs hours, while full QAT of 70B needs about 1.1 TB of training state (16 bytes per parameter) plus data. Measure on your real tasks, including long-context and reasoning evals, not only perplexity. Move to QAT (or distillation-based QAT like Gemma 3’s, or LoRA-based recovery) only if the drop is too large for the product, if you also need 4-bit activations or KV cache, or if the model will be served at such scale that a one-time training cost is negligible.
What is QAT oscillation and how do you reduce it?advanced
A latent weight near a rounding boundary gets a gradient that pushes it across; on the other side the gradient pushes it back, so its quantized value flips every few steps. The final checkpoint can then be noticeably worse than the best one seen. Remedies: decay the learning rate to a small value at the end, freeze weights that oscillate, add a small penalty that pulls latent weights toward their grid points, or re-estimate statistics such as batch-norm after training (Nagel et al., 2022). In the toy of Figure 2, removing the decay leaves the final error at the PTQ level even though the best step reached 14× less.

Exercises

  1. 3-bit symmetric fake quantization with \(s = 0.5\). Compute \(\hat x\) and the STE gradient \(\partial \hat x/\partial x\) for \(x = 1.3\) and \(x = 2.2\).
  2. For the same two values, compute the LSQ gradient \(\partial \hat x / \partial s\) (formula 111.4). Which one asks for a larger scale?
  3. A LLaMA MLP up-projection has \(d_{\text{in}} = 4096\), \(d_{\text{out}} = 11008\). Count the adapter parameters for LoRA and for QA-LoRA with \(r = 64\) and group size 32.
  4. Estimate the memory for full QAT of a 13B model with AdamW and mixed precision, before activations. Why is QA-LoRA or LoftQ attractive here?
  5. In Figure 2, set \(\rho = 0\). Explain, without running anything, why QAT cannot end below the PTQ line.
  6. In Figure 5, what does LoftQ give with rank 0? With \(T = 1\), which part of the LoftQ result is identical to QLoRA?
Answerscheck
1. \(Q = 3\), range \(\pm 1.5\). For \(1.3\): \(1.3/0.5 = 2.6 \to 3\), \(\hat x = 1.5\), gradient 1 (inside the range). For \(2.2\): \(4.4 \to\) clamped to 3, \(\hat x = 1.5\), gradient 0 (clipped). 2. For \(1.3\): \(3 - 2.6 = 0.4\). For \(2.2\): clipped, so \(Q = 3\). The clipped value has the larger gradient; with a positive upstream error it pushes \(s\) up to widen the range. 3. LoRA: \(64 \times (4096 + 11008) = 966{,}656\). QA-LoRA: \(A\) is \(64 \times 128 = 8{,}192\) and \(B\) is \(11008 \times 64 = 704{,}512\), total \(712{,}704\). 4. \(13 \times 10^9 \times 16\) bytes \(\approx 208\) GB: several 80 GB GPUs with sharding. QA-LoRA and LoftQ keep a frozen low-bit base (about 7 GB at 4 bits) and train about 2% of the parameters (rank 64 on every linear layer of LLaMA 13B is about 250M). 5. With \(\Sigma = I\), the error is \(\sum_i (\hat w_i - w_i^*)^2\), a sum of independent terms. Each term is smallest at the nearest grid point, which is exactly what RTN chooses, so RTN is the global optimum for this grid; QAT can only match it or end on a worse point. 6. Rank 0 means no adapter: LoftQ equals QLoRA. With \(T = 1\), \(Q_1 = q(W)\) is the same quantized base as QLoRA; only the adapter differs (\(B_1A_1 = \operatorname{SVD}_r(W - q(W))\) instead of 0).

Key takeaways

  • QAT puts rounding inside training: fake quantization (111.1) in the forward pass on FP32 latent weights, and the straight-through estimator (111.2) in the backward pass.
  • Scales can be learned (LSQ, 111.4); the rounding–clipping trade-off depends on the data, and LLM outliers favour MinMax without clipping (111.6).
  • QAT beats round-to-nearest because weights learn to cancel each other’s errors; decay the learning rate to avoid oscillation.
  • LLM-QAT makes full QAT data-free: the FP16 model generates its own text, and the student matches its full distribution (111.5). It shines where PTQ collapses (W4A8KV4).
  • Adapters on a frozen low-bit base are the cheap form: QLoRA (111.7) trains through the quantized base; QA-LoRA (111.8–111.9) merges into zero points; LoftQ (111.10–111.11) starts \(Q + BA\) close to \(W\).
  • Try PTQ first. Use QAT for 2–3 bits, low-bit activations or KV cache, small models, or models served at very large scale.

Key formulas

\[ \hat x = s\,\operatorname{clamp}\!\left(\left\lfloor \tfrac{x}{s} \right\rceil, -Q, Q\right), \qquad \frac{\partial \mathcal{L}}{\partial x} \approx \frac{\partial \mathcal{L}}{\partial \hat x}\,\mathbb{1}\!\left[\left|\tfrac{x}{s}\right| \le Q\right], \qquad \frac{\partial \hat x}{\partial s} = \lfloor u \rceil - u \]
\[ \mathcal{L}_{\text{CE}} = -\tfrac{1}{n}\sum_i \sum_c p^{\mathcal{T}}_c \log p^{\mathcal{S}}_c, \qquad \beta' = \beta + s\,(BA)_{o,g}, \qquad Q_t = q(W - B_{t-1}A_{t-1}),\ \ B_tA_t = \operatorname{SVD}_r(W - Q_t) \]

References for this session

  1. Bengio et al. (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
  2. Liu et al. (2023). LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  3. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  4. Xu et al. (2023). QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
  5. Li et al. (2023). LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
  6. Jacob et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
  7. Esser et al. (2019). Learned Step Size Quantization
  8. Nagel et al. (2022). Overcoming Oscillations in Quantization-Aware Training
  9. Ma et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  10. Google (Gemma team) (2025). Gemma 3 QAT Models: Bringing state-of-the-Art AI to consumer GPUs
  11. Meta AI (2024). Introducing quantized Llama models with increased speed and a reduced memory footprint
  12. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  13. Hinton et al. (2015). Distilling the Knowledge in a Neural Network
  14. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  15. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  16. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models