QLoRA and training with quantized weights
LoRA freezes the base model, but the frozen weights still fill the GPU. QLoRA stores them in 4 bits with a data type made for normally distributed weights (NF4), compresses the scale factors a second time, computes in BF16, and pages optimizer states to CPU memory when a spike comes. This session builds each piece from the bits up, counts the memory, and shows where the method stops being enough.
What you will learn
- Block-wise absmax quantization: how 16-bit weights become 4-bit codes plus one scale per block, and how they are turned back
- Why NF4 (4-bit NormalFloat) has lower error than INT4 and FP4 on normally distributed weights, and how its 16 values are built from normal quantiles
- Double quantization: compressing the scales from 0.5 to about 0.127 bits per weight, about 3 GB saved on a 65B model
- The QLoRA layer: NF4 storage, BF16 compute, gradients only for the LoRA matrices, and why the frozen weight still takes part in the backward pass
- Paged optimizers: what they move to CPU memory during a spike, and what they cannot save
- A full memory budget for 7B to 70B models, and the evidence (MMLU, perplexity) on how much quality 4-bit storage costs
- When QLoRA is not enough: activations, merging, speed, very low bits, and large changes to the model
Why this matters
In session 57 we froze the base weights and trained only two small matrices per layer. That removed almost all gradients and optimizer states. But one big cost was left: the frozen weights themselves. A 65B model in BF16 is 130 GB, more than any single GPU holds. QLoRA (Dettmers et al., 2023) removes most of that cost: it stores the frozen base in 4 bits and trains LoRA adapters on top, in 16-bit precision.
The numbers are the reason this method became the default for fine-tuning on small hardware. Full 16-bit fine-tuning of LLaMA 65B needs “more than 780 GB of GPU memory”; QLoRA fine-tunes it on a single 48 GB GPU. [Certain] (780 GB is what you get with 12 bytes per parameter: 2 for BF16 weights, 2 for gradients and 8 for two FP32 Adam moments, times 65.3B parameters.) The paper’s best model, Guanaco 65B, reached 99.3% of ChatGPT’s score on the Vicuna benchmark (judged by GPT-4) after about 24 hours of fine-tuning on one GPU. [Certain]
QLoRA is three ideas on top of LoRA. Each one is a common interview topic:
- NF4 storage with BF16 compute. A 4-bit data type whose 16 values are placed where normally distributed weights actually are. Weights are stored in NF4 and turned back into BF16 for each matrix multiply.
- Double quantization. The scale factors needed by 4-bit storage are themselves quantized to 8 bits, which saves about 0.37 bits per parameter.
- Paged optimizers. Optimizer states live in memory that can move to the CPU when the GPU runs out, so a single long batch does not crash the run.
In real systems you meet QLoRA through the Hugging Face stack. This configuration loads a model in NF4:
BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
PEFT then adds the LoRA adapters, and the trainer option optim="paged_adamw_8bit" or "paged_adamw_32bit" selects a paged optimizer. [Certain] Every one of these switches maps to one section of this page.
Step 1: block-wise quantization
Quantization means storing a number with fewer bits by mapping it to one of a small set of allowed values. With 4 bits there are \(2^4 = 16\) allowed values. We call this set the code book \(\{v_0, \dots, v_{15}\}\), and we choose it inside \([-1, 1]\). A stored weight is then only an index (0 to 15) into the code book: 4 bits.
Real weights are not inside \([-1, 1]\); they are small numbers like 0.02. So we first divide by a scale. The simplest choice is the largest absolute value, the absmax. If one scale is used for a whole matrix, a single large weight makes the scale large, and all normal weights are squeezed into the few code values near 0. The fix is block-wise quantization: cut the weights into blocks of \(B\) consecutive values (QLoRA uses \(B = 64\)) and give each block its own scale. [Certain] One outlier then hurts only its own block.
- Read the symbols
- \(w_i\)
- one original weight (BF16 or FP32)
- \(b\), \(B\)
- a block of \(B\) consecutive weights; every weight belongs to exactly one block
- \(c_b\)
- the scale of block \(b\): its absmax, stored in FP32 (32 bits per block)
- \(v_j\)
- the \(j\)-th value of the 4-bit code book, inside \([-1, 1]\)
- \(q_i\)
- the stored 4-bit index: the code value nearest to the normalized weight
- \(\hat{w}_i\)
- the dequantized weight, the value the matrix multiply will use
Formula (58.1) is quantization (done once, when the model is loaded). Formula (58.2) is dequantization (done every time the weight is used). The difference \(\hat{w}_i - w_i\) is the quantization error. It is fixed: in QLoRA the base weights never change, so this error stays for the whole training run. Note that QLoRA uses plain round-to-nearest, with no calibration data, unlike inference methods such as GPTQ (session 106) or AWQ (session 107). [Certain]
The bits per weight are 4 for the index plus the scale cost shared by the block: \(32/B\) bits. With \(B = 64\) that is \(4 + 0.5 = 4.5\) bits. Smaller blocks give smaller error but more scales; we come back to this trade-off with double quantization.
Step 2: NF4, a 4-bit type for normal weights
Which 16 values should the code book contain? The usual answers are INT4 (evenly spaced integers, here \(-7/7, \dots, 7/7\)) and FP4 (a tiny floating-point format; E2M1 has the magnitudes 0, 0.5, 1, 1.5, 2, 3, 4, 6, scaled so that 6 becomes 1). Both ignore something we know: trained weights are roughly normally distributed around zero. [Likely] Most weights are near 0 and few are near the absmax. Evenly spaced levels waste many codes in the empty tails.
QLoRA’s answer is NormalFloat (NF4): put the code values at quantiles of the normal distribution, so that each code is used by about the same number of weights. A data type with this property is called information-theoretically optimal for that distribution: every code carries the same amount of information. [Certain] The paper defines the values from the quantile function \(Q_X\) of the standard normal distribution:
- Read the symbols
- \(k\)
- the number of bits (4 for NF4), so there are \(2^k = 16\) codes
- \(Q_X(p)\)
- the value below which a fraction \(p\) of a standard normal variable falls (for example \(Q_X(0.5) = 0\), \(Q_X(0.977) \approx 2\))
- \(q_i\)
- the \(i\)-th code value, the middle of two neighbouring quantiles; all values are then divided by the largest one, so they lie in \([-1, 1]\)
One more detail matters. A symmetric set of 16 values has no exact zero, but zero is important (padding, pruned weights, and the many weights that are tiny). So NF4 is built asymmetrically: 8 values for the positive side and 7 for the negative side, plus an exact 0. The bitsandbytes code builds the positive half from the normal quantiles at probabilities evenly spaced from 0.5 to 0.9677, the negative half from 7 such quantiles, adds 0, and divides by the largest value. [Certain] Our Python version of this rule reproduces the 16 values printed in the paper’s appendix to \(10^{-7}\):
Look at the spacing: about 0.08 between codes near zero, and 0.28 between the two largest. The levels are dense where weights are dense. Inside a block, the normalized weights \(w_i/c_b\) are not exactly normal (the absmax always maps to ±1, and the rest is a normal shape squeezed by the block’s maximum), but the shape is close enough. [Likely]
How to read Figure 1
With INT4 the blue bars are very uneven: the middle codes are used by many weights and the outer codes by almost none, so the 4 bits carry well under 4 bits of information. With NF4 the bars are much more even and the entropy is the highest of the three. FP4 is in between: its levels are dense near zero but very sparse near ±1. The error in the readout follows the same order. Change the block size: with larger blocks the absmax is larger compared with a typical weight, the histogram gets narrower, and more weights fall into the middle codes.
Take a block of 8 weights (real blocks have 64): \(0.012, -0.031, 0.004, 0.050, -0.018, 0.027, -0.007, 0.021\). The absmax is \(c = 0.050\). Normalized: \(0.24, -0.62, 0.08, 1.00, -0.36, 0.54, -0.14, 0.42\).
Nearest NF4 values: \(0.2461, -0.6962, 0.0796, 1.0, -0.3949, 0.5626, -0.1848, 0.4407\), stored as indices 10, 1, 8, 15, 3, 13, 5, 12. Dequantized (times 0.05): \(0.01231, -0.03481, 0.00398, 0.05, -0.01975, 0.02813, -0.00924, 0.02204\). The largest error is \(0.0038\) on the second weight. The squared error divided by the sum of squared weights is 0.0048.
With INT4 (steps of \(1/7 \approx 0.143\)) the same block gives a relative error of 0.0069, 42% higher. Over a 1024 × 1024 Gaussian matrix with \(B = 64\), the code below measures 0.0116 for INT4, 0.0112 for FP4 and 0.0085 for NF4.
- \(c \leftarrow 0\)
- for each \(w_i\) in the block: \(c \leftarrow \max(c, |w_i|)\) // absmax scale, (58.1)
- for each \(w_i\) in the block:
- \(u_i \leftarrow w_i / c\) // now inside \([-1, 1]\)
- \(q_i \leftarrow \argmin_j |u_i - v_j|\) // nearest code, (58.1)
- pack two \(q_i\) per byte; store \(c\) in FP32
- at use time: \(\hat{w}_i \leftarrow c \cdot v_{q_i}\) // (58.2)
How to read Figure 2
Press Play. First the scan finds the absmax: on the worked example it is 0.050, the fourth weight. Then each weight is divided by \(c\) and the arrow shows it snapping to the nearest code; the live box lists the two nearest code values and their distances, so you can check the choice by hand. On the worked example the codes are 10, 1, 8, 15, 3, 13, 5, 12, as in the text, and the relative error is 0.0048. Switch to INT4 (0.0069) or FP4 (0.0057) at any time: the same weights land on different codes. One block is a small sample: on the seeded block of 16, FP4 happens to beat NF4 (0.0040 against 0.0063); over many blocks NF4 wins, as Figure 1 shows. Now choose “Same block with an outlier”: one weight of 0.150 sets the scale, and the other 15 weights crowd into the 6 NF4 codes from −0.2844 to 0.1609. The block then uses only 7 of its 16 codes.
The paper measured the same order on real models. Mean perplexity on Pile Common Crawl (lower is better) was 34.34 for Int4, 31.07 for Float4 (E2M1), 29.48 for Float4 (E3M0) and 27.41 for NFloat4 with double quantization. [Certain] These were averages over several model families from 125M to 13B parameters. [Likely]
Step 3: double quantization of the scales
Each block of 64 weights needs one FP32 scale: \(32/64 = 0.5\) bits per weight. That sounds small, but 0.5 bits is 12.5% of the 4 bits we pay for the weights themselves. For a 65B model it is about 4 GB. Double quantization treats the scales as just another tensor and quantizes them too: subtract their mean (they are all positive, so centering makes better use of a symmetric code), then quantize them to 8-bit floats in blocks of 256, with one FP32 scale per block of 256 scales. [Certain]
- Read the symbols
- \(k\)
- bits of the weight code (4)
- \(B_1\)
- first-level block size: weights per scale (64)
- \(B_2\)
- second-level block size: first-level scales per FP32 scale (256)
- \(8/B_1\)
- the cost of the 8-bit first-level scales
- \(32/(B_1 B_2)\)
- the cost of the FP32 second-level scales, almost nothing
Dequantization now has two steps. First rebuild the scale of block \(b\) from its 8-bit code \(\hat{c}_{1,b}\), the FP32 scale \(c_2\) of its group of 256, and the mean \(\mu\). Then rebuild the weight as before:
Without it: \(32/64 = 0.5\) bits per weight. With it: \(8/64 + 32/(64 \times 256) = 0.125 + 0.00195 = 0.127\) bits. The saving is \(0.373\) bits per weight. On a 65B model: \(65 \times 10^9 \times 0.373 / 8 \approx 3.03 \times 10^9\) bytes, about 3 GB, the number given in the paper. [Certain] The total storage goes from 4.5 to about 4.127 bits per weight.
LLaMA 65B has 65.29B parameters; 64.76B of them are in the linear layers that get quantized. At 4.127 bits those take 33.4 GB. The embeddings and output head (0.52B parameters) usually stay in 16 bits (1.05 GB). [Likely] So the frozen base takes about 34.5 GB, against 130.6 GB in BF16.
The error added by the second quantization is tiny: in the code below the rebuilt scales differ from the true scales by at most 0.46%. Because one scale serves 64 weights, it is worth spending bits on accuracy there, and double quantization lets us keep \(B = 64\) instead of moving to larger blocks.
- for each block \(b\) of 64 weights: \(c_b \leftarrow \max_{i \in b} |w_i|\) // FP32
- \(\mu \leftarrow \frac{1}{16} \sum_b c_b\)
- \(x_b \leftarrow c_b - \mu\) // centered, both signs
- for each group of \(B_2\) scales:
- \(c_2 \leftarrow \max |x_b|\) over the group // FP32
- for \(b\) in the group: \(k_b \leftarrow \text{round}(127\, x_b / c_2)\) // 8 bits
- \(\hat{c}_b \leftarrow c_2\, k_b / 127 + \mu\) // rebuilt at use time, (58.5)
- bits per weight \(= 4 + (8 \cdot 16 + 32\, G + 32) / 1024\) // (58.4) with \(G\) groups, plus \(\mu\)
How to read Figure 3
Press Play. The bottom bar starts at 512 bits (16 FP32 scales, 4.5 bits per weight). Each time a scale becomes an 8-bit code, 24 bits disappear; each group adds 32 bits for its \(c_2\), and \(\mu\) adds 32 bits once. With \(B_2 = 8\) you end at 224 bits, 4.21875 bits per weight. The live box shows every rounding, for example \(k_0 = \text{round}(127 \times 0.00214 / 0.01063) = 26\). The largest error of a rebuilt scale is 0.119%, and the NF4 error of the 1,024 weights is 0.00846 with the true scales and with the rebuilt ones. Try \(B_2 = 16\): one group and fewer bits (4.1875), with the same largest error here. In a real model a group holds 256 scales and \(\mu\) is shared by millions of weights, which gives the 4.127 bits of (58.4).
How to read Figure 4
Moving left (larger blocks) saves bits but raises the error, because each block’s absmax grows. Without outliers the cost is mild. Tick “Add outlier weights”: now large blocks are much worse, since one outlier squeezes all other weights of its block into the codes near zero, while small blocks keep the damage local. Then tick “Double quantization”: every point moves left (the scales cost 8 bits instead of 32), and the error barely changes. That is why QLoRA can keep the small block of 64 at about 4.13 bits. Without outliers, NF4 has the lowest error at every block size. With outliers and blocks of 512 or more, FP4 catches up: most weights are then squeezed near zero, where FP4 also has dense levels. The y-axis is logarithmic.
Step 4: the QLoRA layer, NF4 storage and BF16 compute
We now have a compact frozen weight. QLoRA separates two kinds of precision. The storage type is how a tensor lives in memory between uses: NF4 for the base weights. The compute type is the precision of the arithmetic: BF16. [Certain] A GPU has no NF4 matrix multiply, so every forward and backward pass first dequantizes the weight to BF16 (58.5), then multiplies. With LoRA matrices \(A \in \R^{r \times d_\text{in}}\) and \(B \in \R^{d_\text{out} \times r}\) in BF16, one layer computes:
- Read the symbols
- \(X\)
- the input activations, shape (tokens, \(d_\text{in}\)), in BF16
- \(\hat{W}\)
- the base weight rebuilt in BF16 from its NF4 codes and double-quantized scales; frozen
- \(A\), \(B\)
- the LoRA matrices, the only trainable part; \(B\) starts at zero so training starts from the quantized base
- \(s\), \(\alpha\), \(r\)
- the LoRA scale, its numerator and the rank, as in session 57
A frequent point of confusion: do gradients go through the 4-bit weights? Yes, for the input; no, for the weight itself. The layer below needs \(\partial L / \partial X\), and that requires \(\hat{W}\). The weight gradient \(\partial L / \partial \hat{W}\) is never computed:
- Read the symbols
- \(G\)
- \(\partial L / \partial Y\), the gradient arriving from the layer above, shape (tokens, \(d_\text{out}\))
- \(G \hat{W}\)
- the part of the input gradient that goes through the frozen weight: needs a second dequantization in the backward pass
- \(X A^{\top}\)
- a small \((\text{tokens} \times r)\) matrix saved in the forward pass
So QLoRA does more work than 16-bit LoRA: the weight is dequantized in the forward pass, again in the backward pass, and again when gradient checkpointing recomputes the forward pass. QLoRA training is therefore slower per step than 16-bit LoRA on the same GPU, when both fit. [Likely] The trade is time for memory.
- \(c_b \leftarrow c_2\, k_b/127 + \mu\) // (58.5), 4 scales
- for each block \(b\): \(\hat{W}[b] \leftarrow \text{bf16}(c_b \cdot \text{NF4}[q])\) // on the fly
- \(Y \leftarrow X\hat{W}^{\top} + s\,(XA^{\top})B^{\top}\); keep \(XA^{\top}\); free \(\hat{W}\) // (58.6)
- \(L \leftarrow \text{mean}\,(Y - Y^*)^2\); \(G \leftarrow 2(Y - Y^*)/64\)
- dequantize again; \(\partial L/\partial X \leftarrow G\hat{W} + s\,GBA\) // (58.7)
- \(\partial L/\partial B \leftarrow s\,G^{\top}(XA^{\top})\), \(\partial L/\partial A \leftarrow s\,(GB)^{\top}X\); no \(\partial L/\partial \hat{W}\) // (58.7)
- Adam updates \(A\), \(B\); zero the gradients; \(q\) is never written
How to read Figure 5
Press Play and watch one training step in 10 sub-steps. The 4 scales are rebuilt from their 8-bit codes, then \(\hat{W}\) is filled block by block (the green bar grows to 32 MiB) and freed after the forward product. At step 1, \(B = 0\), so the output is just \(X\hat{W}^{\top}\) and \(\partial L/\partial A\) is exactly 0. In the backward pass \(\hat{W}\) is built a second time, because the layer below needs \(\partial L/\partial X\). The crossed box never fills: there is no gradient for the frozen weight. Only \(A\) and \(B\) change, and the codes at step 60 are the codes of step 1. Switch to “Whole training steps” and press Play to watch the loss: with rank 2 it falls from 0.32302 to 0.00807 in 60 steps. The green dashed line (0.00987) is the loss of an adapter that learned the change \(0.6\,\mathbf{u}\mathbf{v}^{\top}\) exactly and nothing else. Going below it means the adapter also cancels part of the quantization error, as in the experiment described in the text; rank 4 cancels more (0.00240), rank 1 cannot reach the line (0.01335). In memory, the layer keeps 9.75 MiB between steps, against 34 MiB for 16-bit LoRA; the peaks come from the temporary BF16 weight and the activation gradients.
- input: a pretrained model in BF16, block sizes \(B_1 = 64\), \(B_2 = 256\), LoRA rank \(r\), scale \(\alpha\)
- for each linear layer of the transformer blocks (attention and MLP projections):
- split the weight into blocks of \(B_1\); \(c_b \leftarrow\) absmax of each block // (58.1)
- \(q_i \leftarrow\) index of the nearest NF4 value to \(w_i / c_b\); pack two indices per byte
- \(\mu \leftarrow\) mean of all \(c_b\); quantize \(c_b - \mu\) to 8 bits in groups of \(B_2\), one FP32 scale \(c_2\) per group // (58.4)
- free the BF16 weight; keep \(q\), \(\hat{c}_1\), \(c_2\), \(\mu\) as frozen buffers
- add \(A\) (random, small) and \(B = 0\) in BF16 or FP32, trainable
- keep embeddings, output head and norms in 16 bits (or 32 bits for norms); freeze them
- enable gradient checkpointing; create a paged AdamW over the LoRA parameters only
- input: a batch of token ids, labels with −100 on masked positions
- for each layer in the forward pass:
- \(\hat{W} \leftarrow\) dequant to BF16 (the whole weight of this layer, freed after use) // (58.5)
- \(Y \leftarrow X\hat{W}^{\top} + s\,(XA^{\top})B^{\top}\); discard \(\hat{W}\) // (58.6)
- loss \(\leftarrow\) masked cross-entropy (session 54)
- for each layer in reverse: recompute the forward if checkpointed, dequantize again
- \(\partial L/\partial X \leftarrow G\hat{W} + s\,GBA\); gradients for \(A\), \(B\) only // (58.7)
- clip gradients; paged AdamW updates \(A\), \(B\) // states may be paged in from CPU
- zero gradients; the NF4 codes are never touched
Step 5: the memory budget
Where does the memory of a QLoRA run go? Four parts (see session 38 for a careful count of each):
- Read the symbols
- \(N_q\)
- parameters in the quantized linear layers; \(N_{16}\): parameters kept in 16 bits (embeddings, output head, norms)
- \(P\)
- LoRA parameters; 16 bytes each if the adapter, its gradient and two Adam moments are FP32 (an 8-bit optimizer reduces this)
- \(M_\text{act}\)
- activations saved for the backward pass, plus logits; grows with tokens per batch, and does not shrink with 4-bit weights
- \(M_\text{other}\)
- CUDA context, temporary buffers and allocator fragmentation, often around 1 GB [Likely]
LoRA on all linear layers matters for quality (next section), so \(P\) is not tiny. For LLaMA 7B with \(r = 64\) on all seven projections of each block, \(P = 32 \times 64 \times (4 \times 8192 + 3 \times 15104) \approx\) 159.9M parameters, 2.4% of the model, and 2.6 GB at 16 bytes each. The 4-bit base is 3.9 GB. So in QLoRA the adapter and its optimizer can cost as much as the base. A lower rank or an 8-bit optimizer helps; the paper found that with LoRA on all layers the rank did not noticeably change the final quality. [Certain]
How to read Figure 6
Start at LLaMA 65B and switch methods: full fine-tuning is over 1,000 GB, 16-bit LoRA about 148 GB, QLoRA about 52 GB at rank 64. Drop the rank to 16 and QLoRA falls to about 42 GB, under the 48 GB line: the orange adapter part was 12.8 GB. (The paper’s 65B runs used rank 64 on one 48 GB GPU, with BF16 adapters and paged optimizer states that can leave the GPU; this estimate keeps everything of the adapter in FP32 on the GPU, so it is conservative.) Now move to Llama 3.1 8B and raise the tokens to 16,384: the base is small, but the logits over a 128,256-word vocabulary become one of the largest parts. Untick gradient checkpointing: activations explode. These parts do not care about 4-bit weights, which is the main limit of QLoRA.
Step 6: paged optimizers
Memory use during training is not flat. It depends on the tokens in each batch, and SFT batches vary a lot (session 54): one batch with a very long example can need several GB more activations than the average. If that spike passes the GPU limit, the run dies with an out-of-memory error, often hours in.
Paged optimizers use NVIDIA unified memory: memory that the GPU and the CPU share through one address space, with pages that the driver moves between them on demand. The optimizer states (Adam’s two moments) are allocated in this memory. When the GPU runs out, the driver evicts pages of optimizer state to CPU RAM; when the optimizer step needs them, they are moved back. [Certain] The QLoRA paper reports that with batch size 16 on 65B models and 48 GB GPUs, paged optimizers ran at the same speed as regular ones, and notes that the 33B model “does not quite fit into a 24 GB” GPU without them. [Certain]
The important limit: paging can only move what is pageable. In a QLoRA run that is the optimizer state of the LoRA parameters, not the base weights or the activations. If a spike needs more memory than the optimizer states hold, paging cannot save the step:
- Read the symbols
- \(C\)
- GPU memory capacity
- \(M_\text{fixed}\)
- everything that must stay on the GPU: the 4-bit base, the LoRA weights and gradients, buffers
- \(M_\text{opt}\)
- the optimizer states in paged memory
- \(M_\text{act}(t)\)
- activations of step \(t\), set by the number of tokens in that batch
How to read Figure 7
At 24 GB with rank 16 and no paging, a few long batches cross the line and the run would crash at the first of them. Tick “Paged optimizer”: on those steps the green optimizer part shrinks (pages go to the CPU) and the step fits; the cost is a fraction of a second of transfers. Now choose rank 64: the optimizer states are four times larger, and without paging no step fits at all, which mirrors the paper’s note about 33B on 24 GB. With paging most steps fit, but the longest batches still fail, because they need more memory than the optimizer states can free. Slide the GPU memory to see where each setting starts to work.
Quality versus memory
Does a 4-bit base cost quality? The paper’s main experiment fine-tuned LLaMA 7B to 65B on Alpaca and FLAN v2 and measured 5-shot MMLU (accuracy in %, higher is better): [Certain]
| LLaMA size | BF16 LoRA, Alpaca / FLAN v2 | FP4 QLoRA | NF4 + DQ QLoRA |
|---|---|---|---|
| 7B | 38.4 / 45.6 | 37.2 / 44.0 | 39.0 / 44.5 |
| 13B | 47.2 / 50.6 | 47.3 / 50.0 | 47.5 / 50.7 |
| 33B | 57.7 / 60.5 | 55.9 / 58.5 | 57.3 / 59.2 |
| 65B | 61.8 / 62.5 | 61.3 / 63.3 | 61.8 / 63.9 |
| Mean | 53.0 | 52.2 | 53.1 |
NF4 with double quantization matched 16-bit LoRA on average (53.1 against 53.0), and FP4 was about one point lower. [Certain] Three findings from the same paper matter in practice:
- Adapt all linear layers. LoRA on only the query and value projections did not match 16-bit full fine-tuning; LoRA on all linear layers of the transformer did. [Certain] The paper’s explanation is that the number of adapted layers, not the rank, was the key hyper-parameter. [Certain]
- The recipe. For the Guanaco models: \(r = 64\), \(\alpha = 16\), LoRA on all linear layers, dropout 0.1 up to 13B and 0.05 for 33B and 65B, learning rate \(2 \times 10^{-4}\) for 7B and 13B and \(1 \times 10^{-4}\) for 33B and 65B, constant schedule, Adam \(\beta_2 = 0.999\), maximum gradient norm 0.3. [Certain]
- An honest limit. The authors write that they “did not establish that QLoRA can match full 16-bit finetuning performance at 33B and 65B scales”. [Certain] The match with full fine-tuning was shown at smaller scales; at 33B and 65B the comparison was with 16-bit LoRA.
Our small PyTorch experiment below shows the mechanism in miniature. A 256 × 256 layer is quantized to NF4 and a rank-8 adapter learns a rank-4 change of the original weight. The loss falls from 0.396 to 0.00258. It cannot go to zero: the quantization noise alone gives a loss of 0.00278 on this data. The adapter learned the change and removed a small part of the noise (the part that happens to be low rank), but most of the noise is full rank and stays. This is the general picture: the adapter learns the task, not the quantization error. [Likely]
When QLoRA is not enough
- Activation-bound jobs. Long contexts, reasoning traces, and VLMs or Video-LLMs with thousands of visual tokens per example are dominated by activations and logits (Figure 6). QLoRA does not shrink these. Use gradient checkpointing, chunked loss over the vocabulary, smaller micro-batches, or sequence parallelism (session 48).
- Large changes to the model. Continued pretraining on a new language or domain, or teaching much new knowledge, needs more capacity than a low-rank update and is usually done with full fine-tuning. [Likely] session 62 compares the three directly.
- Merging and serving. The adapter was trained on top of \(\hat{W}\), not \(W\). Merging \(BA\) into the original BF16 \(W\) gives a slightly different model from the one you trained, and quantizing \(\hat{W} + sBA\) again to 4 bits adds new error. Options: serve the NF4 base with an unmerged adapter; dequantize to BF16 and merge into \(\hat{W}\); or use QA-LoRA, which designs the adapter so that the merged model stays in a low-bit format. [Certain] for the options; how much quality each loses depends on the model. [Likely]
- Very low bits. At 2 or 3 bits the quantization error is large and \(B = 0\) starts training from a damaged model. LoftQ initializes \(A\) and \(B\) so that \(\hat{W} + BA\) is close to the original \(W\), which helps most at low bits. [Certain]
- Speed. If the model fits in BF16 on your hardware, 16-bit LoRA is usually faster per step, because it does not dequantize. [Likely] QLoRA is for when memory is the limit.
- Multi-GPU scale. With many GPUs, sharding (ZeRO, FSDP: session 45, session 46) divides the 16-bit costs across devices and allows full fine-tuning. QLoRA is mainly a single-GPU or small-cluster tool. [Likely]
Runnable code: NF4, double quantization and a QLoRA layer
This code builds the three code books, measures their error, double-quantizes the scales, and trains a QLoRA layer written from scratch. It runs on a CPU in a few seconds and prints the numbers used on this page. For clarity it stores one index per byte (real kernels pack two) and uses INT8 for the second-level quantization (bitsandbytes uses an 8-bit float).
import torch, torch.nn as nn torch.manual_seed(0) # 1) three 4-bit code books, all scaled to [-1, 1] def nf4_codebook(offset=0.9677083): # same construction as bitsandbytes pos = torch.special.ndtri(torch.linspace(offset, 0.5, 9)[:-1]) # 8 positive neg = -torch.special.ndtri(torch.linspace(offset, 0.5, 8)[:-1]) # 7 negative v = torch.cat([neg, torch.zeros(1), pos]).sort().values return v / v.max() NF4 = nf4_codebook() INT4 = torch.arange(-7, 8) / 7.0 # symmetric absmax INT4 e2m1 = torch.tensor([0, 0.5, 1, 1.5, 2, 3, 4, 6.0]) FP4 = torch.cat([-e2m1.flip(0)[:-1], e2m1]) / 6.0 # E2M1, one zero print([round(v, 4) for v in NF4[6:10].tolist()], len(NF4), len(INT4), len(FP4)) # [-0.0911, 0.0, 0.0796, 0.1609] 16 15 15 # 2) block-wise absmax quantization, formulas (58.1) and (58.2) def quantize(w, code, block=64): w = w.reshape(-1, block) c = w.abs().amax(dim=1, keepdim=True) # one FP32 scale per block idx = ((w / c).unsqueeze(-1) - code).abs().argmin(dim=-1) # nearest code value return idx.to(torch.uint8), c def dequantize(idx, c, code, shape): return (code[idx.long()] * c).reshape(shape) W = torch.randn(1024, 1024) * 0.02 # Gaussian-like weights for name, code in [("INT4", INT4), ("FP4", FP4), ("NF4", NF4)]: idx, c = quantize(W, code) err = (dequantize(idx, c, code, W.shape) - W).pow(2).mean() / W.pow(2).mean() print(name, f"relative MSE {err.item():.4f}") # INT4 0.0116, FP4 0.0112, NF4 0.0085 # 3) double quantization: quantize the FP32 scales to 8 bits, block 256, formula (58.4) def double_quant(c, block=256): mean = c.mean() # scales are all positive x = (c.flatten() - mean).reshape(-1, block) c2 = x.abs().amax(dim=1, keepdim=True) # FP32, one per 256 scales q = torch.round(x / c2 * 127).to(torch.int8) # 8-bit (int8 here, FP8 in the paper) return q, c2, mean def double_dequant(q, c2, mean): return (q.float() / 127 * c2).flatten()[:, None] + mean idx, c = quantize(W, NF4) q, c2, mean = double_quant(c) c_hat = double_dequant(q, c2, mean) print("scale error", f"{((c_hat - c).abs() / c).max().item():.4f}", "bits/param", 4 + 32 / 64, "->", 4 + 8 / 64 + 32 / (64 * 256)) # scale error 0.0046 bits/param 4.5 -> 4.126953125 # 4) a QLoRA linear layer: frozen NF4 base, BF16 compute, trainable LoRA A, B class QLoRALinear(nn.Module): def __init__(self, base, r=8, alpha=16): super().__init__() self.shape = base.weight.shape idx, c = quantize(base.weight.data, NF4) q, c2, mean = double_quant(c) for k, v in dict(idx=idx, q=q, c2=c2, mean=mean).items(): self.register_buffer(k, v) # buffers: no gradient, no optimizer d_out, d_in = self.shape self.A = nn.Parameter(torch.randn(r, d_in) / d_in ** 0.5) self.B = nn.Parameter(torch.zeros(d_out, r)) # B = 0: start equal to the base self.s = alpha / r def forward(self, x): c = double_dequant(self.q, self.c2, self.mean) # 8-bit -> FP32 scales, (58.5) Wb = dequantize(self.idx, c, NF4, self.shape).to(torch.bfloat16) # NF4 -> BF16 x = x.to(torch.bfloat16) lora = (x @ self.A.to(torch.bfloat16).T) @ self.B.to(torch.bfloat16).T return x @ Wb.T + self.s * lora # formula (58.6) # 5) fine-tune: the target is the original layer plus a small low-rank change base = nn.Linear(256, 256, bias=False) delta = 0.02 * torch.randn(256, 4) @ torch.randn(4, 256) layer = QLoRALinear(base, r=8) opt = torch.optim.AdamW([layer.A, layer.B], lr=1e-2, weight_decay=0.0) X = torch.randn(512, 256) Y = X @ (base.weight.data + delta).T codes_before = layer.idx.clone() for step in range(301): loss = (layer(X).float() - Y).pow(2).mean() opt.zero_grad(); loss.backward(); opt.step() if step in (0, 100, 300): print(step, f"{loss.item():.5f}") # 0.39579, 0.00262, 0.00258 print("codes unchanged:", torch.equal(codes_before, layer.idx), "trainable:", sum(p.numel() for p in layer.parameters()), "frozen:", layer.idx.numel()) # codes unchanged: True trainable: 4096 frozen: 65536 Wq = dequantize(layer.idx, double_dequant(layer.q, layer.c2, layer.mean), NF4, layer.shape) floor = (X @ (Wq - base.weight.data).T).pow(2).mean() # error from quantization alone print("quantization-noise floor:", f"{floor.item():.5f}") # 0.00278
With the real libraries, the same thing is a few lines: load with a BitsAndBytesConfig as in the first section, call PEFT’s prepare_model_for_kbit_training (it freezes the base and enables gradient checkpointing), wrap with a LoraConfig whose target_modules lists all linear projections, and pick a paged optimizer. [Likely] for the exact behavior of prepare_model_for_kbit_training, which has changed across versions.
Where this is used
| Setting | Why QLoRA | What to watch |
|---|---|---|
| Fine-tuning 7B to 70B chat models on one GPU (Guanaco-style SFT) | The base fits in 4 bits; adapters are small | LoRA on all linear layers; rank and optimizer memory (Figure 6) |
| Domain assistants (legal, medical, internal support) | Many small experiments on cheap hardware; one adapter per domain | Merging into BF16 versus serving NF4 with the adapter; forgetting (session 61) |
| Preference tuning (DPO) on a budget | The reference model can be the same NF4 base with the adapter turned off | Both policy and reference pass through dequantization: slower steps (session 68) |
| VLM instruction tuning (session 189) | The language model is most of the weights; the vision encoder is often frozen | Visual tokens make activations dominate; keep the projector trainable in 16 bits |
| Video-LLMs: video QA, key-moment detection, dense captions | Fits a large language backbone next to the long video token sequence | Activations from thousands of frame tokens, not weights, set the memory; checkpointing is required |
| Video retrieval and classification with an LLM head | Cheap adapters per label set or per customer | Short answers, long inputs: the loss is on few tokens, the memory on many |
| On-device or edge personalization | The 4-bit base is also the deployment format | Keep the adapter unmerged, or use QA-LoRA for a low-bit merged model (session 197) |
Common mistakes
- Using FP16 as the compute type. FP16 has a small range (session 37); with 4-bit weights and LoRA, overflows and NaN losses appear. Use
bnb_4bit_compute_dtype=torch.bfloat16on GPUs that support it. If left at FP32, training is just slow. - Adapting only the query and value projections. The paper shows this does not match 16-bit fine-tuning; target all linear layers.
- Expecting 4 bits to fix memory problems from long sequences. Activations and logits are untouched. Turn on gradient checkpointing and measure.
- Merging into the original BF16 weights and expecting the trained model. The adapter was trained against \(\hat{W}\). Merge into the dequantized weights, or evaluate the merged model before you ship it.
- Re-quantizing a merged model and skipping evaluation. Quantizing \(\hat{W} + sBA\) adds new error; it may undo part of the fine-tune.
- Thinking paged optimizers make memory unlimited. Only optimizer states move. A spike bigger than them still crashes, and heavy paging every step is slow (Figure 7).
- Using the full fine-tuning learning rate. LoRA and QLoRA use learning rates around \(10^{-4}\), not \(10^{-5}\); the Guanaco recipe used \(1\)–\(2 \times 10^{-4}\).
- Quantizing the wrong modules. Embeddings, the output head and norms are usually kept in 16 or 32 bits; quantizing them saves little and can hurt. [Likely]
Interview questions
What is QLoRA in one paragraph?basic
What is NF4, and why is it better than INT4 for weights?core
What is the difference between storage type and compute type in QLoRA?basic
Do gradients flow through the 4-bit weights?core
What does double quantization do, and how much does it save?core
Why block-wise quantization instead of one scale per tensor?basic
What are paged optimizers and what can they not do?core
Estimate the GPU memory to QLoRA-fine-tune a 70B model.advanced
You trained a QLoRA adapter. How do you deploy it?advanced
When would you choose 16-bit LoRA or full fine-tuning over QLoRA?advanced
Why does LoRA with \(B = 0\) start from a model that is already “damaged” in QLoRA, and what does LoftQ do about it?advanced
Exercises
- With block size \(B_1 = 128\), how many bits per weight does 4-bit storage cost without double quantization? With double quantization (8-bit scales, \(B_2 = 256\))?
- LLaMA 13B has 13.02B parameters. Treating all of them as quantized at 4.127 bits, how large is the base in GB? How large in BF16?
- A block has absmax 0.05. Quantize the weight \(-0.02\) to NF4 using the table above. What is the dequantized value and the error?
- Count the LoRA parameters for rank 16 on all seven linear layers of LLaMA 7B (\(d = 4096\), MLP width 11,008, 32 layers). How much memory do their FP32 Adam moments take?
- How much does double quantization save on Llama 3.1 70B (70.55B parameters, all treated as quantized)?
- In a QLoRA run the fixed memory is 19.4 GB, the paged optimizer states are 1.0 GB and the GPU has 24 GB. Activations cost 1.24 MB per token. Up to how many tokens per batch fits without paging, and with paging?
Answerscheck
Key takeaways
- QLoRA = LoRA on a frozen 4-bit base. The base is stored in NF4 (block 64, about 4.127 bits per weight with double quantization) and computed in BF16; only the adapters get gradients and optimizer states.
- Block-wise absmax quantization (58.1, 58.2) keeps outliers local. NF4 places its 16 values at normal quantiles (58.3), with an exact zero, and has lower error than INT4 and FP4 on normal weights.
- Double quantization (58.4, 58.5) cuts the cost of the scales from 0.5 to about 0.127 bits per weight: about 3 GB on a 65B model.
- Gradients flow through the dequantized weights to earlier layers (58.7) but never into them; dequantization in both passes makes steps slower than 16-bit LoRA.
- Paged optimizers move optimizer states to CPU RAM during spikes (58.9). They cannot free more than the optimizer states hold.
- NF4 + DQ matched 16-bit LoRA on MMLU (53.1 against 53.0) with LoRA on all linear layers. QLoRA does not shrink activations, changes the merge story, and is not a replacement for full fine-tuning when the model must change a lot.
Key formulas
References for this session
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Hugging Face (code). PEFT library
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Dettmers et al. (2021). 8-bit Optimizers via Block-wise Quantization
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Li et al. (2023). LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
- Xu et al. (2023). QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
- bitsandbytes (code). bitsandbytes: k-bit quantization and 8-bit / paged optimizers for PyTorch
- Belkada et al. (Hugging Face) (2023). Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA