LLM & VLM courseBlog
58Track 4: Post-training and RL · session 6 of 25

QLoRA and training with quantized weights

LoRA freezes the base model, but the frozen weights still fill the GPU. QLoRA stores them in 4 bits with a data type made for normally distributed weights (NF4), compresses the scale factors a second time, computes in BF16, and pages optimizer states to CPU memory when a spike comes. This session builds each piece from the bits up, counts the memory, and shows where the method stops being enough.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 4, 37, 38 and 57

What you will learn

  • Block-wise absmax quantization: how 16-bit weights become 4-bit codes plus one scale per block, and how they are turned back
  • Why NF4 (4-bit NormalFloat) has lower error than INT4 and FP4 on normally distributed weights, and how its 16 values are built from normal quantiles
  • Double quantization: compressing the scales from 0.5 to about 0.127 bits per weight, about 3 GB saved on a 65B model
  • The QLoRA layer: NF4 storage, BF16 compute, gradients only for the LoRA matrices, and why the frozen weight still takes part in the backward pass
  • Paged optimizers: what they move to CPU memory during a spike, and what they cannot save
  • A full memory budget for 7B to 70B models, and the evidence (MMLU, perplexity) on how much quality 4-bit storage costs
  • When QLoRA is not enough: activations, merging, speed, very low bits, and large changes to the model

Why this matters

In session 57 we froze the base weights and trained only two small matrices per layer. That removed almost all gradients and optimizer states. But one big cost was left: the frozen weights themselves. A 65B model in BF16 is 130 GB, more than any single GPU holds. QLoRA (Dettmers et al., 2023) removes most of that cost: it stores the frozen base in 4 bits and trains LoRA adapters on top, in 16-bit precision.

The numbers are the reason this method became the default for fine-tuning on small hardware. Full 16-bit fine-tuning of LLaMA 65B needs “more than 780 GB of GPU memory”; QLoRA fine-tunes it on a single 48 GB GPU. [Certain] (780 GB is what you get with 12 bytes per parameter: 2 for BF16 weights, 2 for gradients and 8 for two FP32 Adam moments, times 65.3B parameters.) The paper’s best model, Guanaco 65B, reached 99.3% of ChatGPT’s score on the Vicuna benchmark (judged by GPT-4) after about 24 hours of fine-tuning on one GPU. [Certain]

QLoRA is three ideas on top of LoRA. Each one is a common interview topic:

  • NF4 storage with BF16 compute. A 4-bit data type whose 16 values are placed where normally distributed weights actually are. Weights are stored in NF4 and turned back into BF16 for each matrix multiply.
  • Double quantization. The scale factors needed by 4-bit storage are themselves quantized to 8 bits, which saves about 0.37 bits per parameter.
  • Paged optimizers. Optimizer states live in memory that can move to the CPU when the GPU runs out, so a single long batch does not crash the run.

In real systems you meet QLoRA through the Hugging Face stack. This configuration loads a model in NF4:

BitsAndBytesConfig(load_in_4bit=True,
                   bnb_4bit_quant_type="nf4",
                   bnb_4bit_use_double_quant=True,
                   bnb_4bit_compute_dtype=torch.bfloat16)

PEFT then adds the LoRA adapters, and the trainer option optim="paged_adamw_8bit" or "paged_adamw_32bit" selects a paged optimizer. [Certain] Every one of these switches maps to one section of this page.

Step 1: block-wise quantization

Quantization means storing a number with fewer bits by mapping it to one of a small set of allowed values. With 4 bits there are \(2^4 = 16\) allowed values. We call this set the code book \(\{v_0, \dots, v_{15}\}\), and we choose it inside \([-1, 1]\). A stored weight is then only an index (0 to 15) into the code book: 4 bits.

Real weights are not inside \([-1, 1]\); they are small numbers like 0.02. So we first divide by a scale. The simplest choice is the largest absolute value, the absmax. If one scale is used for a whole matrix, a single large weight makes the scale large, and all normal weights are squeezed into the few code values near 0. The fix is block-wise quantization: cut the weights into blocks of \(B\) consecutive values (QLoRA uses \(B = 64\)) and give each block its own scale. [Certain] One outlier then hurts only its own block.

\[ c_b = \max_{i \in b} |w_i|, \qquad q_i = \argmin_{j \in \{0,\dots,15\}} \left| \frac{w_i}{c_b} - v_j \right| \tag{58.1} \]
\[ \hat{w}_i = c_b \cdot v_{q_i} \tag{58.2} \]
Read the symbols
\(w_i\)
one original weight (BF16 or FP32)
\(b\), \(B\)
a block of \(B\) consecutive weights; every weight belongs to exactly one block
\(c_b\)
the scale of block \(b\): its absmax, stored in FP32 (32 bits per block)
\(v_j\)
the \(j\)-th value of the 4-bit code book, inside \([-1, 1]\)
\(q_i\)
the stored 4-bit index: the code value nearest to the normalized weight
\(\hat{w}_i\)
the dequantized weight, the value the matrix multiply will use

Formula (58.1) is quantization (done once, when the model is loaded). Formula (58.2) is dequantization (done every time the weight is used). The difference \(\hat{w}_i - w_i\) is the quantization error. It is fixed: in QLoRA the base weights never change, so this error stays for the whole training run. Note that QLoRA uses plain round-to-nearest, with no calibration data, unlike inference methods such as GPTQ (session 106) or AWQ (session 107). [Certain]

The bits per weight are 4 for the index plus the scale cost shared by the block: \(32/B\) bits. With \(B = 64\) that is \(4 + 0.5 = 4.5\) bits. Smaller blocks give smaller error but more scales; we come back to this trade-off with double quantization.

Step 2: NF4, a 4-bit type for normal weights

Which 16 values should the code book contain? The usual answers are INT4 (evenly spaced integers, here \(-7/7, \dots, 7/7\)) and FP4 (a tiny floating-point format; E2M1 has the magnitudes 0, 0.5, 1, 1.5, 2, 3, 4, 6, scaled so that 6 becomes 1). Both ignore something we know: trained weights are roughly normally distributed around zero. [Likely] Most weights are near 0 and few are near the absmax. Evenly spaced levels waste many codes in the empty tails.

QLoRA’s answer is NormalFloat (NF4): put the code values at quantiles of the normal distribution, so that each code is used by about the same number of weights. A data type with this property is called information-theoretically optimal for that distribution: every code carries the same amount of information. [Certain] The paper defines the values from the quantile function \(Q_X\) of the standard normal distribution:

\[ q_i = \frac{1}{2}\left( Q_X\!\left(\frac{i}{2^k + 1}\right) + Q_X\!\left(\frac{i + 1}{2^k + 1}\right) \right) \tag{58.3} \]
Read the symbols
\(k\)
the number of bits (4 for NF4), so there are \(2^k = 16\) codes
\(Q_X(p)\)
the value below which a fraction \(p\) of a standard normal variable falls (for example \(Q_X(0.5) = 0\), \(Q_X(0.977) \approx 2\))
\(q_i\)
the \(i\)-th code value, the middle of two neighbouring quantiles; all values are then divided by the largest one, so they lie in \([-1, 1]\)

One more detail matters. A symmetric set of 16 values has no exact zero, but zero is important (padding, pruned weights, and the many weights that are tiny). So NF4 is built asymmetrically: 8 values for the positive side and 7 for the negative side, plus an exact 0. The bitsandbytes code builds the positive half from the normal quantiles at probabilities evenly spaced from 0.5 to 0.9677, the negative half from 7 such quantiles, adds 0, and divides by the largest value. [Certain] Our Python version of this rule reproduces the 16 values printed in the paper’s appendix to \(10^{-7}\):

−1.0000−0.6962−0.5251−0.3949−0.2844−0.1848−0.09110.00000.07960.16090.24610.33790.44070.56260.72301.0000

Look at the spacing: about 0.08 between codes near zero, and 0.28 between the two largest. The levels are dense where weights are dense. Inside a block, the normalized weights \(w_i/c_b\) are not exactly normal (the absmax always maps to ±1, and the rest is a normal shape squeezed by the block’s maximum), but the shape is close enough. [Likely]

Figure 1. 32,768 seeded Gaussian weights (standard deviation 0.02), cut into blocks and divided by each block’s absmax. Grey: the histogram of the normalized values. Orange ticks: the 16 code values; dashed lines: the edges where rounding switches from one code to the next. Blue bars: the share of weights that land on each code.
How to read Figure 1

With INT4 the blue bars are very uneven: the middle codes are used by many weights and the outer codes by almost none, so the 4 bits carry well under 4 bits of information. With NF4 the bars are much more even and the entropy is the highest of the three. FP4 is in between: its levels are dense near zero but very sparse near ±1. The error in the readout follows the same order. Change the block size: with larger blocks the absmax is larger compared with a typical weight, the histogram gets narrower, and more weights fall into the middle codes.

Worked example: one block by hand

Take a block of 8 weights (real blocks have 64): \(0.012, -0.031, 0.004, 0.050, -0.018, 0.027, -0.007, 0.021\). The absmax is \(c = 0.050\). Normalized: \(0.24, -0.62, 0.08, 1.00, -0.36, 0.54, -0.14, 0.42\).

Nearest NF4 values: \(0.2461, -0.6962, 0.0796, 1.0, -0.3949, 0.5626, -0.1848, 0.4407\), stored as indices 10, 1, 8, 15, 3, 13, 5, 12. Dequantized (times 0.05): \(0.01231, -0.03481, 0.00398, 0.05, -0.01975, 0.02813, -0.00924, 0.02204\). The largest error is \(0.0038\) on the second weight. The squared error divided by the sum of squared weights is 0.0048.

With INT4 (steps of \(1/7 \approx 0.143\)) the same block gives a relative error of 0.0069, 42% higher. Over a 1024 × 1024 Gaussian matrix with \(B = 64\), the code below measures 0.0116 for INT4, 0.0112 for FP4 and 0.0085 for NF4.

  1. \(c \leftarrow 0\)
  2. for each \(w_i\) in the block: \(c \leftarrow \max(c, |w_i|)\) // absmax scale, (58.1)
  3. for each \(w_i\) in the block:
  4. \(u_i \leftarrow w_i / c\) // now inside \([-1, 1]\)
  5. \(q_i \leftarrow \argmin_j |u_i - v_j|\) // nearest code, (58.1)
  6. pack two \(q_i\) per byte; store \(c\) in FP32
  7. at use time: \(\hat{w}_i \leftarrow c \cdot v_{q_i}\) // (58.2)
Figure 2. Formulas (58.1) and (58.2) run on one block in your browser. Top: the weights of the block; the orange c marks the weight that sets the scale. Each cell then shows the normalized value \(u\), the stored code index \(q\) (decimal and 4-bit binary) and, at the end, the dequantized weight \(\hat{w}\). Below: the number line of normalized values with the code values as orange ticks. The blue dot is the current \(u\); the shaded band is the interval that rounds to the chosen code; orange bars count how many weights of the block use each code. The seeded block is 16 Gaussian weights (standard deviation 0.02, seed 5) rounded to 3 decimals.
How to read Figure 2

Press Play. First the scan finds the absmax: on the worked example it is 0.050, the fourth weight. Then each weight is divided by \(c\) and the arrow shows it snapping to the nearest code; the live box lists the two nearest code values and their distances, so you can check the choice by hand. On the worked example the codes are 10, 1, 8, 15, 3, 13, 5, 12, as in the text, and the relative error is 0.0048. Switch to INT4 (0.0069) or FP4 (0.0057) at any time: the same weights land on different codes. One block is a small sample: on the seeded block of 16, FP4 happens to beat NF4 (0.0040 against 0.0063); over many blocks NF4 wins, as Figure 1 shows. Now choose “Same block with an outlier”: one weight of 0.150 sets the scale, and the other 15 weights crowd into the 6 NF4 codes from −0.2844 to 0.1609. The block then uses only 7 of its 16 codes.

The paper measured the same order on real models. Mean perplexity on Pile Common Crawl (lower is better) was 34.34 for Int4, 31.07 for Float4 (E2M1), 29.48 for Float4 (E3M0) and 27.41 for NFloat4 with double quantization. [Certain] These were averages over several model families from 125M to 13B parameters. [Likely]

Step 3: double quantization of the scales

Each block of 64 weights needs one FP32 scale: \(32/64 = 0.5\) bits per weight. That sounds small, but 0.5 bits is 12.5% of the 4 bits we pay for the weights themselves. For a 65B model it is about 4 GB. Double quantization treats the scales as just another tensor and quantizes them too: subtract their mean (they are all positive, so centering makes better use of a symmetric code), then quantize them to 8-bit floats in blocks of 256, with one FP32 scale per block of 256 scales. [Certain]

\[ \text{bits per weight} = k + \frac{32}{B_1} \quad \longrightarrow \quad k + \frac{8}{B_1} + \frac{32}{B_1 B_2} \tag{58.4} \]
Read the symbols
\(k\)
bits of the weight code (4)
\(B_1\)
first-level block size: weights per scale (64)
\(B_2\)
second-level block size: first-level scales per FP32 scale (256)
\(8/B_1\)
the cost of the 8-bit first-level scales
\(32/(B_1 B_2)\)
the cost of the FP32 second-level scales, almost nothing

Dequantization now has two steps. First rebuild the scale of block \(b\) from its 8-bit code \(\hat{c}_{1,b}\), the FP32 scale \(c_2\) of its group of 256, and the mean \(\mu\). Then rebuild the weight as before:

\[ c_b \approx c_2 \cdot \hat{c}_{1,b} + \mu, \qquad \hat{w}_i = c_b \cdot \text{NF4}[q_i] \tag{58.5} \]
Worked example: what double quantization saves

Without it: \(32/64 = 0.5\) bits per weight. With it: \(8/64 + 32/(64 \times 256) = 0.125 + 0.00195 = 0.127\) bits. The saving is \(0.373\) bits per weight. On a 65B model: \(65 \times 10^9 \times 0.373 / 8 \approx 3.03 \times 10^9\) bytes, about 3 GB, the number given in the paper. [Certain] The total storage goes from 4.5 to about 4.127 bits per weight.

LLaMA 65B has 65.29B parameters; 64.76B of them are in the linear layers that get quantized. At 4.127 bits those take 33.4 GB. The embeddings and output head (0.52B parameters) usually stay in 16 bits (1.05 GB). [Likely] So the frozen base takes about 34.5 GB, against 130.6 GB in BF16.

The error added by the second quantization is tiny: in the code below the rebuilt scales differ from the true scales by at most 0.46%. Because one scale serves 64 weights, it is worth spending bits on accuracy there, and double quantization lets us keep \(B = 64\) instead of moving to larger blocks.

  1. for each block \(b\) of 64 weights: \(c_b \leftarrow \max_{i \in b} |w_i|\) // FP32
  2. \(\mu \leftarrow \frac{1}{16} \sum_b c_b\)
  3. \(x_b \leftarrow c_b - \mu\) // centered, both signs
  4. for each group of \(B_2\) scales:
  5. \(c_2 \leftarrow \max |x_b|\) over the group // FP32
  6. for \(b\) in the group: \(k_b \leftarrow \text{round}(127\, x_b / c_2)\) // 8 bits
  7. \(\hat{c}_b \leftarrow c_2\, k_b / 127 + \mu\) // rebuilt at use time, (58.5)
  8. bits per weight \(= 4 + (8 \cdot 16 + 32\, G + 32) / 1024\) // (58.4) with \(G\) groups, plus \(\mu\)
Figure 3. Double quantization run on the 16 scales of 1,024 seeded Gaussian weights (16 blocks of 64, standard deviation 0.02, seed 21). Top: each block’s absmax scale; the green dashed line is their mean \(\mu\); a thin black line across an orange bar is the rebuilt scale \(\hat{c}_b\). Middle: the centered values \(x_b\), dashed amber lines at \(\pm c_2\) of each group (every second group is shaded), and the stored 8-bit codes \(k_b\). Bottom: the bits spent on scales, live. Like the code below, we use INT8 for the second level; bitsandbytes uses an 8-bit float.
How to read Figure 3

Press Play. The bottom bar starts at 512 bits (16 FP32 scales, 4.5 bits per weight). Each time a scale becomes an 8-bit code, 24 bits disappear; each group adds 32 bits for its \(c_2\), and \(\mu\) adds 32 bits once. With \(B_2 = 8\) you end at 224 bits, 4.21875 bits per weight. The live box shows every rounding, for example \(k_0 = \text{round}(127 \times 0.00214 / 0.01063) = 26\). The largest error of a rebuilt scale is 0.119%, and the NF4 error of the 1,024 weights is 0.00846 with the true scales and with the rebuilt ones. Try \(B_2 = 16\): one group and fewer bits (4.1875), with the same largest error here. In a real model a group holds 256 scales and \(\mu\) is shared by millions of weights, which gives the 4.127 bits of (58.4).

Figure 4. A seeded 16 × 4096 Gaussian weight matrix (standard deviation 0.02), quantized in blocks along each row. Each curve is one code book; its points are block sizes 16, 32, 64, 128, 256, 512, 1024 and 4096 (one block per row), from right to left. The x-axis counts all storage: 4 bits per index plus the scales (58.4). “Outlier weights” sets 0.1% of the weights to about ±0.25, more than 12 standard deviations.
How to read Figure 4

Moving left (larger blocks) saves bits but raises the error, because each block’s absmax grows. Without outliers the cost is mild. Tick “Add outlier weights”: now large blocks are much worse, since one outlier squeezes all other weights of its block into the codes near zero, while small blocks keep the damage local. Then tick “Double quantization”: every point moves left (the scales cost 8 bits instead of 32), and the error barely changes. That is why QLoRA can keep the small block of 64 at about 4.13 bits. Without outliers, NF4 has the lowest error at every block size. With outliers and blocks of 512 or more, FP4 catches up: most weights are then squeezed near zero, where FP4 also has dense levels. The y-axis is logarithmic.

Step 4: the QLoRA layer, NF4 storage and BF16 compute

We now have a compact frozen weight. QLoRA separates two kinds of precision. The storage type is how a tensor lives in memory between uses: NF4 for the base weights. The compute type is the precision of the arithmetic: BF16. [Certain] A GPU has no NF4 matrix multiply, so every forward and backward pass first dequantizes the weight to BF16 (58.5), then multiplies. With LoRA matrices \(A \in \R^{r \times d_\text{in}}\) and \(B \in \R^{d_\text{out} \times r}\) in BF16, one layer computes:

\[ Y = X \hat{W}^{\top} + s \, X A^{\top} B^{\top}, \qquad \hat{W} = \text{dequant}(q, \hat{c}_1, c_2, \mu), \qquad s = \frac{\alpha}{r} \tag{58.6} \]
Read the symbols
\(X\)
the input activations, shape (tokens, \(d_\text{in}\)), in BF16
\(\hat{W}\)
the base weight rebuilt in BF16 from its NF4 codes and double-quantized scales; frozen
\(A\), \(B\)
the LoRA matrices, the only trainable part; \(B\) starts at zero so training starts from the quantized base
\(s\), \(\alpha\), \(r\)
the LoRA scale, its numerator and the rank, as in session 57

A frequent point of confusion: do gradients go through the 4-bit weights? Yes, for the input; no, for the weight itself. The layer below needs \(\partial L / \partial X\), and that requires \(\hat{W}\). The weight gradient \(\partial L / \partial \hat{W}\) is never computed:

\[ \frac{\partial L}{\partial X} = G \hat{W} + s\, G B A, \qquad \frac{\partial L}{\partial B} = s\, G^{\top} (X A^{\top}), \qquad \frac{\partial L}{\partial A} = s\, (G B)^{\top} X \tag{58.7} \]
Read the symbols
\(G\)
\(\partial L / \partial Y\), the gradient arriving from the layer above, shape (tokens, \(d_\text{out}\))
\(G \hat{W}\)
the part of the input gradient that goes through the frozen weight: needs a second dequantization in the backward pass
\(X A^{\top}\)
a small \((\text{tokens} \times r)\) matrix saved in the forward pass

So QLoRA does more work than 16-bit LoRA: the weight is dequantized in the forward pass, again in the backward pass, and again when gradient checkpointing recomputes the forward pass. QLoRA training is therefore slower per step than 16-bit LoRA on the same GPU, when both fit. [Likely] The trade is time for memory.

  1. \(c_b \leftarrow c_2\, k_b/127 + \mu\) // (58.5), 4 scales
  2. for each block \(b\): \(\hat{W}[b] \leftarrow \text{bf16}(c_b \cdot \text{NF4}[q])\) // on the fly
  3. \(Y \leftarrow X\hat{W}^{\top} + s\,(XA^{\top})B^{\top}\); keep \(XA^{\top}\); free \(\hat{W}\) // (58.6)
  4. \(L \leftarrow \text{mean}\,(Y - Y^*)^2\); \(G \leftarrow 2(Y - Y^*)/64\)
  5. dequantize again; \(\partial L/\partial X \leftarrow G\hat{W} + s\,GBA\) // (58.7)
  6. \(\partial L/\partial B \leftarrow s\,G^{\top}(XA^{\top})\), \(\partial L/\partial A \leftarrow s\,(GB)^{\top}X\); no \(\partial L/\partial \hat{W}\) // (58.7)
  7. Adam updates \(A\), \(B\); zero the gradients; \(q\) is never written
Figure 5. A QLoRA layer small enough to watch, trained in your browser: an 8 × 8 weight (seed 41, entries with standard deviation 0.5) stored as NF4 codes in 4 blocks of 16, with double-quantized scales; a LoRA adapter of rank \(r\) with \(\alpha = 4\); 8 input tokens. The task is to match a target layer \(W^* = W + 0.6\,\mathbf{u}\mathbf{v}^{\top}\), a rank-1 change of the original weight, with a squared-error loss and Adam (learning rate 0.05, no weight decay). Teal is positive, purple negative; the hex digit in each code cell is the stored 4-bit index. The bars are the memory of the same layer at real size (4096 × 4096, rank 16, FP32 adapter and Adam states, 1,024 tokens, in MiB), following the sub-step shown above.
How to read Figure 5

Press Play and watch one training step in 10 sub-steps. The 4 scales are rebuilt from their 8-bit codes, then \(\hat{W}\) is filled block by block (the green bar grows to 32 MiB) and freed after the forward product. At step 1, \(B = 0\), so the output is just \(X\hat{W}^{\top}\) and \(\partial L/\partial A\) is exactly 0. In the backward pass \(\hat{W}\) is built a second time, because the layer below needs \(\partial L/\partial X\). The crossed box never fills: there is no gradient for the frozen weight. Only \(A\) and \(B\) change, and the codes at step 60 are the codes of step 1. Switch to “Whole training steps” and press Play to watch the loss: with rank 2 it falls from 0.32302 to 0.00807 in 60 steps. The green dashed line (0.00987) is the loss of an adapter that learned the change \(0.6\,\mathbf{u}\mathbf{v}^{\top}\) exactly and nothing else. Going below it means the adapter also cancels part of the quantization error, as in the experiment described in the text; rank 4 cancels more (0.00240), rank 1 cannot reach the line (0.01335). In memory, the layer keeps 9.75 MiB between steps, against 34 MiB for 16-bit LoRA; the peaks come from the temporary BF16 weight and the activation gradients.

Algorithm 1: load a model for QLoRA
  1. input: a pretrained model in BF16, block sizes \(B_1 = 64\), \(B_2 = 256\), LoRA rank \(r\), scale \(\alpha\)
  2. for each linear layer of the transformer blocks (attention and MLP projections):
  3.   split the weight into blocks of \(B_1\); \(c_b \leftarrow\) absmax of each block // (58.1)
  4.   \(q_i \leftarrow\) index of the nearest NF4 value to \(w_i / c_b\); pack two indices per byte
  5.   \(\mu \leftarrow\) mean of all \(c_b\); quantize \(c_b - \mu\) to 8 bits in groups of \(B_2\), one FP32 scale \(c_2\) per group // (58.4)
  6.   free the BF16 weight; keep \(q\), \(\hat{c}_1\), \(c_2\), \(\mu\) as frozen buffers
  7.   add \(A\) (random, small) and \(B = 0\) in BF16 or FP32, trainable
  8. keep embeddings, output head and norms in 16 bits (or 32 bits for norms); freeze them
  9. enable gradient checkpointing; create a paged AdamW over the LoRA parameters only
Algorithm 2: one QLoRA training step
  1. input: a batch of token ids, labels with −100 on masked positions
  2. for each layer in the forward pass:
  3.   \(\hat{W} \leftarrow\) dequant to BF16 (the whole weight of this layer, freed after use) // (58.5)
  4.   \(Y \leftarrow X\hat{W}^{\top} + s\,(XA^{\top})B^{\top}\); discard \(\hat{W}\) // (58.6)
  5. loss \(\leftarrow\) masked cross-entropy (session 54)
  6. for each layer in reverse: recompute the forward if checkpointed, dequantize again
  7.   \(\partial L/\partial X \leftarrow G\hat{W} + s\,GBA\); gradients for \(A\), \(B\) only // (58.7)
  8. clip gradients; paged AdamW updates \(A\), \(B\) // states may be paged in from CPU
  9. zero gradients; the NF4 codes are never touched

Step 5: the memory budget

Where does the memory of a QLoRA run go? Four parts (see session 38 for a careful count of each):

\[ M \approx \underbrace{N_q \frac{4.127}{8} + 2 N_{16}}_{\text{frozen base}} + \underbrace{16\, P}_{\text{LoRA + Adam}} + M_\text{act} + M_\text{other} \tag{58.8} \]
Read the symbols
\(N_q\)
parameters in the quantized linear layers; \(N_{16}\): parameters kept in 16 bits (embeddings, output head, norms)
\(P\)
LoRA parameters; 16 bytes each if the adapter, its gradient and two Adam moments are FP32 (an 8-bit optimizer reduces this)
\(M_\text{act}\)
activations saved for the backward pass, plus logits; grows with tokens per batch, and does not shrink with 4-bit weights
\(M_\text{other}\)
CUDA context, temporary buffers and allocator fragmentation, often around 1 GB [Likely]

LoRA on all linear layers matters for quality (next section), so \(P\) is not tiny. For LLaMA 7B with \(r = 64\) on all seven projections of each block, \(P = 32 \times 64 \times (4 \times 8192 + 3 \times 15104) \approx\) 159.9M parameters, 2.4% of the model, and 2.6 GB at 16 bytes each. The 4-bit base is 3.9 GB. So in QLoRA the adapter and its optimizer can cost as much as the base. A lower rank or an 8-bit optimizer helps; the paper found that with LoRA on all layers the rank did not noticeably change the final quality. [Certain]

Figure 6. An estimate of training memory with formula (58.8). Model shapes are the real LLaMA and Llama 3.1 shapes (hidden size, layers, MLP width, vocabulary, key-value width), and parameter counts are computed from them. Full fine-tuning uses 16 bytes per parameter. Activations count the BF16 inputs of each layer (with checkpointing) or the outputs of every projection in every layer (without), so they are a lower bound; logits are FP32 with their gradient.
How to read Figure 6

Start at LLaMA 65B and switch methods: full fine-tuning is over 1,000 GB, 16-bit LoRA about 148 GB, QLoRA about 52 GB at rank 64. Drop the rank to 16 and QLoRA falls to about 42 GB, under the 48 GB line: the orange adapter part was 12.8 GB. (The paper’s 65B runs used rank 64 on one 48 GB GPU, with BF16 adapters and paged optimizer states that can leave the GPU; this estimate keeps everything of the adapter in FP32 on the GPU, so it is conservative.) Now move to Llama 3.1 8B and raise the tokens to 16,384: the base is small, but the logits over a 128,256-word vocabulary become one of the largest parts. Untick gradient checkpointing: activations explode. These parts do not care about 4-bit weights, which is the main limit of QLoRA.

Step 6: paged optimizers

Memory use during training is not flat. It depends on the tokens in each batch, and SFT batches vary a lot (session 54): one batch with a very long example can need several GB more activations than the average. If that spike passes the GPU limit, the run dies with an out-of-memory error, often hours in.

Paged optimizers use NVIDIA unified memory: memory that the GPU and the CPU share through one address space, with pages that the driver moves between them on demand. The optimizer states (Adam’s two moments) are allocated in this memory. When the GPU runs out, the driver evicts pages of optimizer state to CPU RAM; when the optimizer step needs them, they are moved back. [Certain] The QLoRA paper reports that with batch size 16 on 65B models and 48 GB GPUs, paged optimizers ran at the same speed as regular ones, and notes that the 33B model “does not quite fit into a 24 GB” GPU without them. [Certain]

The important limit: paging can only move what is pageable. In a QLoRA run that is the optimizer state of the LoRA parameters, not the base weights or the activations. If a spike needs more memory than the optimizer states hold, paging cannot save the step:

\[ \text{a step succeeds if} \quad M_\text{fixed} + M_\text{act}(t) \le C, \qquad \text{paged amount} = \max\!\big(0,\ M_\text{fixed} + M_\text{opt} + M_\text{act}(t) - C\big) \le M_\text{opt} \tag{58.9} \]
Read the symbols
\(C\)
GPU memory capacity
\(M_\text{fixed}\)
everything that must stay on the GPU: the 4-bit base, the LoRA weights and gradients, buffers
\(M_\text{opt}\)
the optimizer states in paged memory
\(M_\text{act}(t)\)
activations of step \(t\), set by the number of tokens in that batch
fixed (NF4 base, LoRA weights and grads, 1 GB buffers)optimizer states on the GPUactivationsstep runs out of memory
Figure 7. A toy timeline, not a measurement. LLaMA 33B shape in NF4 (17.4 GB), LoRA on all linear layers with FP32 weights, gradients and Adam moments, gradient checkpointing, and 120 steps whose batches hold two seeded examples of 64 to 2,048 tokens each (log-uniform). Activation memory per token uses the same estimate as Figure 6. Moving pages assumes 25 GB/s over PCIe. The y-axis does not start at zero.
How to read Figure 7

At 24 GB with rank 16 and no paging, a few long batches cross the line and the run would crash at the first of them. Tick “Paged optimizer”: on those steps the green optimizer part shrinks (pages go to the CPU) and the step fits; the cost is a fraction of a second of transfers. Now choose rank 64: the optimizer states are four times larger, and without paging no step fits at all, which mirrors the paper’s note about 33B on 24 GB. With paging most steps fit, but the longest batches still fail, because they need more memory than the optimizer states can free. Slide the GPU memory to see where each setting starts to work.

Quality versus memory

Does a 4-bit base cost quality? The paper’s main experiment fine-tuned LLaMA 7B to 65B on Alpaca and FLAN v2 and measured 5-shot MMLU (accuracy in %, higher is better): [Certain]

LLaMA sizeBF16 LoRA, Alpaca / FLAN v2FP4 QLoRANF4 + DQ QLoRA
7B38.4 / 45.637.2 / 44.039.0 / 44.5
13B47.2 / 50.647.3 / 50.047.5 / 50.7
33B57.7 / 60.555.9 / 58.557.3 / 59.2
65B61.8 / 62.561.3 / 63.361.8 / 63.9
Mean53.052.253.1

NF4 with double quantization matched 16-bit LoRA on average (53.1 against 53.0), and FP4 was about one point lower. [Certain] Three findings from the same paper matter in practice:

  • Adapt all linear layers. LoRA on only the query and value projections did not match 16-bit full fine-tuning; LoRA on all linear layers of the transformer did. [Certain] The paper’s explanation is that the number of adapted layers, not the rank, was the key hyper-parameter. [Certain]
  • The recipe. For the Guanaco models: \(r = 64\), \(\alpha = 16\), LoRA on all linear layers, dropout 0.1 up to 13B and 0.05 for 33B and 65B, learning rate \(2 \times 10^{-4}\) for 7B and 13B and \(1 \times 10^{-4}\) for 33B and 65B, constant schedule, Adam \(\beta_2 = 0.999\), maximum gradient norm 0.3. [Certain]
  • An honest limit. The authors write that they “did not establish that QLoRA can match full 16-bit finetuning performance at 33B and 65B scales”. [Certain] The match with full fine-tuning was shown at smaller scales; at 33B and 65B the comparison was with 16-bit LoRA.

Our small PyTorch experiment below shows the mechanism in miniature. A 256 × 256 layer is quantized to NF4 and a rank-8 adapter learns a rank-4 change of the original weight. The loss falls from 0.396 to 0.00258. It cannot go to zero: the quantization noise alone gives a loss of 0.00278 on this data. The adapter learned the change and removed a small part of the noise (the part that happens to be low rank), but most of the noise is full rank and stays. This is the general picture: the adapter learns the task, not the quantization error. [Likely]

When QLoRA is not enough

  • Activation-bound jobs. Long contexts, reasoning traces, and VLMs or Video-LLMs with thousands of visual tokens per example are dominated by activations and logits (Figure 6). QLoRA does not shrink these. Use gradient checkpointing, chunked loss over the vocabulary, smaller micro-batches, or sequence parallelism (session 48).
  • Large changes to the model. Continued pretraining on a new language or domain, or teaching much new knowledge, needs more capacity than a low-rank update and is usually done with full fine-tuning. [Likely] session 62 compares the three directly.
  • Merging and serving. The adapter was trained on top of \(\hat{W}\), not \(W\). Merging \(BA\) into the original BF16 \(W\) gives a slightly different model from the one you trained, and quantizing \(\hat{W} + sBA\) again to 4 bits adds new error. Options: serve the NF4 base with an unmerged adapter; dequantize to BF16 and merge into \(\hat{W}\); or use QA-LoRA, which designs the adapter so that the merged model stays in a low-bit format. [Certain] for the options; how much quality each loses depends on the model. [Likely]
  • Very low bits. At 2 or 3 bits the quantization error is large and \(B = 0\) starts training from a damaged model. LoftQ initializes \(A\) and \(B\) so that \(\hat{W} + BA\) is close to the original \(W\), which helps most at low bits. [Certain]
  • Speed. If the model fits in BF16 on your hardware, 16-bit LoRA is usually faster per step, because it does not dequantize. [Likely] QLoRA is for when memory is the limit.
  • Multi-GPU scale. With many GPUs, sharding (ZeRO, FSDP: session 45, session 46) divides the 16-bit costs across devices and allows full fine-tuning. QLoRA is mainly a single-GPU or small-cluster tool. [Likely]

Runnable code: NF4, double quantization and a QLoRA layer

This code builds the three code books, measures their error, double-quantizes the scales, and trains a QLoRA layer written from scratch. It runs on a CPU in a few seconds and prints the numbers used on this page. For clarity it stores one index per byte (real kernels pack two) and uses INT8 for the second-level quantization (bitsandbytes uses an 8-bit float).

import torch, torch.nn as nn
torch.manual_seed(0)

# 1) three 4-bit code books, all scaled to [-1, 1]
def nf4_codebook(offset=0.9677083):            # same construction as bitsandbytes
    pos = torch.special.ndtri(torch.linspace(offset, 0.5, 9)[:-1])   # 8 positive
    neg = -torch.special.ndtri(torch.linspace(offset, 0.5, 8)[:-1])  # 7 negative
    v = torch.cat([neg, torch.zeros(1), pos]).sort().values
    return v / v.max()
NF4 = nf4_codebook()
INT4 = torch.arange(-7, 8) / 7.0                               # symmetric absmax INT4
e2m1 = torch.tensor([0, 0.5, 1, 1.5, 2, 3, 4, 6.0])
FP4 = torch.cat([-e2m1.flip(0)[:-1], e2m1]) / 6.0              # E2M1, one zero
print([round(v, 4) for v in NF4[6:10].tolist()], len(NF4), len(INT4), len(FP4))
# [-0.0911, 0.0, 0.0796, 0.1609] 16 15 15

# 2) block-wise absmax quantization, formulas (58.1) and (58.2)
def quantize(w, code, block=64):
    w = w.reshape(-1, block)
    c = w.abs().amax(dim=1, keepdim=True)                       # one FP32 scale per block
    idx = ((w / c).unsqueeze(-1) - code).abs().argmin(dim=-1)  # nearest code value
    return idx.to(torch.uint8), c
def dequantize(idx, c, code, shape):
    return (code[idx.long()] * c).reshape(shape)

W = torch.randn(1024, 1024) * 0.02                             # Gaussian-like weights
for name, code in [("INT4", INT4), ("FP4", FP4), ("NF4", NF4)]:
    idx, c = quantize(W, code)
    err = (dequantize(idx, c, code, W.shape) - W).pow(2).mean() / W.pow(2).mean()
    print(name, f"relative MSE {err.item():.4f}")         # INT4 0.0116, FP4 0.0112, NF4 0.0085

# 3) double quantization: quantize the FP32 scales to 8 bits, block 256, formula (58.4)
def double_quant(c, block=256):
    mean = c.mean()                                             # scales are all positive
    x = (c.flatten() - mean).reshape(-1, block)
    c2 = x.abs().amax(dim=1, keepdim=True)                      # FP32, one per 256 scales
    q = torch.round(x / c2 * 127).to(torch.int8)                # 8-bit (int8 here, FP8 in the paper)
    return q, c2, mean
def double_dequant(q, c2, mean): return (q.float() / 127 * c2).flatten()[:, None] + mean
idx, c = quantize(W, NF4)
q, c2, mean = double_quant(c)
c_hat = double_dequant(q, c2, mean)
print("scale error", f"{((c_hat - c).abs() / c).max().item():.4f}",
      "bits/param", 4 + 32 / 64, "->", 4 + 8 / 64 + 32 / (64 * 256))
# scale error 0.0046 bits/param 4.5 -> 4.126953125

# 4) a QLoRA linear layer: frozen NF4 base, BF16 compute, trainable LoRA A, B
class QLoRALinear(nn.Module):
    def __init__(self, base, r=8, alpha=16):
        super().__init__()
        self.shape = base.weight.shape
        idx, c = quantize(base.weight.data, NF4)
        q, c2, mean = double_quant(c)
        for k, v in dict(idx=idx, q=q, c2=c2, mean=mean).items():
            self.register_buffer(k, v)                          # buffers: no gradient, no optimizer
        d_out, d_in = self.shape
        self.A = nn.Parameter(torch.randn(r, d_in) / d_in ** 0.5)
        self.B = nn.Parameter(torch.zeros(d_out, r))           # B = 0: start equal to the base
        self.s = alpha / r
    def forward(self, x):
        c = double_dequant(self.q, self.c2, self.mean)          # 8-bit -> FP32 scales, (58.5)
        Wb = dequantize(self.idx, c, NF4, self.shape).to(torch.bfloat16)   # NF4 -> BF16
        x = x.to(torch.bfloat16)
        lora = (x @ self.A.to(torch.bfloat16).T) @ self.B.to(torch.bfloat16).T
        return x @ Wb.T + self.s * lora                         # formula (58.6)

# 5) fine-tune: the target is the original layer plus a small low-rank change
base = nn.Linear(256, 256, bias=False)
delta = 0.02 * torch.randn(256, 4) @ torch.randn(4, 256)
layer = QLoRALinear(base, r=8)
opt = torch.optim.AdamW([layer.A, layer.B], lr=1e-2, weight_decay=0.0)
X = torch.randn(512, 256)
Y = X @ (base.weight.data + delta).T
codes_before = layer.idx.clone()
for step in range(301):
    loss = (layer(X).float() - Y).pow(2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
    if step in (0, 100, 300): print(step, f"{loss.item():.5f}")   # 0.39579, 0.00262, 0.00258
print("codes unchanged:", torch.equal(codes_before, layer.idx),
      "trainable:", sum(p.numel() for p in layer.parameters()), "frozen:", layer.idx.numel())
# codes unchanged: True trainable: 4096 frozen: 65536
Wq = dequantize(layer.idx, double_dequant(layer.q, layer.c2, layer.mean), NF4, layer.shape)
floor = (X @ (Wq - base.weight.data).T).pow(2).mean()        # error from quantization alone
print("quantization-noise floor:", f"{floor.item():.5f}")     # 0.00278

With the real libraries, the same thing is a few lines: load with a BitsAndBytesConfig as in the first section, call PEFT’s prepare_model_for_kbit_training (it freezes the base and enables gradient checkpointing), wrap with a LoraConfig whose target_modules lists all linear projections, and pick a paged optimizer. [Likely] for the exact behavior of prepare_model_for_kbit_training, which has changed across versions.

Where this is used

SettingWhy QLoRAWhat to watch
Fine-tuning 7B to 70B chat models on one GPU (Guanaco-style SFT)The base fits in 4 bits; adapters are smallLoRA on all linear layers; rank and optimizer memory (Figure 6)
Domain assistants (legal, medical, internal support)Many small experiments on cheap hardware; one adapter per domainMerging into BF16 versus serving NF4 with the adapter; forgetting (session 61)
Preference tuning (DPO) on a budgetThe reference model can be the same NF4 base with the adapter turned offBoth policy and reference pass through dequantization: slower steps (session 68)
VLM instruction tuning (session 189)The language model is most of the weights; the vision encoder is often frozenVisual tokens make activations dominate; keep the projector trainable in 16 bits
Video-LLMs: video QA, key-moment detection, dense captionsFits a large language backbone next to the long video token sequenceActivations from thousands of frame tokens, not weights, set the memory; checkpointing is required
Video retrieval and classification with an LLM headCheap adapters per label set or per customerShort answers, long inputs: the loss is on few tokens, the memory on many
On-device or edge personalizationThe 4-bit base is also the deployment formatKeep the adapter unmerged, or use QA-LoRA for a low-bit merged model (session 197)

Common mistakes

  • Using FP16 as the compute type. FP16 has a small range (session 37); with 4-bit weights and LoRA, overflows and NaN losses appear. Use bnb_4bit_compute_dtype=torch.bfloat16 on GPUs that support it. If left at FP32, training is just slow.
  • Adapting only the query and value projections. The paper shows this does not match 16-bit fine-tuning; target all linear layers.
  • Expecting 4 bits to fix memory problems from long sequences. Activations and logits are untouched. Turn on gradient checkpointing and measure.
  • Merging into the original BF16 weights and expecting the trained model. The adapter was trained against \(\hat{W}\). Merge into the dequantized weights, or evaluate the merged model before you ship it.
  • Re-quantizing a merged model and skipping evaluation. Quantizing \(\hat{W} + sBA\) adds new error; it may undo part of the fine-tune.
  • Thinking paged optimizers make memory unlimited. Only optimizer states move. A spike bigger than them still crashes, and heavy paging every step is slow (Figure 7).
  • Using the full fine-tuning learning rate. LoRA and QLoRA use learning rates around \(10^{-4}\), not \(10^{-5}\); the Guanaco recipe used \(1\)–\(2 \times 10^{-4}\).
  • Quantizing the wrong modules. Embeddings, the output head and norms are usually kept in 16 or 32 bits; quantizing them saves little and can hurt. [Likely]

Interview questions

What is QLoRA in one paragraph?basic
QLoRA fine-tunes a large model by freezing its base weights in a 4-bit format (NF4, block size 64, with double-quantized scales) and training LoRA adapters in 16-bit precision on top. Every matrix multiply dequantizes the needed weights to BF16, so the arithmetic is BF16 while storage is about 4.13 bits per weight. Gradients flow through the frozen weights to earlier layers, but only the LoRA matrices get weight gradients and optimizer states, which are kept in paged memory. This let a 65B model be fine-tuned on one 48 GB GPU instead of more than 780 GB, and NF4 + DQ matched 16-bit LoRA on MMLU in the paper.
What is NF4, and why is it better than INT4 for weights?core
NF4 is a 4-bit code book whose 16 values sit at quantiles of a standard normal distribution, scaled to \([-1, 1]\), with an exact zero (8 positive values, 7 negative, plus 0). Weights are approximately normal, so equal-probability bins mean each code is used about equally often, which minimizes wasted codes; INT4’s even spacing puts many codes in the sparse tails and too few near zero. With block-wise absmax scaling the measured error is lower: on a Gaussian matrix with block 64, relative MSE 0.0085 for NF4 against 0.0116 for INT4, and the paper reports perplexity 27.41 for NF4 + DQ against 34.34 for Int4.
What is the difference between storage type and compute type in QLoRA?basic
The storage type is how a tensor sits in memory: NF4 codes plus scales for the frozen base. The compute type is the precision of the arithmetic: BF16. GPUs have no NF4 matrix multiply, so weights are dequantized to BF16 just before each use (in the forward pass, in the backward pass and in recomputation) and the BF16 copy is thrown away. The LoRA matrices and activations are in BF16 (or FP32 for the adapter master weights).
Do gradients flow through the 4-bit weights?core
Through them, yes; into them, no. The gradient with respect to the layer input is \(G\hat{W} + sGBA\), which uses the dequantized weight, so earlier layers and their adapters get correct gradients. The gradient with respect to \(\hat{W}\) is never computed and the codes never change. That is why the frozen base costs no gradient or optimizer memory, but still costs a second dequantization in the backward pass.
What does double quantization do, and how much does it save?core
Block-wise quantization with block 64 needs one FP32 scale per 64 weights: 0.5 bits per weight. Double quantization subtracts the mean of the scales and quantizes them to 8-bit floats in blocks of 256, keeping one FP32 scale per 256 scales: \(8/64 + 32/(64 \cdot 256) \approx 0.127\) bits. The saving is 0.373 bits per weight, about 3 GB on a 65B model, with a tiny extra error because the scales are quantized much more finely than the weights. Total storage goes from 4.5 to about 4.127 bits per weight.
Why block-wise quantization instead of one scale per tensor?basic
With one absmax scale per tensor, a single large weight sets the scale and squeezes all normal weights into the few codes near zero, so most of the 16 codes are wasted. With blocks of 64, an outlier only affects its own 63 neighbours. The cost is one scale per block (0.5 bits per weight in FP32, 0.127 with double quantization). Smaller blocks lower the error but cost more bits; 64 is the QLoRA choice.
What are paged optimizers and what can they not do?core
They allocate optimizer states in CUDA unified memory. When the GPU runs out of memory, for example on a batch with a very long sequence, the driver evicts optimizer pages to CPU RAM, and pages them back for the optimizer step. This turns a crash into a short slowdown. They can only free as much memory as the optimizer states take, which in QLoRA is the LoRA parameters’ Adam moments; they do not move the base weights or activations. A spike bigger than that still fails, and if paging happens every step, PCIe transfers slow training noticeably.
Estimate the GPU memory to QLoRA-fine-tune a 70B model.advanced
Llama 3.1 70B has about 70.55B parameters, 68.45B in linear layers. At 4.127 bits those are about 35.3 GB; embeddings and head (2.1B) in BF16 add 4.2 GB: about 39.5 GB for the base. LoRA rank 16 on all linear layers is about 207M parameters; at 16 bytes each about 3.3 GB (less with an 8-bit optimizer or BF16 adapters). Add activations with checkpointing (the BF16 input of each of the 80 layers is \(2 \times 8192 \times 80 \approx 1.3\) MB per token, plus one layer’s working set, about 3.2 GB for 2,048 tokens), FP32 logits over 128,256 words with their gradient (about 1 MB per token, 2.1 GB), and about 1 GB of buffers. Total about 49 GB at 2,048 tokens: an 80 GB GPU, or a 48 GB GPU with short sequences and a lower rank.
You trained a QLoRA adapter. How do you deploy it?advanced
Three options. (1) Serve the NF4 base with the adapter unmerged: exactly the trained model, small memory, extra latency from the adapter path and dequantization, and many adapters can share one base. (2) Dequantize the base to BF16 and merge \(sBA\) into \(\hat{W}\): the trained model up to BF16 rounding, full 16-bit serving cost. Merging into the original \(W\) instead gives a slightly different model. (3) Merge and then quantize again for inference (GPTQ, AWQ, or NF4): cheapest serving, but adds new error, so evaluate it; QA-LoRA was designed so the merged model stays in a low-bit format without this extra step.
When would you choose 16-bit LoRA or full fine-tuning over QLoRA?advanced
16-bit LoRA if the BF16 model fits: steps are faster with no dequantization, and the adapter is trained on the exact weights you may merge into. Full fine-tuning (with ZeRO or FSDP across GPUs) for large changes such as continued pretraining, a new language, or heavy domain shift, where a low-rank update is not enough, and when the best quality matters more than cost. Also note that QLoRA does not help activation-bound workloads such as long-context, reasoning or video SFT; those need checkpointing and parallelism whichever method you use. The QLoRA paper itself did not show a match with full fine-tuning at 33B and 65B.
Why does LoRA with \(B = 0\) start from a model that is already “damaged” in QLoRA, and what does LoftQ do about it?advanced
With \(B = 0\) the starting model is \(\hat{W}\), the quantized base, not the original \(W\). At 4 bits with NF4 the gap is small, so this is fine. At 2 or 3 bits it is large, and the adapter must spend its capacity repairing quantization damage. LoftQ alternates quantization and a low-rank SVD of the residual \(W - \hat{W}\) to choose the quantized weights and the initial \(A, B\) together so that \(\hat{W} + BA \approx W\). Only the low-rank part of the error can be repaired this way; the full-rank part stays.

Exercises

  1. With block size \(B_1 = 128\), how many bits per weight does 4-bit storage cost without double quantization? With double quantization (8-bit scales, \(B_2 = 256\))?
  2. LLaMA 13B has 13.02B parameters. Treating all of them as quantized at 4.127 bits, how large is the base in GB? How large in BF16?
  3. A block has absmax 0.05. Quantize the weight \(-0.02\) to NF4 using the table above. What is the dequantized value and the error?
  4. Count the LoRA parameters for rank 16 on all seven linear layers of LLaMA 7B (\(d = 4096\), MLP width 11,008, 32 layers). How much memory do their FP32 Adam moments take?
  5. How much does double quantization save on Llama 3.1 70B (70.55B parameters, all treated as quantized)?
  6. In a QLoRA run the fixed memory is 19.4 GB, the paged optimizer states are 1.0 GB and the GPU has 24 GB. Activations cost 1.24 MB per token. Up to how many tokens per batch fits without paging, and with paging?
Answerscheck
1. Without DQ: \(4 + 32/128 = 4.25\) bits. With DQ: \(4 + 8/128 + 32/(128 \times 256) = 4.0635\) bits. 2. \(13.02 \times 10^9 \times 4.127/8 \approx 6.71\) GB; in BF16 \(13.02 \times 10^9 \times 2 \approx 26.0\) GB. 3. \(-0.02/0.05 = -0.40\); the nearest NF4 value is \(-0.3949\), so \(\hat{w} = -0.3949 \times 0.05 = -0.01975\), an error of \(0.00025\). 4. Per layer: \(16 \times (4 \times 8192 + 3 \times 15104) = 1{,}249{,}280\); times 32 layers \(= 39{,}976{,}960 \approx 40.0\)M. Two FP32 moments are 8 bytes per parameter: about 0.32 GB. 5. \(70.55 \times 10^9 \times 0.373 / 8 \approx 3.29\) GB. 6. Without paging the free memory is \(24 - 19.4 - 1.0 = 3.6\) GB, so about \(3.6/0.00124 \approx 2{,}900\) tokens. With paging the optimizer states can leave: \(4.6/0.00124 \approx 3{,}700\) tokens. Paging raised the limit by only about 800 tokens, the size of the optimizer states.

Key takeaways

  • QLoRA = LoRA on a frozen 4-bit base. The base is stored in NF4 (block 64, about 4.127 bits per weight with double quantization) and computed in BF16; only the adapters get gradients and optimizer states.
  • Block-wise absmax quantization (58.1, 58.2) keeps outliers local. NF4 places its 16 values at normal quantiles (58.3), with an exact zero, and has lower error than INT4 and FP4 on normal weights.
  • Double quantization (58.4, 58.5) cuts the cost of the scales from 0.5 to about 0.127 bits per weight: about 3 GB on a 65B model.
  • Gradients flow through the dequantized weights to earlier layers (58.7) but never into them; dequantization in both passes makes steps slower than 16-bit LoRA.
  • Paged optimizers move optimizer states to CPU RAM during spikes (58.9). They cannot free more than the optimizer states hold.
  • NF4 + DQ matched 16-bit LoRA on MMLU (53.1 against 53.0) with LoRA on all linear layers. QLoRA does not shrink activations, changes the merge story, and is not a replacement for full fine-tuning when the model must change a lot.

Key formulas

\[ c_b = \max_{i \in b}|w_i|, \quad q_i = \argmin_j \left|\tfrac{w_i}{c_b} - v_j\right|, \quad \hat{w}_i = c_b\, v_{q_i} \]
\[ \text{bits} = 4 + \frac{8}{B_1} + \frac{32}{B_1 B_2} \approx 4.127, \qquad Y = X\hat{W}^{\top} + \frac{\alpha}{r} X A^{\top} B^{\top} \]
\[ \frac{\partial L}{\partial X} = G\hat{W} + sGBA, \qquad M \approx N_q \tfrac{4.127}{8} + 2N_{16} + 16P + M_\text{act} + M_\text{other} \]

References for this session

  1. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  2. Hugging Face (code). PEFT library
  3. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  4. Dettmers et al. (2021). 8-bit Optimizers via Block-wise Quantization
  5. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  6. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  7. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  8. Li et al. (2023). LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
  9. Xu et al. (2023). QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
  10. bitsandbytes (code). bitsandbytes: k-bit quantization and 8-bit / paged optimizers for PyTorch
  11. Belkada et al. (Hugging Face) (2023). Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA