LoRA vs QLoRA vs full fine-tuning
In this lab you fine-tune the same small language model, Qwen2.5-0.5B, in three ways: full fine-tuning, LoRA and QLoRA. You train it for sentiment on IMDB reviews and for one-sentence summaries on XSum, predict and measure memory and speed, score summaries with ROUGE and a position-swapped LLM judge, and end with a short written decision about what you would ship.
What you will learn
- How full fine-tuning, LoRA and QLoRA differ for one linear layer: what is stored, what gets a gradient, and what is served
- How to count LoRA parameters exactly (8,798,208 for Qwen2.5-0.5B at rank 16 on all linear layers) and predict training memory before a run
- What NF4 and double quantization do to weights, and why the math still runs in BF16
- How to fine-tune a decoder model as a document classifier (IMDB) and as a summarizer (XSum) with PEFT and TRL
- How to evaluate summaries with ROUGE and with an LLM judge, and the weaknesses of each
- How to tell a real difference from noise with standard errors and paired comparisons
- How to write a ship decision from your own memory, speed and quality measurements
Why this matters
You have a small language model, one GPU and two tasks: decide if a movie review is positive or negative, and write a one-sentence summary of a news article. You can train all the weights (full fine-tuning), train a small low-rank add-on (LoRA), or do the same on top of a 4-bit copy of the model (QLoRA). In this lab you run all three on both tasks, measure memory, speed and quality yourself, and then write down which one you would ship.
The earlier sessions gave the theory: session 57 explained the low-rank update, session 58 explained 4-bit NF4 weights, and session 61 explained why changing fewer weights can protect old skills. This lab turns that theory into numbers you measured. That matters because the right choice depends on your model size, your GPU, your data and how you will serve the model, and no paper can make that choice for you.
You will:
- Predict the memory of each method before you run it, with a formula you can check (Figure 2).
- Fine-tune
Qwen2.5-0.5Bfor sentiment on IMDB movie reviews and measure accuracy. - Fine-tune it for summarization on XSum, and score the summaries with ROUGE and with an LLM judge.
- Fill one results table with memory, speed and quality for the three methods.
- Write a short “ship memo”: what you would deploy, and why.
Real systems use all three methods. The QLoRA authors fine-tuned a 65B model on a single 48 GB GPU, where regular 16-bit fine-tuning needs more than 780 GB [Certain]. LoRA adapters are the standard way to offer many customized models on one shared base model, and Hugging Face’s PEFT and TRL libraries (used in this lab) are the common open-source tools for it.
Setup
Hardware. One NVIDIA GPU with 16 to 24 GB is enough for every run on this page with the 0.5B model. A 24 GB card (L4, A10G, RTX 4090) is the most comfortable. On a free Colab T4 (16 GB), note that the T4 has no BF16 support, so replace bf16=True with fp16=True [Certain]. QLoRA uses the bitsandbytes library, which needs a CUDA GPU for the 4-bit path we use here [Likely].
pip install "transformers>=4.56" peft trl datasets accelerate bitsandbytes rouge_score
The code on this page was checked against transformers 4.57, peft 0.17 and trl 0.24. These libraries change often; if an argument name fails, check its current documentation. We ran every script end to end on a CPU with a tiny model (SmolLM2-135M, two training steps) to check that the code runs. We did not run the full GPU experiments for you, and this page does not report their results: you will measure them.
Model. Qwen/Qwen2.5-0.5B, the base (not instruct) model. From its config.json: 24 layers, hidden size \(d = 896\), feed-forward size 4,864, 14 query heads and 2 key-value heads of size 64, and a vocabulary of 151,936 tokens with tied input and output embeddings [Certain]. It has 494,032,768 parameters. Of these, 357,826,560 sit in the seven linear layers of the blocks (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) and 136,134,656 in the embedding table. We counted these numbers by building the model from its config.
Data.
| Task | Dataset | Size | We use | Metric |
|---|---|---|---|---|
| Sentiment | stanfordnlp/imdb: movie reviews, label 0 = negative, 1 = positive | 25,000 train, 25,000 test | 5,000 train, 2,000 test (seeded shuffle) | Accuracy |
| Summarization | EdinburghNLP/xsum: BBC articles with a one-sentence summary | 204,045 train, 11,332 validation, 11,334 test | 5,000 train, 500 test | ROUGE-1/2/L and an LLM judge |
The subsets keep each run short (minutes to about an hour on one GPU, depending on the card) [Likely]. Use the same subsets, seeds and number of steps for all three methods. Otherwise you compare runs, not methods.
The table you will fill. Copy it into your notes now. Every empty cell is a number you measure.
| Method | Trainable params | Peak GPU memory (GB) | Train time | IMDB accuracy | XSum ROUGE-L | Judge win rate vs full FT | Saved file size |
|---|---|---|---|---|---|---|---|
| Base model, no training | 0 | n/a | |||||
| Full fine-tuning | 494.0M | 0.5 (itself) | |||||
| LoRA, \(r = 16\), all linear | 8.80M (1.78%) | ||||||
| QLoRA, NF4 + \(r = 16\) | 8.80M (1.78%) |
The trainable counts are for the summarization model. The sentiment model also trains a small new output layer (896 × 2 = 1,792 weights), which LoRA trains fully. Your PEFT printout will show a total parameter count that includes the LoRA weights.
Step 1: three ways to train one layer
Start with one linear layer, the query projection q_proj of one block: a matrix \(W_0\) of shape 896 × 896, so 802,816 weights. The three methods differ only in what they store and what they update for this matrix.
- Full fine-tuning updates \(W_0\) itself. Every weight gets a gradient and two Adam states.
- LoRA freezes \(W_0\) and adds a small trainable path next to it: first a matrix \(A\) that maps the input down to \(r\) numbers, then a matrix \(B\) that maps them back up.
- QLoRA does the same as LoRA, but stores the frozen \(W_0\) in 4 bits. Before each multiplication, a block of 4-bit weights is turned back into BF16 numbers (dequantized).
- Read the symbols
- \(\mathbf{x}, \mathbf{h}\)
- the input vector of the layer (\(d_{in}\) numbers) and its output (\(d_{out}\) numbers)
- \(W_0\)
- the pretrained weight matrix, frozen in LoRA and QLoRA
- \(A, B\)
- the two small trainable matrices; \(A\) starts random and small, \(B\) starts at zero, so \(BA = 0\) and the model is unchanged at step 0
- \(r\)
- the rank: the width of the bottleneck (we use 16)
- \(\alpha\)
- a fixed scale (we use 32); the update is multiplied by \(\alpha / r\), here 2
The number of trainable weights for one adapted layer is the size of \(A\) plus the size of \(B\):
For q_proj with \(r = 16\): \(16 \times (896 + 896) = 28{,}672\) weights instead of 802,816, which is 3.57% of the layer. Step through Figure 1 to see what happens in the forward pass, the backward pass and at serving time for each method.
q_proj layer of Qwen2.5-0.5B (896 × 896) under the three methods, with \(r = 16\). The readout gives the trainable weights and the bytes this one layer needs for weights, gradients and Adam states.How to read Figure 1
Pick a method, then step. Watch step 5 (backward): in LoRA and QLoRA, the gradient still flows through the frozen \(W_0\) back to \(\mathbf{x}\), because the layers below also have adapters that need a gradient. So the activations must still be stored, just as in full fine-tuning. What LoRA saves is the gradient and the Adam states of \(W_0\). In step 6, see why a LoRA model has no extra cost at serving time once \(BA\) is merged into \(W_0\), and why a QLoRA adapter is merged into a 16-bit copy of the base, not into the 4-bit one.
Apply formula (62.2) to all seven linear layers of one Qwen2.5-0.5B block with \(r = 16\). The query and output projections are 896 → 896. The key and value projections are 896 → 128 (2 heads of 64). The gate and up projections are 896 → 4,864, and the down projection is 4,864 → 896.
\(16 \times [\,2(896+896) + 2(896+128) + 3(896+4864)\,] = 16 \times 22{,}912 = 366{,}592\) per block. Times 24 blocks: 8,798,208 trainable weights, which is 1.78% of 494,032,768. PEFT’s print_trainable_parameters() reports the same 8,798,208. If you adapt only q_proj and v_proj (the setting of the original LoRA paper), you get 1,081,344 (0.22%).
Step 2: predict the memory before you run
A training run needs memory for four things: the weights, the gradients, the optimizer states (Adam keeps two numbers per trained weight), and the activations (values saved in the forward pass for the backward pass). Session 38 derived the classic budget. Here it is for our three methods, with \(N\) the number of parameters and \(P\) the number of LoRA parameters.
Full fine-tuning with mixed precision keeps FP32 weights (4 bytes), FP32 gradients (4) and two FP32 Adam states (8), and runs the math in BF16:
LoRA keeps the frozen base in BF16 (2 bytes, no gradient, no Adam states). The small LoRA weights are kept in FP32, so they cost the same 16 bytes each:
QLoRA stores the block linear weights \(N_{lin}\) in 4-bit NF4. Each block of 64 weights also needs a scale; with double quantization (Step 3) the total is 4.127 bits per weight. The other weights \(N_{oth} = N - N_{lin}\) (embeddings, norms) stay in higher precision. PEFT’s helper prepare_model_for_kbit_training casts them to FP32 (we checked its source code in PEFT 0.17), so they cost 4 bytes each:
The activations are about the same for all three methods, because the backward pass must still go through every layer. With gradient checkpointing (store only each block’s input, recompute the rest), a simple estimate is the stored block inputs in BF16 plus the output logits in FP32:
- Read the symbols
- \(L, d, V\)
- the number of blocks, the hidden size and the vocabulary size
- \(b, s\)
- the micro-batch size (sequences per step on the GPU) and the sequence length in tokens
- \(N_{lin}, N_{oth}\)
- parameters in the block linear layers (quantized by QLoRA) and all other parameters
- \(b_4\)
- bytes per weight for NF4 with double quantization
This is a lower bound, not a promise. Real peaks also include the recomputed activations of one block, temporary buffers, memory fragmentation and the CUDA context (often several hundred MB) [Likely]. Use it to rank the methods and to see which term dominates. Then measure the real peak with torch.cuda.max_memory_allocated().
How to read Figure 2
Start with Qwen2.5-0.5B. Full fine-tuning needs 7.9 GB for weights and states, LoRA 1.13 GB, QLoRA 0.87 GB. QLoRA saves little here: the 136M embedding weights are not quantized and are even cast to FP32. Now look at the activation bar: the logits term \(b \cdot s \cdot V \cdot 4\) is 2.49 GB, because the vocabulary is huge (151,936). For small models, the batch and the sequence length decide the memory more than the method does. Switch to Qwen2.5-7B: full fine-tuning needs about 122 GB of states and does not fit on any single GPU listed, LoRA needs about 16 GB, QLoRA about 8.4 GB. This is the size where QLoRA earns its place.
Full: \(16 \times 494{,}032{,}768 = 7.90\) GB. LoRA: \(2 \times 494{,}032{,}768 + 16 \times 8{,}798{,}208 = 0.988 + 0.141 = 1.13\) GB. QLoRA: \(0.5159 \times 357{,}826{,}560 + 4 \times 136{,}206{,}208 + 0.141 = 0.185 + 0.545 + 0.141 = 0.87\) GB.
Activations: block inputs \(24 \times 8 \times 512 \times 896 \times 2 = 0.18\) GB; logits \(8 \times 512 \times 151{,}936 \times 4 = 2.49\) GB. So the estimated totals are 10.6 GB (full), 3.8 GB (LoRA) and 3.5 GB (QLoRA). All three fit on a 16 GB card at this batch size. Now compare with your measured peaks, and explain the gap.
Step 3: what 4 bits do to the weights
QLoRA stores each frozen weight as a 4-bit index into a list of 16 values. Which 16 values? Pretrained weights are roughly bell-shaped (normally distributed) around zero. Evenly spaced values (the INT4 grid) waste levels on the rare large weights. NF4 (4-bit NormalFloat) instead puts its 16 values at quantiles of a normal distribution, so each level is used about equally often [Certain]. The values are scaled to \([-1, 1]\); 0 is exact, there are 7 negative and 8 positive levels.
The weights are cut into blocks of 64. Each block has its own scale \(c\), its largest absolute value (absmax):
- Read the symbols
- \(v_0, \dots, v_{15}\)
- the 16 NF4 values, from \(-1\) through 0 to \(+1\)
- \(q_i\)
- the stored 4-bit index of weight \(i\)
- \(\hat{w}_i\)
- the dequantized BF16 weight that is actually used in the matrix multiplication
Each block scale is one FP32 number for 64 weights: \(32/64 = 0.5\) extra bits per weight. Double quantization quantizes these scales again, to 8-bit numbers in blocks of 256, with one FP32 scale per 256 scales. The overhead drops to \(8/64 + 32/(64 \cdot 256) = 0.127\) bits per weight, which saves 0.373 bits per weight [Certain]. For the 6.5 billion block weights of Qwen2.5-7B, that is about 0.30 GB.
One more point that is easy to miss: the forward and backward passes do not run in 4 bits. Each block is dequantized to BF16 right before it is used, and the gradient flows through the dequantized \(\hat{W}\). The 4 bits only save storage. The extra dequantization work is why a QLoRA step is usually slower than a LoRA step [Likely].
How to read Figure 3
With blocks of 64 and normal weights, NF4 has a smaller error than INT4 (0.0082 versus 0.0109 for the first sample). Turn on outliers: one large weight stretches the scale of its block, and all the small weights in that block lose precision. Then move the block size to 4,096 (one scale for everything): the INT4 error jumps to 0.128 and NF4 to 0.047. Small blocks protect against outliers but cost more bits for scales. Blocks of 64 with double quantization are the QLoRA trade-off.
Step 4: a toy you can train in your browser
Before the real runs, build intuition with a model small enough to train in a fraction of a second. One 32 × 32 “pretrained” matrix \(W_0\) must learn a new task. The new task needs a change of rank \(k\): the target is \(W^\star = W_0 + \Delta\) with \(\operatorname{rank}(\Delta) = k\). We have \(n\) training examples with noisy labels. Full fine-tuning trains all 1,024 weights. LoRA trains \(A\) and \(B\) with rank \(r\), so \(64 r\) weights. QLoRA trains the same LoRA on top of an NF4 copy of \(W_0\). All three use Adam with learning rate 0.01 for 300 steps. We plot the held-out error: the error on new inputs, divided by the error of doing nothing (keeping \(W_0\)). So 1 means “no better than the base”, 0 means perfect.
How to read Figure 4
At the default (\(r = k = 4\), 64 noisy examples), full fine-tuning drops fastest, then gets worse: with 1,024 free weights it fits the noise. LoRA has only 256 weights and a rank limit, which works as a regularizer, so its error stays lower. QLoRA ends a little above LoRA: it must also absorb the 4-bit error of \(W_0\). Now set \(r = 1\) with \(k = 4\): LoRA cannot express the change and stalls near 0.70. Set the noise to 0 as well: full fine-tuning now reaches almost 0, while LoRA with \(r = 1\) stays at 0.63. With no noise, \(n = 512\) and \(r = k = 4\), full fine-tuning and LoRA both reach almost 0, and QLoRA keeps a floor of 0.026. Real LLM results show the same three effects: rank can limit LoRA on hard tasks, small data favors fewer trained weights, and quantization adds a small floor.
The toy is a single matrix with a known answer, so it is honest only about the shape of the trade-offs, not about real LLM numbers. Here is the same experiment in PyTorch, with a from-scratch LoRA layer and NF4 quantizer. It runs on a CPU in about a second and is deterministic.
import torch, torch.nn as nn torch.manual_seed(0) # 1) The 16 NF4 code values (from the QLoRA paper / bitsandbytes) NF4 = torch.tensor([-1.0, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0911, 0.0, 0.0796, 0.1609, 0.2461, 0.3379, 0.4407, 0.5626, 0.7230, 1.0]) def nf4_roundtrip(w, block=64): """Quantize w to NF4 with one absmax scale per block, then dequantize.""" flat = w.reshape(-1, block) scale = flat.abs().amax(dim=1, keepdim=True) # one FP32 number per block idx = ((flat / scale).unsqueeze(-1) - NF4).abs().argmin(-1) # 4-bit index per weight return (NF4[idx] * scale).reshape(w.shape) # 2) A LoRA layer: frozen base weight + trainable low-rank update class LoRALinear(nn.Module): def __init__(self, W0, r=4, alpha=8, quantize=False): super().__init__() d_out, d_in = W0.shape base = nf4_roundtrip(W0) if quantize else W0.clone() self.register_buffer("W", base) # frozen: no gradient, no Adam state self.A = nn.Parameter(torch.randn(r, d_in) / d_in**0.5) # small random self.B = nn.Parameter(torch.zeros(d_out, r)) # zero, so B A = 0 at step 0 self.s = alpha / r def forward(self, x): return x @ self.W.T + self.s * (x @ self.A.T) @ self.B.T class FullLinear(nn.Module): def __init__(self, W0): super().__init__(); self.W = nn.Parameter(W0.clone()) def forward(self, x): return x @ self.W.T # 3) A "pretrained" W0 and a new task that needs a rank-4 change, with noisy labels d, k, n = 32, 4, 64 W0 = torch.randn(d, d) / d**0.5 dW = torch.randn(d, k) @ torch.randn(k, d) W_star = W0 + 0.5 * W0.norm() * dW / dW.norm() X = torch.randn(n, d) Y = X @ W_star.T + 0.5 * torch.randn(n, d) * (0.5 * W0.norm() / d**0.5) def held_out(W): # error on new inputs, relative to "do nothing" (W0) return ((W - W_star).norm()**2 / (W0 - W_star).norm()**2).item() for name, layer in [("full", FullLinear(W0)), ("lora r=4", LoRALinear(W0, r=4)), ("qlora r=4", LoRALinear(W0, r=4, quantize=True))]: opt = torch.optim.Adam([p for p in layer.parameters() if p.requires_grad], lr=1e-2) best = 9.9 for step in range(300): loss = ((layer(X) - Y)**2).mean() opt.zero_grad(); loss.backward(); opt.step() W = layer.W if name == "full" else layer.W + layer.s * layer.B @ layer.A best = min(best, held_out(W.detach())) n_train = sum(p.numel() for p in layer.parameters() if p.requires_grad) print(f"{name:10s} trainable {n_train:5d} train loss {loss.item():.4f} " f"held-out end {held_out(W.detach()):.3f} best {best:.3f}") q = nf4_roundtrip(W0) print("NF4 relative error of W0:", round(((q - W0).norm()**2 / W0.norm()**2).item(), 4))
Output (PyTorch 2.8, CPU):
full trainable 1024 train loss 0.0308 held-out end 0.248 best 0.245 lora r=4 trainable 256 train loss 0.0567 held-out end 0.049 best 0.049 qlora r=4 trainable 256 train loss 0.0624 held-out end 0.076 best 0.076 NF4 relative error of W0: 0.0085
Read it like a lab result. Full fine-tuning has the lowest training loss and the highest held-out error: it memorized the noise. This is why we always score on held-out data, and why a lower training loss is never proof of a better method.
Step 5: sentiment on IMDB
A review is a document, and we want one label for it. A decoder-only model reads left to right, so its last token’s hidden state has seen the whole review. AutoModelForSequenceClassification adds a new linear layer (the “score” head) on top of that last hidden state \(\mathbf{h}_{last}\), and trains with cross-entropy:
Here \(y\) is the true label (0 or 1) and \(W_s\) is the new head. The head is randomly initialized, so it must be trained in all three methods. PEFT does this for you: with task_type="SEQ_CLS" it adds the head (named score in Qwen) to modules_to_save, which means “train this module fully and save it with the adapter” [Certain]. One detail breaks many first runs: the model finds the last real token by looking for the padding token, so model.config.pad_token_id must be set.
- input: base model, method \(\in\) {full, LoRA, QLoRA}, fixed train and test subsets, seed 0
- load the base: FP32 (full), BF16 (LoRA) or NF4 with BF16 compute (QLoRA)
- if method is LoRA or QLoRA: freeze the base; add \(A, B\) with \(r = 16\), \(\alpha = 32\) to all linear layers
- set the learning rate: \(10^{-5}\) for full, \(2 \times 10^{-4}\) for LoRA and QLoRA // LoRA usually needs a \(\sim 10\times\) larger LR [Likely]
- reset the GPU peak-memory counter; start a timer
- train one epoch: cosine schedule, 3% warmup, BF16, gradient checkpointing
- record peak memory, wall time and samples per second
- score on the held-out test subset; save the model or the adapter; record the file size
The shared loader. It is the only place where the three methods differ:
# common.py: load the base model in one of three ways import torch from transformers import BitsAndBytesConfig from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training MODEL = "Qwen/Qwen2.5-0.5B" def load(auto_cls, method, task_type, **kw): if method == "full": # FP32 master weights + BF16 autocast (16 bytes/param) model = auto_cls.from_pretrained(MODEL, dtype=torch.float32, **kw) elif method == "lora": # frozen BF16 base (2 bytes/param) model = auto_cls.from_pretrained(MODEL, dtype=torch.bfloat16, **kw) elif method == "qlora": # frozen NF4 base (about 0.52 bytes/param) bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16) model = auto_cls.from_pretrained(MODEL, quantization_config=bnb, dtype=torch.bfloat16, **kw) model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True) if method in ("lora", "qlora"): cfg = LoraConfig(task_type=task_type, r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear") model = get_peft_model(model, cfg) model.print_trainable_parameters() return model def common_args(method): return dict(learning_rate=1e-5 if method == "full" else 2e-4, num_train_epochs=1, lr_scheduler_type="cosine", warmup_ratio=0.03, bf16=True, gradient_checkpointing=True, gradient_checkpointing_kwargs={"use_reentrant": False}, optim="paged_adamw_32bit" if method == "qlora" else "adamw_torch", logging_steps=25, save_strategy="no", report_to="none", seed=0)
The sentiment script. Run it three times: python train_cls.py full, then lora, then qlora.
# train_cls.py: IMDB sentiment with full FT, LoRA or QLoRA. Usage: python train_cls.py lora import sys, time, json, numpy as np, torch from datasets import load_dataset from transformers import (AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments, DataCollatorWithPadding, set_seed) from common import MODEL, load, common_args method = sys.argv[1]; set_seed(0) tok = AutoTokenizer.from_pretrained(MODEL) if tok.pad_token is None: tok.pad_token = tok.eos_token ds = load_dataset("stanfordnlp/imdb") train = ds["train"].shuffle(seed=0).select(range(5000)) test = ds["test"].shuffle(seed=0).select(range(2000)) enc = lambda b: tok(b["text"], truncation=True, max_length=512) train = train.map(enc, batched=True, remove_columns=["text"]) test = test.map(enc, batched=True, remove_columns=["text"]) model = load(AutoModelForSequenceClassification, method, "SEQ_CLS", num_labels=2) model.config.pad_token_id = tok.pad_token_id # the head reads the last non-pad token def metrics(p): return {"accuracy": float((np.argmax(p.predictions, -1) == p.label_ids).mean())} args = TrainingArguments(output_dir=f"out/cls-{method}", per_device_train_batch_size=8, gradient_accumulation_steps=2, per_device_eval_batch_size=16, **common_args(method)) trainer = Trainer(model=model, args=args, train_dataset=train, eval_dataset=test, data_collator=DataCollatorWithPadding(tok), compute_metrics=metrics) torch.cuda.reset_peak_memory_stats() t0 = time.time(); out = trainer.train(); dt = time.time() - t0 res = {"method": method, "accuracy": trainer.evaluate()["eval_accuracy"], "peak_GB": torch.cuda.max_memory_allocated() / 1e9, "train_minutes": dt / 60, "samples_per_s": out.metrics["train_samples_per_second"]} print(json.dumps(res)); open(f"cls-{method}.json", "w").write(json.dumps(res))
Three remarks on this code. First, full fine-tuning loads the model in FP32 and lets bf16=True run the math in BF16. If you load in BF16 and train all weights, the updates (learning rate \(10^{-5}\) times a small gradient) are often smaller than the BF16 step between two numbers near the weight, and they are lost [Likely]. Second, use_reentrant=False makes gradient checkpointing work when the input embeddings are frozen. Third, paged_adamw_32bit is the paged optimizer of the QLoRA paper (Adam with 32-bit states): its states can move to CPU memory when the GPU is briefly full, instead of crashing [Certain].
Is the difference real? Accuracy on \(n\) test examples is an estimate with a standard error:
With \(\hat{p} = 0.92\) and \(n = 2{,}000\), \(\mathrm{SE} = 0.0061\), so a 95% interval is about \(\pm 1.2\) points. Two methods that differ by less than about one point on 2,000 reviews are not clearly different. Because all methods are scored on the same reviews, a paired comparison is stronger: count the reviews where one method is right and the other is wrong, in each direction, and compare the two counts (McNemar’s test).
Step 6: summarization on XSum
Now the model must write. We turn each article into a prompt and its reference summary into the completion. This is supervised fine-tuning (session 54): next-token cross-entropy, but only on the completion tokens \(C\):
TRL’s SFTTrainer does this when the dataset has prompt and completion columns (completion_only_loss, which is the default for such data), and it adds the end-of-sequence token after the completion [Certain]. That EOS token matters: it is how the model learns to stop after one sentence.
# train_sum.py: XSum one-sentence summaries with TRL's SFTTrainer. Usage: python train_sum.py qlora import sys, time, json, torch from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed from trl import SFTConfig, SFTTrainer from common import MODEL, load, common_args method = sys.argv[1]; set_seed(0) tok = AutoTokenizer.from_pretrained(MODEL) if tok.pad_token is None: tok.pad_token = tok.eos_token def to_pc(ex): # prompt-completion format doc = ex["document"][:3000] # cut the ARTICLE, never the summary return {"prompt": f"Summarize the article in one sentence.\n\nArticle:\n{doc}\n\nSummary:", "completion": " " + ex["summary"]} # TRL appends the EOS token ds = load_dataset("EdinburghNLP/xsum") train = ds["train"].shuffle(seed=0).select(range(5000)).map(to_pc, remove_columns=ds["train"].column_names) model = load(AutoModelForCausalLM, method, "CAUSAL_LM") cfg = SFTConfig(output_dir=f"out/sum-{method}", max_length=1024, completion_only_loss=True, per_device_train_batch_size=4, gradient_accumulation_steps=4, **common_args(method)) trainer = SFTTrainer(model=model, args=cfg, train_dataset=train, processing_class=tok) torch.cuda.reset_peak_memory_stats() t0 = time.time(); out = trainer.train(); dt = time.time() - t0 trainer.save_model(f"out/sum-{method}/final") # full model, or only the adapter for LoRA/QLoRA res = {"method": method, "peak_GB": torch.cuda.max_memory_allocated() / 1e9, "train_minutes": dt / 60, "samples_per_s": out.metrics["train_samples_per_second"]} print(json.dumps(res)); open(f"sum-{method}.json", "w").write(json.dumps(res))
We cut the article to 3,000 characters ourselves. If you let the trainer cut long rows at max_length, it cuts the end of the row, and the end is the summary: the model then trains on rows with no target at all.
Then generate summaries for 500 test articles and score them. Note the left padding: in a batch of prompts of different lengths, a decoder model must have all prompts end at the same position, so the padding goes on the left.
# eval_sum.py: generate summaries for 500 test articles and score them with ROUGE import sys, json, torch from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForCausalLM from peft import PeftModel from rouge_score import rouge_scorer from common import MODEL method = sys.argv[1] # "base", "full", "lora" or "qlora" tok = AutoTokenizer.from_pretrained(MODEL); tok.padding_side = "left" # left-pad for generation if tok.pad_token is None: tok.pad_token = tok.eos_token path = MODEL if method in ("base", "lora", "qlora") else f"out/sum-{method}/final" model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16).to("cuda") if method in ("lora", "qlora"): # BF16 base + adapter, merged = the artifact you ship model = PeftModel.from_pretrained(model, f"out/sum-{method}/final").merge_and_unload() model.eval() test = load_dataset("EdinburghNLP/xsum")["test"].shuffle(seed=0).select(range(500)) prompts = [f"Summarize the article in one sentence.\n\nArticle:\n{d[:3000]}\n\nSummary:" for d in test["document"]] outs = [] for i in range(0, len(prompts), 16): b = tok(prompts[i:i+16], return_tensors="pt", padding=True).to("cuda") with torch.no_grad(): g = model.generate(**b, max_new_tokens=60, do_sample=False, pad_token_id=tok.pad_token_id) for row in tok.batch_decode(g[:, b["input_ids"].shape[1]:], skip_special_tokens=True): outs.append(row.strip().split("\n")[0]) # keep the first line only sc = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True) scores = [sc.score(r, p) for r, p in zip(test["summary"], outs)] res = {k: sum(s[k].fmeasure for s in scores) / len(scores) for k in ["rouge1", "rouge2", "rougeL"]} print(method, json.dumps(res)) json.dump({"method": method, "rouge": res, "preds": outs}, open(f"preds-{method}.json", "w"))
Run eval_sum.py base as well. The untrained base model is your baseline: every method must beat it clearly, or the training did not work.
ROUGE
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) counts word overlap between a candidate summary and the reference. ROUGE-N uses n-grams (sequences of \(N\) words), counting each n-gram at most as often as it appears in the other text:
ROUGE-L uses the longest common subsequence (LCS): the longest list of words that appears in both texts in the same order, with gaps allowed.
ROUGE is cheap, deterministic and useful for comparing runs of the same system. But it only sees words, not meaning. Figure 5 shows the problem with real numbers.
rouge_score package without stemming (lower case, letters and digits only). Green words are matched unigrams; orange words also belong to a matched bigram.How to read Figure 5
The faithful paraphrase (“put out a blaze that burned for 48 hours”) gets ROUGE-1 of only 0.343. The wrong summary says the fire is “still not under control”, the opposite of the reference, and gets ROUGE-1 of 0.842 and ROUGE-L of 0.737. The very short summary has perfect precision but low recall. Try writing your own: you can raise ROUGE a lot by copying words, without being more correct. This is why we add a judge that reads the article.
We verified the three presets with the rouge_score package. In eval_sum.py we turn stemming on (use_stemmer=True), which is common for summarization and gives slightly different numbers. Always report which setting you used.
Step 7: an LLM judge
An LLM judge is a strong instruction-tuned model that reads the article and two summaries and says which is better. Zheng et al. found that strong judges agree with human preferences about as well as humans agree with each other on their benchmarks, but they also found biases [Certain]: position bias (preferring the answer shown first), verbosity bias (preferring longer answers) and self-enhancement bias (preferring answers in their own style). Our script fights the first two directly.
- input: test articles, summaries from system \(a\) and system \(b\), a judge model that is not one of the systems
- for each article:
- \(v_1 \leftarrow \text{judge}(\text{article},\ A = \text{summary of } a,\ B = \text{summary of } b)\) // greedy decoding: deterministic
- \(v_2 \leftarrow \text{judge}(\text{article},\ A = \text{summary of } b,\ B = \text{summary of } a)\), then map the answer back
- if \(v_1 = v_2\): count that verdict else: count a tie // the verdict depended on the order
- return wins \(W\), ties \(T\), losses \(L\), and the win rate (62.13)
# judge.py: pairwise LLM judge with position swap. Usage: python judge.py lora full import sys, json, torch from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForCausalLM a, b = sys.argv[1], sys.argv[2] JUDGE = "Qwen/Qwen2.5-7B-Instruct" # any strong instruct model that is NOT one of the contestants jt = AutoTokenizer.from_pretrained(JUDGE) jm = AutoModelForCausalLM.from_pretrained(JUDGE, dtype=torch.bfloat16, device_map="auto").eval() test = load_dataset("EdinburghNLP/xsum")["test"].shuffle(seed=0).select(range(500)) pa = json.load(open(f"preds-{a}.json"))["preds"]; pb = json.load(open(f"preds-{b}.json"))["preds"] RUBRIC = ("You compare two one-sentence summaries of a news article.\n" "First check faithfulness: a summary that states something the article does not support loses.\n" "Then check coverage of the main point, then clarity. Length is not a quality.\n" "Answer with one word on the last line: A, B or TIE.") def ask(doc, s1, s2): msg = [{"role": "system", "content": RUBRIC}, {"role": "user", "content": f"Article:\n{doc[:3000]}\n\nSummary A: {s1}\n\nSummary B: {s2}"}] enc = jt.apply_chat_template(msg, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(jm.device) with torch.no_grad(): out = jm.generate(**enc, max_new_tokens=200, do_sample=False) words = jt.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).split() last = words[-1].upper().strip(".*:") if words else "TIE" return last if last in ("A", "B", "TIE") else "TIE" win = tie = loss = 0 for doc, x, y in zip(test["document"], pa, pb): v1 = ask(doc, x, y) # a shown first v2 = {"A": "B", "B": "A", "TIE": "TIE"}[ask(doc, y, x)] # b shown first, mapped back v = v1 if v1 == v2 else "TIE" # disagree after the swap: count as a tie win += v == "A"; loss += v == "B"; tie += v == "TIE" n = win + tie + loss print(f"{a} vs {b}: win {win}, tie {tie}, loss {loss}, win rate {(win + 0.5 * tie) / n:.3f}")
A win rate of 0.5 means “no difference”. It has noise too. As an illustration with made-up counts: 180 wins, 170 ties and 150 losses over 500 articles give a win rate of \((180 + 85)/500 = 0.53\), and a rough 95% interval of \(\pm 0.044\) (formula 62.9 with \(n = 500\)). That interval includes 0.5, so this result alone would not show that \(a\) is better. Before you trust any judge, read 20 of its verdicts yourself and check that you agree with most of them.
Step 8: measure speed and fill the table
Record three speed numbers per run: wall time, train_samples_per_second from the trainer, and the peak memory. Here is what to expect for compute, from the FLOP rules of session 25. A training step costs about \(2N\) FLOPs per token forward and \(4N\) backward: \(2N\) for gradients of the inputs (to pass the gradient down) and \(2N\) for gradients of the weights. A frozen weight needs no weight gradient. Gradient checkpointing adds one more forward pass:
So LoRA can save at most about 25% of the compute here (\(6/8 = 0.75\)); the LoRA matrices themselves add only about \(1.78\%\) of the model’s FLOPs at \(r = 16\). In practice, small models are often limited by memory traffic and kernel overheads rather than FLOPs, so the measured speedup can be smaller [Likely]. QLoRA adds dequantization in every forward pass and in the backward pass, so expect it to be slower than LoRA [Likely]. The big gain of LoRA is memory, which lets you use a larger batch or a longer sequence on the same card. That can make it faster in wall time, even if a single step is not.
What the papers found, so you know what to look for in your own table:
- The LoRA paper reports results that match or beat full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 for the tasks it tested, with no extra latency after merging [Certain].
- The QLoRA paper reports that 4-bit NF4 with double quantization matches 16-bit LoRA and 16-bit full fine-tuning on GLUE (RoBERTa-large) and Super-NaturalInstructions (T5 up to 11B; full fine-tuning compared up to 3B), and recovers 16-bit LoRA MMLU results for LLaMA 7B to 65B. It also found that LoRA must be applied to all linear layers to match full fine-tuning; attention-only LoRA fell short [Certain].
- Biderman et al. (“LoRA Learns Less and Forgets Less”) found that on code and math, with large training sets, LoRA is clearly worse than full fine-tuning on the target task, but keeps more of the base model’s other skills [Certain].
Our tasks are much easier than code and math, and our training sets are small. Whether the methods tie on your table is exactly what you are measuring. Do not assume the answer.
Also measure forgetting. A fine-tuned model may lose general skills (session 61). A cheap check: compute the perplexity of each final model on a few hundred sentences of general text (for example WikiText) and compare it with the base model. Put it in an extra column.
Step 9: write down what you would ship
The last step is a short memo, half a page, as if for your team. A decision without written reasons cannot be reviewed. Use this structure:
- Requirement. Example: “Classify reviews at 200 requests per second on one L4; summaries for an internal tool; three more tasks next quarter.”
- Results. Your table, with intervals: accuracy \(\pm\) SE, win rates with counts.
- Decision. One method, and the artifact you ship: merged weights, or base model plus adapter.
- Why. The deciding facts: quality difference versus noise, training cost, serving cost, number of future tasks.
- Risks and checks. What you will monitor, and what would make you change the decision.
Some rules that usually decide it:
- If the quality difference is inside the noise, pick the cheapest method to train and to serve. For a 0.5B model that is usually LoRA in BF16: it fits easily, and QLoRA saves little memory at this size (Figure 2) while running slower.
- If you will serve many tasks or customers, LoRA adapters win: one shared base in GPU memory, a small adapter per task (8.8M weights here, about 35 MB in FP32 or 18 MB in BF16), and serving engines that can batch requests for different adapters together [Likely].
- If the model does not fit in 16-bit on your training GPU, QLoRA is often the only option, and it is a good one.
- If full fine-tuning is clearly better on a task that matters (beyond the interval), and you can afford it, ship it, but check forgetting first.
- Evaluate the exact artifact you ship. A QLoRA adapter was trained against the 4-bit base. Merged into the BF16 base, it gives a slightly different model. Our
eval_sum.pyscores the merged BF16 model for this reason. If you will serve the 4-bit base plus the adapter instead, score that.
Where this is used
| Setting | Usual choice | Why |
|---|---|---|
| One small model (under ~2B), one task, enough GPU | Full fine-tuning or LoRA | Both fit; compare on your data and pick the cheaper one if they tie |
| 7B to 70B model on one or two GPUs | QLoRA | The 16-bit base alone may not fit next to the activations |
| Many customers or tasks on one base model | LoRA adapters, served together | One base in memory, small files per task, fast switching [Likely] |
| Large domain shift with a lot of data (code, math, a new language) | Full fine-tuning or a high-rank LoRA | Low-rank updates can limit learning on such tasks (Biderman et al.) |
| Document classification: sentiment, topic, intent, spam | LoRA plus a trained classification head | Small adapters; the head must be trained fully |
| Video classification and action recognition with a VLM | LoRA on the language model, often a frozen vision tower | Video token sequences are long, so activations dominate memory and states must be small [Likely] |
| Video summaries, key-moment descriptions and captions for Video-RAG | LoRA or QLoRA SFT, scored with ROUGE plus an LLM judge | Same metric problems as text summaries; a judge must see the transcript or the frames [Likely] |
| Video retrieval and fingerprinting embeddings | LoRA on the encoder with a contrastive loss | Keeps the pretrained embedding space close to the original while adapting it (session 60) |
Common mistakes
- Different settings per method. Different data, seeds, steps or sequence lengths make the comparison meaningless. Only the method and its learning rate should change.
- The same learning rate for all three. \(2 \times 10^{-4}\) is a common LoRA rate but usually far too high for full fine-tuning; \(10^{-5}\) is often too low for LoRA. Tune each method briefly on a validation split [Likely].
- No
pad_token_idfor classification. The classifier reads the last non-padding token. Without the padding id it cannot find it and fails for batches larger than one. - Truncating away the target. Cutting long rows at the end removes the summary. Cut the article instead.
- Right padding at generation time. A decoder model then continues from padding tokens. Use
padding_side="left"for batched generation. - Thinking LoRA saves activation memory. It saves gradients and optimizer states of the frozen weights. Activations stay; use gradient checkpointing or a smaller batch.
- Trusting ROUGE alone, or a judge alone. ROUGE misses meaning (Figure 5); judges have position and length biases. Use both, swap positions, and read samples yourself.
- Evaluating a different artifact than you ship. Score the merged model, or the base plus adapter, in the exact precision you will serve.
Interview questions
Why is \(B\) initialized to zero in LoRA, and not \(A\) as well?basic
How many weights does LoRA train with \(r = 16\) on all linear layers of Qwen2.5-0.5B?core
Does LoRA reduce activation memory? Explain.core
In QLoRA, what is stored in 4 bits and what is not? In what precision is the math done?core
Why NF4 instead of plain INT4?advanced
What does double quantization save? Compute it.core
Why is ROUGE not enough to evaluate summaries, and how do you make an LLM judge trustworthy?core
When does LoRA fall short of full fine-tuning?advanced
How do you serve a LoRA model, and what changes for a QLoRA-trained adapter?core
Your QLoRA run is slower per step than LoRA. Why, and when is it still the right choice?advanced
What does the \(\alpha / r\) scale do? What happens if you double \(r\) and keep \(\alpha\)?basic
use_rslora=True.Exercises
- Count the LoRA weights for Qwen2.5-1.5B (28 blocks, \(d = 1{,}536\), feed-forward size 8,960, key and value outputs of size 256) with \(r = 8\) on all linear layers.
- Use formula (62.5) for Qwen2.5-7B with \(r = 16\) on all linear layers: \(N_{lin} = 6{,}525{,}288{,}448\), \(N = 7{,}615{,}616{,}512\), \(P = 40{,}370{,}176\). What if the non-quantized weights stay in BF16 instead of FP32?
- You score 500 reviews and get 90% accuracy. What is the 95% interval?
- Reference: “the cat sat on the mat”. Candidate: “the cat lay on the mat”. Compute ROUGE-1 and ROUGE-2 F1 by hand. Check it in Figure 5.
- In Figure 4, find a setting where LoRA ends above full fine-tuning, and a setting where it ends below. Explain both in one sentence each.
- You must fine-tune Qwen2.5-7B on one 24 GB GPU with \(b = 8\), \(s = 512\). Use Figure 2 to decide which methods may fit. What would you change if LoRA is too tight?
Answerscheck
Key takeaways
- LoRA (62.1) freezes \(W_0\) and trains \(r(d_{in} + d_{out})\) weights per layer (62.2): 8.8M of 494M for Qwen2.5-0.5B at \(r = 16\) on all linear layers.
- Memory: full fine-tuning costs 16 bytes per parameter (62.3), LoRA 2 bytes plus 16 per LoRA weight (62.4), QLoRA about 0.52 bytes per quantized weight (62.5). Activations (62.6) are the same for all three, and for small models with big vocabularies the logits can dominate.
- NF4 (62.7) uses normal quantiles in blocks of 64; double quantization cuts the scale overhead from 0.5 to 0.127 bits per weight. The math still runs in BF16.
- Fewer trained weights can generalize better on small, noisy data, but a low rank can limit hard tasks (Figure 4, Biderman et al.).
- Score with held-out data and intervals (62.9). For summaries, use ROUGE (62.11, 62.12) and a position-swapped LLM judge (62.13); ROUGE alone rewards copying words.
- Ship the cheapest method whose quality is within the noise of the best, and evaluate the exact artifact you serve.
Key formulas
References for this session
- Hugging Face (code). PEFT library
- Hugging Face (code). TRL library
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Biderman et al. (2024). LoRA Learns Less and Forgets Less
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Lin (2004). ROUGE: A Package for Automatic Evaluation of Summaries
- Liu et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Maas et al. (2011). Learning Word Vectors for Sentiment Analysis (the IMDB dataset)
- Narayan, Cohen & Lapata (2018). Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (XSum)
- Qwen Team (2024). Qwen2.5 Technical Report
- Kalajdzievski (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA