LoRA fine-tune a small VLM
In this lab you fine-tune SmolVLM-256M, a small open vision-language model, on ChartQA with LoRA. You compare three choices for the vision tower (frozen, LoRA, fully trained), score the model before and after on held-out data with a paired test, and check that its text-only skills did not get worse.
What you will learn
- How to count what a freeze map trains with formula r(d_in + d_out), and why the vision tower dominates compute in a small tiled VLM
- How SmolVLM turns one chart into 13 tiles and 832 image tokens, and how to put the loss on the 5 answer tokens only
- How to measure a base model first with ChartQA relaxed accuracy, and the blind spots of that metric
- How to train LoRA runs with a frozen, LoRA or fully trained vision tower with PEFT and TRL, and the bugs that silently break them
- When unfreezing the vision tower helps, and why a frozen tower can cause more text drift (a live toy)
- How to compare before and after with a paired interval and an exact McNemar test
- How to check text-only regressions with ARC-Easy, answer length and KL drift
Why this matters
You have a small open vision-language model (VLM) and a set of images from your own domain: charts, forms, product photos, video frames. The model answers questions about them, but not well enough. In this lab you fine-tune it with LoRA, compare three choices for the vision tower (frozen, LoRA, fully trained), measure the score before and after on held-out data, and check that the model did not lose its text-only skills on the way.
The theory is in earlier sessions. Session 189 explained the freeze map (which part to freeze, adapt or train), per-part learning rates and answer-only loss. Session 62 ran the same kind of comparison for a text-only model. Session 192 and session 193 showed how VLM scores can mislead. This lab puts it together on one real model and one real dataset, with numbers you measure yourself.
You will:
- Count what each freeze map trains and what it costs in compute and memory, before you run anything (Figure 1).
- Turn one ChartQA row into a training example and check its loss mask token by token (Figure 2).
- Score the untouched base model on the held-out test set, with the ChartQA metric (Figure 3).
- Train three LoRA runs that differ only in the vision tower, and build intuition with a toy you can train in the browser (Figure 4).
- Compare before and after with a paired test (Figure 5), and run a text-only regression check.
- Fill one results table and write down what you would ship.
Real systems do exactly this. Open VLM families ship fine-tuning tutorials (the SmolVLM model card links one), and Hugging Face’s PEFT and TRL libraries are the common open-source tools for it [Certain]. In industry, the same loop adapts VLMs to receipts, charts, medical images, screenshots and video frames.
Setup
Hardware. One GPU with 16 GB is enough for every run on this page; the small model even trains (slowly) on a CPU. On a free Colab T4, note that the T4 has no BF16 support: load the model in float32 and use fp16=True instead of bf16=True [Certain].
pip install "transformers>=4.56" peft trl datasets accelerate pillow
The code was written for transformers 4.57, peft 0.17 and trl 0.24, and we ran every script end to end on a CPU in a smoke mode (8 training rows, 2 steps, 4 test rows, no image splitting) to check that it runs. We did not run the full GPU experiments, and this page reports no fine-tuning results: you will measure them. These libraries change often; if an argument name fails, check the current documentation.
Model. HuggingFaceTB/SmolVLM-256M-Instruct, Apache 2.0. It has the Idefics3 architecture: a SigLIP vision tower (base size, patch 16, 512 × 512 input), a pixel-shuffle connector, and a SmolLM2-135M language model [Certain, from the model card and config.json]. We built it from its config and counted every weight:
| Part | What it is | Weights |
|---|---|---|
| Vision tower | 12 ViT blocks, width 768, MLP 3,072; 1,024 patches per 512 px tile | 86,433,024 |
| Connector | pixel shuffle (16 neighbor patches → one vector of 12,288) + one linear layer 12,288 → 576 | 7,077,888 |
| LLM blocks | 30 blocks, width 576, 9 query heads and 3 key-value heads of size 64, MLP 1,536 | 106,203,456 |
| Embeddings and output head | vocabulary 49,280, not tied (two separate tables) | 28,385,280 + 28,385,280 |
| Total | matches the 256,484,928 BF16 weights of the released file | 256,484,928 |
The vision tower is a third of the model, which is unusual: in a 7B VLM it is under a tenth. For the next size up, SmolVLM-500M-Instruct (507,482,304 weights, SmolLM2-360M inside), every script works unchanged.
Data. ChartQA (HuggingFaceM4/ChartQA): bar, line and pie charts with questions and short answers [Certain]. Each split has two halves: questions written by people (“human”) and questions generated by a model from the chart’s data table (“augmented”).
| Split | Human | Augmented | Total | We use |
|---|---|---|---|---|
| train | 7,398 | 20,901 | 28,299 | 5,000 rows (seeded shuffle) |
| val | 960 | 960 | 1,920 | to pick the learning rate and the number of steps |
| test | 1,250 | 1,250 | 2,500 | all, only at the end; report human and augmented separately |
In the test set, 76.6% of the answers are numbers, 5.8% are years and 104 answers (8.3% of the human half) are “Yes” or “No”; 2,295 of 2,500 answers are a single word [Certain, we counted them].
The table you will fill. Copy it into your notes now. The trainable counts come from the smoke runs; everything else you measure.
| Run | Vision tower | Trainable weights | Peak GPU memory | Train time | ChartQA human | ChartQA augmented | ARC-Easy (text) | Text KL drift |
|---|---|---|---|---|---|---|---|---|
| Base | n/a | 0 | n/a | n/a | 0 | |||
| A | frozen | 4,884,480 (1.90%) | ||||||
| B | LoRA | 7,538,688 (2.94%) | ||||||
| C | fully trained | 91,317,504 (35.6%) |
Step 1: what each freeze map trains and costs
All three runs put LoRA on the seven linear layers of every LLM block (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), with rank \(r = 16\) and \(\alpha = 32\). They differ only in the vision tower: A freezes it, B adds LoRA to its six linear layers per block (q_proj, k_proj, v_proj, out_proj, fc1, fc2), C trains all of its weights with a 10 times smaller learning rate. The connector stays frozen in all three, so the comparison changes one thing at a time.
A LoRA layer keeps the pretrained matrix \(W_0\) frozen and adds a low-rank path (session 57):
- Read the symbols
- \(\mathbf{x}, \mathbf{h}\)
- the input and output of one linear layer
- \(W_0\)
- the frozen pretrained weight; it gets no gradient and no optimizer state
- \(A, B\)
- the two small trainable matrices; \(B = 0\) at the start, so the fine-tuned model begins exactly at the base model
- \(r, \alpha\)
- the rank (16 here) and a fixed scale (32 here), so the update is multiplied by \(\alpha / r = 2\)
- \(P_{\text{LoRA}}\)
- the number of new trainable weights
LLM, one block. q_proj and o_proj are 576 → 576, k_proj and v_proj are 576 → 192 (3 heads of 64), gate_proj and up_proj are 576 → 1,536, down_proj is 1,536 → 576. So \(\sum (d_{in} + d_{out}) = 2 \cdot 1{,}152 + 2 \cdot 768 + 3 \cdot 2{,}112 = 10{,}176\). Times \(r = 16\) and 30 blocks: 4,884,480. This is run A. PEFT prints trainable params: 4,884,480 || all params: 261,369,408 || trainable%: 1.8688; the “all params” number includes the new LoRA weights, so it is 4,884,480 more than the 256,484,928 of the base model.
Vision tower, one block. Four 768 → 768 attention matrices and the MLP 768 → 3,072 → 768: \(4 \cdot 1{,}536 + 2 \cdot 3{,}840 = 13{,}824\). Times 16 and 12 blocks: 2,654,208. Run B trains \(4{,}884{,}480 + 2{,}654{,}208 = \mathbf{7{,}538{,}688}\). Run C trains the LLM LoRA plus all 86,433,024 tower weights: \(4{,}884{,}480 + 86{,}433{,}024 = \mathbf{91{,}317{,}504}\). All three counts match what PEFT printed in our smoke runs.
The parameter count is not the cost. A VLM has a fixed order: vision tower, then connector, then LLM, then the loss. The backward pass starts at the loss and must go down to the lowest part that has trainable weights. With \(N_p\) weights in part \(p\) and \(t_p\) tokens going through it, a matrix multiply costs about \(2N_p\) FLOPs per token, so per sample:
- Read the symbols
- \(t_p\)
- tokens through part \(p\): 1,024 patches per tile for the vision tower, 64 tokens per tile for the connector, the whole sequence for the LLM
- \([\cdot]\)
- 1 if the condition is true, else 0. The first bracket pays for input gradients, the second for weight gradients, the third for the extra forward pass of gradient checkpointing
Now the surprise. With image splitting, a typical 850 × 600 chart becomes 13 tiles, so the vision tower sees \(13 \times 1{,}024 = 13{,}312\) patches, while the LLM sees only about 895 tokens. One forward pass of the tower costs \(2 \times 86.4\text{M} \times 13{,}312 = 2.30\) TFLOPs, the LLM (blocks and output head) only \(2 \times 134.6\text{M} \times 895 = 0.24\) TFLOPs. In this small VLM, almost all the compute is in the vision tower, even when it is frozen. Explore it in Figure 1.
config.json; the token counts (895 for 13 tiles, 100 for 1 tile, 1,160 for 17 tiles) are from the real processor on one ChartQA test row.How to read Figure 1
Start with run A. The tower is frozen, yet it is still 76% of the compute (18.41 of 24.29 TFLOPs at batch 8): it must run forward for every image. Click run B: the backward pass must now go down through the tower, and the step costs 61.29 TFLOPs, 2.5 times run A, for only 2.7M more weights. Run C adds weight gradients for the tower: 79.70 TFLOPs. Now switch to 1 tile: the compute drops more than 10 times, because the image becomes 64 tokens instead of 832. Image splitting is the biggest cost knob in this lab, bigger than any freeze choice. Memory stays small for the 256M model (run A about 2.4 GB before overheads), so any 16 GB GPU fits; the activations of a trainable tower (6.5 GB in run B) are the largest term.
For memory we use the same simple budget as in session 38: frozen weights in BF16 cost 2 bytes each; every trained weight (LoRA or full) costs 16 bytes (FP32 copy, gradient and two Adam moments); and activations are stored only for parts that the backward pass goes through:
- Read the symbols
- \(L_p, d_p\)
- the number of blocks and the width of part \(p\) (12 and 768 for the tower, 30 and 576 for the LLM)
- \(\kappa\)
- bytes stored per token per block: about 34 without checkpointing (from Korthikanti et al. for a standard block, with the attention-score term removed by fused attention), about 2 with checkpointing (only each block’s BF16 input, plus one block at a time recomputed)
- \(b\, s\, V \cdot 4\)
- the FP32 logits: micro-batch times sequence length times vocabulary size (49,280)
This is a planning estimate [Likely], not a measurement: real peaks add temporary buffers, the CUDA context and fragmentation. Measure the real peak with torch.cuda.max_memory_allocated() and write it in your table.
Step 2: one ChartQA row becomes one training example
Each training example is one short conversation: the user sends a chart and a question, and the assistant answers. We add one fixed line to every question, “Answer with a single word or number.”, so that the base model and the fine-tuned model are asked for the same format. The processor then does three things:
- Resize and split the image. SmolVLM first resizes the image so its longest side is 2,048 pixels (small images are made larger), then cuts it into tiles of 512 × 512, and adds one extra 512 px view of the whole image (the “global” tile).
- Turn tiles into tokens. Each tile gives \(32 \times 32 = 1{,}024\) patches of 16 px. Pixel shuffle merges each 4 × 4 group of patch vectors into one vector, so each tile becomes \(1{,}024 / 16 = 64\) tokens for the LLM (session 178).
- Apply the chat template. Special tokens mark each tile’s row and column, then the question, then “Assistant:”, then the answer, then
<end_of_utterance>.
Here \(W', H'\) are the width and height after resizing. An 850 × 600 chart becomes 2,048 × 1,446, so \(T = 4 \cdot 3 + 1 = 13\) and \(n_{\text{img}} = 832\). A square image gives \(4 \cdot 4 + 1 = 17\) tiles and 1,088 tokens. In the ChartQA test set, 2,178 images give 13 tiles, 299 give 17 and 23 give 9 [Certain, we ran the processor on all 2,500]. With do_image_splitting=False, every image is one 512 px tile: 64 tokens.
Next, the loss mask. The model is trained with next-token cross-entropy, but only on the answer tokens (the set \(\mathcal{A}\)). Everything else gets the label -100, which PyTorch’s cross-entropy ignores:
- Read the symbols
- \(I\)
- the image (its 832 tokens enter the LLM as context)
- \(x_t\)
- the token at position \(t\) of the templated sequence
- \(\mathcal{A}\)
- the positions of the answer tokens, here “ 1 4 <end_of_utterance> \n”: 5 tokens. The end token is in \(\mathcal{A}\) on purpose: it teaches the model to stop
How to read Figure 2
Step to the token view, then to the labels. In “answer only” mode, 5 of 895 tokens carry loss. In “loss on everything but padding” mode, 894 tokens carry loss (the first token is never predicted) and the answer gets only 0.56% of the training signal: the model spends almost all its effort learning to predict <image> placeholders and the fixed template. Masking only the image tokens is better (62 labeled tokens, answer share 8.06%) but still trains the model to predict the question. Note that TRL’s default for a dataset with a messages column is the first mode: its VLM collator masks only padding [Certain, TRL 0.24 source]. The prompt-completion format we use gives the third mode.
- input: rows (image, question, answer), the processor, the chat template
- for each row: prompt \(\leftarrow\) user turn with one image placeholder and the question plus the format line
- completion \(\leftarrow\) assistant turn with the answer; the template adds
<end_of_utterance> - process the prompts with their images // resize, split into tiles, expand each placeholder to \(64 \cdot T\) image tokens
- tokenize the completions; concatenate prompt and completion ids per row
- pad to the longest row; labels \(\leftarrow\) ids; labels[padding] \(\leftarrow -100\); labels[prompt] \(\leftarrow -100\)
- never truncate a row: cutting it would remove the answer, or cut image tokens and break the image-token count
- return input ids, attention mask, pixel values, labels
TRL implements Algorithm 1 for you when the dataset has images, prompt and completion columns. Check it once on a real batch, as in Figure 2; it costs five minutes and catches the most common silent bug in VLM fine-tuning.
# check_batch.py: print which tokens carry loss in one real batch
from trl.trainer.sft_trainer import DataCollatorForVisionLanguageModeling
from common import chartqa, processor, SUFFIX
proc = processor(split_images=True)
collate = DataCollatorForVisionLanguageModeling(proc, completion_only_loss=True)
rows = chartqa("val").select(range(2))
batch = collate([{"images": [r["image"]],
"prompt": [{"role": "user", "content": r["query"] + SUFFIX}],
"completion": [{"role": "assistant", "content": r["label"][0]}]} for r in rows])
for ids, lab in zip(batch["input_ids"], batch["labels"]):
toks = proc.tokenizer.convert_ids_to_tokens(ids)
print(len(toks), "tokens; with loss:", [t for t, l in zip(toks, lab) if l != -100])
On the first two validation rows, our smoke run printed ['ĠBlue', '<end_of_utterance>', 'Ċ'] and ['Ġ', '2', '0', '1', '8', '<end_of_utterance>', 'Ċ']. Note that the tokenizer writes each digit as its own token, so “2018” is four tokens.
Step 3: score the base model first
Before you train, measure the base model on the test set with exactly the prompt, decoding and metric you will use after training. This number is the “before” in “before and after”. It also tests your evaluation code: the SmolVLM-256M model card reports 55.8 on the ChartQA test set (63.2 for the 500M model) [Certain, model card]. Your prompt and answer parsing differ from theirs, so expect your number to differ by a few points [Likely]. If you get 20, your evaluation is broken, not the model.
The ChartQA metric. ChartQA uses relaxed accuracy: a numeric answer is correct if it is within 5% of the gold answer; any other answer must match exactly [Certain, ChartQA paper]. We also ignore case, a final period, commas and a percent sign.
The 5% rule exists because human-written gold values are read off charts and are often slightly off. But it has a blind spot: it treats a year like any other number. For the gold answer 2018, every number from 1917.1 to 2118.9 counts as correct. Years are 144 of the 2,500 test answers. Try it in Figure 3.
| Model output | Gold answer | Exact match | Relaxed |
|---|
How to read Figure 3
At 5%, exact match gives 2 of 9 and relaxed accuracy 7 of 9. Some of the gap is fair (“14.” and “14”, “1,234” and “1234”, 86.3 read from a chart whose table says 86.33). Some is not: “2017” for “2018” is wrong but counts as correct. “0.6” for “0.57” is 5.26% off and fails; move the tolerance to 10% and it passes. “The answer is 14” fails under both rules: format matters, and fine-tuning often “gains” points only by learning to answer in one word. When you report a gain, check how much of it is format.
The evaluation script. It generates greedily (deterministic), uses left padding so that every prompt in a batch ends at the same position, and saves one row per test item with a correct/wrong flag. We need those per-item flags for the paired test in Step 6.
# eval_vlm.py: ChartQA relaxed accuracy. Usage: python eval_vlm.py base|A|B|C
import sys, json, torch
from transformers import AutoModelForImageTextToText
from peft import PeftModel
from common import MODEL, SMOKE, SUFFIX, DEV, DTYPE, chartqa, processor
def norm(s):
return s.strip().rstrip(".").replace(",", "").replace("%", "")
def relaxed_match(pred, gold, tol=0.05):
"""ChartQA rule: numbers may be off by 5% (relative); everything else must match."""
p, g = norm(pred), gold.strip().replace(",", "").replace("%", "")
try:
pv, gv = float(p), float(g)
return abs(pv - gv) <= tol * abs(gv) if gv != 0 else pv == 0
except ValueError:
return p.lower() == g.lower()
def load_model(run):
model = AutoModelForImageTextToText.from_pretrained(MODEL, dtype=DTYPE)
if run != "base":
model = PeftModel.from_pretrained(model, f"out/{run}/adapter")
if run == "C": # the unfrozen tower is saved separately
r = model.load_state_dict(torch.load(f"out/{run}/vision_tower.pt"), strict=False)
assert not r.unexpected_keys, r.unexpected_keys # strict=False hides typos
model = model.merge_and_unload() # evaluate the artifact you would ship
return model.to(DEV).eval()
if __name__ == "__main__":
run = sys.argv[1]
proc = processor(split_images=not SMOKE); proc.tokenizer.padding_side = "left"
model = load_model(run)
test = chartqa("test") # 1,250 human + 1,250 augmented
if SMOKE: test = test.select(range(4))
rows, bs = [], 16
for i in range(0, len(test), bs):
b = test[i:i + bs]
msgs = [[{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": q + SUFFIX}]}]
for q in b["query"]]
texts = [proc.apply_chat_template(m, add_generation_prompt=True) for m in msgs]
enc = proc(text=texts, images=[[im] for im in b["image"]], return_tensors="pt",
padding=True).to(DEV)
with torch.no_grad():
out = model.generate(**enc, max_new_tokens=16, do_sample=False)
preds = proc.batch_decode(out[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)
for q, p, g, h in zip(b["query"], preds, b["label"], b["human_or_machine"]):
rows.append({"q": q, "pred": p.strip(), "gold": g[0], "human": h == 0,
"ok": relaxed_match(p, g[0])})
acc = lambda rs: sum(r["ok"] for r in rs) / max(len(rs), 1)
res = {"run": run, "n": len(rows), "all": acc(rows),
"human": acc([r for r in rows if r["human"]]),
"augmented": acc([r for r in rows if not r["human"]])}
print(json.dumps(res)); json.dump({"summary": res, "rows": rows}, open(f"eval-{run}.json", "w"))
In smoke mode the base model answered the first four test rows with “Corn.”, “190000000.”, “3.” and “Yes.” (gold: 14, 0.57, 3, No). One in four is right; the trailing period on “3.” is why we normalize. Four rows say nothing about accuracy; they only show that the pipeline runs. Read 30 real outputs yourself before you trust any score.
Step 4: train three freeze maps
The shared code. The three runs differ only in load(): which modules get LoRA, and whether the tower is unfrozen.
# common.py: model, data and the three freeze maps import os, torch from datasets import load_dataset from transformers import AutoProcessor, AutoModelForImageTextToText from peft import LoraConfig, get_peft_model MODEL = "HuggingFaceTB/SmolVLM-256M-Instruct" SMOKE = os.environ.get("SMOKE") == "1" # tiny CPU run to check the code DEV = "cuda" if torch.cuda.is_available() else "cpu" DTYPE = torch.bfloat16 if DEV == "cuda" else torch.float32 # T4: use float32 + fp16=True SUFFIX = "\nAnswer with a single word or number." def chartqa(split): # "train", "val" or "test" return load_dataset("HuggingFaceM4/ChartQA", split=split) def processor(split_images=True): # False: 1 tile = 64 tokens per image return AutoProcessor.from_pretrained(MODEL, do_image_splitting=split_images) LLM = r".*text_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)" VIT = r".*vision_model.*\.(q_proj|k_proj|v_proj|out_proj|fc1|fc2)" def load(run): """A: LoRA on the LLM, tower frozen. B: + LoRA on the tower. C: + full tower.""" model = AutoModelForImageTextToText.from_pretrained(MODEL, dtype=DTYPE) target = f"({LLM})|({VIT})" if run == "B" else LLM cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules=target) model = get_peft_model(model, cfg) # freezes everything, then adds A and B if run == "C": # unfreeze the whole vision tower for n, p in model.named_parameters(): if ".vision_model." in n: p.requires_grad = True model.print_trainable_parameters() return model def param_groups(model, lr=2e-4, tower_lr=2e-5): """LoRA weights at lr; fully trained tower weights 10x slower.""" is_tower = lambda n: ".vision_model." in n and "lora_" not in n rest = [p for n, p in model.named_parameters() if p.requires_grad and not is_tower(n)] tower = [p for n, p in model.named_parameters() if p.requires_grad and is_tower(n)] return [{"params": rest, "lr": lr}] + ([{"params": tower, "lr": tower_lr}] if tower else [])
A target_modules string is treated as a regular expression that must match the full module name [Certain, PEFT]. The prefixes text_model and vision_model matter: both parts have layers called q_proj, and a bare list like ["q_proj", "v_proj"] would silently put LoRA into the vision tower too.
# train_vlm.py: LoRA SFT on ChartQA. Usage: python train_vlm.py A|B|C import sys, time, json, torch from transformers import set_seed from trl import SFTConfig, SFTTrainer from common import SMOKE, SUFFIX, DEV, chartqa, processor, load, param_groups run = sys.argv[1]; set_seed(0) proc = processor(split_images=not SMOKE) def to_pc(ex): # prompt-completion format: loss on the completion only return {"images": [ex["image"]], "prompt": [{"role": "user", "content": ex["query"] + SUFFIX}], "completion": [{"role": "assistant", "content": ex["label"][0]}]} train = chartqa("train").shuffle(seed=0).select(range(8 if SMOKE else 5000)) train = train.map(to_pc, remove_columns=train.column_names) model = load(run) opt = torch.optim.AdamW(param_groups(model), weight_decay=0.0) cfg = SFTConfig(output_dir=f"out/{run}", max_length=None, # never cut image tokens or answers completion_only_loss=True, per_device_train_batch_size=2 if SMOKE else 8, gradient_accumulation_steps=1 if SMOKE else 2, num_train_epochs=1, max_steps=2 if SMOKE else -1, lr_scheduler_type="cosine", warmup_ratio=0.03, bf16=(DEV == "cuda"), gradient_checkpointing=not SMOKE, gradient_checkpointing_kwargs={"use_reentrant": False}, logging_steps=1 if SMOKE else 25, save_strategy="no", report_to="none", seed=0, remove_unused_columns=False) trainer = SFTTrainer(model=model, args=cfg, train_dataset=train, processing_class=proc, optimizers=(opt, None)) # our param groups, trainer's scheduler if DEV == "cuda": torch.cuda.reset_peak_memory_stats() t0 = time.time(); out = trainer.train(); dt = time.time() - t0 model.save_pretrained(f"out/{run}/adapter") # saves ONLY the LoRA weights if run == "C": # the unfrozen tower is not in that file torch.save({k: v.detach().cpu() for k, v in model.state_dict().items() if ".vision_model." in k and "lora_" not in k}, f"out/{run}/vision_tower.pt") res = {"run": run, "minutes": dt / 60, "train_loss": out.training_loss, "peak_GB": torch.cuda.max_memory_allocated() / 1e9 if DEV == "cuda" else None} print(json.dumps(res)); json.dump(res, open(f"train-{run}.json", "w"))
Three details in this script each cost someone a day:
max_length=None. TRL’s default is 1,024 tokens, cut from the right [Certain, TRL 0.24]. A square image alone is 1,088 image tokens. With the default, those rows lose their answer, or the number of image tokens no longer matches the image features and the forward pass fails.- Saving run C.
save_pretrainedon a PEFT model saves only the adapter. In our smoke run, the adapter file of run C was 19,597,680 bytes, exactly the size of run A’s: the trained tower was not in it. We save the tower separately (345.8 MB in FP32 on our CPU run, half that in BF16). - One learning rate per part. LoRA weights start at zero and need a large rate (\(2 \times 10^{-4}\) is a common start); a pretrained tower needs a small one. We pass our own AdamW with two parameter groups and let the trainer build the scheduler.
- input: base model, train / val / test splits, a text-only benchmark, seed 0
- score the base model on test (ChartQA) and on the text benchmark; save per-item results
- for run in {A: frozen tower, B: LoRA tower, C: full tower}:
- load the base; apply the freeze map; build the parameter groups
- pick the learning rate and step count on val only // never on test
- train one epoch; record peak memory, time and the gradient norm per part
- merge the adapter; score on test and on the text benchmark; save per-item results
- compare each run with the base, item by item (paired test, Step 6)
- return the filled table and a decision
Step 5: frozen or unfrozen tower, in a toy you can train
Which tower choice wins depends on where the new domain needs a change. If the tower already sees what matters, the LLM only has to learn to answer in a new way, and touching the tower adds risk for nothing. If the tower misses what matters (tiny chart labels, pathology textures), no change in the LLM can recover information the tower never passed on, and the LLM bends in strange ways while trying.
The toy makes this concrete. A “vision tower” \(E_0\) (\(8 \times 16\)) feeds an “LLM” \(L_0\) (\(8 \times 8\)). The new domain needs a change in both: the LLM must answer in a new way (a fixed rank-2 change \(\Delta L\)), and the tower must change by \(g \cdot \Delta E\), where \(g\) is the tower gap you choose. We train on \(n\) noisy examples with Adam for 400 steps. The LLM always gets LoRA with rank \(r\); the tower is frozen, LoRA or full. Optionally, a share of the loss replays “text-only” data: inputs that go straight into the LLM, with the base model’s own outputs as targets. We report three numbers:
- Read the symbols
- \(T\)
- the ideal model for the new domain, \((L_0 + \Delta L)(E_0 + g\,\Delta E)\)
- \(\|\cdot\|_F\)
- the Frobenius norm: the square root of the sum of squared entries
- err
- held-out error on new inputs: 1 means no better than the untouched model, 0 is perfect
- text drift
- how much the LLM changed; for text inputs that skip the tower, this is exactly how much its text outputs changed (for random text inputs)
- tower moved
- how much the shared tower changed; every other image task goes through it
How to read Figure 4
At the default (gap 0.5, 128 examples, rank 4), the frozen tower stalls at an error of 0.303 while LoRA and full towers reach 0.011 and 0.012. Look at the text drift: the frozen run has the largest (0.177), because the LLM is bent to make up for the tower. Set the gap to 0 and the examples to 32: now the frozen tower is best (0.061, against 0.165 and 0.208), because the trainable towers fit the noise. Set the gap to 1 and the examples to 128: frozen stalls at 0.447 with text drift 0.459. Finally, go back to the default and set text replay to 10%: text drift of the LoRA-tower run falls from 0.048 to 0.002, but the tower now moves 0.799 instead of 0.185. Replay protects the LLM by pushing the change into the tower, so check other image skills too.
Here is the same experiment in PyTorch, with a from-scratch LoRA. It runs on a CPU in about three seconds and is deterministic (it uses PyTorch’s random generator, so its numbers differ a little from the figure).
import torch torch.manual_seed(0) DX, DF, DY = 16, 8, 8 # image input, tower output (= LLM input), LLM output def lowrank(a, b, k=2): return torch.randn(a, k) @ torch.randn(k, b) # 1) A "pretrained" VLM: tower E0 and LLM L0. The new domain needs a change in BOTH: # the LLM must answer in a new way (dL) and the tower must see new features (gap * dE). E0, L0 = torch.randn(DF, DX) / DX**0.5, torch.randn(DY, DF) / DF**0.5 dE = lowrank(DF, DX); dE *= 0.6 * E0.norm() / dE.norm() dL = lowrank(DY, DF); dL *= 0.3 * L0.norm() / dL.norm() def make_task(gap, n=128, noise=0.5): T = (L0 + dL) @ (E0 + gap * dE) # the ideal model for the new domain X = torch.randn(n, DX) Y = X @ T.T + noise * torch.randn(n, DY) * T.norm() / (DY * DX)**0.5 return T, X, Y class LoRA(torch.nn.Module): # W = W0 + B A, with B = 0 at the start def __init__(self, W0, r=4): super().__init__() self.register_buffer("W0", W0.clone()) self.A = torch.nn.Parameter(torch.randn(r, W0.shape[1]) / W0.shape[1]**0.5) self.B = torch.nn.Parameter(torch.zeros(W0.shape[0], r)) def weight(self): return self.W0 + self.B @ self.A class Full(torch.nn.Module): def __init__(self, W0): super().__init__(); self.W = torch.nn.Parameter(W0.clone()) def weight(self): return self.W class Frozen(torch.nn.Module): def __init__(self, W0): super().__init__(); self.register_buffer("W", W0.clone()) def weight(self): return self.W def run(tower, task, text_mix=0.0, steps=400): T, X, Y = task torch.manual_seed(1); llm = LoRA(L0) # the LLM always gets LoRA, with the same init opt = torch.optim.Adam(list(tower.parameters()) + list(llm.parameters()), lr=1e-2) for _ in range(steps): L, E = llm.weight(), tower.weight() task_loss = ((X @ (L @ E).T - Y)**2).mean() text_loss = ((L - L0)**2).sum() / DY # text-only replay: keep L t close to L0 t loss = (1 - text_mix) * task_loss + text_mix * text_loss opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): L, E = llm.weight(), tower.weight() err = ((L @ E - T).norm()**2 / (L0 @ E0 - T).norm()**2).item() drift = ((L - L0).norm()**2 / L0.norm()**2).item() moved = ((E - E0).norm()**2 / E0.norm()**2).item() return err, drift, moved for gap in [0.0, 1.0]: task = make_task(gap) # the same data for all three runs for name, make in [("frozen tower", Frozen), ("LoRA tower", LoRA), ("full tower", Full)]: torch.manual_seed(2) err, drift, moved = run(make(E0), task) print(f"gap {gap:.0f} {name:12s} new-domain error {err:.3f} " f"text drift {drift:.3f} tower moved {moved:.3f}")
Output (PyTorch 2.8, CPU):
gap 0 frozen tower new-domain error 0.006 text drift 0.090 tower moved 0.000 gap 0 LoRA tower new-domain error 0.015 text drift 0.081 tower moved 0.020 gap 0 full tower new-domain error 0.017 text drift 0.031 tower moved 0.070 gap 1 frozen tower new-domain error 0.273 text drift 0.701 tower moved 0.000 gap 1 LoRA tower new-domain error 0.005 text drift 0.249 tower moved 0.282 gap 1 full tower new-domain error 0.006 text drift 0.085 tower moved 0.240
Read it like a lab result. With no tower gap, freezing wins: a trainable tower only adds ways to fit noise. With a gap, the frozen run is stuck at 0.273, and its LLM changed the most of all six runs (text drift 0.701). That last number is the one people miss: a frozen tower can cause more text regression, not less, because the LLM is forced to compensate. In the real lab you do not know the gap in advance, which is exactly why you run A, B and C and measure.
Step 6: before and after, on held-out data
After training, run eval_vlm.py for A, B and C. You now have, for every one of the 2,500 test items, a correct/wrong flag for the base model and for each run. Two accuracies alone hide most of the information. Count the four cells of a paired table instead: \(n_{11}\) items both got right, \(n_{10}\) only the base got right, \(n_{01}\) only the tuned model got right, \(n_{00}\) both got wrong.
- Read the symbols
- \(n\)
- the number of test items (2,500 for ChartQA test)
- \(\hat{\Delta}\)
- the change in accuracy, after minus before
- \(\mathrm{SE}\)
- its standard error when both models are scored on the same items; a 95% interval is \(\hat{\Delta} \pm 1.96\,\mathrm{SE}\)
Only the items that changed carry information about the difference. The exact McNemar test asks: if the two models were equally good, how surprising is the split of the changed items? Under that assumption, each changed item is a coin flip:
The paired interval is much narrower than the one you get by treating the two accuracies as independent, because most items are easy or hard for both models. Figure 5 shows both.
compare.py below.How to read Figure 5
“A clear gain” (2,500 items, 120 lost and 260 gained): +5.6 points, paired interval +4.1 to +7.1, unpaired +2.9 to +8.3, \(p < 0.0001\). “A tiny gain” (60 lost, 80 gained): +0.8 points; the paired interval −0.1 to +1.7 includes 0, \(p = 0.108\). Do not report this as an improvement. “A text check” (500 items, 22 lost, 12 gained): −2.0 points, interval −4.3 to +0.3, \(p = 0.121\). This does not prove there is no regression; with 500 items you cannot see a 2-point drop clearly. Use more items (ARC-Easy test has 2,376) or a more sensitive measure such as the KL drift below.
# compare.py: paired comparison of two eval files. Usage: python compare.py base A import sys, json, math a, b = (json.load(open(f"eval-{r}.json"))["rows"] for r in sys.argv[1:3]) assert [r["q"] for r in a] == [r["q"] for r in b], "score the SAME items in the SAME order" for part in ["all", "human", "augmented"]: pairs = [(x["ok"], y["ok"]) for x, y in zip(a, b) if part == "all" or x["human"] == (part == "human")] n = len(pairs) if n == 0: continue n10 = sum(x and not y for x, y in pairs); n01 = sum(y and not x for x, y in pairs) d = (n01 - n10) / n se = math.sqrt(max(n01 + n10 - (n01 - n10)**2 / n, 0)) / n m, k = n01 + n10, min(n01, n10) # exact McNemar (two-sided) p = min(1.0, 2 * sum(math.comb(m, i) for i in range(k + 1)) / 2**m) if m else 1.0 print(f"{part:9s} n={n} change {100*d:+.1f} pts 95% CI [{100*(d-1.96*se):+.1f}, " f"{100*(d+1.96*se):+.1f}] lost {n10} gained {n01} p={p:.4f}")
One caution about independence: the 1,250 human test questions are asked about only 625 charts [Certain, ChartQA paper]. Questions on the same chart are correlated, so the true interval is a little wider than formula (194.9) says [Likely]. A bootstrap that resamples whole charts fixes this.
Step 7: did text-only skills regress?
A VLM is also a chat model. After thousands of steps of “chart in, one number out”, it can forget how to answer plain text questions, or start answering everything with one word. Check three things, each against the base model, on prompts with no image:
- A text benchmark. ARC-Easy (grade-school science questions with 4 choices), scored without generation: the model picks the choice with the highest mean log-probability. 500 test questions, seeded.
- KL drift. Let the base model answer each question, then measure how much the tuned model’s next-token distribution differs on those same tokens. This sees small changes long before a benchmark does.
- Answer length. The mean number of words in free answers. A collapse from sentences to single words is the typical “format leak” of short-answer fine-tuning.
- Read the symbols
- \(S\)
- the positions of the base model’s own greedy answer tokens, over all text prompts
- \(\KL(p \,\|\, q)\)
- \(\sum_v p(v) \log \frac{p(v)}{q(v)}\), in nats: 0 if the two models agree exactly on every next-token distribution
# text_check.py: text-only regression check. Usage: python text_check.py A import sys, json, torch, torch.nn.functional as F from datasets import load_dataset from common import SMOKE, DEV, processor from eval_vlm import load_model run = sys.argv[1] proc = processor(); tok = proc.tokenizer base, tuned = load_model("base"), load_model(run) arc = load_dataset("allenai/ai2_arc", "ARC-Easy", split="test").shuffle(seed=0) arc = arc.select(range(4 if SMOKE else 500)) def ids_of(user, answer=None): # text-only chat prompt, optional answer msgs = [{"role": "user", "content": [{"type": "text", "text": user}]}] p_ids = tok(proc.apply_chat_template(msgs, add_generation_prompt=True), add_special_tokens=False)["input_ids"] a_ids = tok(" " + answer, add_special_tokens=False)["input_ids"] if answer else [] return torch.tensor([p_ids + a_ids], device=DEV), len(p_ids) @torch.no_grad() def answer_logprob(model, user, answer): # mean log p(answer tokens | prompt) ids, n_p = ids_of(user, answer) logp = F.log_softmax(model(input_ids=ids).logits[0, n_p - 1:-1].float(), -1) return logp.gather(-1, ids[0, n_p:, None]).mean().item() @torch.no_grad() def kl_on_base_answer(user): # formula (194.11) for one prompt ids, n_p = ids_of(user) full = base.generate(input_ids=ids, max_new_tokens=48, do_sample=False) lb = F.log_softmax(base(input_ids=full).logits[0, n_p - 1:-1].float(), -1) lt = F.log_softmax(tuned(input_ids=full).logits[0, n_p - 1:-1].float(), -1) return (lb.exp() * (lb - lt)).sum(-1).mean().item() @torch.no_grad() def words(model, user): # does it still write sentences? ids, n_p = ids_of(user) out = model.generate(input_ids=ids, max_new_tokens=48, do_sample=False) return len(tok.decode(out[0, n_p:], skip_special_tokens=True).split()) res = {"run": run} for name, m in [("base", base), (run, tuned)]: ok = 0 for ex in arc: scores = [answer_logprob(m, ex["question"] + "\nAnswer:", c) for c in ex["choices"]["text"]] ok += ex["choices"]["label"][scores.index(max(scores))] == ex["answerKey"] res[f"arc_easy_{name}"] = ok / len(arc) res[f"words_{name}"] = sum(words(m, ex["question"]) for ex in arc) / len(arc) res["kl_drift"] = sum(kl_on_base_answer(ex["question"]) for ex in arc) / len(arc) print(json.dumps(res)); json.dump(res, open(f"text-{run}.json", "w"))
Our smoke run (run A after 2 steps, 4 questions) printed a KL drift of 0.000118 nats per token and the same ARC answers and lengths as the base: two tiny steps barely move the model, as they should. That is a check that the code runs, not a result. Save the per-question ARC flags too if you want the paired test of Step 6 on the text side.
Do not assume LoRA keeps you safe. Biderman et al. found that LoRA forgets less than full fine-tuning, but it still forgets, and more with more training [Certain, “LoRA Learns Less and Forgets Less”]. Measure it.
If you find a regression, the fixes from session 61 apply: mix some text-only instruction data into training (a few percent to tens of percent; tune on val) [Likely], lower the learning rate or the rank, stop earlier, or interpolate the merged weights back toward the base. Note a finding from the SmolVLM paper: reusing the LLM’s own text SFT data in VLM training hurt their small models, and new text data worked better [Certain, SmolVLM paper]. And remember Figure 4: replay can push the change into the vision tower, so re-check an image benchmark the model was good at before.
Step 8: fill the table and decide
Fill every cell of the Setup table. Then write a half-page decision memo, as for your team:
- Goal. Example: “answer chart questions in our reports at 95% of the speed of the base model; keep plain chat working”.
- Results. The table, with paired intervals for every “change” you claim, human and augmented separately.
- Decision. One run, and the artifact: the merged model, or the base plus a small adapter.
- Why. The deciding facts: gain against noise, text regression, training and serving cost.
- Risks. What you will watch in production, and what would make you change the decision.
Rules that usually decide it:
- If B or C do not beat A clearly (paired interval), ship A. It is the cheapest to train, its adapter is 19.6 MB in FP32, and the frozen tower lets you cache image features: with a frozen tower and connector (and no random image augmentation), each training image’s 832 connector outputs can be computed once and reused every epoch, which removes the vision tower’s cost from all epochs after the first [Likely].
- If B or C win clearly, the domain needs a different view of the image. Prefer B unless C is clearly better: B’s adapter is 30.2 MB, while C ships a new 86M-weight tower, which also changes every other image task the model does.
- If text drift is large, fix it before shipping, even if ChartQA went up. A chat product will be judged on its chat.
- Evaluate the exact artifact you ship. We merge the adapter before evaluating for this reason.
Where this is used
| Setting | Usual freeze map | Why |
|---|---|---|
| Charts, tables, documents and receipts with a mid-size or small VLM | LoRA on the LLM; try LoRA on the tower if small text is missed | Most of the skill is reading layout and numbers; high resolution (tiling) often matters more than the freeze map |
| Medical or industrial images far from web photos (pathology, X-ray, defects) | LoRA or full tower with a small learning rate, plus LoRA on the LLM | The tower must learn new visual features; a frozen tower is the usual bottleneck [Likely] |
| A new answer format or label set on familiar images (product attributes, captions in a house style) | Frozen tower, LoRA on the LLM | The change lives in the LLM; cache image features to train fast |
| Many customers, one base model | One LoRA adapter per customer | One shared base in memory, small files per task, adapters can be swapped per request [Likely] |
| Video question answering and action recognition (SmolVLM2 has video variants) | Frozen tower, LoRA on the LLM, few frames per clip | Each frame costs as much as an image, so frames dominate compute and memory; caching frame features pays off [Likely] |
| Key-moment detection and video captions for Video-RAG | LoRA SFT on (frames, timestamp question, answer) data | Answers are short and checkable (a time span, an event), so before/after evaluation is cheap |
| Video classification, retrieval or fingerprinting with VLM embeddings | LoRA on the tower with a contrastive loss | Small changes keep the embedding space compatible with the existing index (session 60) |
Common mistakes
- Loss on the whole sequence. With TRL’s
messagesformat the collator masks only padding, so 894 of 895 tokens carry loss and the answer gets 0.56% of the signal. Use the prompt-completion format, and print one batch (Figure 2). - Truncation. A 1,024-token limit is shorter than one square image (1,088 image tokens). Set
max_length=Noneor a limit you checked against your longest row. - Ambiguous LoRA targets.
target_modules=["q_proj", "v_proj"]matches the vision tower as well as the LLM. Use a regular expression with the part’s prefix and check the trainable count against formula (194.2). - The unfrozen tower is not saved.
save_pretrainedon a PEFT model writes only the adapter. Save the tower yourself, and assert that it loads (no unexpected keys). - Tuning on the test set. Pick the learning rate, the rank and the step count on val. The test set is scored once per run, at the end.
- Comparing two accuracies without pairing. Use formula (194.9) and McNemar on the same items. A gain inside the paired interval is not a gain.
- Counting format as skill. Many “gains” come from learning to answer “14” instead of “The answer is 14”. Give the base model the same format instruction, and read outputs.
- No text-only check. ChartQA up, chat broken is a common outcome of short-answer SFT. Measure ARC-Easy, KL drift and answer length before you ship.
Interview questions
You fine-tune a VLM on a small domain dataset. Do you freeze the vision tower? Why?basic
How many weights does LoRA with \(r = 16\) add to the LLM of SmolVLM-256M?core
In a small VLM with image tiling, where does the training compute go?core
Why put the loss only on the answer tokens? What happens otherwise?core
Your tower is frozen. How can you make training several times faster?advanced
How do you know your fine-tuned model is really better than the base?core
How do you check that a VLM fine-tune did not hurt text-only skills?advanced
What is ChartQA’s relaxed accuracy, and what is wrong with it?basic
After fine-tuning on ChartQA, your VLM answers every chat question with one number. Why, and how do you fix it?advanced
You unfreeze the vision tower and LLM LoRA suddenly learns much more slowly. What could cause it?advanced
Exercises
- Count the LoRA weights for SmolVLM-500M (32 blocks, width 960, key and value outputs 320, MLP 2,560) with \(r = 8\) on all seven LLM linear layers. Check it in Figure 1.
- A 1,024 × 768 photo and a 300 × 200 thumbnail: how many tiles and image tokens does each become with splitting on? What does this say about small images?
- Your counts for run A against the base are \(n_{10} = 60\), \(n_{01} = 80\) on 2,500 items. Compute the change, its paired 95% interval and the McNemar p-value. Is it a gain?
- Under relaxed accuracy, is “0.6” correct for gold “0.57”? Is “1950” correct for gold “2018”? Propose a fix for years.
- In Figure 4, find one setting where the frozen tower is best and one where it is worst. Explain each in one sentence.
- In Figure 4 at the default, set text replay to 10%. What happens to the text drift and to the tower? What does this mean for your real evaluation plan?
- Swap ChartQA for PathVQA (
flaviagiammarino/path-vqa, columnsimage,question,answer). Which lines change, and which metric would you use?
Answerscheck
size={"longest_edge": N*512}) or turn splitting off. 3. Change +0.8 points; SE \(= \sqrt{140 - 400/2500}/2500 = 0.0047\); interval −0.1 to +1.7 points; exact p = 0.108. Not a clear gain. 4. 0.6 is 5.26% off 0.57, so it fails at 5%. 1950 is 3.4% off 2018, so it “passes”. Fix: treat four-digit integers between 1900 and 2100 as exact-match answers (and report the rule you used). 5. Best: gap 0, 32 examples (0.061 against 0.165 and 0.208): no tower change is needed, so trainable towers only fit noise. Worst: gap 1, 128 examples (0.447 against 0.005 and 0.006): the needed information is not in the frozen tower’s output. 6. Text drift for the LoRA-tower run drops from 0.048 to 0.002, but the tower moves 0.799 instead of 0.185 (frozen-tower error rises from 0.303 to 0.326). Replay pushes the change into the tower, so also re-run an image benchmark the base was good at. 7. Only chartqa(), to_pc() and the eval loop’s column names change, and the format line (yes/no or a short phrase). Use accuracy on yes/no questions and exact match (or token F1) on open questions, reported separately; check that the base model has not seen PathVQA.Key takeaways
- Count before you train: LoRA with \(r = 16\) on SmolVLM-256M’s LLM is 4,884,480 weights (194.2); LoRA on the tower adds 2,654,208; a full tower adds 86,433,024.
- In a small tiled VLM, the vision tower dominates compute (194.3): 13,312 patches against 895 LLM tokens. Tiling is the biggest cost knob; unfreezing the tower adds a backward pass through all those patches.
- Train on the answer only (194.6) and never truncate VLM rows. Print one real batch and check which tokens carry loss.
- Whether to unfreeze depends on where the domain gap is. A frozen tower is safest when the tower already sees what matters, but can cause more text drift when it does not, because the LLM compensates (Figure 4).
- Score before and after on the same held-out items with a paired interval and McNemar (194.9, 194.10); know your metric’s blind spots (194.7) and your base model’s training data.
- Check text-only skills with a benchmark, answer length and KL drift (194.11), and evaluate the exact artifact you ship.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Hugging Face (code). PEFT library
- Hugging Face (code). TRL library
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Hugging Face (code). PEFT library
- Hugging Face (code). TRL library
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Laurençon, Marafioti, Sanh & Tronchon (2024). Building and better understanding vision-language models: insights and future directions (Idefics3)
- Masry, Long, Tan, Joty & Hoque (2022). ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Clark et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- He et al. (2020). PathVQA: 30000+ Questions for Medical Visual Question Answering
- Biderman et al. (2024). LoRA Learns Less and Forgets Less