LLM & VLM courseBlog
194Track 7: Vision and VLMs · session 32 of 38

LoRA fine-tune a small VLM

In this lab you fine-tune SmolVLM-256M, a small open vision-language model, on ChartQA with LoRA. You compare three choices for the vision tower (frozen, LoRA, fully trained), score the model before and after on held-out data with a paired test, and check that its text-only skills did not get worse.

Time: 4 to 5 hours (mostly hands-on; the training runs take minutes to about an hour each on one GPU)Level: Intermediate · hands-on labBefore this: VLM SFT and PEFT (session 189), LoRA vs QLoRA vs full fine-tuning (62), Evaluating VLMs (192); Python and PyTorch

What you will learn

  • How to count what a freeze map trains with formula r(d_in + d_out), and why the vision tower dominates compute in a small tiled VLM
  • How SmolVLM turns one chart into 13 tiles and 832 image tokens, and how to put the loss on the 5 answer tokens only
  • How to measure a base model first with ChartQA relaxed accuracy, and the blind spots of that metric
  • How to train LoRA runs with a frozen, LoRA or fully trained vision tower with PEFT and TRL, and the bugs that silently break them
  • When unfreezing the vision tower helps, and why a frozen tower can cause more text drift (a live toy)
  • How to compare before and after with a paired interval and an exact McNemar test
  • How to check text-only regressions with ARC-Easy, answer length and KL drift

Why this matters

You have a small open vision-language model (VLM) and a set of images from your own domain: charts, forms, product photos, video frames. The model answers questions about them, but not well enough. In this lab you fine-tune it with LoRA, compare three choices for the vision tower (frozen, LoRA, fully trained), measure the score before and after on held-out data, and check that the model did not lose its text-only skills on the way.

The theory is in earlier sessions. Session 189 explained the freeze map (which part to freeze, adapt or train), per-part learning rates and answer-only loss. Session 62 ran the same kind of comparison for a text-only model. Session 192 and session 193 showed how VLM scores can mislead. This lab puts it together on one real model and one real dataset, with numbers you measure yourself.

You will:

  1. Count what each freeze map trains and what it costs in compute and memory, before you run anything (Figure 1).
  2. Turn one ChartQA row into a training example and check its loss mask token by token (Figure 2).
  3. Score the untouched base model on the held-out test set, with the ChartQA metric (Figure 3).
  4. Train three LoRA runs that differ only in the vision tower, and build intuition with a toy you can train in the browser (Figure 4).
  5. Compare before and after with a paired test (Figure 5), and run a text-only regression check.
  6. Fill one results table and write down what you would ship.

Real systems do exactly this. Open VLM families ship fine-tuning tutorials (the SmolVLM model card links one), and Hugging Face’s PEFT and TRL libraries are the common open-source tools for it [Certain]. In industry, the same loop adapts VLMs to receipts, charts, medical images, screenshots and video frames.

Setup

Hardware. One GPU with 16 GB is enough for every run on this page; the small model even trains (slowly) on a CPU. On a free Colab T4, note that the T4 has no BF16 support: load the model in float32 and use fp16=True instead of bf16=True [Certain].

pip install "transformers>=4.56" peft trl datasets accelerate pillow

The code was written for transformers 4.57, peft 0.17 and trl 0.24, and we ran every script end to end on a CPU in a smoke mode (8 training rows, 2 steps, 4 test rows, no image splitting) to check that it runs. We did not run the full GPU experiments, and this page reports no fine-tuning results: you will measure them. These libraries change often; if an argument name fails, check the current documentation.

Model. HuggingFaceTB/SmolVLM-256M-Instruct, Apache 2.0. It has the Idefics3 architecture: a SigLIP vision tower (base size, patch 16, 512 × 512 input), a pixel-shuffle connector, and a SmolLM2-135M language model [Certain, from the model card and config.json]. We built it from its config and counted every weight:

PartWhat it isWeights
Vision tower12 ViT blocks, width 768, MLP 3,072; 1,024 patches per 512 px tile86,433,024
Connectorpixel shuffle (16 neighbor patches → one vector of 12,288) + one linear layer 12,288 → 5767,077,888
LLM blocks30 blocks, width 576, 9 query heads and 3 key-value heads of size 64, MLP 1,536106,203,456
Embeddings and output headvocabulary 49,280, not tied (two separate tables)28,385,280 + 28,385,280
Totalmatches the 256,484,928 BF16 weights of the released file256,484,928

The vision tower is a third of the model, which is unusual: in a 7B VLM it is under a tenth. For the next size up, SmolVLM-500M-Instruct (507,482,304 weights, SmolLM2-360M inside), every script works unchanged.

Data. ChartQA (HuggingFaceM4/ChartQA): bar, line and pie charts with questions and short answers [Certain]. Each split has two halves: questions written by people (“human”) and questions generated by a model from the chart’s data table (“augmented”).

SplitHumanAugmentedTotalWe use
train7,39820,90128,2995,000 rows (seeded shuffle)
val9609601,920to pick the learning rate and the number of steps
test1,2501,2502,500all, only at the end; report human and augmented separately

In the test set, 76.6% of the answers are numbers, 5.8% are years and 104 answers (8.3% of the human half) are “Yes” or “No”; 2,295 of 2,500 answers are a single word [Certain, we counted them].

The table you will fill. Copy it into your notes now. The trainable counts come from the smoke runs; everything else you measure.

RunVision towerTrainable weightsPeak GPU memoryTrain timeChartQA humanChartQA augmentedARC-Easy (text)Text KL drift
Basen/a0n/an/a0
Afrozen4,884,480 (1.90%)
BLoRA7,538,688 (2.94%)
Cfully trained91,317,504 (35.6%)

Step 1: what each freeze map trains and costs

All three runs put LoRA on the seven linear layers of every LLM block (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), with rank \(r = 16\) and \(\alpha = 32\). They differ only in the vision tower: A freezes it, B adds LoRA to its six linear layers per block (q_proj, k_proj, v_proj, out_proj, fc1, fc2), C trains all of its weights with a 10 times smaller learning rate. The connector stays frozen in all three, so the comparison changes one thing at a time.

A LoRA layer keeps the pretrained matrix \(W_0\) frozen and adds a low-rank path (session 57):

\[ \mathbf{h} = W_0\mathbf{x} + \frac{\alpha}{r}\,BA\,\mathbf{x}, \qquad A \in \R^{r \times d_{in}},\ B \in \R^{d_{out} \times r},\ B = 0 \text{ at the start} \tag{194.1} \]
\[ P_{\text{LoRA}} = r \sum_{\text{adapted layers}} \left(d_{in} + d_{out}\right) \tag{194.2} \]
Read the symbols
\(\mathbf{x}, \mathbf{h}\)
the input and output of one linear layer
\(W_0\)
the frozen pretrained weight; it gets no gradient and no optimizer state
\(A, B\)
the two small trainable matrices; \(B = 0\) at the start, so the fine-tuned model begins exactly at the base model
\(r, \alpha\)
the rank (16 here) and a fixed scale (32 here), so the update is multiplied by \(\alpha / r = 2\)
\(P_{\text{LoRA}}\)
the number of new trainable weights
Worked example: count the LoRA weights

LLM, one block. q_proj and o_proj are 576 → 576, k_proj and v_proj are 576 → 192 (3 heads of 64), gate_proj and up_proj are 576 → 1,536, down_proj is 1,536 → 576. So \(\sum (d_{in} + d_{out}) = 2 \cdot 1{,}152 + 2 \cdot 768 + 3 \cdot 2{,}112 = 10{,}176\). Times \(r = 16\) and 30 blocks: 4,884,480. This is run A. PEFT prints trainable params: 4,884,480 || all params: 261,369,408 || trainable%: 1.8688; the “all params” number includes the new LoRA weights, so it is 4,884,480 more than the 256,484,928 of the base model.

Vision tower, one block. Four 768 → 768 attention matrices and the MLP 768 → 3,072 → 768: \(4 \cdot 1{,}536 + 2 \cdot 3{,}840 = 13{,}824\). Times 16 and 12 blocks: 2,654,208. Run B trains \(4{,}884{,}480 + 2{,}654{,}208 = \mathbf{7{,}538{,}688}\). Run C trains the LLM LoRA plus all 86,433,024 tower weights: \(4{,}884{,}480 + 86{,}433{,}024 = \mathbf{91{,}317{,}504}\). All three counts match what PEFT printed in our smoke runs.

The parameter count is not the cost. A VLM has a fixed order: vision tower, then connector, then LLM, then the loss. The backward pass starts at the loss and must go down to the lowest part that has trainable weights. With \(N_p\) weights in part \(p\) and \(t_p\) tokens going through it, a matrix multiply costs about \(2N_p\) FLOPs per token, so per sample:

\[ C_p \approx 2 N_p\, t_p \left( 1 + [\text{backward passes through } p] + [p \text{ fully trained}] + [\text{recompute}] \right) \tag{194.3} \]
Read the symbols
\(t_p\)
tokens through part \(p\): 1,024 patches per tile for the vision tower, 64 tokens per tile for the connector, the whole sequence for the LLM
\([\cdot]\)
1 if the condition is true, else 0. The first bracket pays for input gradients, the second for weight gradients, the third for the extra forward pass of gradient checkpointing

Now the surprise. With image splitting, a typical 850 × 600 chart becomes 13 tiles, so the vision tower sees \(13 \times 1{,}024 = 13{,}312\) patches, while the LLM sees only about 895 tokens. One forward pass of the tower costs \(2 \times 86.4\text{M} \times 13{,}312 = 2.30\) TFLOPs, the LLM (blocks and output head) only \(2 \times 134.6\text{M} \times 895 = 0.24\) TFLOPs. In this small VLM, almost all the compute is in the vision tower, even when it is frozen. Explore it in Figure 1.

Model Vision tower Connector LLM Tiles per image
vision tower (compute) / activations (memory)connector (compute) / trained states (memory)LLM (compute)frozen weights (memory)
Figure 1. Trainable weights, compute and memory for one optimizer step, from formula (194.3) and the memory rule (194.4) below. Parameter counts come from building the models from their config.json; the token counts (895 for 13 tiles, 100 for 1 tile, 1,160 for 17 tiles) are from the real processor on one ChartQA test row.
How to read Figure 1

Start with run A. The tower is frozen, yet it is still 76% of the compute (18.41 of 24.29 TFLOPs at batch 8): it must run forward for every image. Click run B: the backward pass must now go down through the tower, and the step costs 61.29 TFLOPs, 2.5 times run A, for only 2.7M more weights. Run C adds weight gradients for the tower: 79.70 TFLOPs. Now switch to 1 tile: the compute drops more than 10 times, because the image becomes 64 tokens instead of 832. Image splitting is the biggest cost knob in this lab, bigger than any freeze choice. Memory stays small for the 256M model (run A about 2.4 GB before overheads), so any 16 GB GPU fits; the activations of a trainable tower (6.5 GB in run B) are the largest term.

For memory we use the same simple budget as in session 38: frozen weights in BF16 cost 2 bytes each; every trained weight (LoRA or full) costs 16 bytes (FP32 copy, gradient and two Adam moments); and activations are stored only for parts that the backward pass goes through:

\[ M \approx 2\,N_{\text{frozen}} + 16\,N_{\text{trained}} + b \sum_{p\ \text{with backward}} L_p\, t_p\, d_p \cdot \kappa + b\, s\, V \cdot 4 \tag{194.4} \]
Read the symbols
\(L_p, d_p\)
the number of blocks and the width of part \(p\) (12 and 768 for the tower, 30 and 576 for the LLM)
\(\kappa\)
bytes stored per token per block: about 34 without checkpointing (from Korthikanti et al. for a standard block, with the attention-score term removed by fused attention), about 2 with checkpointing (only each block’s BF16 input, plus one block at a time recomputed)
\(b\, s\, V \cdot 4\)
the FP32 logits: micro-batch times sequence length times vocabulary size (49,280)

This is a planning estimate [Likely], not a measurement: real peaks add temporary buffers, the CUDA context and fragmentation. Measure the real peak with torch.cuda.max_memory_allocated() and write it in your table.

Step 2: one ChartQA row becomes one training example

Each training example is one short conversation: the user sends a chart and a question, and the assistant answers. We add one fixed line to every question, “Answer with a single word or number.”, so that the base model and the fine-tuned model are asked for the same format. The processor then does three things:

  1. Resize and split the image. SmolVLM first resizes the image so its longest side is 2,048 pixels (small images are made larger), then cuts it into tiles of 512 × 512, and adds one extra 512 px view of the whole image (the “global” tile).
  2. Turn tiles into tokens. Each tile gives \(32 \times 32 = 1{,}024\) patches of 16 px. Pixel shuffle merges each 4 × 4 group of patch vectors into one vector, so each tile becomes \(1{,}024 / 16 = 64\) tokens for the LLM (session 178).
  3. Apply the chat template. Special tokens mark each tile’s row and column, then the question, then “Assistant:”, then the answer, then <end_of_utterance>.
\[ T = \left\lceil \frac{W'}{512} \right\rceil \left\lceil \frac{H'}{512} \right\rceil + 1, \qquad n_{\text{img}} = 64\,T, \qquad \max(W', H') = 2048 \tag{194.5} \]

Here \(W', H'\) are the width and height after resizing. An 850 × 600 chart becomes 2,048 × 1,446, so \(T = 4 \cdot 3 + 1 = 13\) and \(n_{\text{img}} = 832\). A square image gives \(4 \cdot 4 + 1 = 17\) tiles and 1,088 tokens. In the ChartQA test set, 2,178 images give 13 tiles, 299 give 17 and 23 give 9 [Certain, we ran the processor on all 2,500]. With do_image_splitting=False, every image is one 512 px tile: 64 tokens.

Next, the loss mask. The model is trained with next-token cross-entropy, but only on the answer tokens (the set \(\mathcal{A}\)). Everything else gets the label -100, which PyTorch’s cross-entropy ignores:

\[ \mathcal{L} = -\frac{1}{|\mathcal{A}|} \sum_{t \in \mathcal{A}} \log p_\theta\!\left(x_t \mid I,\ x_{\lt t}\right) \tag{194.6} \]
Read the symbols
\(I\)
the image (its 832 tokens enter the LLM as context)
\(x_t\)
the token at position \(t\) of the templated sequence
\(\mathcal{A}\)
the positions of the answer tokens, here “ 1 4 <end_of_utterance> \n”: 5 tokens. The end token is in \(\mathcal{A}\) on purpose: it teaches the model to stop
image tokens (one chip = 64)questionanswertemplate and special tokensthick border = has a label (gives loss)
Figure 2. The real token sequence of ChartQA test row 0 (“How many food item is shown in the bar graph?”, answer “14”) after the SmolVLM processor and chat template, 895 tokens. Each image chip stands for 64 image tokens. Tokens that contain a space are shown in quotes; the space inside the quotes, as in “ many”, is part of the token.
How to read Figure 2

Step to the token view, then to the labels. In “answer only” mode, 5 of 895 tokens carry loss. In “loss on everything but padding” mode, 894 tokens carry loss (the first token is never predicted) and the answer gets only 0.56% of the training signal: the model spends almost all its effort learning to predict <image> placeholders and the fixed template. Masking only the image tokens is better (62 labeled tokens, answer share 8.06%) but still trains the model to predict the question. Note that TRL’s default for a dataset with a messages column is the first mode: its VLM collator masks only padding [Certain, TRL 0.24 source]. The prompt-completion format we use gives the third mode.

Algorithm 1: build one training batch for a VLM
  1. input: rows (image, question, answer), the processor, the chat template
  2. for each row: prompt \(\leftarrow\) user turn with one image placeholder and the question plus the format line
  3.   completion \(\leftarrow\) assistant turn with the answer; the template adds <end_of_utterance>
  4. process the prompts with their images // resize, split into tiles, expand each placeholder to \(64 \cdot T\) image tokens
  5. tokenize the completions; concatenate prompt and completion ids per row
  6. pad to the longest row; labels \(\leftarrow\) ids; labels[padding] \(\leftarrow -100\); labels[prompt] \(\leftarrow -100\)
  7. never truncate a row: cutting it would remove the answer, or cut image tokens and break the image-token count
  8. return input ids, attention mask, pixel values, labels

TRL implements Algorithm 1 for you when the dataset has images, prompt and completion columns. Check it once on a real batch, as in Figure 2; it costs five minutes and catches the most common silent bug in VLM fine-tuning.

# check_batch.py: print which tokens carry loss in one real batch
from trl.trainer.sft_trainer import DataCollatorForVisionLanguageModeling
from common import chartqa, processor, SUFFIX

proc = processor(split_images=True)
collate = DataCollatorForVisionLanguageModeling(proc, completion_only_loss=True)
rows = chartqa("val").select(range(2))
batch = collate([{"images": [r["image"]],
                  "prompt": [{"role": "user", "content": r["query"] + SUFFIX}],
                  "completion": [{"role": "assistant", "content": r["label"][0]}]} for r in rows])
for ids, lab in zip(batch["input_ids"], batch["labels"]):
    toks = proc.tokenizer.convert_ids_to_tokens(ids)
    print(len(toks), "tokens; with loss:", [t for t, l in zip(toks, lab) if l != -100])

On the first two validation rows, our smoke run printed ['ĠBlue', '<end_of_utterance>', 'Ċ'] and ['Ġ', '2', '0', '1', '8', '<end_of_utterance>', 'Ċ']. Note that the tokenizer writes each digit as its own token, so “2018” is four tokens.

Step 3: score the base model first

Before you train, measure the base model on the test set with exactly the prompt, decoding and metric you will use after training. This number is the “before” in “before and after”. It also tests your evaluation code: the SmolVLM-256M model card reports 55.8 on the ChartQA test set (63.2 for the 500M model) [Certain, model card]. Your prompt and answer parsing differ from theirs, so expect your number to differ by a few points [Likely]. If you get 20, your evaluation is broken, not the model.

The ChartQA metric. ChartQA uses relaxed accuracy: a numeric answer is correct if it is within 5% of the gold answer; any other answer must match exactly [Certain, ChartQA paper]. We also ignore case, a final period, commas and a percent sign.

\[ \text{correct}(\hat{y}, y) = \begin{cases} \left| \hat{y} - y \right| \le 0.05\, |y| & \text{if both are numbers} \\ \operatorname{norm}(\hat{y}) = \operatorname{norm}(y) & \text{otherwise} \end{cases} \tag{194.7} \]

The 5% rule exists because human-written gold values are read off charts and are often slightly off. But it has a blind spot: it treats a year like any other number. For the gold answer 2018, every number from 1917.1 to 2118.9 counts as correct. Years are 144 of the 2,500 test answers. Try it in Figure 3.

Model outputGold answerExact matchRelaxed
Figure 3. Exact match (case and spaces ignored) against relaxed accuracy, formula (194.7), on nine output–gold pairs. The gold answers are of the kinds found in the ChartQA test set; the outputs are examples we wrote, not real model outputs.
How to read Figure 3

At 5%, exact match gives 2 of 9 and relaxed accuracy 7 of 9. Some of the gap is fair (“14.” and “14”, “1,234” and “1234”, 86.3 read from a chart whose table says 86.33). Some is not: “2017” for “2018” is wrong but counts as correct. “0.6” for “0.57” is 5.26% off and fails; move the tolerance to 10% and it passes. “The answer is 14” fails under both rules: format matters, and fine-tuning often “gains” points only by learning to answer in one word. When you report a gain, check how much of it is format.

The evaluation script. It generates greedily (deterministic), uses left padding so that every prompt in a batch ends at the same position, and saves one row per test item with a correct/wrong flag. We need those per-item flags for the paired test in Step 6.

# eval_vlm.py: ChartQA relaxed accuracy.  Usage: python eval_vlm.py base|A|B|C
import sys, json, torch
from transformers import AutoModelForImageTextToText
from peft import PeftModel
from common import MODEL, SMOKE, SUFFIX, DEV, DTYPE, chartqa, processor

def norm(s):
    return s.strip().rstrip(".").replace(",", "").replace("%", "")

def relaxed_match(pred, gold, tol=0.05):
    """ChartQA rule: numbers may be off by 5% (relative); everything else must match."""
    p, g = norm(pred), gold.strip().replace(",", "").replace("%", "")
    try:
        pv, gv = float(p), float(g)
        return abs(pv - gv) <= tol * abs(gv) if gv != 0 else pv == 0
    except ValueError:
        return p.lower() == g.lower()

def load_model(run):
    model = AutoModelForImageTextToText.from_pretrained(MODEL, dtype=DTYPE)
    if run != "base":
        model = PeftModel.from_pretrained(model, f"out/{run}/adapter")
        if run == "C":                                 # the unfrozen tower is saved separately
            r = model.load_state_dict(torch.load(f"out/{run}/vision_tower.pt"), strict=False)
            assert not r.unexpected_keys, r.unexpected_keys   # strict=False hides typos
        model = model.merge_and_unload()               # evaluate the artifact you would ship
    return model.to(DEV).eval()

if __name__ == "__main__":
    run = sys.argv[1]
    proc = processor(split_images=not SMOKE); proc.tokenizer.padding_side = "left"
    model = load_model(run)
    test = chartqa("test")                             # 1,250 human + 1,250 augmented
    if SMOKE: test = test.select(range(4))
    rows, bs = [], 16
    for i in range(0, len(test), bs):
        b = test[i:i + bs]
        msgs = [[{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": q + SUFFIX}]}]
                for q in b["query"]]
        texts = [proc.apply_chat_template(m, add_generation_prompt=True) for m in msgs]
        enc = proc(text=texts, images=[[im] for im in b["image"]], return_tensors="pt",
                   padding=True).to(DEV)
        with torch.no_grad():
            out = model.generate(**enc, max_new_tokens=16, do_sample=False)
        preds = proc.batch_decode(out[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)
        for q, p, g, h in zip(b["query"], preds, b["label"], b["human_or_machine"]):
            rows.append({"q": q, "pred": p.strip(), "gold": g[0], "human": h == 0,
                         "ok": relaxed_match(p, g[0])})
    acc = lambda rs: sum(r["ok"] for r in rs) / max(len(rs), 1)
    res = {"run": run, "n": len(rows), "all": acc(rows),
           "human": acc([r for r in rows if r["human"]]),
           "augmented": acc([r for r in rows if not r["human"]])}
    print(json.dumps(res)); json.dump({"summary": res, "rows": rows}, open(f"eval-{run}.json", "w"))

In smoke mode the base model answered the first four test rows with “Corn.”, “190000000.”, “3.” and “Yes.” (gold: 14, 0.57, 3, No). One in four is right; the trailing period on “3.” is why we normalize. Four rows say nothing about accuracy; they only show that the pipeline runs. Read 30 real outputs yourself before you trust any score.

Step 4: train three freeze maps

The shared code. The three runs differ only in load(): which modules get LoRA, and whether the tower is unfrozen.

# common.py: model, data and the three freeze maps
import os, torch
from datasets import load_dataset
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import LoraConfig, get_peft_model

MODEL = "HuggingFaceTB/SmolVLM-256M-Instruct"
SMOKE = os.environ.get("SMOKE") == "1"            # tiny CPU run to check the code
DEV = "cuda" if torch.cuda.is_available() else "cpu"
DTYPE = torch.bfloat16 if DEV == "cuda" else torch.float32   # T4: use float32 + fp16=True
SUFFIX = "\nAnswer with a single word or number."

def chartqa(split):                               # "train", "val" or "test"
    return load_dataset("HuggingFaceM4/ChartQA", split=split)

def processor(split_images=True):                 # False: 1 tile = 64 tokens per image
    return AutoProcessor.from_pretrained(MODEL, do_image_splitting=split_images)

LLM = r".*text_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)"
VIT = r".*vision_model.*\.(q_proj|k_proj|v_proj|out_proj|fc1|fc2)"

def load(run):
    """A: LoRA on the LLM, tower frozen.  B: + LoRA on the tower.  C: + full tower."""
    model = AutoModelForImageTextToText.from_pretrained(MODEL, dtype=DTYPE)
    target = f"({LLM})|({VIT})" if run == "B" else LLM
    cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules=target)
    model = get_peft_model(model, cfg)            # freezes everything, then adds A and B
    if run == "C":                                # unfreeze the whole vision tower
        for n, p in model.named_parameters():
            if ".vision_model." in n:
                p.requires_grad = True
    model.print_trainable_parameters()
    return model

def param_groups(model, lr=2e-4, tower_lr=2e-5):
    """LoRA weights at lr; fully trained tower weights 10x slower."""
    is_tower = lambda n: ".vision_model." in n and "lora_" not in n
    rest = [p for n, p in model.named_parameters() if p.requires_grad and not is_tower(n)]
    tower = [p for n, p in model.named_parameters() if p.requires_grad and is_tower(n)]
    return [{"params": rest, "lr": lr}] + ([{"params": tower, "lr": tower_lr}] if tower else [])

A target_modules string is treated as a regular expression that must match the full module name [Certain, PEFT]. The prefixes text_model and vision_model matter: both parts have layers called q_proj, and a bare list like ["q_proj", "v_proj"] would silently put LoRA into the vision tower too.

# train_vlm.py: LoRA SFT on ChartQA.  Usage: python train_vlm.py A|B|C
import sys, time, json, torch
from transformers import set_seed
from trl import SFTConfig, SFTTrainer
from common import SMOKE, SUFFIX, DEV, chartqa, processor, load, param_groups

run = sys.argv[1]; set_seed(0)
proc = processor(split_images=not SMOKE)

def to_pc(ex):                     # prompt-completion format: loss on the completion only
    return {"images": [ex["image"]],
            "prompt": [{"role": "user", "content": ex["query"] + SUFFIX}],
            "completion": [{"role": "assistant", "content": ex["label"][0]}]}

train = chartqa("train").shuffle(seed=0).select(range(8 if SMOKE else 5000))
train = train.map(to_pc, remove_columns=train.column_names)

model = load(run)
opt = torch.optim.AdamW(param_groups(model), weight_decay=0.0)
cfg = SFTConfig(output_dir=f"out/{run}", max_length=None,     # never cut image tokens or answers
                completion_only_loss=True,
                per_device_train_batch_size=2 if SMOKE else 8,
                gradient_accumulation_steps=1 if SMOKE else 2,
                num_train_epochs=1, max_steps=2 if SMOKE else -1,
                lr_scheduler_type="cosine", warmup_ratio=0.03, bf16=(DEV == "cuda"),
                gradient_checkpointing=not SMOKE,
                gradient_checkpointing_kwargs={"use_reentrant": False},
                logging_steps=1 if SMOKE else 25, save_strategy="no", report_to="none",
                seed=0, remove_unused_columns=False)
trainer = SFTTrainer(model=model, args=cfg, train_dataset=train, processing_class=proc,
                     optimizers=(opt, None))          # our param groups, trainer's scheduler
if DEV == "cuda": torch.cuda.reset_peak_memory_stats()
t0 = time.time(); out = trainer.train(); dt = time.time() - t0
model.save_pretrained(f"out/{run}/adapter")           # saves ONLY the LoRA weights
if run == "C":                                        # the unfrozen tower is not in that file
    torch.save({k: v.detach().cpu() for k, v in model.state_dict().items()
                if ".vision_model." in k and "lora_" not in k}, f"out/{run}/vision_tower.pt")
res = {"run": run, "minutes": dt / 60, "train_loss": out.training_loss,
       "peak_GB": torch.cuda.max_memory_allocated() / 1e9 if DEV == "cuda" else None}
print(json.dumps(res)); json.dump(res, open(f"train-{run}.json", "w"))

Three details in this script each cost someone a day:

  • max_length=None. TRL’s default is 1,024 tokens, cut from the right [Certain, TRL 0.24]. A square image alone is 1,088 image tokens. With the default, those rows lose their answer, or the number of image tokens no longer matches the image features and the forward pass fails.
  • Saving run C. save_pretrained on a PEFT model saves only the adapter. In our smoke run, the adapter file of run C was 19,597,680 bytes, exactly the size of run A’s: the trained tower was not in it. We save the tower separately (345.8 MB in FP32 on our CPU run, half that in BF16).
  • One learning rate per part. LoRA weights start at zero and need a large rate (\(2 \times 10^{-4}\) is a common start); a pretrained tower needs a small one. We pass our own AdamW with two parameter groups and let the trainer build the scheduler.
Algorithm 2: the experiment
  1. input: base model, train / val / test splits, a text-only benchmark, seed 0
  2. score the base model on test (ChartQA) and on the text benchmark; save per-item results
  3. for run in {A: frozen tower, B: LoRA tower, C: full tower}:
  4.   load the base; apply the freeze map; build the parameter groups
  5.   pick the learning rate and step count on val only // never on test
  6.   train one epoch; record peak memory, time and the gradient norm per part
  7.   merge the adapter; score on test and on the text benchmark; save per-item results
  8. compare each run with the base, item by item (paired test, Step 6)
  9. return the filled table and a decision

Step 5: frozen or unfrozen tower, in a toy you can train

Which tower choice wins depends on where the new domain needs a change. If the tower already sees what matters, the LLM only has to learn to answer in a new way, and touching the tower adds risk for nothing. If the tower misses what matters (tiny chart labels, pathology textures), no change in the LLM can recover information the tower never passed on, and the LLM bends in strange ways while trying.

The toy makes this concrete. A “vision tower” \(E_0\) (\(8 \times 16\)) feeds an “LLM” \(L_0\) (\(8 \times 8\)). The new domain needs a change in both: the LLM must answer in a new way (a fixed rank-2 change \(\Delta L\)), and the tower must change by \(g \cdot \Delta E\), where \(g\) is the tower gap you choose. We train on \(n\) noisy examples with Adam for 400 steps. The LLM always gets LoRA with rank \(r\); the tower is frozen, LoRA or full. Optionally, a share of the loss replays “text-only” data: inputs that go straight into the LLM, with the base model’s own outputs as targets. We report three numbers:

\[ \text{err} = \frac{\| L E - T \|_F^2}{\| L_0 E_0 - T \|_F^2}, \qquad \text{text drift} = \frac{\| L - L_0 \|_F^2}{\| L_0 \|_F^2}, \qquad \text{tower moved} = \frac{\| E - E_0 \|_F^2}{\| E_0 \|_F^2} \tag{194.8} \]
Read the symbols
\(T\)
the ideal model for the new domain, \((L_0 + \Delta L)(E_0 + g\,\Delta E)\)
\(\|\cdot\|_F\)
the Frobenius norm: the square root of the sum of squared entries
err
held-out error on new inputs: 1 means no better than the untouched model, 0 is perfect
text drift
how much the LLM changed; for text inputs that skip the tower, this is exactly how much its text outputs changed (for random text inputs)
tower moved
how much the shared tower changed; every other image task goes through it
frozen towerLoRA towerfull tower (dashed)
Figure 4. Live training of a two-matrix toy VLM (seeded, so every run is the same). The y-axis is the held-out error of formula (194.8) on a log scale. Each slider change trains all three tower choices from scratch.
How to read Figure 4

At the default (gap 0.5, 128 examples, rank 4), the frozen tower stalls at an error of 0.303 while LoRA and full towers reach 0.011 and 0.012. Look at the text drift: the frozen run has the largest (0.177), because the LLM is bent to make up for the tower. Set the gap to 0 and the examples to 32: now the frozen tower is best (0.061, against 0.165 and 0.208), because the trainable towers fit the noise. Set the gap to 1 and the examples to 128: frozen stalls at 0.447 with text drift 0.459. Finally, go back to the default and set text replay to 10%: text drift of the LoRA-tower run falls from 0.048 to 0.002, but the tower now moves 0.799 instead of 0.185. Replay protects the LLM by pushing the change into the tower, so check other image skills too.

Here is the same experiment in PyTorch, with a from-scratch LoRA. It runs on a CPU in about three seconds and is deterministic (it uses PyTorch’s random generator, so its numbers differ a little from the figure).

import torch
torch.manual_seed(0)
DX, DF, DY = 16, 8, 8                       # image input, tower output (= LLM input), LLM output

def lowrank(a, b, k=2):
    return torch.randn(a, k) @ torch.randn(k, b)

# 1) A "pretrained" VLM: tower E0 and LLM L0. The new domain needs a change in BOTH:
#    the LLM must answer in a new way (dL) and the tower must see new features (gap * dE).
E0, L0 = torch.randn(DF, DX) / DX**0.5, torch.randn(DY, DF) / DF**0.5
dE = lowrank(DF, DX); dE *= 0.6 * E0.norm() / dE.norm()
dL = lowrank(DY, DF); dL *= 0.3 * L0.norm() / dL.norm()

def make_task(gap, n=128, noise=0.5):
    T = (L0 + dL) @ (E0 + gap * dE)          # the ideal model for the new domain
    X = torch.randn(n, DX)
    Y = X @ T.T + noise * torch.randn(n, DY) * T.norm() / (DY * DX)**0.5
    return T, X, Y

class LoRA(torch.nn.Module):                 # W = W0 + B A, with B = 0 at the start
    def __init__(self, W0, r=4):
        super().__init__()
        self.register_buffer("W0", W0.clone())
        self.A = torch.nn.Parameter(torch.randn(r, W0.shape[1]) / W0.shape[1]**0.5)
        self.B = torch.nn.Parameter(torch.zeros(W0.shape[0], r))
    def weight(self):
        return self.W0 + self.B @ self.A

class Full(torch.nn.Module):
    def __init__(self, W0):
        super().__init__(); self.W = torch.nn.Parameter(W0.clone())
    def weight(self):
        return self.W

class Frozen(torch.nn.Module):
    def __init__(self, W0):
        super().__init__(); self.register_buffer("W", W0.clone())
    def weight(self):
        return self.W

def run(tower, task, text_mix=0.0, steps=400):
    T, X, Y = task
    torch.manual_seed(1); llm = LoRA(L0)    # the LLM always gets LoRA, with the same init
    opt = torch.optim.Adam(list(tower.parameters()) + list(llm.parameters()), lr=1e-2)
    for _ in range(steps):
        L, E = llm.weight(), tower.weight()
        task_loss = ((X @ (L @ E).T - Y)**2).mean()
        text_loss = ((L - L0)**2).sum() / DY  # text-only replay: keep L t close to L0 t
        loss = (1 - text_mix) * task_loss + text_mix * text_loss
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        L, E = llm.weight(), tower.weight()
        err = ((L @ E - T).norm()**2 / (L0 @ E0 - T).norm()**2).item()
        drift = ((L - L0).norm()**2 / L0.norm()**2).item()
        moved = ((E - E0).norm()**2 / E0.norm()**2).item()
    return err, drift, moved

for gap in [0.0, 1.0]:
    task = make_task(gap)                    # the same data for all three runs
    for name, make in [("frozen tower", Frozen), ("LoRA tower", LoRA), ("full tower", Full)]:
        torch.manual_seed(2)
        err, drift, moved = run(make(E0), task)
        print(f"gap {gap:.0f}  {name:12s}  new-domain error {err:.3f}  "
              f"text drift {drift:.3f}  tower moved {moved:.3f}")

Output (PyTorch 2.8, CPU):

gap 0  frozen tower  new-domain error 0.006  text drift 0.090  tower moved 0.000
gap 0  LoRA tower    new-domain error 0.015  text drift 0.081  tower moved 0.020
gap 0  full tower    new-domain error 0.017  text drift 0.031  tower moved 0.070
gap 1  frozen tower  new-domain error 0.273  text drift 0.701  tower moved 0.000
gap 1  LoRA tower    new-domain error 0.005  text drift 0.249  tower moved 0.282
gap 1  full tower    new-domain error 0.006  text drift 0.085  tower moved 0.240

Read it like a lab result. With no tower gap, freezing wins: a trainable tower only adds ways to fit noise. With a gap, the frozen run is stuck at 0.273, and its LLM changed the most of all six runs (text drift 0.701). That last number is the one people miss: a frozen tower can cause more text regression, not less, because the LLM is forced to compensate. In the real lab you do not know the gap in advance, which is exactly why you run A, B and C and measure.

Step 6: before and after, on held-out data

After training, run eval_vlm.py for A, B and C. You now have, for every one of the 2,500 test items, a correct/wrong flag for the base model and for each run. Two accuracies alone hide most of the information. Count the four cells of a paired table instead: \(n_{11}\) items both got right, \(n_{10}\) only the base got right, \(n_{01}\) only the tuned model got right, \(n_{00}\) both got wrong.

\[ \hat{\Delta} = \frac{n_{01} - n_{10}}{n}, \qquad \mathrm{SE}(\hat{\Delta}) = \frac{1}{n}\sqrt{\,n_{01} + n_{10} - \frac{(n_{01} - n_{10})^2}{n}\,} \tag{194.9} \]
Read the symbols
\(n\)
the number of test items (2,500 for ChartQA test)
\(\hat{\Delta}\)
the change in accuracy, after minus before
\(\mathrm{SE}\)
its standard error when both models are scored on the same items; a 95% interval is \(\hat{\Delta} \pm 1.96\,\mathrm{SE}\)

Only the items that changed carry information about the difference. The exact McNemar test asks: if the two models were equally good, how surprising is the split of the changed items? Under that assumption, each changed item is a coin flip:

\[ p = \min\!\left(1,\ 2 \sum_{i=0}^{k} \binom{m}{i} 2^{-m}\right), \qquad m = n_{01} + n_{10}, \quad k = \min(n_{01}, n_{10}) \tag{194.10} \]

The paired interval is much narrower than the one you get by treating the two accuracies as independent, because most items are easy or hard for both models. Figure 5 shows both.

Figure 5. Paired before/after comparison with formulas (194.9) and (194.10). The three presets are hypothetical counts for illustration, not results. Put in your own counts from compare.py below.
How to read Figure 5

“A clear gain” (2,500 items, 120 lost and 260 gained): +5.6 points, paired interval +4.1 to +7.1, unpaired +2.9 to +8.3, \(p < 0.0001\). “A tiny gain” (60 lost, 80 gained): +0.8 points; the paired interval −0.1 to +1.7 includes 0, \(p = 0.108\). Do not report this as an improvement. “A text check” (500 items, 22 lost, 12 gained): −2.0 points, interval −4.3 to +0.3, \(p = 0.121\). This does not prove there is no regression; with 500 items you cannot see a 2-point drop clearly. Use more items (ARC-Easy test has 2,376) or a more sensitive measure such as the KL drift below.

# compare.py: paired comparison of two eval files.  Usage: python compare.py base A
import sys, json, math
a, b = (json.load(open(f"eval-{r}.json"))["rows"] for r in sys.argv[1:3])
assert [r["q"] for r in a] == [r["q"] for r in b], "score the SAME items in the SAME order"
for part in ["all", "human", "augmented"]:
    pairs = [(x["ok"], y["ok"]) for x, y in zip(a, b)
             if part == "all" or x["human"] == (part == "human")]
    n = len(pairs)
    if n == 0: continue
    n10 = sum(x and not y for x, y in pairs); n01 = sum(y and not x for x, y in pairs)
    d = (n01 - n10) / n
    se = math.sqrt(max(n01 + n10 - (n01 - n10)**2 / n, 0)) / n
    m, k = n01 + n10, min(n01, n10)                  # exact McNemar (two-sided)
    p = min(1.0, 2 * sum(math.comb(m, i) for i in range(k + 1)) / 2**m) if m else 1.0
    print(f"{part:9s} n={n}  change {100*d:+.1f} pts  95% CI [{100*(d-1.96*se):+.1f}, "
          f"{100*(d+1.96*se):+.1f}]  lost {n10}  gained {n01}  p={p:.4f}")

One caution about independence: the 1,250 human test questions are asked about only 625 charts [Certain, ChartQA paper]. Questions on the same chart are correlated, so the true interval is a little wider than formula (194.9) says [Likely]. A bootstrap that resamples whole charts fixes this.

Step 7: did text-only skills regress?

A VLM is also a chat model. After thousands of steps of “chart in, one number out”, it can forget how to answer plain text questions, or start answering everything with one word. Check three things, each against the base model, on prompts with no image:

  1. A text benchmark. ARC-Easy (grade-school science questions with 4 choices), scored without generation: the model picks the choice with the highest mean log-probability. 500 test questions, seeded.
  2. KL drift. Let the base model answer each question, then measure how much the tuned model’s next-token distribution differs on those same tokens. This sees small changes long before a benchmark does.
  3. Answer length. The mean number of words in free answers. A collapse from sentences to single words is the typical “format leak” of short-answer fine-tuning.
\[ D_{\text{text}} = \frac{1}{|S|} \sum_{t \in S} \KL\!\left( p_{\text{base}}(\cdot \mid x_{\lt t}) \,\Big\|\, p_{\text{tuned}}(\cdot \mid x_{\lt t}) \right) \tag{194.11} \]
Read the symbols
\(S\)
the positions of the base model’s own greedy answer tokens, over all text prompts
\(\KL(p \,\|\, q)\)
\(\sum_v p(v) \log \frac{p(v)}{q(v)}\), in nats: 0 if the two models agree exactly on every next-token distribution
# text_check.py: text-only regression check.  Usage: python text_check.py A
import sys, json, torch, torch.nn.functional as F
from datasets import load_dataset
from common import SMOKE, DEV, processor
from eval_vlm import load_model

run = sys.argv[1]
proc = processor(); tok = proc.tokenizer
base, tuned = load_model("base"), load_model(run)
arc = load_dataset("allenai/ai2_arc", "ARC-Easy", split="test").shuffle(seed=0)
arc = arc.select(range(4 if SMOKE else 500))

def ids_of(user, answer=None):                   # text-only chat prompt, optional answer
    msgs = [{"role": "user", "content": [{"type": "text", "text": user}]}]
    p_ids = tok(proc.apply_chat_template(msgs, add_generation_prompt=True),
                add_special_tokens=False)["input_ids"]
    a_ids = tok(" " + answer, add_special_tokens=False)["input_ids"] if answer else []
    return torch.tensor([p_ids + a_ids], device=DEV), len(p_ids)

@torch.no_grad()
def answer_logprob(model, user, answer):         # mean log p(answer tokens | prompt)
    ids, n_p = ids_of(user, answer)
    logp = F.log_softmax(model(input_ids=ids).logits[0, n_p - 1:-1].float(), -1)
    return logp.gather(-1, ids[0, n_p:, None]).mean().item()

@torch.no_grad()
def kl_on_base_answer(user):                     # formula (194.11) for one prompt
    ids, n_p = ids_of(user)
    full = base.generate(input_ids=ids, max_new_tokens=48, do_sample=False)
    lb = F.log_softmax(base(input_ids=full).logits[0, n_p - 1:-1].float(), -1)
    lt = F.log_softmax(tuned(input_ids=full).logits[0, n_p - 1:-1].float(), -1)
    return (lb.exp() * (lb - lt)).sum(-1).mean().item()

@torch.no_grad()
def words(model, user):                          # does it still write sentences?
    ids, n_p = ids_of(user)
    out = model.generate(input_ids=ids, max_new_tokens=48, do_sample=False)
    return len(tok.decode(out[0, n_p:], skip_special_tokens=True).split())

res = {"run": run}
for name, m in [("base", base), (run, tuned)]:
    ok = 0
    for ex in arc:
        scores = [answer_logprob(m, ex["question"] + "\nAnswer:", c) for c in ex["choices"]["text"]]
        ok += ex["choices"]["label"][scores.index(max(scores))] == ex["answerKey"]
    res[f"arc_easy_{name}"] = ok / len(arc)
    res[f"words_{name}"] = sum(words(m, ex["question"]) for ex in arc) / len(arc)
res["kl_drift"] = sum(kl_on_base_answer(ex["question"]) for ex in arc) / len(arc)
print(json.dumps(res)); json.dump(res, open(f"text-{run}.json", "w"))

Our smoke run (run A after 2 steps, 4 questions) printed a KL drift of 0.000118 nats per token and the same ARC answers and lengths as the base: two tiny steps barely move the model, as they should. That is a check that the code runs, not a result. Save the per-question ARC flags too if you want the paired test of Step 6 on the text side.

Do not assume LoRA keeps you safe. Biderman et al. found that LoRA forgets less than full fine-tuning, but it still forgets, and more with more training [Certain, “LoRA Learns Less and Forgets Less”]. Measure it.

If you find a regression, the fixes from session 61 apply: mix some text-only instruction data into training (a few percent to tens of percent; tune on val) [Likely], lower the learning rate or the rank, stop earlier, or interpolate the merged weights back toward the base. Note a finding from the SmolVLM paper: reusing the LLM’s own text SFT data in VLM training hurt their small models, and new text data worked better [Certain, SmolVLM paper]. And remember Figure 4: replay can push the change into the vision tower, so re-check an image benchmark the model was good at before.

Step 8: fill the table and decide

Fill every cell of the Setup table. Then write a half-page decision memo, as for your team:

  1. Goal. Example: “answer chart questions in our reports at 95% of the speed of the base model; keep plain chat working”.
  2. Results. The table, with paired intervals for every “change” you claim, human and augmented separately.
  3. Decision. One run, and the artifact: the merged model, or the base plus a small adapter.
  4. Why. The deciding facts: gain against noise, text regression, training and serving cost.
  5. Risks. What you will watch in production, and what would make you change the decision.

Rules that usually decide it:

  • If B or C do not beat A clearly (paired interval), ship A. It is the cheapest to train, its adapter is 19.6 MB in FP32, and the frozen tower lets you cache image features: with a frozen tower and connector (and no random image augmentation), each training image’s 832 connector outputs can be computed once and reused every epoch, which removes the vision tower’s cost from all epochs after the first [Likely].
  • If B or C win clearly, the domain needs a different view of the image. Prefer B unless C is clearly better: B’s adapter is 30.2 MB, while C ships a new 86M-weight tower, which also changes every other image task the model does.
  • If text drift is large, fix it before shipping, even if ChartQA went up. A chat product will be judged on its chat.
  • Evaluate the exact artifact you ship. We merge the adapter before evaluating for this reason.

Where this is used

SettingUsual freeze mapWhy
Charts, tables, documents and receipts with a mid-size or small VLMLoRA on the LLM; try LoRA on the tower if small text is missedMost of the skill is reading layout and numbers; high resolution (tiling) often matters more than the freeze map
Medical or industrial images far from web photos (pathology, X-ray, defects)LoRA or full tower with a small learning rate, plus LoRA on the LLMThe tower must learn new visual features; a frozen tower is the usual bottleneck [Likely]
A new answer format or label set on familiar images (product attributes, captions in a house style)Frozen tower, LoRA on the LLMThe change lives in the LLM; cache image features to train fast
Many customers, one base modelOne LoRA adapter per customerOne shared base in memory, small files per task, adapters can be swapped per request [Likely]
Video question answering and action recognition (SmolVLM2 has video variants)Frozen tower, LoRA on the LLM, few frames per clipEach frame costs as much as an image, so frames dominate compute and memory; caching frame features pays off [Likely]
Key-moment detection and video captions for Video-RAGLoRA SFT on (frames, timestamp question, answer) dataAnswers are short and checkable (a time span, an event), so before/after evaluation is cheap
Video classification, retrieval or fingerprinting with VLM embeddingsLoRA on the tower with a contrastive lossSmall changes keep the embedding space compatible with the existing index (session 60)

Common mistakes

  1. Loss on the whole sequence. With TRL’s messages format the collator masks only padding, so 894 of 895 tokens carry loss and the answer gets 0.56% of the signal. Use the prompt-completion format, and print one batch (Figure 2).
  2. Truncation. A 1,024-token limit is shorter than one square image (1,088 image tokens). Set max_length=None or a limit you checked against your longest row.
  3. Ambiguous LoRA targets. target_modules=["q_proj", "v_proj"] matches the vision tower as well as the LLM. Use a regular expression with the part’s prefix and check the trainable count against formula (194.2).
  4. The unfrozen tower is not saved. save_pretrained on a PEFT model writes only the adapter. Save the tower yourself, and assert that it loads (no unexpected keys).
  5. Tuning on the test set. Pick the learning rate, the rank and the step count on val. The test set is scored once per run, at the end.
  6. Comparing two accuracies without pairing. Use formula (194.9) and McNemar on the same items. A gain inside the paired interval is not a gain.
  7. Counting format as skill. Many “gains” come from learning to answer “14” instead of “The answer is 14”. Give the base model the same format instruction, and read outputs.
  8. No text-only check. ChartQA up, chat broken is a common outcome of short-answer SFT. Measure ARC-Easy, KL drift and answer length before you ship.

Interview questions

You fine-tune a VLM on a small domain dataset. Do you freeze the vision tower? Why?basic
Start frozen. It is the cheapest (no backward pass through the tower, no tower activations, image features can be cached), it cannot damage other image skills, and with little data a trainable tower mostly fits noise (Figure 4 at gap 0 and 32 examples: frozen 0.061 against 0.165 and 0.208). Then try LoRA on the tower as a second run. Unfreeze only if it wins clearly on held-out data, which usually means the images are far from what the tower was trained on.
How many weights does LoRA with \(r = 16\) add to the LLM of SmolVLM-256M?core
Per block, \(r \sum (d_{in} + d_{out})\) over q (576 → 576), k and v (576 → 192), o (576 → 576), gate and up (576 → 1,536) and down (1,536 → 576): \(16 \times 10{,}176 = 162{,}816\). Times 30 blocks: 4,884,480, which is 1.90% of the 256,484,928 base weights. Adding LoRA to the tower’s six linear layers adds 2,654,208 more.
In a small VLM with image tiling, where does the training compute go?core
Mostly into the vision tower. A 13-tile chart is 13,312 patches through an 86.4M-weight tower: about 2.30 TFLOPs per forward pass, against 0.24 TFLOPs for the 134.6M-weight LLM on its 895 tokens. Even with a frozen tower it is about three quarters of the step. So the number of tiles (resolution) is the main cost knob, and unfreezing the tower is expensive because it adds a backward pass through those 13,312 tokens.
Why put the loss only on the answer tokens? What happens otherwise?core
The image and the question are inputs, not things the model should learn to produce. With loss on everything, almost all the loss is on predicting image placeholders and template tokens (894 labeled tokens, 5 of them answer tokens, so the answer gets 0.56% of the signal). Training is slower and noisier, and the model can learn to “generate questions”. Keep the end-of-turn token in the answer labels so the model learns to stop.
Your tower is frozen. How can you make training several times faster?advanced
Cache the image features. If the tower and the connector are frozen and the images are not randomly augmented, the connector outputs for each image never change. Compute them once (832 vectors of size 576 per 13-tile chart), store them, and feed them to the LLM in later epochs. That removes the tower’s cost, which Figure 1 shows is most of the step. You can no longer use random crops or flips, and you must cache again if you change the resolution or the tiling.
How do you know your fine-tuned model is really better than the base?core
Score both on the same held-out test items with the same prompt, decoding and metric, then compare item by item. The change is \((n_{01} - n_{10})/n\), with the paired standard error of formula (194.9), and an exact McNemar test on the changed items. Report the interval, not only the two accuracies, and report subsets separately (ChartQA human and augmented). Also check the base model is not already trained on your “domain” data, and that the gain is not only answer format.
How do you check that a VLM fine-tune did not hurt text-only skills?advanced
Run text-only checks against the base: a benchmark scored by log-likelihood (ARC-Easy), the mean answer length on open questions, and the KL drift of formula (194.11): the KL from the base to the tuned model’s next-token distributions, on the base model’s own answers. KL sees small changes before benchmark accuracy moves, and it needs no labels. If there is a regression, add text instruction data to the mix, lower the learning rate or rank, stop earlier, or interpolate toward the base.
What is ChartQA’s relaxed accuracy, and what is wrong with it?basic
A numeric answer is correct if it is within 5% of the gold value; other answers need an exact match. It forgives small reading errors on charts. But it treats years like amounts: for gold 2018, anything from 1917.1 to 2118.9 is “correct”, and 144 of the 2,500 test answers are years. It also gives no credit for “The answer is 14”, so format changes can look like skill changes.
After fine-tuning on ChartQA, your VLM answers every chat question with one number. Why, and how do you fix it?advanced
Every training answer was one or two words, so the model learned a strong prior for very short answers, and the LLM LoRA applies to text-only prompts too. It is a format leak, a form of forgetting. Fixes: mix in text-only instruction data with long answers; keep a system prompt or instruction that marks chart questions so the short format is tied to it; lower the LR or rank, or stop earlier; or keep the adapter separate and apply it only to chart requests. Measure it with answer length and KL drift on text prompts.
You unfreeze the vision tower and LLM LoRA suddenly learns much more slowly. What could cause it?advanced
Global gradient clipping. The trainer clips the norm of all gradients together. The fully trained tower has a much larger gradient (in our smoke run 428.7 against 22.6 for LoRA alone), so clipping to 1.0 scales every gradient down, including the LoRA ones. Log the norm per parameter group; fix it with a lower tower learning rate, per-group clipping, or a higher clipping threshold.

Exercises

  1. Count the LoRA weights for SmolVLM-500M (32 blocks, width 960, key and value outputs 320, MLP 2,560) with \(r = 8\) on all seven LLM linear layers. Check it in Figure 1.
  2. A 1,024 × 768 photo and a 300 × 200 thumbnail: how many tiles and image tokens does each become with splitting on? What does this say about small images?
  3. Your counts for run A against the base are \(n_{10} = 60\), \(n_{01} = 80\) on 2,500 items. Compute the change, its paired 95% interval and the McNemar p-value. Is it a gain?
  4. Under relaxed accuracy, is “0.6” correct for gold “0.57”? Is “1950” correct for gold “2018”? Propose a fix for years.
  5. In Figure 4, find one setting where the frozen tower is best and one where it is worst. Explain each in one sentence.
  6. In Figure 4 at the default, set text replay to 10%. What happens to the text drift and to the tower? What does this mean for your real evaluation plan?
  7. Swap ChartQA for PathVQA (flaviagiammarino/path-vqa, columns image, question, answer). Which lines change, and which metric would you use?
Answerscheck
1. Per block: q and o \(960 + 960 = 1{,}920\) each, k and v \(960 + 320 = 1{,}280\) each, gate and up \(960 + 2{,}560 = 3{,}520\) each, down 3,520: sum 16,960. Times 8 and 32 blocks: 4,341,760. 2. The photo is resized to 2,048 × 1,536: \(4 \cdot 3 + 1 = 13\) tiles, 832 tokens. The thumbnail is resized up to 2,048 × 1,365: also 13 tiles and 832 tokens (we checked both with the processor). Small images are upscaled and cost as much as large ones; for small images set a smaller longest edge (the model card suggests size={"longest_edge": N*512}) or turn splitting off. 3. Change +0.8 points; SE \(= \sqrt{140 - 400/2500}/2500 = 0.0047\); interval −0.1 to +1.7 points; exact p = 0.108. Not a clear gain. 4. 0.6 is 5.26% off 0.57, so it fails at 5%. 1950 is 3.4% off 2018, so it “passes”. Fix: treat four-digit integers between 1900 and 2100 as exact-match answers (and report the rule you used). 5. Best: gap 0, 32 examples (0.061 against 0.165 and 0.208): no tower change is needed, so trainable towers only fit noise. Worst: gap 1, 128 examples (0.447 against 0.005 and 0.006): the needed information is not in the frozen tower’s output. 6. Text drift for the LoRA-tower run drops from 0.048 to 0.002, but the tower moves 0.799 instead of 0.185 (frozen-tower error rises from 0.303 to 0.326). Replay pushes the change into the tower, so also re-run an image benchmark the base was good at. 7. Only chartqa(), to_pc() and the eval loop’s column names change, and the format line (yes/no or a short phrase). Use accuracy on yes/no questions and exact match (or token F1) on open questions, reported separately; check that the base model has not seen PathVQA.

Key takeaways

  • Count before you train: LoRA with \(r = 16\) on SmolVLM-256M’s LLM is 4,884,480 weights (194.2); LoRA on the tower adds 2,654,208; a full tower adds 86,433,024.
  • In a small tiled VLM, the vision tower dominates compute (194.3): 13,312 patches against 895 LLM tokens. Tiling is the biggest cost knob; unfreezing the tower adds a backward pass through all those patches.
  • Train on the answer only (194.6) and never truncate VLM rows. Print one real batch and check which tokens carry loss.
  • Whether to unfreeze depends on where the domain gap is. A frozen tower is safest when the tower already sees what matters, but can cause more text drift when it does not, because the LLM compensates (Figure 4).
  • Score before and after on the same held-out items with a paired interval and McNemar (194.9, 194.10); know your metric’s blind spots (194.7) and your base model’s training data.
  • Check text-only skills with a benchmark, answer length and KL drift (194.11), and evaluate the exact artifact you ship.

Key formulas

\[ \mathbf{h} = W_0\mathbf{x} + \tfrac{\alpha}{r}BA\mathbf{x}, \quad P = r\textstyle\sum(d_{in} + d_{out}), \quad C_p \approx 2N_p t_p(1 + [\text{bwd}] + [\text{full}]) \]
\[ T = \lceil W'/512 \rceil \lceil H'/512 \rceil + 1, \quad n_{\text{img}} = 64T, \quad \mathcal{L} = -\tfrac{1}{|\mathcal{A}|}\textstyle\sum_{t \in \mathcal{A}} \log p_\theta(x_t \mid I, x_{\lt t}) \]
\[ \hat{\Delta} = \frac{n_{01} - n_{10}}{n}, \quad \mathrm{SE} = \frac{\sqrt{n_{01} + n_{10} - (n_{01} - n_{10})^2/n}}{n}, \quad D_{\text{text}} = \overline{\KL(p_{\text{base}} \,\|\, p_{\text{tuned}})} \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
  2. Hugging Face (code). PEFT library
  3. Hugging Face (code). TRL library
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
  5. Hugging Face (code). PEFT library
  6. Hugging Face (code). TRL library
  7. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  8. Laurençon, Marafioti, Sanh & Tronchon (2024). Building and better understanding vision-language models: insights and future directions (Idefics3)
  9. Masry, Long, Tan, Joty & Hoque (2022). ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
  10. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  11. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  12. Clark et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
  13. He et al. (2020). PathVQA: 30000+ Questions for Medical Visual Question Answering
  14. Biderman et al. (2024). LoRA Learns Less and Forgets Less