LLM & VLM courseBlog
189Track 7: Vision and VLMs · session 27 of 38

VLM SFT and PEFT

Supervised fine-tuning (SFT) turns a pretrained vision-language model into an assistant, or into a specialist for your own images. For each of its three parts (vision encoder, connector, LLM) you decide: freeze it, train a small LoRA adapter, or train it fully, and with which learning rate. This session shows how to make those choices with real parameter counts and memory numbers, how QLoRA fits a 7B VLM on one small GPU, and how to keep the text skills the model already had.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: LoRA and QLoRA (sessions 57 and 58), forgetting and data mixing (61), and VLM training stages (186); batching and masking for multimodal sequences (188) helps

What you will learn

  • What VLM SFT optimizes: next-token loss on the answer tokens only, with image tokens and prompts masked out
  • The freeze map: frozen, LoRA or full for the vision tower, the connector and the LLM, and what each choice costs in memory and compute
  • Different learning rates per part with AdamW parameter groups, with the real LLaVA-1.5, LLaVA-OneVision and Molmo values
  • Where to put LoRA (LLM, connector, vision tower), how many weights each target adds, and what LLaVA-1.5 LoRA scored against full fine-tuning
  • QLoRA for VLMs: NF4 at 4.127 bits per weight, what to keep in 16 bits (vision tower, connector, embeddings, lm_head), and a 7B VLM in under 10 GB before activations
  • How to keep text-only skills: measure them, mix text data in, prefer LoRA or a smaller learning rate, stop earlier, or interpolate back toward the base
  • A runnable PyTorch freeze map with LoRA, per-part learning rates, answer-only loss and a merge check

Why this matters

After pretraining, a vision-language model (VLM) can describe images, but it does not yet follow instructions well, and it knows nothing about your images: your product photos, your medical scans, your video frames. Supervised fine-tuning (SFT) fixes both. You show the model many examples of “image + question → good answer” and train it to produce the answer.

In session 186 you saw that SFT is the third stage of building a VLM. In session 188 you saw how to pack samples and put the loss only on the answer. This session is about the decisions you make when you run SFT, either as the last stage of building a VLM or as a fine-tune of an open model such as LLaVA-1.5 or Qwen2-VL:

  1. What to train. A VLM has three parts: the vision tower (the vision encoder, for example a ViT), the connector (a small network that maps image features into the LLM’s input space) and the LLM. Each one can be frozen, adapted with a small add-on (PEFT, parameter-efficient fine-tuning, such as LoRA), or trained fully.
  2. How fast each part moves. The parts were trained on very different data, so they need different learning rates.
  3. How to fit it on your GPU. Full fine-tuning of a 7B VLM needs more than 100 GB before activations. LoRA and QLoRA bring this below 20 GB and 10 GB.
  4. How not to break what already works. A VLM is also a chat model. If SFT sees only images, the model can forget text skills (and even some visual skills it had).

The real recipes we will use: LLaVA-1.5 (CLIP ViT-L/14-336, a 2-layer MLP connector, Vicuna-7B or 13B), which ships both a full fine-tuning script and a LoRA script with published scores; Qwen2-VL, which freezes its vision encoder during SFT; and Idefics2, which trains its pretrained parts with LoRA and DoRA. [Certain]

What SFT optimizes in a VLM

One SFT sample is a short conversation with one or more images. The chat template turns it into one token sequence: image tokens (the connector outputs), then the user’s text, then the assistant’s answer. The model is trained with the usual next-token loss, but only on the answer tokens. The image tokens and the user’s words are context, not targets:

\[ \mathcal{L}_{\text{SFT}}(\theta) = -\frac{1}{\sum_t m_t} \sum_{t=1}^{T} m_t \log p_\theta\!\left(x_t \mid I,\ x_{\lt t}\right) \tag{189.1} \]
Read the symbols
\(\theta\)
all weights of the VLM: vision tower, connector and LLM
\(I\)
the image (or images, or video frames) of the sample
\(x_1, \dots, x_T\)
the text tokens of the conversation after the chat template
\(m_t\)
the loss mask: 1 if token \(x_t\) belongs to an assistant answer, 0 for prompts, image placeholders, padding and template tokens
\(p_\theta(x_t \mid I, x_{\lt t})\)
the probability the model gives to the correct next token

In code, the mask is the label value -100, which PyTorch’s cross-entropy ignores. Idefics2 states the same rule: “we only compute the loss on the tokens of the answers in the Q/A pairs”. [Certain] Formula (189.1) is the same loss as text SFT (session 54). What is new in a VLM is the question which part of \(\theta\) the gradient is allowed to change.

The freeze map: three parts, three choices

For each part you choose one of three modes. We call the result the freeze map.

  • Frozen: the weights never change. The part needs memory only for its weights (2 bytes each in BF16), and no weight gradients are computed for it.
  • LoRA: the original weights stay frozen, and small low-rank matrices are added to some of its linear layers and trained (session 57).
  • Full: every weight is trained. With mixed-precision AdamW this costs 16 bytes per weight: BF16 weight 2, BF16 gradient 2, FP32 master copy 4, Adam moments \(m\) and \(v\) 4 + 4 (session 4).

What do real recipes do in SFT? LLaVA-1.5 freezes the vision tower and trains the connector and the LLM fully (or the LLM with LoRA) [Certain, from the training code]. Qwen2-VL: “we lock the ViT parameters and perform exclusive fine-tuning of the LLM” [Certain]. LLaVA-OneVision trains everything, with a smaller learning rate for the vision tower [Certain]. Idefics2 trains the new parts fully and the pretrained vision encoder and LLM with LoRA (DoRA in its instruction tuning) [Certain].

Compute: where the backward pass must go

Freezing saves more than memory. In a VLM the order is fixed: vision tower, then connector, then LLM, then the loss. The backward pass starts at the loss and must travel back to the lowest trainable part. Parts below it need only a forward pass. Parts above it need the gradient with respect to their inputs (to pass it down), and only trained weights need gradients with respect to the weights. For a part with \(N_p\) weights, each pass costs about \(2N_p\) FLOPs per token (one multiply and one add per weight):

\[ C_p \approx \underbrace{2N_p}_{\text{forward}} + \underbrace{2N_p \cdot [\text{backward passes through } p]}_{\text{input gradients}} + \underbrace{2N_p \cdot [p \text{ is fully trained}]}_{\text{weight gradients}} \ \text{ FLOPs per token} \tag{189.2} \]

The brackets are 1 if the condition is true and 0 otherwise. A LoRA part has a frozen base, so it pays \(4N_p\) (plus a tiny amount for the adapters); a fully trained LLM pays \(6N_p\), the familiar training rule. A frozen vision tower below a trainable connector pays only \(2N_p\) per image token and stores no activations for the backward pass. That is why LLaVA runs the vision tower under torch.no_grad() [Certain, from the code].

Figure 1 puts the three choices together for two real models. The parameter counts are computed from the models’ config.json files on Hugging Face and match the size of the released weights exactly: LLaVA-1.5-7B has 7,063,427,072 weights (vision tower 303.5M, connector 21.0M, LLM 6,738.9M) and Qwen2-VL-7B has 8,291,375,616 (vision tower 631.2M, merger 44.6M, LLM 7,615.6M). [Certain]

Model Vision tower Connector LLM LoRA in the LLM
frozenLoRAfull trainingmemory: frozen weightsmemory: trainable weights, gradients, FP32 copy, Adam m and v (16 bytes each)
Figure 1. A freeze map and its memory cost before activations, from formula (189.6). Parameter counts are computed from the Hugging Face configs of LLaVA-1.5-7B and Qwen2-VL-7B. LoRA on the vision tower and connector covers all their linear layers. A dashed border means the LLM’s linear layers are stored in NF4 (4.127 bits per weight); embeddings and the output layer stay in BF16. The dashed orange arrow shows how far the backward pass must travel.
How to read Figure 1

Start with “LLaVA-1.5 full SFT”: 6.76B trainable weights and 108.77 GB, more than one 80 GB GPU, which is why the official script uses DeepSpeed ZeRO-3 to shard the states over 8 GPUs. Click “LLaVA-1.5 LoRA” (the official LoRA recipe: rank 128 on all linear layers of the LLM, full connector): 340.8M trainable weights (4.82%), 19.54 GB. Turn on 4-bit: 9.93 GB, because the 6.48B linear weights of the LLM drop from 2 bytes to 0.516 bytes. Now change the rank: memory grows slowly, from 14.74 GB at rank 8 to 19.54 GB at rank 128, because the frozen base dominates. Finally unfreeze the vision tower fully: the backward pass now reaches the image, and 303.5M more weights cost 16 bytes each.

Different learning rates for different parts

When several parts train together, they rarely share one learning rate. The rule of thumb: the more a part already knows, and the broader the data it learned from, the more gently it should move. New or small parts can move fast. In AdamW this is done with parameter groups: each group has its own learning rate \(\eta_g\) and its own weight decay \(\lambda_g\), and the same update rule:

\[ \theta_g \leftarrow \theta_g - \eta_g(t)\left( \frac{\hat{\mathbf{m}}_g}{\sqrt{\hat{\mathbf{v}}_g} + \varepsilon} + \lambda_g\, \theta_g \right), \qquad \eta_g(t) = \eta_g \cdot s(t), \qquad g \in \{\text{vision}, \text{connector}, \text{LLM}, \text{LoRA}\} \tag{189.3} \]
Read the symbols
\(\theta_g\)
the trainable weights of group \(g\)
\(\eta_g\)
the peak learning rate of the group
\(s(t)\)
the shared schedule, a number between 0 and 1 (warmup, then cosine decay)
\(\hat{\mathbf{m}}_g, \hat{\mathbf{v}}_g\)
Adam’s bias-corrected moment estimates for those weights
\(\lambda_g\)
the weight decay of the group (often 0 for LoRA, norms and biases)

Real values, all with a cosine schedule:

RecipeVision towerConnectorLLM
LLaVA-1.5 full SFTfrozen\(2\times10^{-5}\)\(2\times10^{-5}\) (full)
LLaVA-1.5 LoRA SFT (\(r = 128\), \(\alpha = 256\))frozen\(2\times10^{-5}\) (full)\(2\times10^{-4}\) (LoRA factors)
LLaVA-OneVision (later stages)\(2\times10^{-6}\)\(1\times10^{-5}\)\(1\times10^{-5}\)
Molmo (pretraining)\(6\times10^{-6}\)\(2\times10^{-4}\)\(2\times10^{-5}\)

The LLaVA-1.5 rows come from its finetune.sh and finetune_lora.sh scripts (1 epoch, warmup 3%, weight decay 0) [Certain]. Molmo also uses a 200-step warmup for the connector against 2,000 for the ViT and the LLM, and clips gradients separately for each part [Certain]. The OneVision values are from its paper [Certain, as reported in session 186].

Notice the LLaVA LoRA row. The LoRA factors get a learning rate 10 times larger than the full-training rate, while the connector keeps the small one. This is typical: a LoRA update starts at zero (\(B = 0\)) and lives in a small subspace, so it needs bigger steps to move as far; the connector is a full weight matrix and keeps the full-training rate. The script sets it with a separate flag, --mm_projector_lr 2e-5, which creates a separate parameter group in the optimizer [Certain, from the code].

A softer version of the same idea, inside the vision tower, is layer-wise learning-rate decay: block \(l\) of \(L\) gets \(\eta \cdot \gamma^{L - l}\) with \(\gamma\) around 0.8 to 0.9, so the early blocks (general edges and textures) move least. It comes from fine-tuning ViTs for classification (BEiT, MAE) and is used in some VLM recipes. [Likely]

Algorithm 1: build the freeze map and the optimizer
  1. input: VLM with parts vision, connector, LLM; a mode per part (frozen, LoRA, full); learning rates \(\eta_g\)
  2. set requires_grad = False for every weight // start from "all frozen"
  3. for each part \(p\) with mode full: set requires_grad = True for its weights; put them in group \(p\)
  4. for each part \(p\) with mode LoRA: wrap the chosen linear layers, \(W_0 x \to W_0 x + \tfrac{\alpha}{r} B A x\); put \(A, B\) in group "LoRA of \(p\)"
  5. split every group into weights that get decay (matrices) and weights that do not (biases, norms)
  6. build AdamW with one entry per group: \(\{\text{params}, \eta_g, \lambda_g\}\) // formula (189.3)
  7. if the lowest trainable part is above the vision tower: run the vision tower under no_grad // formula (189.2)
  8. print the trainable count per part, and check it against your plan before you start a run

LoRA in a VLM: on the LLM, the connector or the vision tower

LoRA replaces the output of a frozen linear layer \(W_0 \in \R^{d_{\text{out}} \times d_{\text{in}}}\) by

\[ \mathbf{h} = W_0 \mathbf{x} + \frac{\alpha}{r}\, B A \mathbf{x}, \qquad A \in \R^{r \times d_{\text{in}}},\ \ B \in \R^{d_{\text{out}} \times r},\ \ B_{\text{init}} = 0 \tag{189.4} \]
Read the symbols
\(r\)
the rank, much smaller than \(d_{\text{in}}\) and \(d_{\text{out}}\) (common values 8 to 128)
\(\alpha\)
a scale; the update is multiplied by \(\alpha / r\) (rsLoRA uses \(\alpha / \sqrt{r}\))
\(A, B\)
the trainable factors; \(A\) starts random, \(B\) starts at zero, so at step 0 the model is unchanged

Each adapted layer adds \(r(d_{\text{in}} + d_{\text{out}})\) trainable weights. For a whole part, sum over the adapted layers:

\[ N_{\text{LoRA}} = r \sum_{\ell \in \text{targets}} \left( d_{\text{in}}^{(\ell)} + d_{\text{out}}^{(\ell)} \right) \tag{189.5} \]

Where should the adapters go? Each of the three parts gives a different answer.

LoRA on the LLM (the default)

Most of the weights are here, and most of the new behavior (answer format, task skill, domain words) is language behavior. LLaVA-1.5 adds LoRA to every linear layer of the LLM except lm_head, and its code explicitly skips any module whose name contains mm_projector or vision_tower [Certain, from the code]. The QLoRA paper found that adapting all linear layers of each transformer block is needed to match full fine-tuning; adapting only the query and value projections is not enough [Certain]. Figure 1 shows the price: for LLaVA-1.5-7B at rank 8, “q and v only” adds 4.2M weights and “all 7 linear layers” adds 20.0M. Both are tiny next to the 14 GB of frozen weights.

Full training (not LoRA) for the connector

The connector is small (21.0M weights in LLaVA-1.5, 0.30% of the model; 44.6M in Qwen2-VL) and it is the part that most needs to change when the image distribution changes. LoRA on it would save almost nothing. So recipes train it fully: LLaVA-1.5 LoRA trains the projector fully at \(2\times10^{-5}\) [Certain], and Idefics2 uses “standard full fine-tuning” for its newly initialized parts [Certain]. With Hugging Face PEFT you list it in modules_to_save, which makes it trainable and saves it with the adapter. Freeze the connector only if you want the visual features to reach the LLM exactly as before (for example, a text-only style fine-tune).

LoRA on the vision tower (when the images are new)

Freeze the vision tower when your images look like web photos: CLIP and SigLIP have seen billions of them. Adapt it when the pixels are unusual: X-rays, satellite images, thermal video, dense documents at a new resolution, or a new position-embedding size. Idefics2 does exactly this: it interpolates the encoder’s position embeddings to a higher resolution and trains “the vision encoder with LoRA parameters to adapt to these modifications” [Certain]. LoRA is a safer way to change the vision tower than full training, because the change is small and can be scaled down or removed. In Figure 1, LoRA rank 8 on all linear layers of the LLaVA vision tower adds only 3.5M weights, but it makes the backward pass go all the way down to the pixels, so activation memory and compute for the image tokens go up.

Does LoRA lose quality for VLM SFT?

LLaVA-1.5 released both versions trained on the same 665K-sample mix for one epoch, so we can compare them directly. Figure 2 shows the differences. For the 7B model, LoRA was higher on 8 of 11 benchmarks, lower on 2 and equal on 1, with a mean difference of +0.6 points [Certain, from the published table]. Each number comes from a single run, and differences of about 1 point may be noise. [Likely] The text-only skills were not measured in that table.

LoRA higherfull fine-tuning higher
Figure 2. LLaVA-1.5 with full fine-tuning of the LLM (learning rate \(2\times10^{-5}\)) and with LoRA (rank 128, \(\alpha\) = 256, learning rate \(2\times10^{-4}\)), both with a frozen vision tower and a fully trained connector. Scores are copied from the LLaVA Model Zoo. Bars show LoRA minus full, in points; the labels give both scores.
How to read Figure 2

Green bars are benchmarks where the LoRA model scored higher, orange where full fine-tuning did. For 7B, most bars are green and small (under 2.5 points); VizWiz is the largest loss (−2.2). Switch to 13B: the picture is mixed (4 higher, 5 lower, 2 equal) and the biggest bars go in both directions (VizWiz +5.3, LLaVA-Bench −3.0). The lesson is not “LoRA is better” but “on this data, LoRA and full fine-tuning are about equal, at about one fifth of the memory”.

This matches what is known for text LLMs: on instruction-following data, which mostly teaches format and style, LoRA usually matches full fine-tuning; on data that must teach a lot of new knowledge (for example a large amount of code or math), full fine-tuning learns more. That is the finding of “LoRA Learns Less and Forgets Less” [Certain]. For a VLM, the second case is learning a new visual domain or a new language from scratch; there, more trainable capacity (higher rank, more targets, or full training) helps. [Likely]

DoRA (weight-decomposed LoRA) splits each weight into a length and a direction and trains the length separately (session 57). Idefics2 used it for instruction tuning [Certain]. In PEFT it is one flag, use_dora=True; it is slower per step than LoRA, so merge it before serving.

QLoRA for VLMs

QLoRA (session 58) stores the frozen base weights in 4 bits and trains LoRA adapters in 16 bits on top. In each forward pass a block of 4-bit weights is turned back into BF16 (dequantized), multiplied, and thrown away. Gradients flow through the frozen 4-bit weights into the adapters. The QLoRA paper fine-tuned a 65B model on one 48 GB GPU this way [Certain]. Its storage cost per weight, with blocks of 64 weights, 8-bit block scales, and one FP32 scale for every 256 blocks (“double quantization”), is

\[ b_{\text{NF4}} = 4 + \frac{8}{64} + \frac{32}{64 \cdot 256} = 4.127 \text{ bits} = 0.516 \text{ bytes per weight} \tag{189.6} \]

So the memory of a VLM training run, before activations, is

\[ M \approx \sum_{p\ \text{frozen part}} b_p\, N_p + 16\, N_{\text{train}} \ \text{ bytes}, \qquad b_p = 2 \ (\text{BF16}) \ \text{or}\ 0.516 \ (\text{NF4}) \tag{189.7} \]
Read the symbols
\(N_p\)
the number of frozen base weights in part \(p\) (also the frozen base of a LoRA part)
\(b_p\)
bytes per frozen weight: 2 in BF16, 0.516 in NF4 with double quantization
\(N_{\text{train}}\)
all trainable weights: LoRA factors plus fully trained parts, 16 bytes each with mixed-precision AdamW

What to quantize in a VLM, and what not

  • The LLM’s linear layers: yes. They hold 96.1% of LLaVA-1.5’s LLM weights and 85.7% of Qwen2-VL’s (Qwen2-VL has a large vocabulary: its input and output embeddings are 1.09B weights). This is where the savings are.
  • Embeddings and lm_head: no. They are not nn.Linear layers inside blocks, the output layer is sensitive, and in Hugging Face it is skipped by default. [Likely]
  • The connector: no. It is small and you usually train it fully, and a fully trained part cannot be stored in 4 bits. LLaVA’s QLoRA path passes llm_int8_skip_modules=["mm_projector"] to the quantization config for exactly this reason [Certain, from the code].
  • The vision tower: usually no. Quantizing it saves little: \(303.5\text{M} \times (2 - 0.516) = 0.45\) GB for LLaVA-1.5, 0.94 GB for Qwen2-VL. And errors in the image features reach every visual token the LLM reads. Most QLoRA VLM recipes keep it in BF16. [Likely]
Worked example: LLaVA-1.5-7B, three ways

Full SFT (vision frozen, connector and LLM full): frozen \(2 \times 303.5\text{M} = 0.61\) GB, trainable \(16 \times 6{,}759.9\text{M} = 108.16\) GB. Total 108.77 GB.

LoRA (rank 128 on all 7 linear layers of 32 blocks, connector full): LoRA adds \(128 \times 32 \times (4 \cdot 8{,}192 + 3 \cdot 15{,}104) = 319.8\)M weights; with the connector, \(N_{\text{train}} = 340.8\)M (4.82% of 7.06B). Memory: \(2 \times 7{,}042.4\text{M} = 14.08\) GB frozen + \(16 \times 340.8\text{M} = 5.45\) GB. Total 19.54 GB.

QLoRA (the same, with the LLM’s 6,476.0M linear weights in NF4): those weights drop from 12.95 GB to 3.34 GB, saving 9.61 GB. Total 9.93 GB. With a smaller rank (8) and the connector frozen as well, Qwen2-VL-7B needs 7.22 GB.

Activations come on top, and in a VLM they grow with the number of image tokens: 576 per image for LLaVA-1.5 at 336 px, and many more for high-resolution or video inputs. Gradient checkpointing (session 39) is almost always on.

Two practical warnings. First, merging: after QLoRA training, do not add \(\frac{\alpha}{r}BA\) into the 4-bit weights; the sum would be rounded back to the 4-bit grid and much of the update lost. Load the base model in BF16, merge there, and quantize again for serving if you need to. [Likely] Second, speed: dequantizing on every forward pass makes QLoRA steps slower than BF16 LoRA steps. Use it when memory, not time, is the limit. [Likely]

Keeping text-only skills while learning vision tasks

A VLM is used as a chat model too: users ask it plain questions, ask for code, or keep talking after the image is forgotten. If SFT trains only on image data, the LLM’s weights drift toward that data and its text skills drop. This is catastrophic forgetting (session 61). It also happens inside vision: Zhai et al. treated several fine-tuned VLMs as image classifiers and found that “almost all evaluated MLLMs fail to retain the same performance levels as their vision encoders” [Certain]. Fine-tuning LLaVA further on one image dataset helped other image datasets at first, then the model began to hallucinate and generalize worse, even with the image encoder frozen [Certain].

First, measure it. Before training, score the base model on a small text suite that matches your users (for example MMLU for knowledge, GSM8K for math, IFEval for instruction following, plus your own prompts). After training, the forgetting on that suite is

\[ F = \text{score}_{\text{text}}(\theta_0) - \text{score}_{\text{text}}(\theta_{\text{SFT}}) \tag{189.8} \]

where \(\theta_0\) are the weights before SFT. Track \(F\) during training, not only at the end; it often grows with the number of steps.

Then use one or more of five remedies.

  1. Mix text-only data into SFT. With a share \(\lambda\) of text samples, the loss becomes
    \[ \mathcal{L} = (1 - \lambda)\, \mathcal{L}_{\text{SFT}}^{\text{image}} + \lambda\, \mathcal{L}_{\text{SFT}}^{\text{text}} \tag{189.9} \]
    LLaVA-1.5 includes about 40K ShareGPT text conversations in its 665K mix (\(\lambda \approx 6.0\%\)), and the authors report that the model learned from them to answer in the user’s language, a behavior that “is transferred to visual conversations” [Certain]. Qwen2-VL’s SFT data “encompasses not only pure text-based dialogue data but also multimodal conversational data”, and Idefics2 adds text-only instruction datasets [Certain]. Good text samples are ones close to what the base model was tuned on.
  2. Change fewer weights. Freeze the LLM or use LoRA on it. LoRA constrains the update to rank \(r\), and “LoRA Learns Less and Forgets Less” found it keeps more of the base model’s skills outside the target domain than full fine-tuning [Certain]. Figure 3 shows the same trade-off in a toy.
  3. Move gently. A smaller learning rate, one epoch instead of three, and early stopping on the text suite.
  4. Interpolate back. Mix the fine-tuned weights with the original ones, as in WiSE-FT:
    \[ \theta(\beta) = (1 - \beta)\, \theta_0 + \beta\, \theta_{\text{SFT}}, \qquad \text{for LoRA: } W(\beta) = W_0 + \beta\, \frac{\alpha}{r} B A, \qquad 0 \le \beta \le 1 \tag{189.10} \]
    Sweep \(\beta\) and pick the value with the best combined score. For LoRA this is free: scale the adapter at merge time.
  5. Keep the chat format. Use the base model’s own chat template and system prompt. Many “forgetting” reports are really a format mismatch: the model was trained on short benchmark answers (“Answer with a single word”) and now answers chat questions in one word. LLaVA-1.5 attaches an explicit format prompt to each short-answer dataset for this reason [Certain], and Idefics2 adds a short “chatty” stage on long dialogues because its answers had become too short [Certain].
image-task losstext loss (forgetting)one method, at the chosen step
Figure 3. A toy of forgetting, trained live in your browser. The “LLM” is one 8 × 8 linear layer that starts at the pretrained weights and is perfect on text. Training changes it by \(E\). Text inputs and image-question inputs use partly the same directions, so the text loss is \(\sum_{ij} E_{ij}^2 a_j\) and the image-task loss is \(\sum_{ij} (E_{ij} - \Delta_{ij})^2 b_j\), where \(\Delta\) is the seeded change the image task needs (a strong rank-1 part, a weaker rank-1 part and a small full-rank part). 300 steps of gradient descent with learning rate 0.05 on formula (189.9); LoRA uses \(\alpha = 2\) and the same seeded \(A\).
How to read Figure 3

With full fine-tuning and no text data, the image-task loss falls to almost 0 (0.001 at step 300), and the text loss climbs to 2.390: the model learned the new task and paid with its old one. Move the text share to 10%: the text loss falls to 1.494 (37% less forgetting), while the image-task loss is still only 0.041. Now try LoRA \(r = 1\): it can only learn the strongest direction of the change, so the image loss stays at 1.478 (29% of the start) but the text loss is only 0.537. Higher ranks move toward full fine-tuning: they learn more and forget more. Drag the step slider back to 30: full fine-tuning already reached 0.083 on the image task with half the forgetting (1.254), so early stopping is also a remedy. The early bump of the LoRA text loss comes from the random start of \(A\).

The toy is linear, so do not read its numbers as predictions for a real VLM. It shows the shape of the trade-off: every remedy moves you along a curve between “learned the new task” and “kept the old one”, and mixing text data moves the whole curve down. [Likely]

Putting it together: an SFT step with a freeze map

Algorithm 2: one VLM SFT training step
  1. input: batch of (images, conversations); model with freeze map; optimizer from Algorithm 1; schedule \(s(t)\)
  2. build token ids with the model’s chat template; insert image placeholders
  3. labels \(\leftarrow\) ids; set labels to \(-100\) on image tokens, user turns, system prompt and padding // \(m_t\) of formula (189.1)
  4. visual tokens \(\leftarrow\) connector(vision tower(images)) // vision tower under no_grad if frozen
  5. logits \(\leftarrow\) LLM([visual tokens ; text embeddings])
  6. loss \(\leftarrow\) cross-entropy(logits shifted by one, labels) // formula (189.1)
  7. loss.backward(); clip the gradient norm (per part, if you follow Molmo)
  8. set each group’s rate to \(\eta_g \cdot s(t)\); optimizer step; zero gradients // formula (189.3)
  9. every few hundred steps: evaluate the image task and the text suite // forgetting \(F\), formula (189.8)

The PyTorch version below builds a tiny VLM (a 2-block vision tower, an MLP connector, a 2-block LLM), freezes the vision tower, trains the connector fully, puts LoRA on all linear layers of the LLM blocks, gives the two groups different learning rates, masks the loss to the answer, and checks that merging the adapters does not change the outputs. It runs on a CPU in a few seconds and prints the same numbers every time (PyTorch 2.x).

import math, re, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

class LoRALinear(nn.Module):
    """h = W0 x + (alpha/r) B A x, with W0 frozen (formula 189.4)."""
    def __init__(self, base: nn.Linear, r=4, alpha=8):
        super().__init__()
        self.base, self.s = base, alpha / r
        base.weight.requires_grad_(False)
        if base.bias is not None: base.bias.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(r, base.in_features) / math.sqrt(base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))     # B = 0: the model starts unchanged
    def forward(self, x):
        return self.base(x) + self.s * (x @ self.A.T) @ self.B.T
    def merge(self):                                                 # fold the adapter into W0 for serving
        self.base.weight.data += self.s * self.B @ self.A
        return self.base

class Block(nn.Module):                                              # a tiny pre-norm transformer block
    def __init__(self, d):
        super().__init__()
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.q_proj, self.k_proj, self.v_proj, self.o_proj = (nn.Linear(d, d) for _ in range(4))
        self.up_proj, self.down_proj = nn.Linear(d, 4 * d), nn.Linear(4 * d, d)
    def forward(self, x, causal):
        h = self.n1(x); B_, T, d = h.shape
        q, k, v = self.q_proj(h), self.k_proj(h), self.v_proj(h)
        a = F.scaled_dot_product_attention(q, k, v, is_causal=causal)
        x = x + self.o_proj(a)
        return x + self.down_proj(F.gelu(self.up_proj(self.n2(x))))

class TinyVLM(nn.Module):
    def __init__(self, dv=32, dl=64, vocab=100, nv=2, nl=2):
        super().__init__()
        self.vision_tower = nn.ModuleList(Block(dv) for _ in range(nv))
        self.patch = nn.Linear(48, dv)                                    # 4x4x3 pixel patches
        self.connector = nn.Sequential(nn.Linear(dv, dl), nn.GELU(), nn.Linear(dl, dl))
        self.embed = nn.Embedding(vocab, dl)
        self.llm = nn.ModuleList(Block(dl) for _ in range(nl))
        self.lm_head = nn.Linear(dl, vocab, bias=False)
    def forward(self, patches, ids):
        v = self.patch(patches)
        for blk in self.vision_tower: v = blk(v, causal=False)
        x = torch.cat([self.connector(v), self.embed(ids)], dim=1)       # [image tokens | text tokens]
        for blk in self.llm: x = blk(x, causal=True)
        return self.lm_head(x)

model = TinyVLM()
n_all = sum(p.numel() for p in model.parameters())

# 1) freeze everything, then choose per part (the "freeze map")
model.requires_grad_(False)
for p in model.connector.parameters(): p.requires_grad_(True)       # connector: full training
targets = re.compile(r"llm\.\d+\.(q_proj|k_proj|v_proj|o_proj|up_proj|down_proj)")
for name, mod in list(model.named_modules()):                       # LLM: LoRA on all linear layers
    if targets.fullmatch(name):
        parent, child = name.rsplit(".", 1)
        setattr(model.get_submodule(parent), child, LoRALinear(mod, r=4, alpha=8))
n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"trainable {n_train} of {n_all} ({100 * n_train / n_all:.2f}%)")

# 2) one learning rate per part (formula 189.3): LoRA factors high, the full connector lower
lora = [p for n, p in model.named_parameters() if p.requires_grad and n.endswith((".A", ".B"))]
conn = list(model.connector.parameters())
opt = torch.optim.AdamW([{"params": lora, "lr": 2e-3}, {"params": conn, "lr": 2e-4}], weight_decay=0.0)

# 3) SFT batch: 16 image tokens, a prompt, then the answer; loss only on the answer (formula 189.1)
patches = torch.randn(8, 16, 48)
ids = torch.randint(0, 100, (8, 12))
labels = torch.full((8, 16 + 12), -100)                            # -100 = ignored by cross_entropy
labels[:, 16 + 6:] = ids[:, 6:]                                     # the last 6 text tokens are the answer
model.eval()                                                       # no dropout here; keeps the check exact
for step in range(30):
    logits = model(patches, ids)
    loss = F.cross_entropy(logits[:, :-1].reshape(-1, 100), labels[:, 1:].reshape(-1))  # shift by one
    opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
    if step in (0, 29): print(f"step {step:2d}  answer loss {loss.item():.3f}")

# 4) no gradient ever reached the frozen vision tower
print("vision grads:", all(p.grad is None for p in model.vision_tower.parameters()))

# 5) merge the adapters and check the outputs do not change
with torch.no_grad():
    before = model(patches, ids)
    for name, mod in list(model.named_modules()):
        if isinstance(mod, LoRALinear):
            parent, child = name.rsplit(".", 1)
            setattr(model.get_submodule(parent), child, mod.merge())
    after = model(patches, ids)
print("max change after merge:", f"{(before - after).abs().max().item():.1e}")

Output: trainable 15488 of 146016 (10.61%), the answer loss goes from 4.885 at step 0 to 2.890 at step 29, vision grads: True, and the merge changes the outputs by about \(10^{-6}\) (float rounding). Check the count by hand: the connector has \(32 \cdot 64 + 64 + 64 \cdot 64 + 64 = 6{,}272\) weights; LoRA with \(r = 4\) adds \(4 \cdot (4 \cdot 128 + 2 \cdot 320) = 4{,}608\) per block, 9,216 for two blocks; the total is 15,488.

With Hugging Face libraries the same freeze map is a configuration. The sketch below is not run here (it needs a CUDA GPU, bitsandbytes, peft and the model download); the module names are those of the Hugging Face LLaVA implementation and differ between models, so print model.named_modules() first.

from transformers import LlavaForConditionalGeneration, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,
                         llm_int8_skip_modules=["vision_tower", "multi_modal_projector", "lm_head"])
model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf", quantization_config=bnb)
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)

cfg = LoraConfig(r=128, lora_alpha=256, lora_dropout=0.05,
                 # a regex: LoRA only inside the language model, on all 7 projections
                 target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)",
                 modules_to_save=["multi_modal_projector"])       # connector: full training, saved with the adapter
model = get_peft_model(model, cfg)
model.print_trainable_parameters()

Where this is used

ApplicationTypical freeze mapWhy
General visual assistant (stage 3 of building a VLM)Vision frozen or slow; connector and LLM full; text-only data in the mixLarge, varied data justifies full training; text data protects chat skills (LLaVA-1.5, Qwen2-VL)
Domain specialist: documents, charts, receiptsLoRA on the LLM, connector full; vision LoRA if resolution changesMostly new answer formats and words; fine text needs higher resolution, which the encoder must adapt to
Medical, satellite, thermal or industrial imagesLoRA on the vision tower and the LLM, connector fullThe pixels are unlike web photos, so the encoder itself must change, but gently
Video classification, action recognition and key-moment detection with a video LLMVision frozen (features can be cached per frame), connector full, LoRA on the LLMMany frames per sample make the vision backward pass very expensive; a frozen encoder lets you precompute features once
Video-RAG and video retrieval answersLoRA on the LLM onlyThe model must learn to read retrieved frames and cite timestamps; the visual features are already good
Fine-tuning on one consumer GPUQLoRA on the LLM, everything else frozen or full in BF16A 7B VLM fits in about 7 to 10 GB before activations (Figure 1)
Many customers or tasks on one serverOne frozen base model, one LoRA adapter per taskAdapters are a few hundred MB and can be swapped or batched together (session 57)

Common mistakes

  • Training on the prompt and the image tokens. Forgetting the loss mask (labels \(-100\)) makes the model learn to predict user questions and spends gradient on placeholders. Print one sample with its labels before every new run.
  • LoRA landing in the wrong part. A target list like ["q_proj", "v_proj"] matches by name suffix, so it may also hit the vision tower’s attention layers if they use the same names. Use a regex or exclude_modules, and print the trainable count per part.
  • The connector is silently frozen. With PEFT, everything not adapted is frozen, including the connector you meant to train. Put it in modules_to_save and check requires_grad.
  • One learning rate for everything. \(2\times10^{-4}\) is fine for LoRA factors and harmful for full weights of the LLM or the vision tower. Use parameter groups.
  • Quantizing the connector or the vision tower in QLoRA. The savings are small (under 1 GB for 7B models) and the quality cost lands on every image token. Skip them in the quantization config.
  • Measuring only the image benchmark. Without a text suite you will not see forgetting until users report it. Evaluate both, from step 0.
  • Changing the chat template. Fine-tuning with a different template than the base model used turns a good chat model into a confused one. Use the processor’s own template.
  • Merging a QLoRA adapter into 4-bit weights. Merge into a BF16 copy of the base, then quantize again if needed.

Interview questions

Which parts of a VLM do you train in SFT, and why?basic
It depends on data size and how new the images are. The common default is: vision tower frozen (it was trained on billions of images, and next-token loss on a few hundred thousand samples can only make it less general), connector fully trained (small and closest to the change), LLM fully trained or with LoRA. Qwen2-VL locks the ViT in SFT; LLaVA-1.5 freezes it and trains the connector and LLM. Unfreeze the vision tower (preferably with LoRA or a 5 to 10 times smaller learning rate) when the pixels are unusual or the resolution changes, as Idefics2 does.
Why use different learning rates for different parts? Give real numbers.core
Parts differ in how much they already know and how many weights they have. A well-trained vision tower should move slowly to keep general features; a new or small connector can move fast; LoRA factors start at zero and live in a small subspace, so they need larger steps. Examples: LLaVA-1.5 LoRA uses \(2\times10^{-4}\) for LoRA and \(2\times10^{-5}\) for the connector; LLaVA-OneVision uses \(2\times10^{-6}\) for the vision tower and \(1\times10^{-5}\) for the rest; Molmo pretraining uses \(6\times10^{-6}\) (ViT), \(2\times10^{-4}\) (connector), \(2\times10^{-5}\) (LLM). In PyTorch these are AdamW parameter groups.
How much memory does SFT of LLaVA-1.5-7B need with full fine-tuning, LoRA and QLoRA?core
Use frozen weights × bytes + 16 bytes per trainable weight. Full (vision frozen): \(2 \times 0.30\text{B} + 16 \times 6.76\text{B} = 108.8\) GB, so it must be sharded. LoRA rank 128 on all LLM linear layers with a full connector: 340.8M trainable, 19.5 GB. QLoRA (LLM linears in NF4 at 0.516 bytes): 9.9 GB. All before activations, which grow with the image tokens (576 per image for LLaVA-1.5) and are reduced with gradient checkpointing.
Where would you put LoRA in a VLM?core
On all linear layers of the LLM blocks (attention and MLP, not lm_head); the QLoRA paper found that adapting all linear layers is needed to match full fine-tuning. The connector is small, so train it fully instead (modules_to_save). Add LoRA to the vision tower only when the image domain or resolution is new, accepting that the backward pass then runs through the encoder, which costs activation memory and compute for every image token.
If the LLM is frozen and only the connector is trained, is the LLM's backward pass free?advanced
No. The gradient must flow from the loss through all LLM layers down to the connector, so the LLM still computes input gradients (about \(2N\) FLOPs per token extra) and must keep its activations. You save only the weight gradients (another \(2N\)) and all optimizer state. Per token the LLM costs about \(4N\) instead of \(6N\). Parts below the lowest trainable part, such as a frozen vision tower under a trainable connector, need only the forward pass and can run under no_grad.
What do you quantize and what do you keep in 16 bits when you QLoRA a VLM?core
Quantize the LLM’s linear layers to NF4: they hold most of the weights (96% of LLaVA-1.5’s LLM). Keep in BF16: the connector (small and usually fully trained, LLaVA passes it in llm_int8_skip_modules), the vision tower (small savings, about 0.45 GB for CLIP ViT-L, and errors would reach every visual token), the embeddings and lm_head, and the LoRA adapters. Compute runs in BF16 after dequantizing each block.
Your fine-tuned VLM is great on your image task but worse at plain chat. What happened, and what do you do?core
Catastrophic forgetting, or a format shift. First check the format: short-answer training data (“answer in one word”) teaches short answers everywhere; add format prompts and long dialogue data, as LLaVA-1.5 and Idefics2 did. Then measure a text suite against the base model and reduce forgetting: mix text-only SFT data in (LLaVA-1.5 used about 6%), use LoRA or freeze the LLM, lower the learning rate and train fewer steps, or interpolate the weights back toward the base (WiSE-FT, or scale the LoRA adapter by \(\beta \lt 1\)) and pick the best combined score.
Does LoRA lose quality compared with full fine-tuning for VLM instruction tuning?core
Often not. LLaVA-1.5-7B with LoRA (rank 128, lr \(2\times10^{-4}\)) scored higher than full fine-tuning on 8 of 11 benchmarks in the released model zoo, mean +0.6 points; for 13B the mean was +0.2 points. Instruction data mostly teaches format and style, which fits in a low-rank update. When the data must teach a lot of new knowledge, full fine-tuning learns more (“LoRA Learns Less and Forgets Less”), so raise the rank or train fully in that case.
Why does QLoRA not lose the 4-bit rounding of the update, and why must you be careful when merging?advanced
During training, the update lives in separate BF16 adapter matrices \(A, B\); only the frozen base is 4-bit, and it is dequantized for each matmul. The update is never rounded. At merge time, adding \(\frac{\alpha}{r}BA\) into the 4-bit weights would round the sum to the NF4 grid, and the small update could mostly vanish. Merge into a BF16 copy of the base instead, then quantize the merged model again (for example with GPTQ or AWQ) if you serve in 4 bits.
You fine-tune a video LLM for action recognition with 32 frames per clip. How do you set the freeze map?advanced
Freeze the vision tower: with 32 frames, a trainable encoder would need a backward pass and stored activations for every frame’s patches, which dominates cost. A frozen encoder also lets you precompute and cache the frame features once and reuse them every epoch. Train the connector fully (it decides how frames are pooled into tokens) and put LoRA on the LLM so it learns the label vocabulary and answer format. Unfreeze the encoder with LoRA only if the frames are far from its training data, such as thermal or surgical video.

Exercises

  1. LLaVA-1.5-13B has an LLM with 40 blocks, hidden size 5,120 and MLP size 13,824. How many weights does LoRA rank 128 on all 7 linear layers add?
  2. Using formula (189.2), compare the LLM compute per token for (a) connector-only training, (b) LoRA on the LLM, (c) full LLM training.
  3. A 7B VLM’s LLM has 6.5B linear weights. How much memory does NF4 with double quantization save over BF16 for these weights?
  4. In Figure 3, set full fine-tuning and find the text share \(\lambda\) where the text loss at step 300 first drops below 1.0. What is the image-task loss there?
  5. Your SFT mix has 500K image samples. How many text-only samples do you add for a text share of 6%?
  6. Write the PyTorch parameter groups for: vision tower LoRA at \(5\times10^{-5}\), connector full at \(1\times10^{-5}\), LLM LoRA at \(2\times10^{-4}\), no weight decay on biases and norms.
Answerscheck
1. Per block: \(128 \times (4 \times (5{,}120 + 5{,}120) + 3 \times (5{,}120 + 13{,}824)) = 128 \times 97{,}792 = 12{,}517{,}376\); times 40 blocks: 500,695,040, about 500.7M. 2. (a) The backward pass goes through the LLM for input gradients only: about \(4N\). (b) The same \(4N\), plus a small LoRA cost. (c) \(6N\). Freezing saves weight gradients and optimizer states, not the backward pass. 3. \(6.5\text{B} \times (2 - 0.516) = 9.65\) GB (from 13.0 GB to 3.35 GB). 4. At \(\lambda = 20\%\) the text loss is 1.108 (still above 1). At \(\lambda = 25\%\) it is 0.978, with an image-task loss of 0.145 (2.8% of the start). Compare \(\lambda = 0\): 2.390 and 0.001. 5. \(0.06 = T / (500{,}000 + T)\), so \(T = 30{,}000 / 0.94 \approx 31{,}915\), about 32K. 6. Collect parameters by name (for example names containing vision_tower and lora_), and for each part split into p.ndim >= 2 (decay) and p.ndim < 2 (no decay), giving up to six groups, each with its own lr and weight_decay; then pass the list to torch.optim.AdamW.

Key takeaways

  • VLM SFT is next-token loss on the answer tokens only (189.1). The new decision is the freeze map: frozen, LoRA or full for the vision tower, the connector and the LLM.
  • The backward pass must reach the lowest trainable part (189.2). Frozen parts above it still pay for input gradients and activations; frozen parts below it pay only the forward pass.
  • Use parameter groups (189.3): slow for the vision tower, faster for the connector, about 10 times faster for LoRA factors than for full weights (LLaVA-1.5: \(2\times10^{-4}\) vs \(2\times10^{-5}\)).
  • Put LoRA (189.4, 189.5) on all linear layers of the LLM, train the connector fully, and adapt the vision tower only when the images are new. LLaVA-1.5 LoRA matched full fine-tuning on its benchmarks.
  • QLoRA stores the LLM’s linear layers at 4.127 bits (189.6); with (189.7), LLaVA-1.5-7B goes from 108.8 GB (full) to 19.5 GB (LoRA) to 9.9 GB (QLoRA). Keep the vision tower, connector, embeddings and lm_head in BF16.
  • Measure forgetting on a text suite (189.8). Fix it with text data in the mix (189.9), fewer changed weights, gentler training, weight interpolation (189.10), and the base model’s chat format.

Key formulas

\[ \mathcal{L}_{\text{SFT}} = -\frac{\sum_t m_t \log p_\theta(x_t \mid I, x_{\lt t})}{\sum_t m_t}, \qquad \mathbf{h} = W_0\mathbf{x} + \frac{\alpha}{r} B A \mathbf{x}, \qquad N_{\text{LoRA}} = r\sum_\ell \big(d_{\text{in}}^{(\ell)} + d_{\text{out}}^{(\ell)}\big) \]
\[ M \approx \sum_{p} b_p N_p + 16\, N_{\text{train}}, \quad b_{\text{NF4}} = 4 + \tfrac{8}{64} + \tfrac{32}{64\cdot 256} = 4.127 \text{ bits}, \quad \mathcal{L} = (1-\lambda)\mathcal{L}^{\text{image}} + \lambda \mathcal{L}^{\text{text}}, \quad \theta(\beta) = (1-\beta)\theta_0 + \beta\theta_{\text{SFT}} \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
  2. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  3. Hugging Face (code). PEFT library
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
  5. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  6. Hugging Face (code). PEFT library
  7. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  8. Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation
  9. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  10. Laurençon et al. (2024). What matters when building vision-language models?
  11. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  12. Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
  13. Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer
  14. Biderman et al. (2024). LoRA Learns Less and Forgets Less
  15. Zhai, Tong, Li, Cai, Qu, Lee & Ma (2023). Investigating the Catastrophic Forgetting in Multimodal Large Language Models
  16. Wortsman et al. (2022). Robust fine-tuning of zero-shot models
  17. Liu et al. (2023). LLaVA code base: v1.5 fine-tuning scripts (finetune.sh, finetune_lora.sh) and Model Zoo