VLM SFT and PEFT
Supervised fine-tuning (SFT) turns a pretrained vision-language model into an assistant, or into a specialist for your own images. For each of its three parts (vision encoder, connector, LLM) you decide: freeze it, train a small LoRA adapter, or train it fully, and with which learning rate. This session shows how to make those choices with real parameter counts and memory numbers, how QLoRA fits a 7B VLM on one small GPU, and how to keep the text skills the model already had.
What you will learn
- What VLM SFT optimizes: next-token loss on the answer tokens only, with image tokens and prompts masked out
- The freeze map: frozen, LoRA or full for the vision tower, the connector and the LLM, and what each choice costs in memory and compute
- Different learning rates per part with AdamW parameter groups, with the real LLaVA-1.5, LLaVA-OneVision and Molmo values
- Where to put LoRA (LLM, connector, vision tower), how many weights each target adds, and what LLaVA-1.5 LoRA scored against full fine-tuning
- QLoRA for VLMs: NF4 at 4.127 bits per weight, what to keep in 16 bits (vision tower, connector, embeddings, lm_head), and a 7B VLM in under 10 GB before activations
- How to keep text-only skills: measure them, mix text data in, prefer LoRA or a smaller learning rate, stop earlier, or interpolate back toward the base
- A runnable PyTorch freeze map with LoRA, per-part learning rates, answer-only loss and a merge check
Why this matters
After pretraining, a vision-language model (VLM) can describe images, but it does not yet follow instructions well, and it knows nothing about your images: your product photos, your medical scans, your video frames. Supervised fine-tuning (SFT) fixes both. You show the model many examples of “image + question → good answer” and train it to produce the answer.
In session 186 you saw that SFT is the third stage of building a VLM. In session 188 you saw how to pack samples and put the loss only on the answer. This session is about the decisions you make when you run SFT, either as the last stage of building a VLM or as a fine-tune of an open model such as LLaVA-1.5 or Qwen2-VL:
- What to train. A VLM has three parts: the vision tower (the vision encoder, for example a ViT), the connector (a small network that maps image features into the LLM’s input space) and the LLM. Each one can be frozen, adapted with a small add-on (PEFT, parameter-efficient fine-tuning, such as LoRA), or trained fully.
- How fast each part moves. The parts were trained on very different data, so they need different learning rates.
- How to fit it on your GPU. Full fine-tuning of a 7B VLM needs more than 100 GB before activations. LoRA and QLoRA bring this below 20 GB and 10 GB.
- How not to break what already works. A VLM is also a chat model. If SFT sees only images, the model can forget text skills (and even some visual skills it had).
The real recipes we will use: LLaVA-1.5 (CLIP ViT-L/14-336, a 2-layer MLP connector, Vicuna-7B or 13B), which ships both a full fine-tuning script and a LoRA script with published scores; Qwen2-VL, which freezes its vision encoder during SFT; and Idefics2, which trains its pretrained parts with LoRA and DoRA. [Certain]
What SFT optimizes in a VLM
One SFT sample is a short conversation with one or more images. The chat template turns it into one token sequence: image tokens (the connector outputs), then the user’s text, then the assistant’s answer. The model is trained with the usual next-token loss, but only on the answer tokens. The image tokens and the user’s words are context, not targets:
- Read the symbols
- \(\theta\)
- all weights of the VLM: vision tower, connector and LLM
- \(I\)
- the image (or images, or video frames) of the sample
- \(x_1, \dots, x_T\)
- the text tokens of the conversation after the chat template
- \(m_t\)
- the loss mask: 1 if token \(x_t\) belongs to an assistant answer, 0 for prompts, image placeholders, padding and template tokens
- \(p_\theta(x_t \mid I, x_{\lt t})\)
- the probability the model gives to the correct next token
In code, the mask is the label value -100, which PyTorch’s cross-entropy ignores. Idefics2 states the same rule: “we only compute the loss on the tokens of the answers in the Q/A pairs”. [Certain] Formula (189.1) is the same loss as text SFT (session 54). What is new in a VLM is the question which part of \(\theta\) the gradient is allowed to change.
The freeze map: three parts, three choices
For each part you choose one of three modes. We call the result the freeze map.
- Frozen: the weights never change. The part needs memory only for its weights (2 bytes each in BF16), and no weight gradients are computed for it.
- LoRA: the original weights stay frozen, and small low-rank matrices are added to some of its linear layers and trained (session 57).
- Full: every weight is trained. With mixed-precision AdamW this costs 16 bytes per weight: BF16 weight 2, BF16 gradient 2, FP32 master copy 4, Adam moments \(m\) and \(v\) 4 + 4 (session 4).
What do real recipes do in SFT? LLaVA-1.5 freezes the vision tower and trains the connector and the LLM fully (or the LLM with LoRA) [Certain, from the training code]. Qwen2-VL: “we lock the ViT parameters and perform exclusive fine-tuning of the LLM” [Certain]. LLaVA-OneVision trains everything, with a smaller learning rate for the vision tower [Certain]. Idefics2 trains the new parts fully and the pretrained vision encoder and LLM with LoRA (DoRA in its instruction tuning) [Certain].
Compute: where the backward pass must go
Freezing saves more than memory. In a VLM the order is fixed: vision tower, then connector, then LLM, then the loss. The backward pass starts at the loss and must travel back to the lowest trainable part. Parts below it need only a forward pass. Parts above it need the gradient with respect to their inputs (to pass it down), and only trained weights need gradients with respect to the weights. For a part with \(N_p\) weights, each pass costs about \(2N_p\) FLOPs per token (one multiply and one add per weight):
The brackets are 1 if the condition is true and 0 otherwise. A LoRA part has a frozen base, so it pays \(4N_p\) (plus a tiny amount for the adapters); a fully trained LLM pays \(6N_p\), the familiar training rule. A frozen vision tower below a trainable connector pays only \(2N_p\) per image token and stores no activations for the backward pass. That is why LLaVA runs the vision tower under torch.no_grad() [Certain, from the code].
Figure 1 puts the three choices together for two real models. The parameter counts are computed from the models’ config.json files on Hugging Face and match the size of the released weights exactly: LLaVA-1.5-7B has 7,063,427,072 weights (vision tower 303.5M, connector 21.0M, LLM 6,738.9M) and Qwen2-VL-7B has 8,291,375,616 (vision tower 631.2M, merger 44.6M, LLM 7,615.6M). [Certain]
How to read Figure 1
Start with “LLaVA-1.5 full SFT”: 6.76B trainable weights and 108.77 GB, more than one 80 GB GPU, which is why the official script uses DeepSpeed ZeRO-3 to shard the states over 8 GPUs. Click “LLaVA-1.5 LoRA” (the official LoRA recipe: rank 128 on all linear layers of the LLM, full connector): 340.8M trainable weights (4.82%), 19.54 GB. Turn on 4-bit: 9.93 GB, because the 6.48B linear weights of the LLM drop from 2 bytes to 0.516 bytes. Now change the rank: memory grows slowly, from 14.74 GB at rank 8 to 19.54 GB at rank 128, because the frozen base dominates. Finally unfreeze the vision tower fully: the backward pass now reaches the image, and 303.5M more weights cost 16 bytes each.
Different learning rates for different parts
When several parts train together, they rarely share one learning rate. The rule of thumb: the more a part already knows, and the broader the data it learned from, the more gently it should move. New or small parts can move fast. In AdamW this is done with parameter groups: each group has its own learning rate \(\eta_g\) and its own weight decay \(\lambda_g\), and the same update rule:
- Read the symbols
- \(\theta_g\)
- the trainable weights of group \(g\)
- \(\eta_g\)
- the peak learning rate of the group
- \(s(t)\)
- the shared schedule, a number between 0 and 1 (warmup, then cosine decay)
- \(\hat{\mathbf{m}}_g, \hat{\mathbf{v}}_g\)
- Adam’s bias-corrected moment estimates for those weights
- \(\lambda_g\)
- the weight decay of the group (often 0 for LoRA, norms and biases)
Real values, all with a cosine schedule:
| Recipe | Vision tower | Connector | LLM |
|---|---|---|---|
| LLaVA-1.5 full SFT | frozen | \(2\times10^{-5}\) | \(2\times10^{-5}\) (full) |
| LLaVA-1.5 LoRA SFT (\(r = 128\), \(\alpha = 256\)) | frozen | \(2\times10^{-5}\) (full) | \(2\times10^{-4}\) (LoRA factors) |
| LLaVA-OneVision (later stages) | \(2\times10^{-6}\) | \(1\times10^{-5}\) | \(1\times10^{-5}\) |
| Molmo (pretraining) | \(6\times10^{-6}\) | \(2\times10^{-4}\) | \(2\times10^{-5}\) |
The LLaVA-1.5 rows come from its finetune.sh and finetune_lora.sh scripts (1 epoch, warmup 3%, weight decay 0) [Certain]. Molmo also uses a 200-step warmup for the connector against 2,000 for the ViT and the LLM, and clips gradients separately for each part [Certain]. The OneVision values are from its paper [Certain, as reported in session 186].
Notice the LLaVA LoRA row. The LoRA factors get a learning rate 10 times larger than the full-training rate, while the connector keeps the small one. This is typical: a LoRA update starts at zero (\(B = 0\)) and lives in a small subspace, so it needs bigger steps to move as far; the connector is a full weight matrix and keeps the full-training rate. The script sets it with a separate flag, --mm_projector_lr 2e-5, which creates a separate parameter group in the optimizer [Certain, from the code].
A softer version of the same idea, inside the vision tower, is layer-wise learning-rate decay: block \(l\) of \(L\) gets \(\eta \cdot \gamma^{L - l}\) with \(\gamma\) around 0.8 to 0.9, so the early blocks (general edges and textures) move least. It comes from fine-tuning ViTs for classification (BEiT, MAE) and is used in some VLM recipes. [Likely]
- input: VLM with parts vision, connector, LLM; a mode per part (frozen, LoRA, full); learning rates \(\eta_g\)
- set
requires_grad = Falsefor every weight // start from "all frozen" - for each part \(p\) with mode full: set
requires_grad = Truefor its weights; put them in group \(p\) - for each part \(p\) with mode LoRA: wrap the chosen linear layers, \(W_0 x \to W_0 x + \tfrac{\alpha}{r} B A x\); put \(A, B\) in group "LoRA of \(p\)"
- split every group into weights that get decay (matrices) and weights that do not (biases, norms)
- build AdamW with one entry per group: \(\{\text{params}, \eta_g, \lambda_g\}\) // formula (189.3)
- if the lowest trainable part is above the vision tower: run the vision tower under
no_grad// formula (189.2) - print the trainable count per part, and check it against your plan before you start a run
LoRA in a VLM: on the LLM, the connector or the vision tower
LoRA replaces the output of a frozen linear layer \(W_0 \in \R^{d_{\text{out}} \times d_{\text{in}}}\) by
- Read the symbols
- \(r\)
- the rank, much smaller than \(d_{\text{in}}\) and \(d_{\text{out}}\) (common values 8 to 128)
- \(\alpha\)
- a scale; the update is multiplied by \(\alpha / r\) (rsLoRA uses \(\alpha / \sqrt{r}\))
- \(A, B\)
- the trainable factors; \(A\) starts random, \(B\) starts at zero, so at step 0 the model is unchanged
Each adapted layer adds \(r(d_{\text{in}} + d_{\text{out}})\) trainable weights. For a whole part, sum over the adapted layers:
Where should the adapters go? Each of the three parts gives a different answer.
LoRA on the LLM (the default)
Most of the weights are here, and most of the new behavior (answer format, task skill, domain words) is language behavior. LLaVA-1.5 adds LoRA to every linear layer of the LLM except lm_head, and its code explicitly skips any module whose name contains mm_projector or vision_tower [Certain, from the code]. The QLoRA paper found that adapting all linear layers of each transformer block is needed to match full fine-tuning; adapting only the query and value projections is not enough [Certain]. Figure 1 shows the price: for LLaVA-1.5-7B at rank 8, “q and v only” adds 4.2M weights and “all 7 linear layers” adds 20.0M. Both are tiny next to the 14 GB of frozen weights.
Full training (not LoRA) for the connector
The connector is small (21.0M weights in LLaVA-1.5, 0.30% of the model; 44.6M in Qwen2-VL) and it is the part that most needs to change when the image distribution changes. LoRA on it would save almost nothing. So recipes train it fully: LLaVA-1.5 LoRA trains the projector fully at \(2\times10^{-5}\) [Certain], and Idefics2 uses “standard full fine-tuning” for its newly initialized parts [Certain]. With Hugging Face PEFT you list it in modules_to_save, which makes it trainable and saves it with the adapter. Freeze the connector only if you want the visual features to reach the LLM exactly as before (for example, a text-only style fine-tune).
LoRA on the vision tower (when the images are new)
Freeze the vision tower when your images look like web photos: CLIP and SigLIP have seen billions of them. Adapt it when the pixels are unusual: X-rays, satellite images, thermal video, dense documents at a new resolution, or a new position-embedding size. Idefics2 does exactly this: it interpolates the encoder’s position embeddings to a higher resolution and trains “the vision encoder with LoRA parameters to adapt to these modifications” [Certain]. LoRA is a safer way to change the vision tower than full training, because the change is small and can be scaled down or removed. In Figure 1, LoRA rank 8 on all linear layers of the LLaVA vision tower adds only 3.5M weights, but it makes the backward pass go all the way down to the pixels, so activation memory and compute for the image tokens go up.
Does LoRA lose quality for VLM SFT?
LLaVA-1.5 released both versions trained on the same 665K-sample mix for one epoch, so we can compare them directly. Figure 2 shows the differences. For the 7B model, LoRA was higher on 8 of 11 benchmarks, lower on 2 and equal on 1, with a mean difference of +0.6 points [Certain, from the published table]. Each number comes from a single run, and differences of about 1 point may be noise. [Likely] The text-only skills were not measured in that table.
How to read Figure 2
Green bars are benchmarks where the LoRA model scored higher, orange where full fine-tuning did. For 7B, most bars are green and small (under 2.5 points); VizWiz is the largest loss (−2.2). Switch to 13B: the picture is mixed (4 higher, 5 lower, 2 equal) and the biggest bars go in both directions (VizWiz +5.3, LLaVA-Bench −3.0). The lesson is not “LoRA is better” but “on this data, LoRA and full fine-tuning are about equal, at about one fifth of the memory”.
This matches what is known for text LLMs: on instruction-following data, which mostly teaches format and style, LoRA usually matches full fine-tuning; on data that must teach a lot of new knowledge (for example a large amount of code or math), full fine-tuning learns more. That is the finding of “LoRA Learns Less and Forgets Less” [Certain]. For a VLM, the second case is learning a new visual domain or a new language from scratch; there, more trainable capacity (higher rank, more targets, or full training) helps. [Likely]
DoRA (weight-decomposed LoRA) splits each weight into a length and a direction and trains the length separately (session 57). Idefics2 used it for instruction tuning [Certain]. In PEFT it is one flag, use_dora=True; it is slower per step than LoRA, so merge it before serving.
QLoRA for VLMs
QLoRA (session 58) stores the frozen base weights in 4 bits and trains LoRA adapters in 16 bits on top. In each forward pass a block of 4-bit weights is turned back into BF16 (dequantized), multiplied, and thrown away. Gradients flow through the frozen 4-bit weights into the adapters. The QLoRA paper fine-tuned a 65B model on one 48 GB GPU this way [Certain]. Its storage cost per weight, with blocks of 64 weights, 8-bit block scales, and one FP32 scale for every 256 blocks (“double quantization”), is
So the memory of a VLM training run, before activations, is
- Read the symbols
- \(N_p\)
- the number of frozen base weights in part \(p\) (also the frozen base of a LoRA part)
- \(b_p\)
- bytes per frozen weight: 2 in BF16, 0.516 in NF4 with double quantization
- \(N_{\text{train}}\)
- all trainable weights: LoRA factors plus fully trained parts, 16 bytes each with mixed-precision AdamW
What to quantize in a VLM, and what not
- The LLM’s linear layers: yes. They hold 96.1% of LLaVA-1.5’s LLM weights and 85.7% of Qwen2-VL’s (Qwen2-VL has a large vocabulary: its input and output embeddings are 1.09B weights). This is where the savings are.
- Embeddings and
lm_head: no. They are not nn.Linear layers inside blocks, the output layer is sensitive, and in Hugging Face it is skipped by default. [Likely] - The connector: no. It is small and you usually train it fully, and a fully trained part cannot be stored in 4 bits. LLaVA’s QLoRA path passes
llm_int8_skip_modules=["mm_projector"]to the quantization config for exactly this reason [Certain, from the code]. - The vision tower: usually no. Quantizing it saves little: \(303.5\text{M} \times (2 - 0.516) = 0.45\) GB for LLaVA-1.5, 0.94 GB for Qwen2-VL. And errors in the image features reach every visual token the LLM reads. Most QLoRA VLM recipes keep it in BF16. [Likely]
Full SFT (vision frozen, connector and LLM full): frozen \(2 \times 303.5\text{M} = 0.61\) GB, trainable \(16 \times 6{,}759.9\text{M} = 108.16\) GB. Total 108.77 GB.
LoRA (rank 128 on all 7 linear layers of 32 blocks, connector full): LoRA adds \(128 \times 32 \times (4 \cdot 8{,}192 + 3 \cdot 15{,}104) = 319.8\)M weights; with the connector, \(N_{\text{train}} = 340.8\)M (4.82% of 7.06B). Memory: \(2 \times 7{,}042.4\text{M} = 14.08\) GB frozen + \(16 \times 340.8\text{M} = 5.45\) GB. Total 19.54 GB.
QLoRA (the same, with the LLM’s 6,476.0M linear weights in NF4): those weights drop from 12.95 GB to 3.34 GB, saving 9.61 GB. Total 9.93 GB. With a smaller rank (8) and the connector frozen as well, Qwen2-VL-7B needs 7.22 GB.
Activations come on top, and in a VLM they grow with the number of image tokens: 576 per image for LLaVA-1.5 at 336 px, and many more for high-resolution or video inputs. Gradient checkpointing (session 39) is almost always on.
Two practical warnings. First, merging: after QLoRA training, do not add \(\frac{\alpha}{r}BA\) into the 4-bit weights; the sum would be rounded back to the 4-bit grid and much of the update lost. Load the base model in BF16, merge there, and quantize again for serving if you need to. [Likely] Second, speed: dequantizing on every forward pass makes QLoRA steps slower than BF16 LoRA steps. Use it when memory, not time, is the limit. [Likely]
Keeping text-only skills while learning vision tasks
A VLM is used as a chat model too: users ask it plain questions, ask for code, or keep talking after the image is forgotten. If SFT trains only on image data, the LLM’s weights drift toward that data and its text skills drop. This is catastrophic forgetting (session 61). It also happens inside vision: Zhai et al. treated several fine-tuned VLMs as image classifiers and found that “almost all evaluated MLLMs fail to retain the same performance levels as their vision encoders” [Certain]. Fine-tuning LLaVA further on one image dataset helped other image datasets at first, then the model began to hallucinate and generalize worse, even with the image encoder frozen [Certain].
First, measure it. Before training, score the base model on a small text suite that matches your users (for example MMLU for knowledge, GSM8K for math, IFEval for instruction following, plus your own prompts). After training, the forgetting on that suite is
where \(\theta_0\) are the weights before SFT. Track \(F\) during training, not only at the end; it often grows with the number of steps.
Then use one or more of five remedies.
- Mix text-only data into SFT. With a share \(\lambda\) of text samples, the loss becomes
\[ \mathcal{L} = (1 - \lambda)\, \mathcal{L}_{\text{SFT}}^{\text{image}} + \lambda\, \mathcal{L}_{\text{SFT}}^{\text{text}} \tag{189.9} \]LLaVA-1.5 includes about 40K ShareGPT text conversations in its 665K mix (\(\lambda \approx 6.0\%\)), and the authors report that the model learned from them to answer in the user’s language, a behavior that “is transferred to visual conversations” [Certain]. Qwen2-VL’s SFT data “encompasses not only pure text-based dialogue data but also multimodal conversational data”, and Idefics2 adds text-only instruction datasets [Certain]. Good text samples are ones close to what the base model was tuned on.
- Change fewer weights. Freeze the LLM or use LoRA on it. LoRA constrains the update to rank \(r\), and “LoRA Learns Less and Forgets Less” found it keeps more of the base model’s skills outside the target domain than full fine-tuning [Certain]. Figure 3 shows the same trade-off in a toy.
- Move gently. A smaller learning rate, one epoch instead of three, and early stopping on the text suite.
- Interpolate back. Mix the fine-tuned weights with the original ones, as in WiSE-FT:
\[ \theta(\beta) = (1 - \beta)\, \theta_0 + \beta\, \theta_{\text{SFT}}, \qquad \text{for LoRA: } W(\beta) = W_0 + \beta\, \frac{\alpha}{r} B A, \qquad 0 \le \beta \le 1 \tag{189.10} \]Sweep \(\beta\) and pick the value with the best combined score. For LoRA this is free: scale the adapter at merge time.
- Keep the chat format. Use the base model’s own chat template and system prompt. Many “forgetting” reports are really a format mismatch: the model was trained on short benchmark answers (“Answer with a single word”) and now answers chat questions in one word. LLaVA-1.5 attaches an explicit format prompt to each short-answer dataset for this reason [Certain], and Idefics2 adds a short “chatty” stage on long dialogues because its answers had become too short [Certain].
How to read Figure 3
With full fine-tuning and no text data, the image-task loss falls to almost 0 (0.001 at step 300), and the text loss climbs to 2.390: the model learned the new task and paid with its old one. Move the text share to 10%: the text loss falls to 1.494 (37% less forgetting), while the image-task loss is still only 0.041. Now try LoRA \(r = 1\): it can only learn the strongest direction of the change, so the image loss stays at 1.478 (29% of the start) but the text loss is only 0.537. Higher ranks move toward full fine-tuning: they learn more and forget more. Drag the step slider back to 30: full fine-tuning already reached 0.083 on the image task with half the forgetting (1.254), so early stopping is also a remedy. The early bump of the LoRA text loss comes from the random start of \(A\).
The toy is linear, so do not read its numbers as predictions for a real VLM. It shows the shape of the trade-off: every remedy moves you along a curve between “learned the new task” and “kept the old one”, and mixing text data moves the whole curve down. [Likely]
Putting it together: an SFT step with a freeze map
- input: batch of (images, conversations); model with freeze map; optimizer from Algorithm 1; schedule \(s(t)\)
- build token ids with the model’s chat template; insert image placeholders
- labels \(\leftarrow\) ids; set labels to \(-100\) on image tokens, user turns, system prompt and padding // \(m_t\) of formula (189.1)
- visual tokens \(\leftarrow\) connector(vision tower(images)) // vision tower under no_grad if frozen
- logits \(\leftarrow\) LLM([visual tokens ; text embeddings])
- loss \(\leftarrow\) cross-entropy(logits shifted by one, labels) // formula (189.1)
- loss.backward(); clip the gradient norm (per part, if you follow Molmo)
- set each group’s rate to \(\eta_g \cdot s(t)\); optimizer step; zero gradients // formula (189.3)
- every few hundred steps: evaluate the image task and the text suite // forgetting \(F\), formula (189.8)
The PyTorch version below builds a tiny VLM (a 2-block vision tower, an MLP connector, a 2-block LLM), freezes the vision tower, trains the connector fully, puts LoRA on all linear layers of the LLM blocks, gives the two groups different learning rates, masks the loss to the answer, and checks that merging the adapters does not change the outputs. It runs on a CPU in a few seconds and prints the same numbers every time (PyTorch 2.x).
import math, re, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) class LoRALinear(nn.Module): """h = W0 x + (alpha/r) B A x, with W0 frozen (formula 189.4).""" def __init__(self, base: nn.Linear, r=4, alpha=8): super().__init__() self.base, self.s = base, alpha / r base.weight.requires_grad_(False) if base.bias is not None: base.bias.requires_grad_(False) self.A = nn.Parameter(torch.randn(r, base.in_features) / math.sqrt(base.in_features)) self.B = nn.Parameter(torch.zeros(base.out_features, r)) # B = 0: the model starts unchanged def forward(self, x): return self.base(x) + self.s * (x @ self.A.T) @ self.B.T def merge(self): # fold the adapter into W0 for serving self.base.weight.data += self.s * self.B @ self.A return self.base class Block(nn.Module): # a tiny pre-norm transformer block def __init__(self, d): super().__init__() self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d) self.q_proj, self.k_proj, self.v_proj, self.o_proj = (nn.Linear(d, d) for _ in range(4)) self.up_proj, self.down_proj = nn.Linear(d, 4 * d), nn.Linear(4 * d, d) def forward(self, x, causal): h = self.n1(x); B_, T, d = h.shape q, k, v = self.q_proj(h), self.k_proj(h), self.v_proj(h) a = F.scaled_dot_product_attention(q, k, v, is_causal=causal) x = x + self.o_proj(a) return x + self.down_proj(F.gelu(self.up_proj(self.n2(x)))) class TinyVLM(nn.Module): def __init__(self, dv=32, dl=64, vocab=100, nv=2, nl=2): super().__init__() self.vision_tower = nn.ModuleList(Block(dv) for _ in range(nv)) self.patch = nn.Linear(48, dv) # 4x4x3 pixel patches self.connector = nn.Sequential(nn.Linear(dv, dl), nn.GELU(), nn.Linear(dl, dl)) self.embed = nn.Embedding(vocab, dl) self.llm = nn.ModuleList(Block(dl) for _ in range(nl)) self.lm_head = nn.Linear(dl, vocab, bias=False) def forward(self, patches, ids): v = self.patch(patches) for blk in self.vision_tower: v = blk(v, causal=False) x = torch.cat([self.connector(v), self.embed(ids)], dim=1) # [image tokens | text tokens] for blk in self.llm: x = blk(x, causal=True) return self.lm_head(x) model = TinyVLM() n_all = sum(p.numel() for p in model.parameters()) # 1) freeze everything, then choose per part (the "freeze map") model.requires_grad_(False) for p in model.connector.parameters(): p.requires_grad_(True) # connector: full training targets = re.compile(r"llm\.\d+\.(q_proj|k_proj|v_proj|o_proj|up_proj|down_proj)") for name, mod in list(model.named_modules()): # LLM: LoRA on all linear layers if targets.fullmatch(name): parent, child = name.rsplit(".", 1) setattr(model.get_submodule(parent), child, LoRALinear(mod, r=4, alpha=8)) n_train = sum(p.numel() for p in model.parameters() if p.requires_grad) print(f"trainable {n_train} of {n_all} ({100 * n_train / n_all:.2f}%)") # 2) one learning rate per part (formula 189.3): LoRA factors high, the full connector lower lora = [p for n, p in model.named_parameters() if p.requires_grad and n.endswith((".A", ".B"))] conn = list(model.connector.parameters()) opt = torch.optim.AdamW([{"params": lora, "lr": 2e-3}, {"params": conn, "lr": 2e-4}], weight_decay=0.0) # 3) SFT batch: 16 image tokens, a prompt, then the answer; loss only on the answer (formula 189.1) patches = torch.randn(8, 16, 48) ids = torch.randint(0, 100, (8, 12)) labels = torch.full((8, 16 + 12), -100) # -100 = ignored by cross_entropy labels[:, 16 + 6:] = ids[:, 6:] # the last 6 text tokens are the answer model.eval() # no dropout here; keeps the check exact for step in range(30): logits = model(patches, ids) loss = F.cross_entropy(logits[:, :-1].reshape(-1, 100), labels[:, 1:].reshape(-1)) # shift by one opt.zero_grad(set_to_none=True); loss.backward(); opt.step() if step in (0, 29): print(f"step {step:2d} answer loss {loss.item():.3f}") # 4) no gradient ever reached the frozen vision tower print("vision grads:", all(p.grad is None for p in model.vision_tower.parameters())) # 5) merge the adapters and check the outputs do not change with torch.no_grad(): before = model(patches, ids) for name, mod in list(model.named_modules()): if isinstance(mod, LoRALinear): parent, child = name.rsplit(".", 1) setattr(model.get_submodule(parent), child, mod.merge()) after = model(patches, ids) print("max change after merge:", f"{(before - after).abs().max().item():.1e}")
Output: trainable 15488 of 146016 (10.61%), the answer loss goes from 4.885 at step 0 to 2.890 at step 29, vision grads: True, and the merge changes the outputs by about \(10^{-6}\) (float rounding). Check the count by hand: the connector has \(32 \cdot 64 + 64 + 64 \cdot 64 + 64 = 6{,}272\) weights; LoRA with \(r = 4\) adds \(4 \cdot (4 \cdot 128 + 2 \cdot 320) = 4{,}608\) per block, 9,216 for two blocks; the total is 15,488.
With Hugging Face libraries the same freeze map is a configuration. The sketch below is not run here (it needs a CUDA GPU, bitsandbytes, peft and the model download); the module names are those of the Hugging Face LLaVA implementation and differ between models, so print model.named_modules() first.
from transformers import LlavaForConditionalGeneration, BitsAndBytesConfig from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training import torch bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16, llm_int8_skip_modules=["vision_tower", "multi_modal_projector", "lm_head"]) model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf", quantization_config=bnb) model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True) cfg = LoraConfig(r=128, lora_alpha=256, lora_dropout=0.05, # a regex: LoRA only inside the language model, on all 7 projections target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)", modules_to_save=["multi_modal_projector"]) # connector: full training, saved with the adapter model = get_peft_model(model, cfg) model.print_trainable_parameters()
Where this is used
| Application | Typical freeze map | Why |
|---|---|---|
| General visual assistant (stage 3 of building a VLM) | Vision frozen or slow; connector and LLM full; text-only data in the mix | Large, varied data justifies full training; text data protects chat skills (LLaVA-1.5, Qwen2-VL) |
| Domain specialist: documents, charts, receipts | LoRA on the LLM, connector full; vision LoRA if resolution changes | Mostly new answer formats and words; fine text needs higher resolution, which the encoder must adapt to |
| Medical, satellite, thermal or industrial images | LoRA on the vision tower and the LLM, connector full | The pixels are unlike web photos, so the encoder itself must change, but gently |
| Video classification, action recognition and key-moment detection with a video LLM | Vision frozen (features can be cached per frame), connector full, LoRA on the LLM | Many frames per sample make the vision backward pass very expensive; a frozen encoder lets you precompute features once |
| Video-RAG and video retrieval answers | LoRA on the LLM only | The model must learn to read retrieved frames and cite timestamps; the visual features are already good |
| Fine-tuning on one consumer GPU | QLoRA on the LLM, everything else frozen or full in BF16 | A 7B VLM fits in about 7 to 10 GB before activations (Figure 1) |
| Many customers or tasks on one server | One frozen base model, one LoRA adapter per task | Adapters are a few hundred MB and can be swapped or batched together (session 57) |
Common mistakes
- Training on the prompt and the image tokens. Forgetting the loss mask (labels \(-100\)) makes the model learn to predict user questions and spends gradient on placeholders. Print one sample with its labels before every new run.
- LoRA landing in the wrong part. A target list like
["q_proj", "v_proj"]matches by name suffix, so it may also hit the vision tower’s attention layers if they use the same names. Use a regex orexclude_modules, and print the trainable count per part. - The connector is silently frozen. With PEFT, everything not adapted is frozen, including the connector you meant to train. Put it in
modules_to_saveand checkrequires_grad. - One learning rate for everything. \(2\times10^{-4}\) is fine for LoRA factors and harmful for full weights of the LLM or the vision tower. Use parameter groups.
- Quantizing the connector or the vision tower in QLoRA. The savings are small (under 1 GB for 7B models) and the quality cost lands on every image token. Skip them in the quantization config.
- Measuring only the image benchmark. Without a text suite you will not see forgetting until users report it. Evaluate both, from step 0.
- Changing the chat template. Fine-tuning with a different template than the base model used turns a good chat model into a confused one. Use the processor’s own template.
- Merging a QLoRA adapter into 4-bit weights. Merge into a BF16 copy of the base, then quantize again if needed.
Interview questions
Which parts of a VLM do you train in SFT, and why?basic
Why use different learning rates for different parts? Give real numbers.core
How much memory does SFT of LLaVA-1.5-7B need with full fine-tuning, LoRA and QLoRA?core
Where would you put LoRA in a VLM?core
If the LLM is frozen and only the connector is trained, is the LLM's backward pass free?advanced
What do you quantize and what do you keep in 16 bits when you QLoRA a VLM?core
Your fine-tuned VLM is great on your image task but worse at plain chat. What happened, and what do you do?core
Does LoRA lose quality compared with full fine-tuning for VLM instruction tuning?core
Why does QLoRA not lose the 4-bit rounding of the update, and why must you be careful when merging?advanced
You fine-tune a video LLM for action recognition with 32 frames per clip. How do you set the freeze map?advanced
Exercises
- LLaVA-1.5-13B has an LLM with 40 blocks, hidden size 5,120 and MLP size 13,824. How many weights does LoRA rank 128 on all 7 linear layers add?
- Using formula (189.2), compare the LLM compute per token for (a) connector-only training, (b) LoRA on the LLM, (c) full LLM training.
- A 7B VLM’s LLM has 6.5B linear weights. How much memory does NF4 with double quantization save over BF16 for these weights?
- In Figure 3, set full fine-tuning and find the text share \(\lambda\) where the text loss at step 300 first drops below 1.0. What is the image-task loss there?
- Your SFT mix has 500K image samples. How many text-only samples do you add for a text share of 6%?
- Write the PyTorch parameter groups for: vision tower LoRA at \(5\times10^{-5}\), connector full at \(1\times10^{-5}\), LLM LoRA at \(2\times10^{-4}\), no weight decay on biases and norms.
Answerscheck
vision_tower and lora_), and for each part split into p.ndim >= 2 (decay) and p.ndim < 2 (no decay), giving up to six groups, each with its own lr and weight_decay; then pass the list to torch.optim.AdamW.Key takeaways
- VLM SFT is next-token loss on the answer tokens only (189.1). The new decision is the freeze map: frozen, LoRA or full for the vision tower, the connector and the LLM.
- The backward pass must reach the lowest trainable part (189.2). Frozen parts above it still pay for input gradients and activations; frozen parts below it pay only the forward pass.
- Use parameter groups (189.3): slow for the vision tower, faster for the connector, about 10 times faster for LoRA factors than for full weights (LLaVA-1.5: \(2\times10^{-4}\) vs \(2\times10^{-5}\)).
- Put LoRA (189.4, 189.5) on all linear layers of the LLM, train the connector fully, and adapt the vision tower only when the images are new. LLaVA-1.5 LoRA matched full fine-tuning on its benchmarks.
- QLoRA stores the LLM’s linear layers at 4.127 bits (189.6); with (189.7), LLaVA-1.5-7B goes from 108.8 GB (full) to 19.5 GB (LoRA) to 9.9 GB (QLoRA). Keep the vision tower, connector, embeddings and lm_head in BF16.
- Measure forgetting on a text suite (189.8). Fix it with text data in the mix (189.9), fewer changed weights, gentler training, weight interpolation (189.10), and the base model’s chat format.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Hugging Face (code). PEFT library
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Hugging Face (code). PEFT library
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Laurençon et al. (2024). What matters when building vision-language models?
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
- Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer
- Biderman et al. (2024). LoRA Learns Less and Forgets Less
- Zhai, Tong, Li, Cai, Qu, Lee & Ma (2023). Investigating the Catastrophic Forgetting in Multimodal Large Language Models
- Wortsman et al. (2022). Robust fine-tuning of zero-shot models
- Liu et al. (2023). LLaVA code base: v1.5 fine-tuning scripts (finetune.sh, finetune_lora.sh) and Model Zoo