VLM training stages
A vision-language model is built from two trained parts, a vision encoder and an LLM, joined by a new connector with random weights. It is not trained in one go. First only the connector learns to speak the LLM's language; then more parts are unfrozen on large multimodal data; then the model learns to follow instructions; and often a last stage of preference tuning or RL fixes hallucination and reasoning. This session explains each stage with the real recipes of LLaVA-1.5, LLaVA-OneVision, Qwen2-VL, InternVL 2.5, Idefics2, Molmo and MM1, and shows with numbers what to freeze, when, and why.
What you will learn
- Why VLM training is split into stages: a random connector sends noise into a trained LLM, and big gradients through that noise damage the LLM
- Stage 1 (alignment): train only the projector on image-caption pairs, with a high learning rate; LLaVA-1.5 trains 21.0M of 7.06B weights (0.30%) for about 2,180 steps
- Stage 2 (multimodal pretraining): unfreeze the LLM and often the vision encoder, with smaller learning rates for the encoder, more data, higher resolution and some text-only data
- Stage 3 (instruction tuning) and the optional stage 4 (preference tuning such as DPO and MPO, or RL with verifiable rewards)
- What to freeze, when and why: memory (2 bytes per frozen weight, 16 per trained weight), compute (about 4N instead of 6N FLOPs per token in stage 1), stability, and forgetting
- What the ablations say, including the ones that disagree (Prismatic vs Cambrian-1 on stage 1 and on unfreezing the vision encoder)
- How to write a stage switcher in PyTorch with requires_grad, parameter groups and per-module learning rates
Why this matters
In session 185 you glued a vision encoder to a small language model with a connector and trained it. It worked, but you had to decide many things on the way: which parts to train, with which learning rate, on which data, and in which order. Real vision-language models (VLMs) answer these questions with a sequence of training stages. This session explains that sequence.
A VLM is made of three parts (see session 177): a vision encoder that turns an image into patch vectors, a connector that maps those vectors into the LLM’s input space, and a language model that reads the visual tokens together with the text. Two of the three parts are already trained, on billions of images and trillions of words. The connector is new and random. If you train everything at once with one learning rate, the random connector sends noise into the LLM, the gradients are large, and the LLM can lose part of what it knew. So almost every recipe starts gently and unfreezes more parts later. [Certain]
The standard plan has three stages and an optional fourth:
- Stage 1, alignment: train only the connector (projector) on image–caption pairs.
- Stage 2, multimodal pretraining: unfreeze more (the LLM, often the vision encoder) and train on large, mixed data.
- Stage 3, instruction tuning: teach the model to follow user requests in a chat format.
- Stage 4 (optional), preference tuning or RL: make answers more truthful, less hallucinated, and better at reasoning.
The names are not fixed. Some papers merge stages 1 and 2, some call stage 2 “stage 1.5”, and some skip stage 1 entirely. The book calls the whole idea the “when” of training, separate from the “how” (supervised, self-supervised, contrastive) [VLM ch. 3]. What stays the same is the logic: which parameters may change, how fast, and on what data. If you fine-tune a VLM for your own product (document reading, video search, a robot), you will make these same decisions, usually with a smaller budget.
The starting point: two trained parts and one new bridge
Let us write the model down. An image \(I\) goes through the vision encoder \(E_\phi\) (weights \(\phi\)) and gives \(N_v\) patch vectors. The connector \(C_\psi\) (weights \(\psi\)) maps each one to the LLM width \(d\). The LLM \(f_\omega\) (weights \(\omega\)) reads the visual tokens followed by the text tokens:
- Read the symbols
- \(\theta = (\phi, \psi, \omega)\)
- all weights: vision encoder, connector, LLM
- \(C_\psi(E_\phi(I))\)
- the visual tokens, \(N_v\) vectors of width \(d\) (576 for LLaVA-1.5 at 336 px, 729 for SigLIP at 384 px)
- \(\mathrm{Emb}_\omega(x_t)\)
- the embedding of text token \(x_t\); the embedding table is part of the LLM
- \([\,\cdot\,;\,\cdot\,]\)
- joining the two lists of vectors into one sequence
Every stage trains this model with the same basic loss: next-token prediction. But we only want the model to learn to produce the parts it will produce at test time: the caption, or the assistant’s answer. The image tokens, the system prompt and the user’s question are input, not output. So we use a loss mask \(m_t\) (see session 18):
In stage 1 the targets are the caption tokens. In stage 3 they are the assistant’s answers in a multi-turn chat. In PyTorch you set the label of every non-target position to −100, which cross_entropy ignores.
What changes between stages is which of \(\phi, \psi, \omega\) are updated, with which learning rate. We write this as one rule. For each part \(j\) and stage \(s\) there is a learning rate \(\eta_j^{(s)}\), and \(\eta_j^{(s)} = 0\) means “frozen”:
In practice “frozen” is stronger than “learning rate 0”: you set requires_grad = False, so PyTorch does not even compute the gradient, and the optimizer does not keep any state for those weights. This saves both compute and memory, as we will count below.
Why train in stages?
Think about the first training step. The connector is random, so the visual tokens are random vectors. The LLM has never seen such vectors. Its loss on the caption is high, and the gradient is large. Where does that gradient go?
Gradients flow backward from the loss. To reach the connector, they must pass through the LLM: by the chain rule (see session 4),
Here \(\mathbf{h}_{\text{vis}}\) are the visual tokens. The first factor is computed by backpropagating through all LLM layers, even if the LLM is frozen. If the LLM is not frozen, the same large, noisy gradient also updates \(\omega\). The LLM then changes its weights to cope with noise, instead of the connector learning to produce useful vectors. Three bad things can happen:
- Forgetting. The LLM’s language skill gets worse (it starts to repeat, or loses knowledge). This is the same catastrophic forgetting as in session 61.
- Instability. Loss spikes or divergence. Idefics2 reports that unfreezing everything in their fully autoregressive model “would yield loss divergences”, and they could not fix it even by “aggressively lowering the learning rate or gradually unfreezing” [Certain]. They switched to LoRA for the pretrained parts.
- Wasted compute. Billions of LLM weights move to fix a problem that a 21M-parameter connector could fix alone.
The staged recipe avoids this. Stage 1 freezes everything except the connector, so the only thing that can move is the connector. Once the visual tokens make sense to the LLM, the gradients are smaller and more meaningful, and it is safe to let the LLM and the encoder change, slowly. Figure 1 walks through the stages of one real recipe, LLaVA-OneVision-7B.
How to read Figure 1
Follow the colors. In stage 1 only the connector is orange: 0.25% of the weights learn, with a learning rate 100 times higher than later. From stage 1.5 on, everything is orange, but the vision encoder’s learning rate stays 5 times smaller than the LLM’s. Notice also the grey text in the data box: the number of visual tokens per image grows from 729 to up to 7,290. Cheap, low-resolution data comes first; expensive, high-resolution data comes later, when the model can use it.
Stage 1: alignment (projector only)
Goal: teach the connector to map vision features into vectors that the frozen LLM already “understands”, so that the LLM, given the visual tokens, can write a caption. Nothing else may change.
Data: simple image–caption pairs. LLaVA-1.5 uses 558K pairs from LAION, CC and SBU, re-captioned by BLIP (the “LCS-558K” set) [Certain]. The prompt is just the image; the target is one sentence. Short targets make this stage cheap.
Trainable: only \(\psi\). In LLaVA-1.5 the connector is a 2-layer MLP from 1024 (CLIP ViT-L/14 width) to 4096 (Vicuna width), so it has \(1024 \cdot 4096 + 4096 + 4096 \cdot 4096 + 4096 = 20{,}979{,}712\) parameters (see session 178).
Learning rate: high, \(10^{-3}\), with batch 256, one epoch, cosine decay and 3% warmup [Certain]. That is 50 times higher than the \(2\times10^{-5}\) of stage 2. A high rate is safe here because only a small, freshly initialized network moves, and the frozen LLM cannot be damaged. The original LLaVA used \(2\times10^{-3}\) with a linear projector; LLaVA-1.5 halved it for the MLP [Certain].
Parameter counts (computed from the model configs): CLIP ViT-L/14-336 vision tower \(303{,}507{,}456\); MLP connector \(20{,}979{,}712\); Vicuna-7B (Llama 2 architecture) \(6{,}738{,}415{,}616\). Total \(7{,}062{,}902{,}784 \approx 7.06\)B.
Trainable in stage 1: \(20{,}979{,}712 / 7{,}062{,}902{,}784 = 0.30\%\). Steps: \(558{,}000 / 256 = 2{,}179.7\), so about 2,180 steps. Stage 2 (665K samples, batch 128) is about 5,196 steps and trains \(95.7\%\) of the weights. The paper reports about 6 hours for stage 1 and 20 hours for stage 2 on 8 A100 GPUs [Certain].
Stage 1 is cheap in memory, but not free in compute
Training only 0.3% of the weights does not make each step 300 times cheaper. Let \(N\) be the LLM size. A training step costs about \(2N\) FLOPs per token forward, \(2N\) to compute gradients with respect to activations, and \(2N\) for gradients with respect to weights (the \(6N\) rule of session 4). With the LLM frozen, we skip only the weight gradients, but we still need the activation gradients to reach the connector (formula 186.4):
So stage 1 costs about \(4/6 = 67\%\) of a full step per token. The vision encoder is cheaper: nothing before it trains, so it needs a forward pass only (you can even pre-compute its features). The large savings are in memory: no gradients and no Adam states for 99.7% of the weights (Figure 3).
Is stage 1 needed at all?
Papers disagree, and this is a good interview topic.
- Prismatic (Karamcheti et al.) trained LLaVA-style models with and without stage 1 and found that skipping it gave equal or better scores while saving 20–25% of training cost [Certain]. They use single-stage training afterwards.
- Cambrian-1 repeated the test with 0, 0.5M and 1.2M adapter (stage 1) samples and found that “pre-training the connector first enhances model performance and that more adapter data further improves performance” [Certain].
- Molmo skips the connector-only stage but keeps its purpose: the connector gets a 10 times higher learning rate than the LLM (\(2\times10^{-4}\) vs \(2\times10^{-5}\)) and a short warmup of 200 steps, against 2,000 steps for the ViT and the LLM [Certain].
- InternVL 2.5 keeps an “MLP warmup” stage and says it improves training stability [Certain].
A fair reading: with a small budget and good data the effect is small, and the result depends on the encoder and on how much stage-1 data you have. [Likely] The core idea that survives everywhere is: the new connector must move fast while the pretrained parts move slowly. A separate stage, a higher learning rate, or a shorter warmup are three ways to get that.
Stage 2: multimodal pretraining (unfreeze more)
Goal: give the model real visual knowledge: reading text in images (OCR), charts, documents, counting, many objects, long and detailed descriptions, and several images per sequence. A connector alone cannot learn this; the LLM (and often the encoder) must change.
Data: much more and much richer than stage 1. Examples:
- LLaVA-OneVision stage 1.5: 4M “high-quality knowledge” samples, mostly detailed re-captions, OCR and document data [Certain].
- Idefics2: interleaved web documents (OBELICS), image–text pairs and PDF pages, about 1.5B images and 225B text tokens in total [Certain].
- Qwen2-VL: about 600B tokens in its first stage and 800B more in its second, 1.4T in total [Certain].
- MM1: 45% interleaved documents, 45% image–text pairs and 10% text-only, for 200K steps (about 400B tokens) [Certain].
Trainable: the connector and the LLM, and in many recent models the vision encoder too. Resolution usually grows here: Idefics2 goes from 384 px in its first pretraining part to up to 980 px in the second, when PDFs enter the mix, because text must be large enough to read [Certain] (see session 180).
Different learning rates for different parts
When several parts train together, they rarely share one learning rate. The rule of thumb: the better a part is already trained, the more carefully it should move. In AdamW you write this with parameter groups:
Why is the vision encoder the most careful? It was trained contrastively on billions of images and its features are general. Next-token loss on a few million samples can easily bend them toward the training set (for example, toward documents) and hurt other skills. A 5 to 10 times smaller rate keeps the features close to where they were. InternVL 2.5 says this directly: a lower learning rate in its ViT stage “to prevent catastrophic forgetting” [Certain]. Molmo goes one step further and clips gradients separately for the ViT, the connector and the LLM [Certain].
How large should the LLM’s rate be? MM1 fitted the best peak rate against model size on small models and extrapolated:
where \(N\) is the number of (non-embedding) LLM parameters. For \(N = 3\times10^{10}\) this gives \(\eta^* = 2.2\times10^{-5}\), the value used for MM1-30B [Certain]; for \(N = 7\times10^9\) it gives \(4.1\times10^{-5}\). The rate falls as models grow: bigger models need gentler steps. Treat the formula as specific to MM1’s setup, not as a law. [Likely]
Keep some text-only data
A VLM is still a language model, and users ask it pure-text questions too. If stage 2 sees only image data, the LLM drifts away from its text skills. The fix is to mix text-only samples into the batch. With \(K\) data sources and mixing weights \(w_k\):
MM1 found that interleaved data helps few-shot and text-only scores, while caption data helps zero-shot captioning [Certain]. Qwen2-VL says pure text “continues to play a crucial role” in its second stage, and Idefics2 adds text-only instruction data in SFT [Certain]. You will explore this trade-off in Figure 4.
Unfreeze fully, or use LoRA?
Two different answers work in practice:
- Full unfreezing with small rates (MM1, LLaVA-OneVision, Qwen2-VL, InternVL 2.5). Most of the strongest open models do this. It needs sharded training, because every weight now has gradients and optimizer states.
- LoRA on the pretrained parts, full training for the new parts (Idefics2). LoRA (see session 57) adds small low-rank matrices and trains only those. In their ablation, this added 12.9 points over frozen backbones for the fully autoregressive model, and it was stable where full unfreezing diverged [Certain]. LoRA can be merged back, so it costs nothing at inference.
Reuse the trained encoder
InternVL 2.5 trains its vision encoder once, with a smaller LLM (stage 1.5), and then plugs it into larger LLMs, which skip that stage. They report that their 78B model saw only about 120B training tokens, against 1.4T for Qwen2-VL [Certain]. Training the encoder is expensive; doing it once and reusing it is a cheap trick worth remembering.
Stage 3: instruction tuning
Goal: turn a model that can describe images into an assistant that follows requests: answer a question in one word, give a bounding box, read a table into JSON, compare two images, explain a chart step by step. This is supervised fine-tuning (SFT, see session 54) with images.
Data format: multi-turn chats in the model’s chat template (see session 9). The image appears as a placeholder that is replaced by the visual tokens. The loss mask (186.2) keeps only the assistant’s turns. LLaVA-1.5’s 665K mix is a good small example [Certain]:
| Source | Samples | What it teaches |
|---|---|---|
| LLaVA conversations (GPT-generated) | 158K | open chat about an image |
| ShareGPT (text only) | 40K | keeps the language skill |
| VQAv2, GQA, OKVQA, A-OKVQA | 83K + 72K + 9K + 66K | short answers, multiple choice |
| OCRVQA, TextCaps | 80K + 22K | reading text in images |
| RefCOCO, Visual Genome | 48K + 86K | regions and box coordinates |
| Total | 665K |
Two details from that recipe matter a lot. First, academic VQA data has very short answers, so LLaVA-1.5 adds a format prompt such as “Answer the question using a single word or phrase.” Without it the model learns to answer everything in one word. Molmo does the same with style tags such as “vqa2:” [Certain]. Second, several QA pairs about the same image are merged into one conversation, so the image is encoded once.
Trainable: at least the connector and the LLM. LLaVA-1.5 keeps the encoder frozen; LLaVA-OneVision, MM1 and InternVL 2.5 train everything; Qwen2-VL locks the ViT again in this stage [Certain]. The learning rate is small (\(10^{-5}\) to \(2\times10^{-5}\)), usually one epoch.
Data quality matters more here than anywhere else. The LLM is trainable and it writes the final answer. InternVL 2.5 reports that “even a small amount of noisy data (e.g., a few thousand samples) can lead to abnormal model behavior, like repetitive output” [Certain]. Idefics2 adds a short extra stage (“chatty”, a few hundred steps on long dialogue data) because benchmark-style SFT made its answers too short for real users [Certain]. LLaVA-OneVision ends with a short mixed phase of 1.6M single-image, multi-image and video samples, and that phase is where its video skill comes from [Certain].
Stage 4 (optional): preference tuning or RL
After SFT the model follows instructions, but two problems remain. It hallucinates: it describes objects that are not in the image, because the LLM’s language prior is strong (see session 156). And its step-by-step reasoning on charts and math is weak: SFT shows it correct answers but never tells it which of its own answers are bad. A fourth stage fixes this with preference data or rewards.
Preference tuning. For each prompt (image plus question), you have a chosen answer \(y_c\) and a rejected answer \(y_r\). DPO (see session 68) raises the likelihood of \(y_c\) relative to \(y_r\), compared with a frozen copy \(\pi_0\) of the SFT model:
- Read the symbols
- \(x\)
- the input: image tokens and the question
- \(\pi_\theta,\ \pi_0\)
- the model being trained and the frozen reference (the SFT model)
- \(\beta\)
- how strongly the model is kept close to \(\pi_0\); MPO uses 0.1
- \(\sigma\)
- the sigmoid function
Where do the pairs come from? For hallucination, the rejected answer can be the model’s own description and the chosen one a human correction of it (RLHF-V). [Likely] For reasoning, the MPO paper (Wang et al., on InternVL2-8B) builds about 3M pairs (the MMPR set) in two ways: when the true answer is known, chosen answers are sampled answers that reach it and rejected ones are those that do not; when it is not known, the model completes a half-written answer without the image, and that completion is the rejected answer (“Dropout NTP”), since it shows what the model writes when it is not looking [Certain].
MPO also found that DPO alone did little for chain-of-thought answers, while adding the plain SFT loss helped [Certain]. Its final loss is a mix:
Here \(\mathcal{L}_{\text{BCO}}\) is a “quality” loss that scores each answer on its own (good or bad), and \(\mathcal{L}_{\text{SFT}}(y_c)\) is formula (186.2) on the chosen answer. All parameters train with learning rate \(5\times10^{-6}\) for one epoch. On MathVista, InternVL2-8B went from 58.3 to 67.0 [Certain].
RL with verifiable rewards. For tasks with checkable answers (math on a figure, counting, a box that must overlap the object, a chart value), you can sample several answers and reward the correct ones with GRPO (see session 70). Several 2025 VLMs added such a stage for visual reasoning. [Likely] As in text LLMs, this stage uses a very small learning rate and a KL penalty or reference model so the model does not drift far from the SFT model.
Suppose, with \(\beta = 0.1\), the trained model makes the chosen answer 2 nats more likely than the reference does (\(\log \pi_\theta/\pi_0 = +2\)) and the rejected answer 1 nat less likely (\(-1\)). The margin is \(0.1 \cdot (2 - (-1)) = 0.3\), and \(\mathcal{L}_{\text{DPO}} = -\log\sigma(0.3) = 0.554\). At the start of training both log-ratios are 0, so the loss is \(-\log\sigma(0) = \ln 2 = 0.693\).
Real recipes side by side
Figure 2 puts eight published recipes in one grid. Each row is a stage; each cell says whether the vision encoder, the connector and the LLM train in that stage. The differences are large, and every one of these models is good. There is no single correct recipe, only trade-offs.
How to read Figure 2
Look down the “Vision” column. LLaVA-1.5 and Prismatic never train the encoder; MM1 and Molmo train it from the start; LLaVA-OneVision and InternVL 2.5 start frozen and unfreeze it later; Qwen2-VL trains it first and freezes it at the end. Now look at the “Connector” column of the first row: in most recipes it trains alone in stage 1, but Molmo, MM1 and Idefics2 train it together with everything else and protect the pretrained parts with small learning rates or LoRA instead.
What to freeze, when, and why
There are four reasons to freeze a part, and you should be able to name them.
- Protection. A random part upstream sends noise; freezing the pretrained parts until it is trained protects them (stage 1).
- Memory. A frozen weight needs only its BF16 value (2 bytes). A trained weight with mixed-precision AdamW needs 16 bytes: BF16 weight 2, BF16 gradient 2, FP32 master copy 4, Adam \(m\) 4, Adam \(v\) 4.
- Compute. No weight gradients for frozen parts (formula 186.5), and no backward pass at all through parts that sit before every trainable part.
- Generality. A part trained on much broader data than yours (the vision encoder, trained on billions of images) can only lose generality on your narrow data.
And there are two reasons to unfreeze: the frozen part lacks a skill you need (an encoder trained at 224 px cannot read small text in a 1344 px page), or the data is good and large enough to improve it.
Memory, by formula
Let \(N_{\text{all}}\) be all parameters and \(N_{\text{train}}\) the trainable ones. Before activations:
(2 bytes for every weight, plus 14 more for each trainable one.) For LLaVA-1.5-7B: stage 1 needs \(2 \times 7.06\text{B} + 14 \times 21.0\text{M} = 14.4\) GB; stage 2 needs \(2 \times 7.06\text{B} + 14 \times 6.76\text{B} = 108.8\) GB, more than one 80 GB GPU. This is exactly why LLaVA-1.5 uses DeepSpeed ZeRO-2 for stage 1 and ZeRO-3 (which shards weights, gradients and optimizer states across GPUs) for stage 2 [Certain]. Activation memory comes on top and grows with the number of visual tokens. Try it in Figure 3.
How to read Figure 3
The default is stage 1: only the connector trains, and the bar is almost all blue (frozen BF16 weights, 14.4 GB). Switch the LLM to Full: this is stage 2, and the orange part (optimizer states) dominates, 108.8 GB. Switch to LoRA with rank 8: the LLM gains only 20.0M trainable weights and the total drops to 14.7 GB. That is the memory argument for Idefics2’s LoRA recipe. Training the vision encoder adds about 4.2 GB (14 bytes × 303.5M).
What the evidence says about the vision encoder
| Study | Setting | Finding |
|---|---|---|
| Prismatic | LLaVA-1.5 data (665K), 7B | Fine-tuning the encoder hurt, most on localization (RefCOCO, OCID-Ref) [Certain] |
| Cambrian-1 | 23 encoders, 1.2M adapter + 737K SFT | Unfreezing helped almost everywhere [Certain] |
| ShareGPT4V | 1.2M detailed GPT-4V-style captions in pretraining | Unfreezing the last half of the encoder blocks was best, with a learning rate of \(2\times10^{-5}\) [Certain] |
| MM1 | SFT at different resolutions | Frozen encoder better at low resolution (+2.2 points); unfrozen better at 1344 px (+2.9 points) [Certain] |
| Idefics2 | Fully autoregressive, 8B | Full unfreezing diverged; LoRA on the backbones was stable and +12.9 points over frozen [Certain] |
The pattern: unfreezing the encoder helps when (a) the data is large and detailed, (b) the input differs from what the encoder was trained on (higher resolution, documents), and (c) its learning rate is small or it uses LoRA. It hurts when the data is small and noisy. ShareGPT4V puts it simply: earlier works kept the encoder frozen because their captions were low quality, and training on them “might degrade its visual knowledge extraction ability” [Certain].
See it in a toy
Figure 4 is a toy model small enough to train in your browser, but with the same structure. The “LLM” is a matrix \(W\), pretrained as \(W_0\). The “connector” is a matrix \(P\), started at random. The image task needs the right \(P\) and also a small change of the LLM (\(W_0 \to W_0 + \Delta\)), just as real VLMs need the LLM to learn new skills. We measure two things: the image-task loss, and the drift \(\lVert W - W_0 \rVert^2 / \lVert W_0 \rVert^2\), a stand-in for how much the LLM’s text behavior has changed. The text-only share \(f\) adds the loss \(\lVert W - W_0\rVert^2\) with weight \(f\), like mixing in text data (formula 186.8).
How to read Figure 4
With the defaults (LLM rate 0.01, no text data), staged training ends with image loss 0.016 and drift 0.026. Joint training ends with a worse image loss (0.046) and 4.3 times more drift (0.114); its drift peaks at 0.372 early on, when the random connector pushes the LLM around. Raise the LLM rate to 0.06 in joint mode: it diverges in the first steps, as Idefics2 saw at full scale. Staged training stays stable. Then raise the text-only share: drift falls in both recipes, and the image loss rises a little. That is the trade-off of formula (186.8).
- input: pretrained encoder \(E_\phi\), pretrained LLM \(f_\omega\), new connector \(C_\psi\), data sources, GPU budget
- if \(C_\psi\) is random then train only \(\psi\) on captions with a high rate (stage 1), or give \(\psi\) a 10× higher rate and a short warmup in a joint stage
- if the target images differ from the encoder’s training images (resolution, documents, medical, video frames) and there are millions of good samples then unfreeze \(\phi\) with \(\eta_\phi \approx \eta_\omega / 5\), or LoRA
- else keep \(\phi\) frozen // cheaper, and safer on small or noisy data
- if memory \(2N_{\text{all}} + 14N_{\text{train}}\) does not fit then shard (ZeRO-3 / FSDP), or switch \(\omega\) (and \(\phi\)) to LoRA
- mix 5–20% text-only data into every stage that trains \(\omega\) // formula (186.8); exact share is a tuning choice
- for SFT, mask all non-answer tokens (186.2) and add format prompts for short-answer data
- after SFT, if hallucination or reasoning is the weak point then add DPO / MPO on preference pairs or RL with verifiable rewards, with a small rate and a reference model
- evaluate text-only benchmarks after every stage to catch forgetting early
Putting it together: a stage switcher in PyTorch
All stages share one training loop. Only two things change per stage: which parameters have requires_grad, and the optimizer’s parameter groups with their learning rates.
- input: model \((\phi, \psi, \omega)\), a list of stages, each with a learning rate per part (or “frozen”), data and number of steps
- for each stage \(s\) do
- for each part \(j\): set
requires_grad\(\leftarrow (\eta_j^{(s)} \ne 0)\); put frozen parts with dropout in eval mode - build a new AdamW with one group per trainable part, learning rate \(\eta_j^{(s)}\) // formula (186.3), (186.6)
- build a new warmup + cosine schedule for this stage
- for each batch: build labels with −100 on non-target tokens; loss (186.2); backward; clip; step
- save a checkpoint; evaluate image and text-only benchmarks
- end for
The code below is a tiny but complete version. The “vision encoder” is an MLP over 16 patch vectors, the “LLM” a 2-layer causal transformer. It runs on a CPU in a few seconds and prints the same numbers every time:
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, D, DV, NP = 100, 64, 32, 16 # vocab, LLM width, vision width, image tokens class TinyVLM(nn.Module): def __init__(self): super().__init__() self.vision = nn.Sequential(nn.Linear(48, DV), nn.GELU(), nn.Linear(DV, DV)) # "pretrained" encoder self.connector = nn.Sequential(nn.Linear(DV, D), nn.GELU(), nn.Linear(D, D)) # new, random self.embed = nn.Embedding(V, D) layer = nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0.0, batch_first=True, norm_first=True) self.blocks = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False) self.head = nn.Linear(D, V, bias=False) def llm_params(self): return [*self.embed.parameters(), *self.blocks.parameters(), *self.head.parameters()] def forward(self, patches, ids): img = self.connector(self.vision(patches)) # (B, NP, D): visual tokens h = torch.cat([img, self.embed(ids)], dim=1) # formula (186.1): image first, then text T = h.size(1) causal = torch.triu(torch.full((T, T), float("-inf")), diagonal=1) h = self.blocks(h, mask=causal) return self.head(h[:, NP - 1:-1]) # position i predicts text token i # which part trains in which stage, and with which learning rate (None = frozen) STAGES = { "1-align": {"vision": None, "connector": 1e-3, "llm": None}, "2-pretrain": {"vision": 2e-6, "connector": 1e-5, "llm": 1e-5}, "3-sft": {"vision": None, "connector": 1e-5, "llm": 1e-5}, } def set_stage(model, stage): parts = {"vision": list(model.vision.parameters()), "connector": list(model.connector.parameters()), "llm": model.llm_params()} groups = [] for name, params in parts.items(): lr = STAGES[stage][name] for p in params: p.requires_grad_(lr is not None) # frozen: no gradient, no optimizer state if lr is not None: groups.append({"params": params, "lr": lr, "name": name}) return torch.optim.AdamW(groups, betas=(0.9, 0.95), weight_decay=0.0) # formula (186.3) def masked_loss(logits, ids, n_prompt): labels = ids.clone() labels[:, :n_prompt] = -100 # formula (186.2): no loss on the prompt return F.cross_entropy(logits.reshape(-1, V), labels.reshape(-1), ignore_index=-100) model = TinyVLM() patches = torch.randn(8, NP, 48) ids = torch.randint(0, V, (8, 12)) for stage in STAGES: opt = set_stage(model, stage) # a NEW optimizer per stage n_train = sum(p.numel() for g in opt.param_groups for p in g["params"]) n_all = sum(p.numel() for p in model.parameters()) before = model.vision[0].weight.detach().clone() for step in range(30): loss = masked_loss(model(patches, ids), ids, n_prompt=4) opt.zero_grad(set_to_none=True) loss.backward() opt.step() moved = (model.vision[0].weight - before).abs().max().item() print(f"{stage:11s} trainable {n_train:6d}/{n_all} ({100*n_train/n_all:4.1f}%) " f"loss {loss.item():.3f} vision moved {moved:.1e}") # 1-align trainable 6272/121664 ( 5.2%) loss 4.577 vision moved 0.0e+00 # 2-pretrain trainable 121664/121664 (100.0%) loss 4.493 vision moved 6.2e-05 # 3-sft trainable 119040/121664 (97.8%) loss 4.412 vision moved 0.0e+00
The output (PyTorch 2.8, CPU) confirms the mechanics: in stage 1 and stage 3 the vision weights do not move at all, and in stage 2 they move only slightly because of their small rate. The data here is random, so the loss stays near \(\ln 100 = 4.61\); the point is the freezing logic, not the learning. In a real run you would also use a warmup + cosine schedule per stage, gradient clipping, BF16, and real image–text data, and you would put frozen modules with dropout or batch norm in eval() mode.
Where this is used
| Setting | Typical staging | Why |
|---|---|---|
| Building a general open VLM (LLaVA, Qwen-VL, InternVL families) | Connector warmup → full pretraining with small encoder rate → SFT → optional DPO / MPO / RL | The recipes of Figure 2; budget decides how much is unfrozen |
| Adapting a VLM to documents, charts, screenshots | Continue stage 2 at higher resolution with OCR / PDF data, encoder unfrozen at a low rate or with LoRA; then task SFT | The frozen encoder cannot read small text (MM1: unfreezing helps at 1344 px) |
| Video understanding (video QA, captioning, action recognition) | Start from an image VLM; a short video SFT phase with frames as image tokens, often with the encoder frozen | LLaVA-OneVision gets its video skill from a 1.6M-sample final mixed phase [Certain]; frozen encoders keep frame features stable [Likely] |
| Temporal grounding and key-moment detection in long video | Freeze the encoder; train the connector and the LLM (often LoRA) on data with timestamps in the answers | The new skill is in the language side: writing “from 0:42 to 0:51” [Likely] |
| Video retrieval and Video-RAG | Keep the retrieval encoder frozen (CLIP / SigLIP embeddings in the index); fine-tune only the answering VLM | Changing the encoder would force you to re-embed the whole video index [Likely] |
| A small product fine-tune on one GPU (for example, defect inspection) | Freeze encoder, LoRA on the LLM, train connector; one SFT stage; maybe DPO against hallucinated defects | Memory (186.11) and small data; a frozen encoder generalizes better [Likely] |
| Vision-language-action models for robots | Start from a trained VLM, then train on robot trajectories with action tokens | The VLM stages give world knowledge; the last stage adds a new output type (see session 220) [Likely] |
Common mistakes
- Training everything at once with one high learning rate. A random connector plus a trainable LLM is the classic way to get loss spikes and a model that forgot how to write (Figure 4). Train the connector first, or give each part its own rate.
- Reusing the optimizer across stages. Adam’s \(m\) and \(v\) from stage 1 belong to the connector only; parameters added in stage 2 have no state, and the schedule is wrong. Build a new optimizer and schedule per stage.
- “Frozen” parts that still change. Setting
requires_grad=Falsedoes not stop dropout or batch-norm running statistics; and passing frozen weights to the optimizer with weight decay can still shrink them in some setups. Put frozen modules ineval()and leave them out of the optimizer. - No loss mask. Training on the prompt and the image placeholder tokens wastes capacity and teaches the model to generate user questions. Label every non-answer token −100.
- Only image data in stages that train the LLM. Text-only benchmarks drop. Mix in text data (MM1 used 10% in pretraining) and check text benchmarks after each stage.
- Unfreezing the vision encoder on small or noisy data. Its features get worse, especially for localization (Prismatic). Unfreeze with a 5–10× smaller rate, only with enough good data, or use LoRA.
- Short-answer SFT only. The model scores well on VQA benchmarks but answers users in one word. Add format prompts and long-form chat data.
- Noisy SFT data. A few thousand bad samples can cause repetition (InternVL 2.5). Filter SFT data harder than pretraining data.
Interview questions
Why does LLaVA train only the projector in stage 1?basic
If only 0.3% of weights train in stage 1, is each step 300 times cheaper?core
Should you freeze the vision encoder? Give evidence both ways.core
What changes from stage 1 to stage 2?basic
Why do VLM recipes mix text-only data into multimodal training?core
Idefics2 could not stabilize full unfreezing. What did they do and why does it work?advanced
Design the SFT stage for a VLM. What makes it work?core
What does a preference or RL stage add to a VLM after SFT?core
How can you create rejected answers for VLM preference data without humans?advanced
Stage 2 of LLaVA-1.5-7B does not fit on one 80 GB GPU. Why, and what are your options?advanced
Prismatic says stage 1 is unnecessary; Cambrian-1 says it helps. How do you reconcile them?advanced
Exercises
- LLaVA-1.5-13B uses the same vision tower (303,507,456), an MLP from 1024 to 5120, and a 13,015,864,320-parameter LLM. Compute the connector size and the trainable fraction in stage 1.
- LLaVA-OneVision-0.5B uses batch 512 in stage 1 on 558K samples for one epoch. How many steps is that?
- Using formula (186.11), how much memory does stage 2 of LLaVA-1.5-13B need (LLM and connector trained, encoder frozen), before activations? Check your answer in Figure 3.
- Use MM1’s formula (186.7) to estimate the peak learning rate for \(N = 10^9\).
- Your VLM was trained with a frozen 384 px encoder. You must now read small text in 1344 px invoices, and you have 2M good document samples. Which stage do you add, and which learning rates do you use?
- In Figure 4, choose joint training with LLM rate 0.03. Read the peak drift with text-only share 0% and with 30%. What does this tell you?
- Compute the DPO loss (186.9) with \(\beta = 0.1\) when \(\log \pi_\theta/\pi_0\) is +1 for the chosen and +1 for the rejected answer. Why does the loss not change from its starting value?
Answerscheck
Key takeaways
- A VLM joins two pretrained parts with a random connector (186.1) and trains them all with masked next-token loss (186.2); stages differ in which parts train and how fast (186.3).
- Stage 1 (alignment) trains only the connector with a high rate: 0.30% of LLaVA-1.5-7B, about 2,180 steps. It saves memory, but compute only drops to about \(4N\) per token (186.5).
- Stage 2 unfreezes the LLM and often the encoder, with per-part rates (encoder 5–10× lower, 186.6), more data, higher resolution and some text-only data (186.8). LoRA is a stable alternative to full unfreezing.
- Stage 3 is SFT on chat data with loss only on answers; data quality and answer format matter most. Stage 4 adds DPO / MPO (186.9, 186.10) or RL against hallucination and for reasoning.
- Freeze for protection, memory (2 vs 16 bytes per weight, 186.11), compute and generality; unfreeze when the part lacks a needed skill and there is enough good data. The ablations disagree, so test on your own data.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Laurençon et al. (2024). What matters when building vision-language models?
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Liu et al. (2023). Visual Instruction Tuning
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Laurençon et al. (2024). What matters when building vision-language models?
- Karamcheti, Nair, Balakrishna, Liang, Kollar & Sadigh (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
- Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Chen et al. (2023). ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
- McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
- Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Chen et al. (2024). Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (InternVL 2.5)
- Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
- Wang et al. (2024). Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
- Yu et al. (2023). RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation
- Laurençon et al. (2023). OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents