LLM & VLM courseBlog
185Track 7: Vision and VLMs · session 23 of 38

build a mini-VLM

In this lab you build a small vision-language model from three parts: a frozen SigLIP image encoder, a two-layer MLP projector, and a frozen 135M language model (SmolLM2). You train only the projector on Flickr8k captions, then add LoRA and teach the model to answer questions about CIFAR-10 images, and you check every step with a wrong-image control. At the end you read the attention from text tokens to image tokens and see how little of it goes to the image.

Time: 3 to 4 hours (reading 1.5 hours; the scripts run in about 30 to 40 minutes on a laptop GPU)Level: Intermediate · hands-on labBefore this: Sessions 177 to 183 (VLM anatomy, connectors, fusion, case studies, visual token efficiency), session 57 (LoRA), the linear-probe lab (171); Python and PyTorch

What you will learn

  • How image features enter a language model: patch tokens, a 2 × 2 merge, an MLP, and inputs_embeds next to text embeddings
  • How to build the training sequence with the chat template and a label mask, so that the loss is on the answer only
  • Stage 1: training only the projector on captions, and why a wrong-image control is needed to show that the model uses the image
  • Stage 2: instruction tuning with LoRA on a small VQA set, its parameter count, and the forgetting it causes
  • Whether the alignment stage is needed: our single-stage run matched or beat two stages, as the Prismatic study found
  • How to read attention from text to image tokens per layer, and why sinks make raw attention maps misleading

Why this matters

In the last sessions you read how vision-language models are built: a vision encoder, a connector, and a language model, trained in stages. In this lab you build one. You take a frozen SigLIP image encoder, a small frozen language model (SmolLM2-135M-Instruct), and a two-layer MLP between them. You train the MLP alone on image–caption pairs, then add LoRA adapters and teach the model to answer questions about images. At the end you open the model and look at where its text tokens attend in the image.

This is the LLaVA recipe at toy scale. LLaVA connected a frozen CLIP encoder to a language model with a trainable projection, trained only the projection on 595K image–caption pairs, and then trained the projection and the language model on 158K instruction examples [Certain]. LLaVA-1.5 replaced the linear projection with a two-layer MLP and used 558K and 665K examples for the two stages [Certain] (session 181). The same parts at almost the same size are inside a real model: SmolVLM-256M uses a SigLIP B/16 encoder of about 93M parameters and SmolLM2-135M as its language model [Certain]. So what you build here is not a cartoon; it is a small cousin of a released model.

The lab answers four practical questions that papers often skip:

  1. What exactly goes into the language model? You will build the input sequence by hand: text embeddings, image embeddings, and a label mask (Figure 1).
  2. Can a tiny MLP, trained alone, make a frozen LLM “see”? You will train it and measure the caption loss with the right image, a wrong image and no image (Figure 2).
  3. What does instruction tuning with LoRA add on top? You will measure question-answering accuracy before and after, with and without Stage 1, and with shuffled images (Figure 3).
  4. Where do the text tokens look? You will read the attention weights from text positions to image positions, layer by layer (Figures 4 and 5).

We ran every script on this page ourselves, on an Apple laptop GPU (PyTorch MPS backend) that was shared with other jobs at the time, and every number in the tables and figures is our measurement. On a CUDA GPU the losses and accuracies will be close but not identical, because the floating-point kernels differ.

Setup

Hardware. Any machine with a GPU and 16 GB of memory. Nothing here needs more than one small GPU: the largest model has 135M parameters, and the vision encoder runs only once over the data.

pip install torch transformers datasets numpy

We used PyTorch 2.15 and transformers 5.23.

The three parts. We read the sizes from the model config files and counted the parameters by loading the models.

Part (Hub id)What it isSizeStage 1Stage 2
Vision encoder
google/siglip-base-patch16-224
ViT-B/16, 12 layers, width 768, trained with the sigmoid image–text loss (session 167)92,884,224 parameters (7,087,104 of them in the pooling head, which we do not use)frozenfrozen
Projector (ours)\(2 \times 2\) patch merge, then Linear(\(3072 \rightarrow 576\)), GELU, Linear(\(576 \rightarrow 576\))2,102,400 parameterstrainedtrained
Language model
HuggingFaceTB/SmolLM2-135M-Instruct
Llama-style decoder: 30 layers, width 576, 9 query heads and 3 key-value heads (grouped-query attention), vocabulary 49,152, tied input and output embeddings134,515,008 parametersfrozenfrozen + LoRA (460,800)

The data. Stage 1 uses Flickr8k (jxie/flickr8k on the Hub): 8,000 photos of people and animals, each with 5 captions written by people, split into 6,000 training, 1,000 validation and 1,000 test images. Stage 2 uses a small visual question answering (VQA) set that we generate from CIFAR-10: 500 images per class for training and 100 per class for testing, three questions per image, so 15,000 training and 3,000 test questions. The answers come from the class labels, so they are always correct and easy to score. Real instruction data is far more varied; this set is chosen so that the result is easy to measure on a laptop.

The table you will fill. Copy it into your notes and fill it with your own run. Our values are in Steps 3 to 5.

MeasurementBefore trainingAfter Stage 1After Stage 2
Caption loss, right image (nats per token)
Caption loss, wrong image
Caption loss, no image—
VQA accuracy: “What is in this image?”—
VQA accuracy: “Animal or vehicle?”—
VQA accuracy: “Is there a … ?”—
Attention on image tokens at the answer position—

Step 1: what goes into the language model

A language model only takes one kind of input: a sequence of vectors, one per position, each of width \(d_{\text{LLM}} = 576\) for SmolLM2. Normally these vectors come from the embedding table: token id in, row of the table out. A VLM of the LLaVA type adds a second source. The image is turned into a few vectors of the same width, and they are placed in the sequence where text embeddings would go. The language model does not know that they are not words. This design is called the unified sequence or adapter approach (session 179).

The image path has three steps. First, SigLIP cuts the \(224 \times 224\) image into \(16 \times 16\) patches and returns one vector per patch:

\[ Z = f_{\text{SigLIP}}(x) \in \R^{N \times d_v}, \qquad N = \left(\frac{224}{16}\right)^2 = 14^2 = 196, \quad d_v = 768 \tag{185.1} \]

Second, we merge each \(2 \times 2\) block of neighbouring patches into one vector by putting the four vectors side by side. This is the “pixel shuffle” (also called space-to-depth) that SmolVLM and InternVL use to cut the number of image tokens [Certain] (session 183). It loses no information, it only reshapes:

\[ \tilde{\mathbf{z}}_{(i,j)} = \big[\, \mathbf{z}_{(2i,2j)};\ \mathbf{z}_{(2i,2j+1)};\ \mathbf{z}_{(2i+1,2j)};\ \mathbf{z}_{(2i+1,2j+1)} \,\big] \in \R^{4 d_v}, \qquad 0 \le i, j \lt 7 \tag{185.2} \]

Third, a two-layer MLP maps each merged vector to the LLM’s width:

\[ \mathbf{h}_{(i,j)} = W_2\, \operatorname{GELU}\!\big(W_1 \tilde{\mathbf{z}}_{(i,j)} + \mathbf{b}_1\big) + \mathbf{b}_2, \qquad W_1 \in \R^{576 \times 3072},\ W_2 \in \R^{576 \times 576} \tag{185.3} \]
Read the symbols
\(x\)
the input image, resized to \(224 \times 224\)
\(Z\)
SigLIP’s output: one row \(\mathbf{z}_{(r,c)}\) of 768 numbers per patch at row \(r\) and column \(c\) of the \(14 \times 14\) grid
\(\tilde{\mathbf{z}}_{(i,j)}\)
the four patches of block \((i,j)\) concatenated: 3,072 numbers; there are \(7 \times 7 = 49\) blocks
\(W_1, \mathbf{b}_1, W_2, \mathbf{b}_2\)
the projector’s weights and biases: the only weights trained in Stage 1
\(\mathbf{h}_{(i,j)}\)
one image token: a vector of 576 numbers that the LLM reads like a word embedding

Why merge? Every image token costs the LLM as much compute as a word, in every layer. With 196 tokens per image, the image would be about 90% of a short training sequence. LLaVA-1.5 passes all 576 patch tokens of its 336 px CLIP encoder to the LLM [Certain]; small models such as SmolVLM merge aggressively because the LLM is the expensive part [Certain]. Here the merge makes the lab four times cheaper and still keeps a \(7 \times 7\) spatial map, which is enough to see where the model looks.

Then we build the sequence. SmolLM2-Instruct was trained with the ChatML template: every turn starts with <|im_start|> and the role, and ends with <|im_end|>. We put the 49 image tokens right after the user header, before the question:

\[ S = \big[\ E(\text{header}),\ \ \mathbf{h}_{1}, \dots, \mathbf{h}_{49},\ \ E(\text{question} + \text{turn switch}),\ \ E(\text{answer} + \text{end})\ \big] \tag{185.4} \]
Read the symbols
\(E(\cdot)\)
the frozen token embedding table of SmolLM2, applied to the token ids of a piece of text
header
<|im_start|>user and a newline: 3 tokens
\(\mathbf{h}_1, \dots, \mathbf{h}_{49}\)
the image tokens from (185.3), in row order of the \(7 \times 7\) grid
turn switch
<|im_end|>, a newline, <|im_start|>assistant and a newline: 6 tokens
end
<|im_end|>, which SmolLM2 uses as its end-of-sequence token (id 2)

The model is trained with the normal next-token loss, but only on the answer. Every other position gets the label −100, which PyTorch’s cross-entropy skips. We do not want the model to spend capacity predicting the chat template or the question, and it cannot predict image tokens at all, because they are not in its vocabulary. Figure 1 shows the whole path.

image tokenstext tokens (no loss)answer tokens (loss) and trained parts
Figure 1. Top: the three parts; a thick border means trained in this stage, a dashed border means frozen. Bottom: the sequence the LLM sees for one example (formula 185.4), with the label of each block underneath (−100 means no loss). Token counts are the real SmolLM2 tokenizer counts.
How to read Figure 1

Click a block of the sequence to see what it is and where it comes from. Switch the stage: in Stage 1 only the projector is trained (2,102,400 parameters, under 1% of the whole model); in Stage 2 the LoRA matrices inside the LLM are trained too. In both stages the loss is only on the answer block. Notice the image share: the 49 image tokens are most of a short example (65% of the positions in the caption example, 69% in the question example). With 196 unmerged tokens they would be about 90%.

Algorithm 1: build one training example
  1. input: cached SigLIP features \(Z \in \R^{196 \times 768}\), question text, answer text
  2. \(H \leftarrow \text{Projector}(Z)\) // merge \(2 \times 2\) (185.2), MLP (185.3): \(49 \times 576\)
  3. \(P \leftarrow E(\text{header})\) // 3 tokens
  4. \(Q \leftarrow E(\text{question} + \text{turn switch})\)
  5. \(A \leftarrow E(\text{answer} + \text{end})\)
  6. \(S \leftarrow [P; H; Q; A]\) // formula 185.4
  7. labels \(\leftarrow [-100] \times (|P| + 49 + |Q|)\) followed by the answer token ids
  8. return \(S\), labels // batches are padded on the right with mask 0 and label −100
Worked example: count the parameters and the positions

Projector. \(W_1\): \(3072 \times 576 = 1{,}769{,}472\) weights plus 576 biases; \(W_2\): \(576 \times 576 = 331{,}776\) plus 576. Total \(1{,}769{,}472 + 576 + 331{,}776 + 576 = 2{,}102{,}400\). Without the merge (input 768) it would be \(442{,}368 + 576 + 331{,}776 + 576 = 775{,}296\).

Share trained in Stage 1. All parameters: \(92{,}884{,}224 + 2{,}102{,}400 + 134{,}515{,}008 = 229{,}501{,}632\). We train \(2{,}102{,}400\), which is 0.92% of them.

Positions. For the question “What is in this image? Answer with one word.” the tokenizer gives 3 header tokens, then 49 image tokens, then 17 tokens for the question and the assistant header: 69 positions before the answer. The answer “cat” is 1 token plus <|im_end|>. Image tokens are \(49 / 69 = 71\%\) of the context. With all 196 patches they would be \(196 / 216 = 90.7\%\).

Step 2: run the vision encoder once

The vision encoder is frozen in both stages, so its output for an image never changes. Run it once over all images and store the result, exactly as in the linear-probe lab (session 171). Training then never touches SigLIP; each step reads \(196 \times 768\) numbers per image from disk.

Algorithm 2: extract and cache the image features
  1. input: images, the SigLIP vision tower in eval() mode, its image processor
  2. for each batch of 64 images do
  3.   resize to \(224 \times 224\) and normalize with the processor // SigLIP maps pixels to \([-1, 1]\)
  4.   \(Z \leftarrow\) last_hidden_state with gradients off // \(196 \times 768\), after SigLIP’s final LayerNorm
  5.   store \(Z\) in FP16 together with the captions or labels, in the same order
  6. end for

We keep the patch tokens, not the pooled vector. The pooled output of SigLIP’s attention-pooling head is one vector per image, which is good for retrieval but throws away where things are. A VLM needs the spatial map, so LLaVA-style models read the patch tokens of a late layer [Certain]. (LLaVA uses the second-to-last layer of CLIP [Likely]; we use the last layer of SigLIP for simplicity. Exercise 5 asks you to compare.)

Worked example: how big is the cache?

Each image gives \(196 \times 768 = 150{,}528\) numbers, 301 KB in FP16. Flickr8k’s 8,000 images take \(8{,}000 \times 150{,}528 \times 2 = 2.41\) GB; the 6,000 training images alone 1.81 GB, which matches the file we saved. The 6,000 CIFAR-10 images take another 1.81 GB. Load the big files with torch.load(..., mmap=True) so the operating system reads only the rows each batch needs.

Step 3: Stage 1, align the projector on captions

At the start the projector is random, so the image tokens are noise to the LLM. Stage 1 has one job: make the image tokens mean something to the frozen LLM. The task is captioning, because a caption is the simplest text that depends on the image. The question is always “Describe the image.” and the answer is one of the five human captions. The LLM and SigLIP stay frozen; gradients flow back through the frozen LLM into the projector, but only the projector’s weights change. LLaVA calls this stage “pre-training for feature alignment” [Certain].

The loss is the average next-token cross-entropy over the answer tokens of a batch:

\[ \mathcal{L}(\theta) = -\frac{1}{\sum_b |A_b|} \sum_{b} \sum_{t \in A_b} \log p_{\text{LLM}}\big(s_{b,t+1} \mid S_{b,\le t}\big), \qquad S_b = \big[P;\ \text{Proj}_\theta(Z_b);\ Q;\ A_b\big] \tag{185.5} \]
Read the symbols
\(\theta\)
the projector’s weights, the only trainable parameters in Stage 1
\(b\)
an example in the batch (32 per batch)
\(A_b\)
the set of positions \(t\) whose next token \(s_{b,t+1}\) belongs to the answer (caption tokens and the final <|im_end|>); these are the positions whose label is not −100
\(S_{b,\le t}\)
the sequence up to position \(t\): header, image tokens, question, and the answer so far
\(p_{\text{LLM}}\)
the frozen LLM’s next-token probability

One practical detail saves a lot of memory. The full logits tensor for a batch is \(B \times L \times V\) numbers: with 32 examples, about 75 positions and a vocabulary of 49,152, that is 118 million numbers, 472 MB in FP32, plus the same again for its gradient. But only the answer positions need logits. So we run the LLM body, pick the hidden states at the answer positions, and apply the output layer only there. The loss is the same; the memory for logits falls by a factor of about five.

Algorithm 3: Stage 1 training loop
  1. input: cached features of 6,000 training images, 5 captions each; projector \(\theta\) (random); frozen LLM
  2. freeze every LLM parameter (requires_grad = False); optimizer AdamW on \(\theta\) only, learning rate \(10^{-3}\), weight decay 0
  3. for epoch in 1, 2 do // each epoch: every image once, with one of its 5 captions chosen at random
  4.   for each batch of 32 do
  5.     build the sequences and labels (Algorithm 1) with question “Describe the image.”
  6.     \(\mathcal{L} \leftarrow\) formula (185.5); backpropagate; AdamW step // warmup 20 steps, then cosine decay
  7.     every 25 steps: validation loss on 256 held-out images
  8.   end for
  9. end for // \(2 \times 6{,}000 / 32 = 375\) steps

This is close to LLaVA-1.5’s Stage 1 recipe (AdamW, learning rate \(10^{-3}\), cosine decay with warmup, no weight decay, one epoch [Certain]), with a smaller batch (32 against 256) because our dataset is 93 times smaller (6,000 against 558K images). The run took 931 seconds on our shared laptop GPU.

Does the model use the image? The shuffled-image control

A falling loss is not yet proof. The projector could learn to say “this is a Flickr caption” (short sentences, “A dog …”, “A man in a …”) without passing any content of the photo. To separate the two, evaluate the same captions three ways: with the right image, with a wrong image (every image swapped with its neighbour in the list), and with no image at all (the image tokens removed). The difference between wrong and right is what the image content is worth:

\[ \Delta_{\text{img}} = \mathcal{L}_{\text{wrong image}} - \mathcal{L}_{\text{right image}}, \qquad \frac{\operatorname{PPL}_{\text{wrong}}}{\operatorname{PPL}_{\text{right}}} = e^{\Delta_{\text{img}}} \tag{185.6} \]
Read the symbols
\(\mathcal{L}\)
the mean caption loss (185.5) over the 1,000 validation images, in nats per token
\(\operatorname{PPL} = e^{\mathcal{L}}\)
perplexity: roughly, among how many equally likely tokens the model is choosing at each step

Our measurements on the 1,000 validation images (first caption of each):

InputLoss before Stage 1Loss after Stage 1Perplexity after
Right image4.2772.60713.6
Wrong image (swapped)4.2743.40230.0
No image4.3534.35377.7

Read the table in three steps.

  • No image gives exactly the same loss before and after (4.353). This is a good sanity check: without image tokens, nothing in the computation changed, because the LLM is frozen.
  • Wrong image falls from 4.274 to 3.402, by 0.876 nats. This part of the gain needs an image but not the image: the projector learned to signal “write a short photo caption”. That is style, not seeing.
  • Right image falls further, to 2.607. The content gain (185.6) is \(\Delta_{\text{img}} = 3.402 - 2.607 = 0.795\) nats per token: the perplexity with the right image is \(e^{0.795} = 2.2\) times lower. A frozen 135M LLM now reads a photo through 49 vectors and 2.1M trained weights.

Before training, right and wrong images give almost the same loss (4.277 against 4.274): a random projector carries no usable content. Figure 2 shows the whole run.

training loss (thin: per batch, thick: smoothed)validation loss, 256 images, right imagefinal validation loss with wrong images
Figure 2. Our Stage 1 run: 375 steps of 32 captions, only the projector trained. The loss is in nats per caption token (formula 185.5). The dashed line is the loss with every image swapped for another one after training: the gap between it and the validation curve is the content of the image (formula 185.6).
How to read Figure 2

Move the slider. Most of the drop happens in the first 50 steps: the validation loss goes from 4.18 to 2.91. It is already below the dashed “wrong image” line (3.40) at the step-25 check (3.07), so the projector starts to pass image content very early, not only style. The slow part after step 50 adds another 0.37 nats. The learning rate (in the readout) warms up for 20 steps and then follows a cosine down to zero. The validation curve flattens near 2.54; this suggests that a third epoch would add little at this data size [Speculation].

Here are the first five generated captions on test images (greedy decoding, never seen in training), next to the first human caption:

Our model (after Stage 1)Human reference
Two dogs are playing in the snow .The dogs are in the snow in front of a fence .
A dog is swimming in a pool .a brown and white dog swimming towards some in the pool
A group of people wearing bright colors and hats are dancing in a parade .A man and a woman in festive costumes dancing .
A woman is sitting on a bench in a park .A couple of people sit outdoors at a table with an umbrella and talk .
A football player is wearing a helmet .A man is wearing a Sooners red football shirt and helmet .

The captions are fluent and mostly right about the main subject (dogs, snow, pool, dancing, a football helmet), and vague or wrong about details (one bench instead of a table with an umbrella). The space before the full stop is copied from Flickr8k’s tokenized captions: the model learned the style of its data exactly.

Step 4: Stage 2, instruction tuning with LoRA

After Stage 1 the model can see, but it cannot follow instructions about what it sees. Ask it “What is in this image? Answer with one word.” and it answers like a captioner. Our Stage-1 model’s first five answers on CIFAR-10 test images were “The image is a dog”, “The image is a beach”, “The dog is a toy”, “The image is a car” and “The bird is in the”. Some are even right about the content, but none is one word, so a strict scorer gives 0.0%. To the yes/no question it answered “Yes” every time.

Stage 2 teaches the format and the skill of answering. LLaVA unfreezes the whole LLM here [Certain]. We use LoRA instead (session 57): the LLM’s weights stay frozen, and a small trainable low-rank update is added to some of its matrices. The projector keeps training too.

\[ W' \mathbf{x} = W \mathbf{x} + \frac{\alpha}{r}\, B A\, \mathbf{x}, \qquad A \in \R^{r \times d_{\text{in}}},\ \ B \in \R^{d_{\text{out}} \times r},\ \ B_{\text{init}} = 0 \tag{185.7} \]
Read the symbols
\(W\)
a frozen weight matrix of the LLM; we adapt the query projection q_proj (\(576 \rightarrow 576\)) and the value projection v_proj (\(576 \rightarrow 192\), because SmolLM2 has only 3 key-value heads of width 64) in all 30 layers
\(A, B\)
the trainable LoRA matrices; \(A\) starts random and \(B\) starts at zero, so at step 0 the model is exactly the Stage 1 model
\(r\)
the rank, 8 here
\(\alpha\)
a scale, 16 here, so the update is multiplied by \(\alpha / r = 2\)

The number of LoRA parameters is

\[ n_{\text{LoRA}} = L \cdot r \cdot \big[(d + d) + (d + d_{kv})\big] = 30 \cdot 8 \cdot \big[(576 + 576) + (576 + 192)\big] = 460{,}800 \tag{185.8} \]

With the projector, Stage 2 trains \(2{,}102{,}400 + 460{,}800 = 2{,}563{,}200\) parameters, about 1.1% of the model.

The VQA set

For every CIFAR-10 image we write three questions whose answers come from the label:

  • “What is in this image? Answer with one word.” The class name (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). Chance: 10%.
  • “Is this an animal or a vehicle? Answer with one word.” Six classes are animals and four are vehicles; on our balanced test set, always saying “animal” scores 60%.
  • “Is there a … in this image? Answer yes or no.” With probability one half the named class is the true one, otherwise a random other class. On our test set “yes” is the right answer 50.0% of the time.

That gives 15,000 training questions (5,000 images) and 3,000 test questions (1,000 different images). We train one epoch: 468 steps of 32, AdamW, learning rate \(2 \times 10^{-4}\) with 20 warmup steps and cosine decay, for the LoRA matrices and the projector together. The answer is scored by its first word, lower-cased, with punctuation removed. CIFAR-10 images are \(32 \times 32\) pixels, so SigLIP sees them blown up to \(224 \times 224\): a hard, blurry input, but every answer is checkable.

Algorithm 4: Stage 2 with LoRA
  1. input: the Stage 1 projector, the frozen LLM, 15,000 (image, question, answer) triples
  2. wrap q_proj and v_proj of every layer with LoRA (185.7): \(A \sim\) random, \(B = 0\)
  3. trainable \(\leftarrow\) all \(A, B\) and the projector // 2,563,200 parameters
  4. for each shuffled batch of 32 do
  5.   build sequences (Algorithm 1) with the question and the short answer; loss (185.5) on the answer and <|im_end|> only
  6.   AdamW step // learning rate \(2 \times 10^{-4}\), warmup, cosine
  7. end for
  8. evaluate with greedy decoding, left padding, at most 5 new tokens; also with every image swapped for another

Results

Our measurements on the 1,000 test images (1,000 questions of each type). The run took 601 seconds on our shared laptop GPU.

ModelWhat is it?Animal or vehicle?Is there a …?
After Stage 1 only0.0%0.1%50.0%
Stage 1, then Stage 294.3%99.0%86.6%
Same model, each image swapped with another8.9%51.4%49.4%
Stage 2 only (random projector, no Stage 1)94.4%99.5%95.0%
Best guess without the image10%60%50.0%

Four things to read from it.

  • Stage 2 works, and it uses the image. 94.3% on a ten-way question about 32-pixel images, from a 135M LLM that never saw CIFAR-10. With swapped images every question type falls to chance: 8.9% (chance 10%), 51.4% and 49.4%. For “animal or vehicle?” with random pairs, the expected score is \(0.6^2 + 0.4^2 = 52\%\), which is what we see. So the answers come from the image, not from the question.
  • Yes/no is the hardest. 86.6% against 94.3% for naming the class, although it seems easier. The model must compare the image with a class named in the question, and the answer token does not contain the class: a check, not a lookup. “Is there a … in the image?” is also the exact format of POPE, the standard test for object hallucination in VLMs [Certain] (session 193).
  • Stage 1 did not help here, and this surprised us. We expected the run without Stage 1 to be clearly worse. It was not: from a random projector, Stage 2 reached 94.4%, 99.5% and 95.0%, equal on the first two questions and 8.4 points better on yes/no. The Stage 1 projector only gave a faster start: smoothed training loss 1.05 against 1.97 after 25 steps, the same by step 100 (Figure 3, “Loss curves”). This agrees with the Prismatic study, which found that the projector-only alignment stage of LLaVA-1.5 is unnecessary and that single-stage training even improves the average score, while saving 20 to 25% of the training cost [Certain]. Our task is also easy for single-stage training: a closed set of short answers with 15,000 examples. Why the Stage 1 projector does worse on yes/no we do not know; one guess is that its caption-shaped features plus the Stage 1 model’s “Yes” habit make the yes/no check harder to learn in one epoch [Speculation]. This is one seed per setting; the 8.4-point gap is much larger than the test noise (about 1 point), but a different seed could change it [Likely].
  • Stage 2 makes the model forget how to caption. We measured the Flickr8k caption loss again with the Stage 2 weights: 3.572, against 2.607 after Stage 1, which is worse than Stage 1 with a wrong image (3.402). 468 steps of one-word answers taught the model that answers are one word. This is why real Stage 2 data are a mixture: LLaVA-1.5’s 665K examples combine conversations, detailed descriptions, short-answer VQA with a format prompt, and other tasks [Certain] (session 186).
Figure 3. Stage 2 results on 3,000 test questions about 1,000 held-out CIFAR-10 images. “Accuracy” compares five variants for one question type; the dashed line is the best score without looking at the image. “Loss curves” shows the Stage 2 training loss when the projector starts from Stage 1 or from random weights.
How to read Figure 3

Pick a question type. The bar “Stage 2 from the Stage 1 projector” is far above the dashed line; the bar “Stage 1 + 2, images swapped” sits on it: that gap is the proof that the model reads the image. Compare it with “Stage 2 from a random projector”: equal or higher. The two top bars are near zero for the first two questions because the untuned models answer in sentences, not in one word. Switch to “Loss curves”: the run from the Stage 1 projector starts lower and falls faster; read the readout for the values at a few steps.

Step 5: where do the text tokens look?

The image enters the LLM only through attention: a text position can use an image token only by attending to it. So the attention weights show, in a rough way, how much each text position reads from the image. In layer \(\ell\), head \(h\), the attention row of position \(t\) is a probability distribution over the positions \(j \le t\) (session 17). We average it over the 9 heads and add up the part that falls on the 49 image positions:

\[ m^{\ell}_{t} = \sum_{j \in \mathcal{I}} \bar{a}^{\ell}_{t j}, \qquad \bar{a}^{\ell}_{t j} = \frac{1}{H} \sum_{h=1}^{H} \softmax_j\!\Big(\frac{\mathbf{q}^{\ell,h}_t \cdot \mathbf{k}^{\ell,h}_j}{\sqrt{d_h}}\Big) \tag{185.9} \]
Read the symbols
\(m^{\ell}_t\)
the image attention mass: the share of position \(t\)’s attention in layer \(\ell\) that goes to image tokens, between 0 and 1
\(\mathcal{I}\)
the 49 positions of the image tokens (positions 3 to 51)
\(\mathbf{q}^{\ell,h}_t, \mathbf{k}^{\ell,h}_j\)
the query of position \(t\) and the key of position \(j\) in head \(h\); \(d_h = 64\)
\(H\)
the number of query heads, 9

Which position \(t\)? The one that predicts the word we care about: to study the word “dogs”, take the row of the token just before it. The baseline is the share of image positions in the context: if attention were spread evenly, \(m\) would equal that share (49 of 69 positions, 71%, for the class question). To get attention weights from Hugging Face, load the model with attn_implementation="eager"; the fused SDPA kernel does not return them.

Algorithm 5: attention from text to image
  1. load the LLM with eager attention; build the sequence (Algorithm 1) with the model’s own caption or the question
  2. one forward pass with output_attentions=True // 30 tensors of shape \((1, 9, L, L)\)
  3. average over heads; for each word, take the row \(t\) that predicts it
  4. \(m^\ell_t \leftarrow\) the sum of that row over the image positions (185.9); the 49 values themselves, reshaped to \(7 \times 7\), are the map
  5. compare with the baseline \(49 / L\), with \(L\) the number of context positions, and with the attention on the first token

Maps while the model writes a caption

Figure 4 uses the Stage 1 model on four Flickr8k test images. For each word of its own caption you see the attention to the \(7 \times 7\) image tokens. We looked at the raw maps first, and they are disappointing in an instructive way: for almost every word and layer band, the strongest image token is the first one, the top-left corner, in row 1, column 1, whatever the word. The maps for different words of the same caption are very similar (average cosine similarity between 0.90 and 0.98 over all layers, depending on the image). The first image position behaves like a small extra attention sink, the same effect as the large attention to the first token of a sequence that StreamingLLM described [Certain] (session 116).

So the figure has a second mode, “Minus caption average”: from each word’s map we subtract the average map of all the words of that caption and keep the positive part. What remains is what is special about this word. Now “dogs” lights up the two dogs, and “woman” the people under the umbrella, while “snow” and “the” go to background tokens. This is a qualitative look at four images, not a measurement.

One number is clearer. In all four captions, the word that receives the most image attention (all layers) is the noun that names the main subject: “dogs” (20.7%), “woman” (16.2%), “dog” (18.7%) and “man” (19.2%). Function words such as “of” and “and” get about 10% to 12%. The model reads the image most when it has to name what is in it.

Figure 4. Attention from the position that writes each caption word to the 49 image tokens (formula 185.9), Stage 1 model, Flickr8k test images, averaged over heads and over the chosen band of layers. Left: the image (a 112 px copy) with the strongest tokens outlined. Right: the \(7 \times 7\) map. The captions are the model’s own greedy output. Each word button is one token, shown without its leading space (the token for “dogs” is really “ dogs”).
How to read Figure 4

Choose an image, then click the words of the caption. In “Raw” mode most maps peak at row 1, column 1: a positional sink, not content. Switch to “Minus caption average” and click “dogs” in image 1 or “woman” in image 2: the remaining attention sits on the subject. Try the layer bands: the band 19–24 carries the most image attention in three of the four captions (16.8% to 21.9% on average); in image 4 the band 1–6 is slightly higher (15.0% against 13.7%). Image 4 is also a failure case: the caption says “a black and white shirt”, but the man wears a yellow shirt. The model knows there is a man and a shirt; the colour comes from the language prior of Flickr captions.

How much attention goes to the image, layer by layer

Now take the Stage 2 model and the position that predicts the one-word answer, averaged over 100 test images. Figure 5 shows the image mass (185.9) in each layer, next to the mass on the first token and on the text.

image tokens (dashed: their share of positions)first tokenquestion and template textStage 1 model, while writing captions: image tokens
Figure 5. Where the answer position attends, per layer (formula 185.9, mean over 9 heads and 100 CIFAR-10 test images), for the Stage 2 model. The three solid lines of one question add up to 100% in each layer.
How to read Figure 5

The dashed line is where the image line would be if attention ignored content: 71%. The real image line is far below it in almost every layer: 17.1% on average over the 30 layers for the class question, 12.4% for animal or vehicle. Layers 2 and 3 read almost nothing from the image (3.2% and 4.6% for the class question). For the class question there is a clear peak in layers 22 to 24 (48.9%, 58.9%, 45.4%); the animal-or-vehicle question does not have it (10.9% to 13.7% there). In the deep layers the first token takes up to 84%: it is the attention sink, a place to put attention that is not needed.

Three lessons from Figure 5.

  • Image tokens get far less attention than their number suggests. They are 71% of the positions and get 17% of the attention on average. This is the observation behind FastV, which found that image tokens receive little attention after the first layers of LLaVA-style models and drops many of them at inference [Certain]. In our model layers 2 and 3 nearly ignore the image; we have not tested how many tokens could be dropped (Exercise 6).
  • The reading is concentrated in a few layers. For the class question, three layers (22 to 24) carry half or more of their attention to the image. For the coarse animal-or-vehicle question those layers stay low. One possible reason is that the coarse decision needs less detail, so it is read earlier or from fewer tokens [Speculation]; attention alone cannot prove it. A causal test is to block attention to image tokens in only those layers and measure the accuracy drop.
  • Attention is not explanation. The first-token sink, the corner-token sink of Figure 4 and the head averaging all mix with content. Use attention mass as a cheap diagnostic (“does the answer read the image at all, and where?”), and use interventions, such as the shuffled-image control or blocking attention, to make claims.

The full program

These are the five scripts exactly as we ran them, in this order: extract.py, s1.py, s2.py s1, s3.py, s2.py scratch. Each stage reads the files the previous one saved. All randomness is seeded (seed_all(0) and fixed seeds for the data order), so a rerun on the same machine gives the same numbers; on another GPU the losses and accuracies will differ slightly. Set DEV = "cuda" in common.py on an NVIDIA GPU.

common.py: shared pieces: projector, LoRA, sequence builder, loss, generation

import torch, torch.nn as nn, math, random, numpy as np
from transformers import AutoModelForCausalLM, AutoTokenizer
DEV = "mps"
LLM = "HuggingFaceTB/SmolLM2-135M-Instruct"
GRID, MERGE, D_VIS = 14, 2, 768          # SigLIP: 14 x 14 patches of 768; merge 2 x 2 neighbours
N_IMG = (GRID // MERGE) ** 2              # 49 image tokens go into the LLM

def seed_all(s=0):
    random.seed(s); np.random.seed(s); torch.manual_seed(s)

class Projector(nn.Module):
    """Pixel-shuffle 2 x 2 neighbouring patches into one vector (4 x 768 = 3072), then a
    LLaVA-1.5 style MLP: Linear -> GELU -> Linear. 196 patches in, 49 tokens of width 576 out."""
    def __init__(self, d_vis=768, d_llm=576, r=MERGE):
        super().__init__()
        self.r = r
        self.net = nn.Sequential(nn.Linear(d_vis * r * r, d_llm), nn.GELU(), nn.Linear(d_llm, d_llm))
    def forward(self, v):                                   # v: (B, 196, 768)
        B, N, d = v.shape; g = int(N ** 0.5); r = self.r
        v = v.view(B, g // r, r, g // r, r, d).permute(0, 1, 3, 2, 4, 5)   # (B, 7, 7, 2, 2, 768)
        return self.net(v.reshape(B, (g // r) ** 2, r * r * d))           # (B, 49, 576)

class LoRALinear(nn.Module):
    def __init__(self, base, r=8, alpha=16):
        super().__init__()
        self.base = base; self.r = r; self.scale = alpha / r
        self.A = nn.Parameter(torch.randn(r, base.in_features) / math.sqrt(base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
    def forward(self, x):
        return self.base(x) + (x @ self.A.t() @ self.B.t()) * self.scale

def load_llm(attn="sdpa"):
    tok = AutoTokenizer.from_pretrained(LLM)
    llm = AutoModelForCausalLM.from_pretrained(LLM, torch_dtype=torch.float32, attn_implementation=attn).to(DEV)
    for p in llm.parameters(): p.requires_grad_(False)
    return tok, llm

def add_lora(llm, r=8, alpha=16, targets=("q_proj", "v_proj")):
    for layer in llm.model.layers:
        for t in targets:
            setattr(layer.self_attn, t, LoRALinear(getattr(layer.self_attn, t), r, alpha).to(DEV))
    return [p for n, p in llm.named_parameters() if n.endswith(".A") or n.endswith(".B")]

class Builder:
    """Turns (image features, question, answer) into one sequence of embeddings + labels."""
    def __init__(self, tok):
        self.tok = tok
        self.pre = tok("<|im_start|>user\n").input_ids            # 3 tokens, image goes right after
    def ids(self, q, a=None):
        suf = self.tok(q + "<|im_end|>\n<|im_start|>assistant\n").input_ids
        ans = self.tok(a + "<|im_end|>").input_ids if a is not None else []
        return suf, ans
    def batch(self, llm, proj, feats, qs, ans, use_image=True, left_pad=False):
        """feats: (B,196,768) fp16 on cpu. Returns inputs_embeds, attention_mask, labels."""
        E = llm.get_input_embeddings()
        B = len(qs); rows = []
        img = proj(feats.to(DEV).float()) if use_image else None
        for i in range(B):
            suf, a = self.ids(qs[i], ans[i] if ans is not None else None)
            pre_e = E(torch.tensor(self.pre, device=DEV))
            txt_e = E(torch.tensor(suf + a, device=DEV))
            parts = [pre_e] + ([img[i]] if use_image else []) + [txt_e]
            emb = torch.cat(parts)
            n_ctx = len(self.pre) + (N_IMG if use_image else 0) + len(suf)
            lab = [-100] * n_ctx + a
            rows.append((emb, lab))
        L = max(r[0].shape[0] for r in rows); d = rows[0][0].shape[1]
        emb = torch.zeros(B, L, d, device=DEV); mask = torch.zeros(B, L, dtype=torch.long, device=DEV)
        labels = torch.full((B, L), -100, dtype=torch.long, device=DEV)
        for i, (e, lab) in enumerate(rows):
            n = e.shape[0]
            s = L - n if left_pad else 0
            emb[i, s:s+n] = e; mask[i, s:s+n] = 1
            labels[i, s:s+n] = torch.tensor(lab, device=DEV)
        return emb, mask, labels

def answer_loss(llm, emb, mask, lab, reduction="mean"):
    """Next-token cross-entropy on answer tokens only. We run the LM head only at the positions
    that predict an answer token, so we never build the full (B, L, 49152) logits tensor."""
    h = llm.model(inputs_embeds=emb, attention_mask=mask).last_hidden_state   # (B, L, 576)
    tgt = lab[:, 1:]; keep = tgt != -100                                        # position t predicts token t+1
    logits = llm.lm_head(h[:, :-1][keep])                                       # (n_answer_tokens, 49152)
    return nn.functional.cross_entropy(logits, tgt[keep], reduction=reduction), int(keep.sum())

@torch.no_grad()
def mean_loss(llm, proj, bld, feats, qs, ans, use_image=True, bs=32):
    tot, cnt = 0.0, 0
    for i in range(0, len(qs), bs):
        emb, mask, lab = bld.batch(llm, proj, feats[i:i+bs], qs[i:i+bs], ans[i:i+bs], use_image)
        l, n = answer_loss(llm, emb, mask, lab, reduction="sum")
        tot += l.item(); cnt += n
    return tot / cnt

@torch.no_grad()
def generate(llm, proj, bld, feats, qs, max_new=30, bs=32, use_image=True):
    outs = []
    for i in range(0, len(qs), bs):
        emb, mask, _ = bld.batch(llm, proj, feats[i:i+bs], qs[i:i+bs], None, use_image, left_pad=True)
        g = llm.generate(inputs_embeds=emb, attention_mask=mask, max_new_tokens=max_new, do_sample=False,
                         eos_token_id=2, pad_token_id=2)
        outs += [bld.tok.decode(x, skip_special_tokens=True).strip() for x in g]
    return outs

def cosine_lr(step, total, base, warm):
    if step < warm: return base * (step + 1) / warm
    return base * 0.5 * (1 + math.cos(math.pi * (step - warm) / max(1, total - warm)))

extract.py: Step 2: run SigLIP once and cache the patch features

# Step 1: run the frozen SigLIP vision tower once, cache 196 patch features per image (fp16).
import torch, numpy as np, time, json, io, base64
from datasets import load_dataset
from transformers import SiglipVisionModel, AutoProcessor
torch.manual_seed(0)
dev = "mps"
enc = SiglipVisionModel.from_pretrained("google/siglip-base-patch16-224").to(dev).eval()
proc = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")
print("vision params", sum(p.numel() for p in enc.parameters()))

@torch.no_grad()
def encode(images, bs=64):
    out = []
    for i in range(0, len(images), bs):
        px = proc(images=images[i:i+bs], return_tensors="pt")["pixel_values"].to(dev)
        out.append(enc(pixel_values=px).last_hidden_state.half().cpu())
    return torch.cat(out)

info = {}
fl = load_dataset("jxie/flickr8k")
for split in ["train", "validation", "test"]:
    d = fl[split]; t0 = time.time()
    imgs = [im.convert("RGB") for im in d["image"]]
    F = encode(imgs); info[f"flickr_{split}_s"] = round(time.time() - t0, 1)
    caps = [[d[f"caption_{k}"][i] for k in range(5)] for i in range(len(d))]
    torch.save({"feats": F, "caps": caps}, f"flickr_{split}.pt"); print(split, F.shape, info)

cf = load_dataset("uoft-cs/cifar10")
names = cf["train"].features["label"].names
for split, per, seed in [("train", 500, 0), ("test", 100, 1)]:
    lab = np.array(cf[split]["label"]); rng = np.random.default_rng(seed)
    idx = rng.permutation(np.concatenate([rng.permutation(np.where(lab == c)[0])[:per] for c in range(10)]))
    imgs = [cf[split][int(i)]["img"].convert("RGB") for i in idx]
    F = encode(imgs)
    torch.save({"feats": F, "labels": torch.tensor(lab[idx]), "idx": idx.tolist(), "names": names}, f"cifar_{split}.pt")
    print(split, F.shape)
json.dump(info, open("extract.json", "w"))

s1.py: Step 3: Stage 1, train the projector on Flickr8k captions

# Stage 1: train only the projector on Flickr8k image-caption pairs. LLM and SigLIP frozen.
import torch, json, time, random, sys
from common import *
seed_all(0)
tok, llm = load_llm()
bld = Builder(tok)
proj = Projector().to(DEV)
R = {"n_proj": sum(p.numel() for p in proj.parameters()), "n_llm": sum(p.numel() for p in llm.parameters())}
tr = torch.load("flickr_train.pt", mmap=True); va = torch.load("flickr_validation.pt"); te = torch.load("flickr_test.pt")
Q = "Describe the image."
vq = [Q] * 1000; va_caps = [c[0] for c in va["caps"]]
def val_all(tag):
    shuf = torch.roll(va["feats"], 1, 0)
    R[tag] = {"match": mean_loss(llm, proj, bld, va["feats"], vq, va_caps),
              "shuffled": mean_loss(llm, proj, bld, shuf, vq, va_caps),
              "noimg": mean_loss(llm, proj, bld, va["feats"], vq, va_caps, use_image=False)}
    print(tag, R[tag], flush=True)
val_all("val_before")
EPOCHS, BS, LR = 2, 32, 1e-3
order = []
rng = random.Random(0)
for ep in range(EPOCHS):
    idx = list(range(6000)); rng.shuffle(idx); order += [(i, rng.randrange(5)) for i in idx]
steps = len(order) // BS
opt = torch.optim.AdamW(proj.parameters(), lr=LR, weight_decay=0.0)
curve, vcurve = [], []
t0 = time.time()
for s in range(steps):
    b = order[s*BS:(s+1)*BS]
    feats = tr["feats"][[i for i, _ in b]]
    caps = [tr["caps"][i][k] for i, k in b]
    for g in opt.param_groups: g["lr"] = cosine_lr(s, steps, LR, 20)
    emb, mask, lab = bld.batch(llm, proj, feats, [Q] * BS, caps)
    loss, _ = answer_loss(llm, emb, mask, lab)
    opt.zero_grad(); loss.backward(); opt.step()
    curve.append(round(loss.item(), 4))
    if s % 25 == 0 or s == steps - 1:
        vl = mean_loss(llm, proj, bld, va["feats"][:256], vq[:256], va_caps[:256])
        vcurve.append([s, round(vl, 4)])
        print(s, steps, round(loss.item(), 3), round(vl, 3), round(time.time() - t0), flush=True)
R["s1_time_s"] = round(time.time() - t0); R["s1_steps"] = steps
R["curve"] = curve; R["vcurve"] = vcurve
val_all("val_after")
R["test_caps_gen"] = generate(llm, proj, bld, te["feats"][:12], [Q] * 12)
R["test_caps_ref"] = [c[0] for c in te["caps"][:12]]
for g, r in zip(R["test_caps_gen"], R["test_caps_ref"]): print("GEN:", g, "| REF:", r)
torch.save(proj.state_dict(), "proj_s1.pt")
json.dump(R, open("s1.json", "w"))

s2.py: Step 4: Stage 2, LoRA + projector on the CIFAR-10 VQA set; run with argument s1 or scratch

# Stage 2: instruction-tune on a small VQA set built from CIFAR-10, with LoRA (q_proj, v_proj) + projector.
import torch, json, time, random, sys, re
from common import *
INIT = sys.argv[1] if len(sys.argv) > 1 else "s1"      # "s1" = start from the stage-1 projector, "scratch" = random projector
seed_all(0)
tok, llm = load_llm()
bld = Builder(tok)
proj = Projector().to(DEV)
if INIT == "s1": proj.load_state_dict(torch.load("proj_s1.pt"))
tr = torch.load("cifar_train.pt", mmap=True); te = torch.load("cifar_test.pt")
names = tr["names"]
ANIMAL = {"bird", "cat", "deer", "dog", "frog", "horse"}
def art(w): return ("an " if w[0] in "aeiou" else "a ") + w
def make(labels, seed):
    rng = random.Random(seed); ex = []
    for i, c in enumerate(labels.tolist()):
        n = names[c]
        ex.append((i, "what", "What is in this image? Answer with one word.", n))
        ex.append((i, "kind", "Is this an animal or a vehicle? Answer with one word.", "animal" if n in ANIMAL else "vehicle"))
        yes = rng.random() < 0.5
        x = n if yes else names[rng.choice([k for k in range(10) if k != c])]
        ex.append((i, "yesno", f"Is there {art(x)} in this image? Answer yes or no.", "yes" if yes else "no"))
    return ex
train_ex = make(tr["labels"], 0); test_ex = make(te["labels"], 1)
def norm(s): return re.sub(r"[^a-z]", "", s.lower().split()[0]) if s.strip() else ""
def evaluate(tag, use_image=True, shuffle=False):
    feats = te["feats"]
    if shuffle: feats = torch.roll(feats, 1, 0)
    res = {}
    for qt in ["what", "kind", "yesno"]:
        ex = [e for e in test_ex if e[1] == qt]
        gen = generate(llm, proj, bld, feats[[e[0] for e in ex]], [e[2] for e in ex], max_new=5, use_image=use_image)
        acc = sum(norm(g) == e[3] for g, e in zip(gen, ex)) / len(ex)
        res[qt] = round(acc, 4); res[qt + "_sample"] = gen[:5]
    print(tag, {k: v for k, v in res.items() if "sample" not in k}, flush=True)
    R[tag] = res
R = {"init": INIT, "n_train": len(train_ex), "n_test": len(test_ex)}
evaluate("before")
lora = add_lora(llm)
R["n_lora"] = sum(p.numel() for p in lora)
for p in proj.parameters(): p.requires_grad_(True)
params = lora + list(proj.parameters())
EPOCHS, BS, LR = 1, 32, 2e-4
rng = random.Random(0); order = train_ex[:]; rng.shuffle(order)
steps = len(order) // BS
opt = torch.optim.AdamW(params, lr=LR, weight_decay=0.0)
curve = []; t0 = time.time()
for s in range(steps):
    b = order[s*BS:(s+1)*BS]
    for g in opt.param_groups: g["lr"] = cosine_lr(s, steps, LR, 20)
    emb, mask, lab = bld.batch(llm, proj, tr["feats"][[e[0] for e in b]], [e[2] for e in b], [e[3] for e in b])
    loss, _ = answer_loss(llm, emb, mask, lab)
    opt.zero_grad(); loss.backward(); opt.step()
    curve.append(round(loss.item(), 4))
    if s % 50 == 0: print(s, steps, round(loss.item(), 3), round(time.time() - t0), flush=True)
R["s2_time_s"] = round(time.time() - t0); R["curve"] = curve; R["steps"] = steps
evaluate("after")
evaluate("after_shuffled", shuffle=True)
# forgetting check: stage-1 caption loss on Flickr val, with the stage-2 weights
va = torch.load("flickr_validation.pt")
R["flickr_val_after_s2"] = mean_loss(llm, proj, bld, va["feats"], ["Describe the image."] * 1000, [c[0] for c in va["caps"]])
print("flickr val after s2", R["flickr_val_after_s2"])
torch.save({"proj": proj.state_dict(), "lora": {n: p.detach().cpu() for n, p in llm.named_parameters() if n.endswith(".A") or n.endswith(".B")}}, f"s2_{INIT}.pt")
json.dump(R, open(f"s2_{INIT}.json", "w"))

s3.py: Step 5: attention from text to image

# Step 4: where do text tokens look? Attention from text positions to the 196 image tokens.
import torch, json, base64, io, numpy as np
from common import *
from datasets import load_dataset
seed_all(0)
tok, llm = load_llm(attn="eager")
bld = Builder(tok)
P0 = len(bld.pre)                      # image tokens live at positions P0 .. P0+195
R = {}

def attn_rows(emb, mask):
    out = llm(inputs_embeds=emb, attention_mask=mask, output_attentions=True)
    return torch.stack([a[0].mean(0) for a in out.attentions]).cpu()      # (30, L, L), mean over 9 heads

# ---- A. Stage-1 model on Flickr8k test: attention while it writes each caption word ----
proj = Projector().to(DEV); proj.load_state_dict(torch.load("proj_s1.pt"))
te = torch.load("flickr_test.pt"); fl = load_dataset("jxie/flickr8k")["test"]
PICK = [0, 3, 7, 11]
maps, thumbs, words_all, frac_s1, caps = [], [], [], [], []
Q = "Describe the image."
with torch.no_grad():
    for i in PICK:
        cap = generate(llm, proj, bld, te["feats"][i:i+1], [Q], max_new=24)[0]
        emb, mask, lab = bld.batch(llm, proj, te["feats"][i:i+1], [Q], [cap])
        A = attn_rows(emb, mask)
        ans_pos = (lab[0] != -100).nonzero().flatten().tolist()         # positions whose label is a caption token
        ids = lab[0, ans_pos].tolist()
        words, M = [], []
        for p, t in zip(ans_pos, ids):
            if t == 2: continue
            q = p - 1                                                   # the row that PREDICTS this token
            m = A[:, q, P0:P0+N_IMG]                                     # (30, 196)
            frac_s1.append(m.sum(-1).tolist())
            words.append(tok.decode([t]))
            M.append(m)
        M = torch.stack(M)                                              # (T, 30, 196)
        maps.append(M); words_all.append(words)
        img = fl[i]["image"].convert("RGB").resize((112, 112))
        b = io.BytesIO(); img.save(b, "JPEG", quality=70); thumbs.append(base64.b64encode(b.getvalue()).decode())
        caps.append(cap); print(i, cap, flush=True)
R["caps"] = caps
R["words"] = words_all; R["thumbs"] = thumbs
# bands of layers: 1-6, 7-12, 13-18, 19-24, 25-30, all
def bands(M):
    out = [M[:, k*6:(k+1)*6].mean(1) for k in range(5)] + [M.mean(1)]
    return torch.stack(out, 1)                                          # (T, 6, 196)
enc = []
for M in maps:
    Bm = bands(M)
    Bm = Bm / Bm.amax(-1, keepdim=True).clamp_min(1e-12)                 # each map scaled to its own max
    enc.append(base64.b64encode((Bm * 255).round().clamp(0, 255).to(torch.uint8).numpy().tobytes()).decode())
    R.setdefault("bandmass", []).append(bands(M).sum(-1).tolist())       # (T, 6) mass on image before scaling
R["maps_b64"] = enc
F = np.array(frac_s1)                                                    # (all words, 30)
R["s1_frac_layer"] = F.mean(0).round(4).tolist()

# ---- B. which patches get attention regardless of the word? (sink patches) ----
allm = torch.cat([M.mean(1) for M in maps])                              # (T_total, 196)
top = allm.mean(0).topk(5)
R["sink_patches"] = top.indices.tolist(); R["sink_mass"] = top.values.round(decimals=4).tolist()

# ---- C. Stage-2 model on CIFAR VQA: fraction of attention on image, per layer, at the answer position ----
ck = torch.load("s2_s1.pt")
lora = add_lora(llm)
proj.load_state_dict(ck["proj"])
sd = dict(llm.named_parameters())
for n, p in ck["lora"].items(): sd[n].data.copy_(p.to(DEV))
cf = torch.load("cifar_test.pt"); names = cf["names"]
qs = {"what": "What is in this image? Answer with one word.",
      "kind": "Is this an animal or a vehicle? Answer with one word."}
res = {}
with torch.no_grad():
    for qt, q in qs.items():
        fr, fs, ft, L0 = [], [], [], None
        for j in range(100):
            emb, mask, lab = bld.batch(llm, proj, cf["feats"][j:j+1], [q], None)
            A = attn_rows(emb, mask)
            row = A[:, -1]                                               # last context position predicts the answer
            fr.append(row[:, P0:P0+N_IMG].sum(-1).tolist())
            fs.append(row[:, 0].tolist())                                # first token (attention sink)
            ft.append((row[:, 1:P0].sum(-1) + row[:, P0+N_IMG:].sum(-1)).tolist())
            L0 = emb.shape[1]
        res[qt] = {"img": np.array(fr).mean(0).round(4).tolist(), "first": np.array(fs).mean(0).round(4).tolist(),
                   "text": np.array(ft).mean(0).round(4).tolist(), "L": L0}
        print(qt, L0, np.array(fr).mean(), flush=True)
R["s2_frac"] = res
json.dump(R, open("s3.json", "w"))
print("done")

Where this is used

System or taskWhat corresponds to this lab
LLaVA and LLaVA-1.5The same recipe at large scale: frozen CLIP ViT-L, a linear (LLaVA) or two-layer MLP (LLaVA-1.5) projector trained alone on 595K or 558K caption pairs, then instruction tuning on 158K or 665K examples with the LLM unfrozen [Certain].
SmolVLMNearly our parts: SigLIP B/16 (about 93M) and SmolLM2-135M in the 256M model, with pixel shuffle to cut image tokens [Certain]. It adds image splitting for high resolution and video frames (session 180).
Domain VLMs (medical, retail, documents)Teams often start from an open VLM and only re-run Stage 2 with LoRA on their own question–answer data, exactly as in Step 4 (session 189).
Video question answeringSample frames, encode each with the frozen encoder, merge or pool tokens per frame, and concatenate them in time order before the question. Token merging (formula 185.2) matters even more, because frames multiply the image tokens.
Video retrieval and key-moment detectionThe cached SigLIP features of Step 2 are what a retrieval index stores per frame; the VLM is then called only on the few frames the index returns (Video-RAG).
Captioning for search and accessibilityStage 1 alone already gives a captioner (our Flickr8k samples); real systems use it to write alt text or to index photo libraries.
Debugging and interpretabilityShuffled-image controls (Step 3) and attention-to-image measurements (Step 5) are how papers such as FastV found that image tokens get little attention in deep layers [Certain], which led to token pruning at inference (session 183).

Common mistakes

  • Putting the loss on every position. If the labels are not −100 outside the answer, the model spends its capacity predicting the chat template and the question, and the loss numbers mean something else. Check the label mask on one printed example before training.
  • Off-by-one in the shift. Position \(t\) predicts token \(t+1\). If you compare logits and labels without the shift, the model learns to copy its input and the loss falls fast for the wrong reason. Hugging Face shifts inside the model when you pass labels; if you compute the loss yourself (as we do, to save memory), shift yourself.
  • No shuffled-image control. A falling caption loss does not prove the model looks at the image: it may only learn caption style. Always report the loss or accuracy with each image swapped for another one. The gap is the evidence.
  • Wrong feature layer or normalization. Using SigLIP’s pooled vector instead of the patch tokens, or feeding images normalized for another encoder, fails silently: training runs, but the image carries less.
  • Learning rate from LLM fine-tuning for the projector. A fresh projector needs a large rate (LLaVA-1.5 uses \(10^{-3}\) [Certain]); with \(2 \times 10^{-5}\) it hardly moves in one epoch. In Stage 2, the LoRA rate should be lower (we use \(2 \times 10^{-4}\)).
  • Left padding forgotten at generation. For batched generation the prompts must be padded on the left, so that every row ends at the position that predicts the first answer token. Right padding makes rows continue after pad tokens.
  • Deciding a recipe question from one task and one seed. In our lab Stage 1 did not help on closed-set VQA; on open-ended captioning or with a fully unfrozen LLM it may. Test recipe choices on the evaluation you care about, with at least two seeds.
  • Reading raw attention maps as explanations. Head-averaged attention mixes sink tokens, position effects and real content. Report the attention mass with a baseline (the share of image tokens in the sequence) and look at many examples before you draw a conclusion.

Interview questions

Describe the architecture of a LLaVA-style VLM. How does the image get into the language model?basic
Three parts: a pretrained vision encoder (CLIP or SigLIP ViT), a small connector (a linear layer or a two-layer MLP, sometimes after merging neighbouring patches), and a decoder-only LLM. The encoder’s patch tokens go through the connector and come out with the LLM’s embedding width. They are inserted into the input sequence as embeddings, at the place of an image placeholder in the chat template, next to the ordinary token embeddings, and passed with inputs_embeds. The LLM then treats them like words; text positions read them through normal causal self-attention. In our lab: 196 SigLIP patches, merged to 49 tokens, MLP to width 576.
Why freeze both the encoder and the LLM in Stage 1, and train only the projector?basic
At the start the projector is random, so the image tokens are noise. If the LLM were trainable, its weights would be pushed around by gradients that come from noise, which can damage its language ability. Training only the projector is also cheap (2.1M parameters here, under 1% of the model) and the task is simple: map one feature space into another that already exists. Caption pairs are plentiful, so this alignment uses cheap data. LLaVA and LLaVA-1.5 freeze both in Stage 1 for these reasons.
Why are the labels −100 everywhere except the answer?basic
−100 is the default ignore_index of PyTorch’s cross-entropy, so those positions add no loss and no gradient. The image positions cannot have a label at all (image tokens are not vocabulary items), and the template and question are given, not generated, so training on them wastes capacity and makes the loss harder to interpret. The loss is then the average over answer tokens only. Remember the shift: position \(t\) predicts the token at \(t+1\).
Your caption loss went down. How do you show that the model actually uses the image?core
Evaluate the same text with the right image, a wrong image (shuffle images across examples) and no image. The difference between wrong and right is the image content. In our run the loss fell from 4.28 to 2.61 with the right image, but also from 4.27 to 3.40 with a wrong image: about half of the gain was caption style that any image triggers. The content gain was 0.795 nats per token, a 2.2 times lower perplexity. For question answering, the same control dropped accuracy from 94.3% to 8.9%, which is chance.
How does LoRA work, how many parameters did it add here, and why does it start from the Stage 1 model exactly?core
LoRA keeps \(W\) frozen and adds \(\frac{\alpha}{r} B A\) with \(A \in \R^{r \times d_{in}}\), \(B \in \R^{d_{out} \times r}\). With \(r = 8\) on q_proj (\(576 \rightarrow 576\)) and v_proj (\(576 \rightarrow 192\), because of 3 key-value heads) in 30 layers, that is \(30 \cdot 8 \cdot (1152 + 768) = 460{,}800\) parameters. \(B\) is initialized to zero, so \(BA = 0\) and the model at step 0 is exactly the Stage 1 model; training only moves away from it as needed. The update can be merged into \(W\) after training, so inference costs nothing extra.
Why merge \(2 \times 2\) patches before the projector? What do you lose?core
Every image token costs the LLM as much as a word in every layer, in compute and in KV cache. Merging (pixel shuffle) concatenates four neighbouring patch vectors into one, so 196 tokens become 49 with no information thrown away at the input: the projector sees all 3,072 numbers. What you lose is spatial resolution in the LLM’s attention (it can point only to \(7 \times 7\) cells) and possibly fine detail, since the MLP must compress four patches into one 576-number vector. Small models such as SmolVLM merge aggressively; LLaVA-1.5 does not merge and passes 576 tokens.
After Stage 1 the model captions well but scores 0% on “What is in this image? Answer with one word.” Why?core
Stage 1 only taught the model to describe images; it never saw an instruction about the answer format. It answered “The image is a dog”, often right in content but not one word, so a strict scorer gives 0. Seeing and following instructions are different skills: Stage 2 (visual instruction tuning) teaches the second. A fair evaluation either fixes the format with a prompt and a scorer that extracts the answer, or tunes the model on the format.
After Stage 2 the caption loss got worse than with a wrong image. What happened, and how do real recipes avoid it?core
Catastrophic forgetting through narrow data: 468 steps of one-word answers taught the projector and the LoRA weights that answers are short, so long captions became unlikely (loss 3.572 against 2.607). Real Stage 2 data are a mixture of task types and answer lengths; LLaVA-1.5 mixes conversations, detailed descriptions, short VQA with an explicit format instruction, and more. Other fixes: replay some Stage 1 caption data, use a lower learning rate or fewer steps, keep the projector frozen in Stage 2, or keep separate LoRA adapters per task.
How would you measure where a VLM looks, and what are the traps?advanced
Run with eager attention and output_attentions=True, take the row of the position that predicts the word of interest, average over heads, and sum over image positions to get the image attention mass per layer, or reshape the image part into the patch grid for a map. Compare with the baseline share of image positions. Traps: attention sinks (the first token took up to 84% in deep layers; the first image token was a sink inside the image), head averaging hides specialized heads, and attention weight is not influence, because value vectors differ in size. Use interventions (shuffled images, masking attention to image tokens in some layers) to make causal claims.
Image tokens were 71% of the positions but got 17% of the attention. What would you do with this observation?advanced
It suggests that many image tokens are not needed in many layers, which is what FastV used: it drops a large part of the image tokens after layer 2 based on their attention scores, with small accuracy loss on LLaVA-style models. Before relying on it, measure: drop tokens in some layers and check accuracy, per task, because the class question in our model read the image heavily in layers 22 to 24 while the coarse question did not. For training-time savings, merge tokens before the LLM (pixel shuffle, pooling, a resampler) instead.
How would you turn this mini-VLM into a video question-answering model?advanced
Sample frames (for example 8 to 32 per clip), run the frozen encoder per frame and cache the features, merge or pool tokens per frame so that the total fits the context budget (\(32 \text{ frames} \times 49 \text{ tokens} = 1{,}568\) tokens here), add time information (frame order, timestamps in text, or temporal position ids), and train Stage 1 on video–caption pairs and Stage 2 on video QA, including questions about order and key moments. Keep the shuffled control: shuffle frames in time to test if the model uses order at all.
In this lab, Stage 2 from a random projector matched or beat the two-stage model. Does that mean Stage 1 is useless?advanced
No, it means it was not needed for this task. The task was a closed set of short answers with 15,000 clean examples, and LoRA plus a trainable projector learned it from scratch in one epoch (94.4%, 99.5%, 95.0% against 94.3%, 99.0%, 86.6% with Stage 1). Stage 1 gave a faster start but no better end. Prismatic found the same at real scale: single-stage training was better on average and 20 to 25% cheaper. Stage 1 can still matter when the LLM is fully unfrozen, when instruction data are scarce, or for open-ended generation such as captioning, which our VQA test does not measure. The right answer is to run both on your own evaluation, with more than one seed.

Exercises

  1. Without the \(2 \times 2\) merge, the projector input is 768 and there are 196 image tokens. How many projector parameters and how many context positions does the class question have then? What share of the context is image?
  2. Add LoRA (\(r = 8\)) to k_proj and o_proj as well. How many LoRA parameters do you get?
  3. Use the Stage 1 table: what were \(\Delta_{\text{img}}\) and the perplexity ratio (185.6) before training? What does it tell you?
  4. With every image swapped for its neighbour in the test list, what accuracy do you expect on the class question if the model answers from the image it sees? Our test set has 100 images per class in random order. Compare with our 8.9%.
  5. Change the extraction to use SigLIP’s second-to-last layer (output_hidden_states=True, hidden_states[-2]) and rerun Stage 1. Predict first: will the content gain \(\Delta_{\text{img}}\) be larger or smaller?
  6. Measure what the attention analysis suggests: in the Stage 2 model, block attention from text positions to image tokens in layers 2 and 3 only, then in layers 22 to 24 only, and measure the class accuracy in both cases.
  7. In Figure 5, at layer 23 for the class question, divide the image mass by the baseline share. Do the same with the 30-layer average. Interpret the two numbers.
Answerscheck
1. \(768 \times 576 + 576 + 576 \times 576 + 576 = 775{,}296\) parameters; \(3 + 196 + 17 = 216\) positions, image share \(196 / 216 = 90.7\%\) (against 71% with the merge). 2. Per layer: q \(8 \times 1152 = 9{,}216\), k \(8 \times 768 = 6{,}144\), v \(6{,}144\), o \(9{,}216\), total \(30{,}720\); times 30 layers \(= 921{,}600\), twice the q-and-v setting. 3. \(4.274 - 4.277 = -0.003\) nats, a perplexity ratio of \(e^{-0.003} \approx 1.00\): a random projector carries no usable image content, so the wrong image is as good as the right one. 4. The answer is right only if the neighbour has the same class. In a random order with 100 of 1,000 per class, that happens with probability \(99 / 999 \approx 9.9\%\), so about 10%; our 8.9% (89 of 1,000) is within the noise of such a count (standard error about 0.9 points). 5. This is an experiment; we did not run it. LLaVA chose the second-to-last CLIP layer because the last layer is shaped most by the contrastive objective; for SigLIP the difference may be small. Report the right, wrong and no-image losses as in Step 3. 6. Also an experiment we did not run. Implement it with a custom 4D attention mask per layer, or a forward hook that zeroes the attention to image positions and renormalizes. If attention mass tracks use, blocking layers 2 and 3 should hurt little and blocking 22 to 24 more, but only the measurement can say. 7. \(0.589 / 0.710 = 0.83\): even in the most image-heavy layer the image gets less than its share of positions. \(0.171 / 0.710 = 0.24\) on average: per position, an image token gets about a quarter of the attention of an average position.

Key takeaways

  • A LLaVA-style VLM is three parts: a frozen vision encoder, a small projector, and an LLM. Image tokens enter the LLM as input embeddings (185.4); the loss is only on the answer (185.5).
  • Stage 1 trains only the projector (2.1M parameters, 0.92% of our model) on captions. It cut the caption loss from 4.28 to 2.61 nats per token.
  • Always run the wrong-image control: about half of the Stage 1 gain was style; the content gain was 0.795 nats (185.6), a 2.2 times lower perplexity.
  • Stage 2 with LoRA (460,800 parameters, 185.7–185.8) taught instruction following: 94.3%, 99.0% and 86.6% on three question types, falling to chance with wrong images. Skipping Stage 1 gave the same or better accuracy here (as Prismatic found at scale); it only slowed the start.
  • Narrow Stage 2 data cause forgetting: the caption loss rose to 3.572. Real recipes mix many task types.
  • Image tokens were 71% of the positions but got 17% of the attention, concentrated in a few layers; sinks dominate raw maps. Attention is a diagnostic, not an explanation.

Key formulas

\[ \mathbf{h}_{(i,j)} = W_2\, \operatorname{GELU}\big(W_1 [\mathbf{z}_{2i,2j}; \mathbf{z}_{2i,2j+1}; \mathbf{z}_{2i+1,2j}; \mathbf{z}_{2i+1,2j+1}] + \mathbf{b}_1\big) + \mathbf{b}_2 \]
\[ \mathcal{L} = -\frac{1}{|A|}\sum_{t \in A} \log p(s_{t+1} \mid S_{\le t}), \qquad \Delta_{\text{img}} = \mathcal{L}_{\text{wrong}} - \mathcal{L}_{\text{right}} \]
\[ W' = W + \frac{\alpha}{r} B A, \quad n_{\text{LoRA}} = L\, r \sum_{\text{adapted}} (d_{\text{in}} + d_{\text{out}}), \qquad m^{\ell}_t = \sum_{j \in \mathcal{I}} \bar{a}^{\ell}_{tj} \]

References for this session

  1. Liu et al. (2023). Visual Instruction Tuning
  2. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
  4. Liu et al. (2023). Visual Instruction Tuning
  5. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  6. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  7. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  8. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  9. Ben Allal et al. (2025). SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
  10. Hodosh, Young and Hockenmaier (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
  11. Krizhevsky (2009). Learning Multiple Layers of Features from Tiny Images
  12. Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  13. Chen et al. (2024). An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
  14. Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
  15. Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
  16. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
  17. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models