Case studies 1: LLaVA, BLIP-2, Flamingo, IDEFICS
Four models shaped almost every open vision-language model that came after them. LLaVA joined CLIP to an LLM with one small projection and trained it on instruction data written by GPT-4. BLIP-2 froze both big models and trained only a small Q-Former between them. Flamingo read images and text mixed in one document and learned new tasks from a few examples in the prompt. Idefics2 then ran careful ablations to find out which of these choices really matter. This session takes each one apart, with the real numbers from the papers.
What you will learn
- LLaVA: a CLIP encoder, a linear (later MLP) projector and Vicuna, trained in two stages, and how GPT-4 wrote 158K instruction samples from captions and boxes without seeing the images
- LLaVA-1.5: the small changes (MLP, 336 px, academic VQA data with response-format prompts) that took it to the top of 11 benchmarks in about one day on 8 A100s
- BLIP-2: a 188M Q-Former with 32 learned queries between a frozen image encoder and a frozen LLM, its three stage-1 losses and their attention masks, and why stage 1 prevents forgetting
- Flamingo: the Perceiver resampler, gated cross-attention layers that start at zero, image-causal masking, and few-shot prompting with interleaved images and text
- IDEFICS and Idefics2: an open Flamingo, then six ablation findings (language model first, frozen vs LoRA, 64 pooled tokens, aspect ratio, image splitting)
- How to compare designs by trainable parameters, visual tokens in the context and training data, with numbers you can check
Why this matters
Between 2022 and 2024, four papers set the patterns that most open vision-language models still follow. If you can explain these four, you can place almost any new model on the map in a few minutes.
In session 177 we took a VLM apart into three parts: a vision encoder, a connector and a language model. We saw two ways to join them: put image tokens into the LLM’s sequence, or let the text attend to the image through new cross-attention layers. This session is the history behind that map. Each case study answers one big question:
- LLaVA (2023): how simple can the bridge be? Answer: one linear layer (later a small MLP), if you have good instruction data. LLaVA got that data by asking a text-only GPT-4 to write conversations about images it never saw.
- BLIP-2 (2023): can we keep both big models frozen and train only a small module in the middle? Answer: yes, with a 188M-parameter Q-Former trained in two stages.
- Flamingo (2022): can a VLM learn a new task from a few examples in the prompt, like GPT-3? Answer: yes, if it is trained on web pages where images and text are interleaved.
- IDEFICS and Idefics2 (2023, 2024): an open copy of Flamingo, then a careful study of what matters when you build a VLM, with controlled ablations.
These are not museum pieces. LLaVA’s recipe (encoder, MLP, LLM, two stages) is the starting point of Qwen-VL, InternVL and many others (session 182). The Q-Former and the Perceiver resampler live on as token compressors, which matter a lot for video, where you have many frames (session 183). Flamingo’s gated cross-attention returned in Llama 3.2 Vision [Likely]. And Idefics2’s ablations are the best public evidence for what to spend your compute on.
One frame for four models
Each model makes three choices. Which parts are trained? How many visual tokens does the LLM see per image, and where? What data teaches the connection? We will keep coming back to these three. Figure 1 lets you compare the four models on the first two.
How to read Figure 1
In “Parameters”, compare BLIP-2 and Flamingo-80B: both freeze the LLM, but BLIP-2 trains 108M weights (0.9% of 12.1B) and Flamingo trains about 10.2B (13% of 80B), mostly in its new cross-attention layers. LLaVA-1.5 trains a 21M MLP first, then also the whole 6.7B LLM. In “Visual tokens”, move the slider to 5: LLaVA-1.5 puts 2,880 image tokens in the context, Idefics2 320 (or 1,600 with image splitting), BLIP-2 160, Flamingo zero. This is why Flamingo-style and pooled designs were the first to handle many images, and why video models compress tokens.
LLaVA: the simplest bridge, and data written by GPT-4
The architecture
LLaVA (Large Language and Vision Assistant) takes a pretrained CLIP ViT-L/14 image encoder \(g\) and a pretrained LLM (Vicuna, a chat-tuned LLaMA). Between them sits one trainable matrix \(W\). The image \(X_v\) goes through CLIP; the grid of patch features \(Z_v\) is multiplied by \(W\) so that each patch becomes a vector of the same size as a word embedding:
- Read the symbols
- \(X_v\)
- the input image
- \(g\)
- the frozen CLIP vision encoder; LLaVA reads the grid features of a late layer (before or after the last one; the second-to-last worked best on ScienceQA)
- \(Z_v\)
- one 1,024-dimensional vector per image patch
- \(W\)
- the trainable projection, \(1024 \to d_{\text{LLM}}\)
- \(H_v\)
- the visual tokens: vectors the LLM reads exactly like word embeddings
That is the whole connector. The visual tokens are placed in the sequence where the image appears, and the LLM does the rest. LLaVA-1.5 replaced the single matrix with a two-layer MLP, inspired by self-supervised learning, where an MLP head beats a linear one:
For LLaVA-1.5-7B, \(W_1\) is \(1024 \times 4096\) and \(W_2\) is \(4096 \times 4096\): 20,979,712 parameters, about 21M. The image is 336×336 pixels and the patch is 14 pixels, so CLIP gives \(24 \times 24 = 576\) patch tokens (the CLS token is dropped). The Hugging Face config confirms it: GELU activation, features from layer −2 [Certain].
Figure 2 runs this projector in your browser on a toy image. The image is cut into a grid of patches; each patch has a 2-number CLIP feature (top patches look like sky, bottom patches like ground). A tiny MLP with the shape of (181.2), \(2 \to 3 \to 3\), turns each patch into one visual token. Watch the sequence at the bottom: one patch gives one token, so the token count is set by the grid.
- input: image \(X_v\); projector \(W_1, b_1, W_2, b_2\)
- \(Z_v \leftarrow g(X_v)\) // one vector \(\mathbf{z}_p\) per patch
- for \(p = 1 \dots N_{\text{patch}}\):
- \(\mathbf{a} \leftarrow W_1 \mathbf{z}_p + b_1\)
- \(\mathbf{u} \leftarrow \mathrm{GELU}(\mathbf{a})\)
- \(\mathbf{h}_p \leftarrow W_2 \mathbf{u} + b_2\) // (181.2)
- append \(\mathbf{h}_p\) to the LLM sequence
- LLM reads [text] + \(\mathbf{h}_1 \dots \mathbf{h}_{N_{\text{patch}}}\) + [text]
How to read Figure 2
Press Play or Step. For patch 1, \(\mathbf{z} = (1.1, -0.2)\) gives \(\mathbf{a} = (1.080, -0.050, -0.740)\), GELU gives \(\mathbf{u} = (0.929, -0.024, -0.170)\): the large positive value passes almost unchanged, the negative ones shrink toward 0. Then \(\mathbf{h}_1 = (0.713, -0.283, 0.133)\) joins the sequence. With 3 × 3 patches the LLM reads 9 visual tokens + 7 text tokens; switch to 4 × 4 and it reads 16. LLaVA-1.5 has 24 × 24 patches, so 576 tokens per image. The projector never looks at two patches together: it is a lookup from “CLIP words” to “LLM words”, applied to each patch.
The data: GPT-4 writes about images it cannot see
In early 2023 there was almost no data of the form “image, instruction, good long answer”. LLaVA’s key idea was to create it with a text-only GPT-4 (and ChatGPT). For a COCO image, the authors gave GPT-4 a symbolic description: the human-written captions and the object bounding boxes (class name plus coordinates). GPT-4 never saw pixels. From that text it wrote three kinds of responses [Certain]:
- Conversation: a multi-turn chat with questions a person might ask about the image (what objects, how many, where, what are they doing). 58K samples.
- Detailed description: a long, complete description, prompted from a list of instructions such as “Describe the following image in detail”. 23K samples.
- Complex reasoning: questions that need step-by-step reasoning beyond what is visible (“What challenges might these people face?”). 77K samples.
Total: 158K instruction samples, released as LLaVA-Instruct-158K. A few hand-written examples were placed in each prompt as few-shot demonstrations, so GPT-4 knew the style.
- input: images with captions \(C\) and boxes \(B\) (COCO), a text-only LLM (GPT-4), a few hand-written seed examples per type
- for each image \(X_v\) do
- context \(\leftarrow\) the captions \(C\) and the boxes \(B\) as plain text // e.g. “person: [0.68, 0.24, 0.77, 0.69]”
- type \(\leftarrow\) one of {conversation, detailed description, complex reasoning}
- prompt \(\leftarrow\) system message for that type + seed examples + context
- response \(\leftarrow\) GPT-4(prompt) // GPT-4 never sees the pixels
- store \((X_v, \text{response})\) as one training sample
- return the instruction dataset (158K samples)
Two-stage training and the loss
Every training sample is turned into one token sequence. LLaVA uses a chat format: a system message, then turns of “Human: … <STOP> Assistant: … <STOP>”, with the image tokens placed in the first human turn. The model is trained with the normal next-token loss, but only on the answer tokens. For an answer \(X_a\) of length \(L\):
- Read the symbols
- \(x_i\)
- the \(i\)-th token of the assistant’s answers (all turns)
- \(X_{\text{instruct}, \lt i}\), \(X_{a, \lt i}\)
- the instructions and the answers that come before token \(i\): the model can see earlier turns
- \(\theta\)
- the trainable weights: \(W\) only in stage 1; \(W\) and the LLM in stage 2
The image tokens, the system message and the human turns are part of the input, but they get no loss: their label is set to −100 in PyTorch, so cross-entropy ignores them. Figure 3 shows which tokens are trained.
- Stage 1, feature alignment. 595K image–caption pairs filtered from CC3M. Each pair becomes a one-turn sample: a short random instruction (“Describe the image briefly”) and the caption as the answer. Encoder and LLM are frozen; only \(W\) is trained. One epoch, learning rate \(2 \times 10^{-3}\), batch 128, under 4 hours on 8 A100s [Certain]. The paper calls this “training a compatible visual tokenizer for the frozen LLM”.
- Stage 2, end-to-end fine-tuning. The 158K instruction samples. Encoder still frozen; \(W\) and the whole LLM are trained. Three epochs, learning rate \(2 \times 10^{-5}\), batch 32, about 10 hours [Certain].
How to read Figure 3
In stage 1, only the caption and the stop marker (14 tokens) get loss, out of 605 tokens: 2.3%. Most of the sequence is image tokens, which teach nothing directly; they only condition the answer. In the two-turn chat, both assistant answers get loss, and the second answer can see the first question and answer. In the LLaVA-1.5 VQA sample, the instruction ends with “Answer the question using a single word or phrase.” and the answer is one or two tokens. This format prompt is what let LLaVA-1.5 learn from short-answer VQA datasets without forgetting how to write long answers.
Take the two-turn chat of Figure 3: 576 image tokens + 68 tokens of system message, questions and role markers + 119 tokens of answers and stop markers. Sequence length 763; loss on 119 tokens (15.6%). The 576 image tokens are 75% of the sequence. With five images it would be \(5 \times 576 = 2{,}880\) image tokens, which is exactly the number Idefics2 reports for LLaVA-NeXT-13B per image, because LLaVA-NeXT splits one image into up to four crops plus a downsized full view [Likely]. The projector for the 13B model (\(d = 5120\)) has \(1024 \cdot 5120 + 5120 + 5120 \cdot 5120 + 5120 = 31{,}467{,}520\) weights.
LLaVA-1.5: small changes, large gains
The first LLaVA was a good chatbot but weak on academic benchmarks with short answers, such as VQAv2. LLaVA-1.5 (October 2023) kept the design and changed four things: an MLP projector (formula 181.2), CLIP at 336 pixels, Vicuna v1.5, and a 665K instruction mix that adds academic VQA, OCR and region data, each with a response-format prompt. Its ablation table shows how each step helped (7B unless noted) [Certain]:
| Step | Resolution | GQA | MME | MM-Vet |
|---|---|---|---|---|
| LLaVA (original) | 224 | – | 809.6 | 25.5 |
| + VQAv2 | 224 | 47.0 | 1197.0 | 27.7 |
| + format prompt | 224 | 46.8 | 1323.8 | 26.3 |
| + MLP connector | 224 | 47.3 | 1355.2 | 27.8 |
| + OKVQA, OCR data | 224 | 50.0 | 1377.6 | 29.6 |
| + region-level VQA | 224 | 50.3 | 1426.5 | 30.8 |
| + resolution 336 | 336 | 51.4 | 1450 | 30.3 |
| + GQA data | 336 | 62.0 | 1469.2 | 30.7 |
| + ShareGPT (text-only chat) | 336 | 62.0 | 1510.7 | 31.1 |
| + 13B LLM | 336 | 63.3 | 1531.3 | 36.1 |
The GQA jump from 51.4 to 62.0 comes from training on GQA itself, so it is not zero-shot; the paper marks such numbers [Certain]. The final model used 558K pretraining pairs and 665K instruction samples, about 1.2M in total, and the 13B model trained in about one day on one 8-A100 node [Certain]. The abstract reports state-of-the-art results on 11 benchmarks at the time [Certain]. The lesson that spread across the field: data and format matter more than a clever connector. LLaVA-1.5 also changed hyper-parameters very little: the stage-1 learning rate was halved to \(10^{-3}\) because the MLP is deeper.
BLIP-2: freeze both, train the Q-Former
The idea
Training a large VLM end to end is expensive, and updating the LLM can make it forget its language skills (catastrophic forgetting). BLIP-2 keeps both the image encoder and the LLM frozen. The only trained part is a Querying Transformer (Q-Former) in the middle. Its job is to read the frozen image features and produce a short, fixed set of vectors that the frozen LLM can understand.
The Q-Former is a BERT-base-sized transformer (initialized from BERT-base) with one extra ingredient: 32 learned query vectors of dimension 768. These queries are parameters, not inputs. They pass through self-attention layers (where they can also see text tokens, depending on the task) and through cross-attention layers into the frozen image features, inserted in every other block. The cross-attention layers start from random weights. In total, 188M parameters [Certain].
- Read the symbols
- \(Q^{(\ell)}\)
- the 32 query vectors after layer \(\ell\), a \(32 \times 768\) matrix (before layer 1: the learned queries)
- \(F\)
- the frozen image features, for example \(257 \times 1024\) for CLIP ViT-L/14 (256 patches plus CLS); BLIP-2 uses the second-to-last layer of the ViT
- \(W_Q, W_K, W_V\)
- the cross-attention weights, trained
The output \(Z\) is only \(32 \times 768 = 24{,}576\) numbers, versus \(257 \times 1024 = 263{,}168\) for the ViT-L features: about 11 times fewer. The paper calls this a bottleneck: it forces the queries to keep only the visual information that is useful for text.
Figure 4 runs one Q-Former cross-attention layer, formula (181.4), on the same kind of toy image. Each learned query computes a score with every image feature, turns the scores into weights with a softmax, and adds the weighted sum of the features to itself. To keep the numbers readable, \(W_Q = W_K = W_V = I\) and \(d = 2\). A last linear layer \(P\) (\(2 \to 3\)) maps each query to the LLM’s size. Change the patch grid: the output stays at one token per query.
- input: frozen features \(F\) (\(N_{\text{patch}} \times d\)), learned queries \(Q\) (\(n_q \times d\)), projection \(P\)
- for each query \(\mathbf{q}_i\):
- \(s_{ij} \leftarrow \mathbf{q}_i \cdot \mathbf{f}_j / \sqrt{d}\) for every feature \(j\)
- \(\mathbf{w}_i \leftarrow \softmax(\mathbf{s}_i)\)
- \(\mathbf{q}_i \leftarrow \mathbf{q}_i + \sum_j w_{ij} \mathbf{f}_j\) // (181.4)
- return \(P \mathbf{q}_1, \dots, P \mathbf{q}_{n_q}\): \(n_q\) tokens, prepended to the text
How to read Figure 4
With 3 × 3 patches and 2 queries: \(Q_1 = (2, 0)\) “looks for sky”. Its largest score is 1.980 on patch 3, whose feature is \((1.4, -0.4)\); after the softmax that patch gets weight 0.258 and each bottom-row (ground) patch gets less than 0.05. Q1 reads \(\mathbf{o} = (0.986, -0.018)\) and becomes \(Q_1' = (2.986, -0.018)\). \(Q_2 = (0, 2)\) looks for ground and puts its largest weight, 0.243, on patch 7. Now switch to 4 × 4: 16 features go in, still 2 tokens come out. BLIP-2 does this with 32 queries over 257 features, in several layers, with trained \(W_Q, W_K, W_V\).
Stage 1: three losses, three attention masks
In stage 1 the Q-Former is trained with the frozen image encoder but without the LLM, using three losses inherited from BLIP (session 173). The same network handles all three; only the self-attention mask between queries and text changes:
- Image–text contrastive (ITC). Pull matching image and text embeddings together, push others apart. Queries and text must not see each other (a unimodal mask), or the task would leak. The image has 32 output vectors \(z_1, \dots, z_{32}\) and the text has one, the [CLS] output \(t\). The similarity is the best match:
\[ s(I, T) = \max_{q = 1, \dots, 32} \; z_q^\top t \tag{181.5} \]Because the encoder is frozen, BLIP-2 fits more samples per GPU and uses in-batch negatives instead of BLIP’s momentum queue [Certain].
- Image–text matching (ITM). A binary classifier: does this text describe this image? Here everything sees everything (a bidirectional mask). Each of the 32 query outputs goes through a two-class linear layer, and the 32 logits are averaged. Hard negatives are picked as in ALBEF [Likely].
- Image-grounded text generation (ITG). Generate the caption. Text tokens see all queries and the earlier text tokens; queries do not see the text (a multimodal causal mask, as in UniLM). The text cannot see the image directly, so all visual information must flow through the queries. The [CLS] token is replaced by a [DEC] token.
The stage-1 loss is the sum \(\mathcal{L}_1 = \mathcal{L}_{\text{ITC}} + \mathcal{L}_{\text{ITM}} + \mathcal{L}_{\text{ITG}}\). Figure 5 shows the three masks.
How to read Figure 5
ITC allows 41 of 81 pairs: the two blocks on the diagonal only, so the image summary and the text summary are computed separately and can be compared by a dot product. ITM allows all 81, so the queries can check the text word by word. ITG allows 51: the text rows see all queries and a lower triangle of text, while the query rows still see only queries. Click the “[DEC]” row in ITG: it sees the 4 queries and itself, so the first word is generated from the image alone.
Take 4 query outputs \(z_1 = (1, 0)\), \(z_2 = (0, 1)\), \(z_3 = (0.6, 0.8)\), \(z_4 = (-1, 0)\) and a text embedding \(t = (0.6, 0.8)\), all unit length. The dot products are \(0.6, 0.8, 1.0, -0.6\). Formula (181.5) takes the maximum, 1.0: one query that found the right content is enough. The mean would be 0.45 and would punish the image for queries that look at other things, which is exactly what we want queries to be free to do.
Stage 2: plug in the frozen LLM
Now the Q-Former, still attached to the frozen image encoder, is connected to a frozen LLM. A single fully connected layer maps the 32 query outputs from 768 to the LLM’s embedding size (2,560 for OPT-2.7B, so \(768 \cdot 2560 + 2560 = 1{,}968{,}640\) weights). The 32 projected vectors are prepended to the text embeddings as a soft visual prompt. For a decoder-only LLM (OPT), the loss is plain language modeling of the text. For an encoder–decoder LLM (FlanT5), the text is split: the first part goes with the visual prompt into the encoder, the rest is the target for the decoder.
Why not skip stage 1? The paper tested it. Without stage-1 representation learning, zero-shot VQA was much lower for both LLM types, and with OPT the performance fell as training went on: catastrophic forgetting [Certain]. Stage 1 gives the LLM visual prompts that already carry text-relevant information, so the frozen LLM does not have to adapt to raw features.
- input: frozen image encoder \(E\) (ViT-L/14 from CLIP or ViT-g/14 from EVA-CLIP), frozen LLM, 129M images (BLIP’s data, with CapFilt captions)
- initialize the Q-Former from BERT-base; cross-attention and the 32 queries random
- Stage 1, 250k steps: for each batch \((I, T)\): \(F \leftarrow E(I)\) // no gradient into E
- \(\mathcal{L}_1 \leftarrow \mathcal{L}_{\text{ITC}}\)(unimodal mask) + \(\mathcal{L}_{\text{ITM}}\)(bidirectional mask) + \(\mathcal{L}_{\text{ITG}}\)(causal mask); update the Q-Former
- Stage 2, 80k steps: add a linear layer \(P: 768 \to d_{\text{LLM}}\)
- for each batch: prompt \(\leftarrow P(\text{QFormer}(E(I)))\), 32 vectors; \(\mathcal{L}_2 \leftarrow\) LM loss of the frozen LLM on \(T\) given the prompt
- update the Q-Former and \(P\) only
- return Q-Former, \(P\)
The cost was modest for 2023: the largest model (ViT-g with FlanT5-XXL) took under 6 days for stage 1 and under 3 days for stage 2 on one machine with 16 A100 (40 GB) GPUs, with frozen weights stored in FP16 (BF16 for FlanT5) [Certain]. Images are 224×224.
The results, and the famous comparison
On zero-shot VQAv2 (test-dev), BLIP-2 with ViT-g and FlanT5-XXL scored 65.0, against 56.3 for Flamingo-80B: 8.7 points higher, with 188M trainable parameters against Flamingo’s 10.2B, which is 54 times fewer [Certain]. On OK-VQA, which needs world knowledge, Flamingo-80B stayed ahead (50.6 vs 45.9); the authors credit Flamingo’s 70B language model [Certain]. Two lessons: a good bottleneck can replace a lot of trainable capacity, and knowledge lives in the LLM, so a bigger LLM still wins on knowledge tasks.
BLIP-2’s weakness is that its frozen LLM was never taught to follow visual instructions, and 32 tokens are a tight budget for reading text in images. InstructBLIP added instruction tuning and made the Q-Former also read the instruction; LLaVA-style designs with more tokens later won on OCR-heavy tasks [Likely].
Flamingo: interleaved images and text, and few-shot learning
The goal
GPT-3 showed that a large LM can learn a task from a few examples in the prompt (in-context learning, session 143). Flamingo (DeepMind, 2022) asked for the same with images: give the model “(image, caption), (image, caption), image, ?” and it should continue in the same style. To do that, the model must read sequences of images and text mixed together, and it must keep the knowledge of a large frozen LM. Flamingo-80B is built on the frozen 70B Chinchilla LM and a frozen NFNet-F6 image encoder (pretrained contrastively, like CLIP).
The Perceiver resampler
The vision encoder gives a variable number of features (more for larger images, many more for video). The Perceiver resampler turns any number of features into exactly 64 visual tokens. Like the Q-Former, it has learned latent queries; each layer lets the latents attend to the image features and to themselves:
- Read the symbols
- \(X\)
- the 64 latent vectors (learned at the start, updated by each layer)
- \(X_f\)
- the flattened visual features; for video, frames are sampled at 1 frame per second, each frame is encoded, and a learned time embedding is added before flattening
- \([X_f; X]\)
- features and latents stacked: the keys and values
Flamingo uses 6 resampler layers of width 1,536 with 16 heads, 194M parameters, the same for all model sizes [Certain]. In its ablation, the Perceiver resampler beat an MLP and a plain transformer of similar budget, and was faster.
Gated cross-attention: start at zero
The visual tokens do not enter the LM’s sequence. Instead, new GATED XATTN-DENSE blocks are inserted between the frozen LM layers. In each one, the text hidden states \(y\) attend to the visual tokens \(x\) (queries from text, keys and values from the image), then pass through a new feed-forward layer. Both results are multiplied by a tanh gate with a learned scalar \(\alpha\):
Both \(\alpha\) start at 0, and \(\tanh(0) = 0\). So at the first step, every new block is an identity: the model is exactly the frozen LM, and its text skills are intact. As training goes on, the gates open where the image helps. The paper plots \(|\tanh \alpha|\) over training: the gates grow quickly from 0 in all layers, and more in deeper layers [Certain]. Without the gating the overall ablation score fell from 70.7 to 66.5 [Certain].
Cross-attention blocks are expensive, so Flamingo adds one before every layer in the 3B model, every 4th layer in the 9B model, and every 7th layer in the 80B model (12 blocks in 80 layers) [Certain]. These blocks are most of the trainable weights: 10B of the 80B model. Adding them every 4th layer instead of every layer made training 66% faster and lowered the overall score by only 1.9% [Certain].
Image-causal masking
A web page can have several images. Which image should a text token look at? Flamingo’s answer: only the last image before it. Formally, the model is trained to maximize
- Read the symbols
- \(y_\ell\)
- the \(\ell\)-th text token
- \(x_{\le \ell}\)
- the images and videos that come before token \(\ell\) in the interleaved sequence
In practice, the cross-attention mask lets token \(\ell\) see only the 64 visual tokens of the immediately preceding image. Earlier images still influence it indirectly, because the LM’s own self-attention sees all earlier text, and that text has already looked at its own images. Text before the first image sees no image. This makes the number of cross-attention scores per text token constant (64) no matter how many images the prompt has, and it let Flamingo train on up to 5 images per sequence and then run with 32 shots at test time. Try it in Figure 6, which also runs the gated update (181.7) on toy numbers.
- for each text token \(\ell\) with hidden state \(\mathbf{y}\):
- \(m \leftarrow\) the last image before \(\ell\) // image-causal mask (181.8)
- if there is none: \(\mathbf{a} \leftarrow \mathbf{0}\)
- else \(s_j \leftarrow \mathbf{y} \cdot \mathbf{x}_j / \sqrt{d}\) over the visual tokens of \(m\); \(\mathbf{w} \leftarrow \softmax(\mathbf{s})\); \(\mathbf{a} \leftarrow \sum_j w_j \mathbf{x}_j\)
- \(\mathbf{y} \leftarrow \mathbf{y} + \tanh(\alpha_{\text{xattn}})\, \mathbf{a}\) // (181.7)
- \(\mathbf{y} \leftarrow\) frozen LM block\((\mathbf{y})\) // not simulated
How to read Figure 6
Step to the token “dog” (or click it). With image-causal masking it sees only image 2; its weights are 0.33, 0.26, 0.25, 0.16 and the message is \(\mathbf{a} = (-0.094, 0.922)\), close to the “dog” image tokens. At \(\alpha = 0.5\) the gate is 0.462, so \(\mathbf{y}\) moves from \((-0.1, 0.9)\) to \((-0.143, 1.326)\). Switch to “all previous images”: the same row now reads 8 tokens, the cat image pulls the message to \((0.268, 0.644)\), and the final “A” reads 12 tokens instead of 4. With 32 shots of 64 tokens each it would read 2,112 (32 support images plus the query image). Now move the slider to 0: the gate is 0, the orange step disappears and the hidden state is exactly what the frozen LM computes. At \(\alpha = 2\) the gate is 0.964, nearly fully open. A negative \(\alpha\) gives a negative gate, which subtracts the visual message; training decides the sign. In every case the LM sequence holds 11 text tokens and no visual tokens.
Data and the training loss
Few-shot skill comes from the data. Flamingo was trained on a mixture of four datasets [Certain]:
- M3W (MultiModal MassiveWeb): about 43 million web pages with 185M images and 182 GB of text, with <image> tags inserted where the images appear in the page layout. For each page, a random 256-token span with up to the first 5 images is taken.
- ALIGN: 1.8 billion images with alt-text (noisy).
- LTIP (Long Text and Image Pairs): 312M image–text pairs with longer, better descriptions.
- VTP (Video and Text Pairs): 27M short videos (about 22 seconds on average) with descriptions.
The loss is a weighted sum of the per-dataset negative log-likelihoods:
The weights \(\lambda_m\) were tuned and are “key to performance”. Gradients are accumulated over all datasets in every step, which beat a round-robin schedule (one dataset per step): 70.7 vs 62.9 on the ablation score [Certain]. Removing M3W was the single most damaging change: the score fell from 70.7 to 53.4 [Certain]. The largest run used 1,536 TPU chips for 15 days [Certain].
Few-shot results
At test time there is no training. You build a prompt of \(k\) support examples and the query, and the model continues the text. Flamingo-80B on VQAv2 (test-dev) and COCO captions (CIDEr) [Certain]:
| Shots | VQAv2 | COCO CIDEr |
|---|---|---|
| 0 | 56.3 | 84.3 |
| 4 | 63.1 | 103.2 |
| 32 | 67.6 | 113.8 |
| Fine-tuned state of the art (at the time) | 80.2 | 143.3 |
More shots help steadily. Across 16 benchmarks, Flamingo set a new few-shot state of the art, and on 6 of them it beat the fine-tuned state of the art using only 32 examples [Certain]. For classification with many classes, it chose the support examples by visual similarity to the query (RICES, retrieval-based in-context example selection) [Certain]. Flamingo was never released; that gap is what IDEFICS filled.
IDEFICS and Idefics2: what matters when building a VLM
IDEFICS: an open Flamingo
Hugging Face first built OBELICS, an open dataset of interleaved web documents: 141 million pages from Common Crawl, 353 million images and 115 billion text tokens [Certain]. On it they trained IDEFICS (9B and 80B), which follows Flamingo’s architecture closely: frozen LLaMA, frozen OpenCLIP encoder, a Perceiver resampler and new cross-attention blocks. In IDEFICS-80B, 14B of the 80B weights are trained: 13.9B in cross-attention, 126M in the resampler [Certain]. It showed that Flamingo’s recipe can be reproduced with open data.
Idefics2: six findings from controlled ablations
For the next model, the authors asked a sharper question: with everything else fixed, which choices actually matter? Each ablation trains for 6,000 steps and reports the average 4-shot score over VQAv2, TextVQA, OK-VQA and COCO [Certain]. The six findings, with their numbers:
- The language model matters more than the vision encoder. Replacing LLaMA-1-7B with Mistral-7B gave +5.1 (62.5 → 67.6). Replacing CLIP-ViT-H with SigLIP-SO400M gave +3.3 (57.4 → 60.7). EVA-CLIP-5B, ten times bigger than SigLIP, scored about the same (60.2), which suggests it is under-trained [Certain].
- Frozen backbones favor cross-attention; trainable backbones favor the unified sequence. With frozen backbones: cross-attention 66.7, fully autoregressive (image tokens in the sequence) 60.3. With LoRA on the backbones: fully autoregressive 69.5, cross-attention 67.3, even though cross-attention has 1.3B more trainable parameters [Certain].
- Unfreezing everything in the fully autoregressive design diverged. LoRA (session 57) both stabilized training and added enough freedom [Certain]. Note that Prismatic VLMs had found that unfreezing the encoder hurts; Idefics2 suggests parameter-efficient training is the difference.
- Learned pooling to 64 tokens helps. A Perceiver resampler that pools SigLIP’s 729 tokens to 64 raised the score by 8.5 points (51.8 → 60.3) and cut the tokens 11 times. 128 tokens was not better than 64 (71.2 vs 71.7) [Certain].
- Keeping the aspect ratio costs little and saves compute. Passing images at their own size and aspect ratio (378 to 768 pixels) scored 72.1 versus 73.1 for 768-pixel squares, while saving memory and time [Certain] (session 180).
- Image splitting trades compute for accuracy. Splitting into 4 crops plus the full image (5 × 64 = 320 tokens) helps most on text reading; it is used only in the instruction stage, and you can turn it on or off at inference [Certain].
How to read Figure 7
In the Flamingo view, the longest drops are removing M3W (−17.3), training the LM from random weights (−12.9) and a single cross-attention block in the middle (−10.9). Unfreezing the pretrained LM also hurt (−8.0): this is catastrophic forgetting measured directly. In the Idefics2 view, the architecture group is measured against the frozen fully autoregressive model (60.3): LoRA on its backbones adds +9.2, and removing the perceiver pooling costs −8.5. Small differences (around 1 point) after only 6,000 steps should not be over-read.
The Idefics2 recipe
The final Idefics2-8B: SigLIP-SO400M encoder, a modality projection plus a Perceiver resampler to 64 tokens per image (or 320 with splitting), Mistral-7B-v0.1, fully autoregressive, images up to 980×980 at their native aspect ratio. Pretraining has two stages on OBELICS, image–text pairs with synthetic captions (LAION COCO) and, in the second stage, PDF documents for OCR; the whole model is trained, with LoRA on the pretrained backbones, at learning rate \(10^{-4}\) for about 2 epochs: about 1.5 billion images and 225 billion text tokens [Certain]. Synthetic captions beat alt-text in their ablation (52.9 vs 49.8) [Certain]. Instruction tuning uses The Cauldron, 50 vision–language datasets in one question–answer format, plus text-only instruction data, with DoRA and loss only on the answers [Certain].
Results with only 64 tokens per image: MMMU 43.5 (val), MathVista 51.6, TextVQA 70.4, MMBench 76.8; with 320 tokens TextVQA rises to 73.0 [Certain]. For comparison in the same table, LLaVA-NeXT-13B uses 2,880 tokens per image and scores 36.2 on MMMU and 67.1 on TextVQA [Certain]. As a base model with 8 in-context examples, Idefics2 scored 70.3 on VQAv2, against 58.0 for Flamingo-9B and 56.4 for IDEFICS-9B [Certain].
The four designs compared
| LLaVA-1.5 | BLIP-2 | Flamingo | Idefics2 | |
|---|---|---|---|---|
| Vision encoder | CLIP ViT-L/14, 336 px, frozen | EVA-CLIP ViT-g/14 (or CLIP ViT-L), 224 px, frozen | NFNet-F6, frozen | SigLIP-SO400M, LoRA |
| Connector | 2-layer MLP, 21M (7B) | Q-Former 188M + linear | Perceiver resampler 194M | MLP + Perceiver resampler |
| Language model | Vicuna-7B/13B, trained in stage 2 | OPT or FlanT5, frozen | Chinchilla 1.4B/7B/70B, frozen | Mistral-7B, LoRA |
| How the LLM sees the image | 576 tokens in the sequence | 32 tokens in the sequence | 64 tokens via gated cross-attention | 64 (or 320) tokens in the sequence |
| Key data | 558K captions + 665K instructions | 129M images, CapFilt captions | M3W, ALIGN, LTIP, VTP | OBELICS, LAION COCO, PDFs, The Cauldron |
| Best at (in its time) | chat, cheap to reproduce | zero-shot VQA, captioning | few-shot, many images, video | efficiency per token, OCR, open data |
The connector types are studied in depth in session 178 and the two families in session 179. The broad trend after 2024: the unified sequence with an MLP won for general models, because when the LLM can be trained, it is simpler and as good or better (Idefics2 finding 2; MM1 found the connector type matters little). Compression modules (Q-Former, resampler, pixel shuffle) survive where tokens are expensive: many images, video and phones.
A prompt with 5 images and 256 text tokens. LLaVA-1.5: \(5 \times 576 + 256 = 3{,}136\) tokens in the sequence, so self-attention computes about \(3{,}136^2 \approx 9.8\) million scores per head per layer at prefill. Idefics2 (64 per image): \(5 \times 64 + 256 = 576\) tokens, about 0.33 million scores, 30 times fewer. Flamingo: 256 tokens in the sequence; each cross-attention block scores \(256 \times 64 = 16{,}384\) pairs, the same for 1 or 32 images thanks to image-causal masking (it still encodes and resamples every image). With 32 images LLaVA-1.5 would need \(32 \times 576 + 256 = 18{,}688\) tokens, more than Vicuna v1.5’s 4,096-token context.
Code: the four key mechanisms in PyTorch
The file below runs on a CPU in about a second and is deterministic. It builds the LLaVA projector and label mask, the three Q-Former masks, the ITC loss with the max over queries, a Flamingo gated cross-attention block with image-causal masking, and a Perceiver resampler.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) LLaVA-1.5: MLP projector + loss only on the answer, formulas (181.2) and (181.3) d_vis, d_llm, n_vis = 1024, 4096, 576 proj = nn.Sequential(nn.Linear(d_vis, d_llm), nn.GELU(), nn.Linear(d_llm, d_llm)) print("projector params:", sum(p.numel() for p in proj.parameters())) # 20,979,712 Zv = torch.randn(1, n_vis, d_vis) # CLIP layer -2 features, CLS dropped Hv = proj(Zv) # 576 visual tokens in the LLM's space prompt_ids, answer_ids = torch.arange(10, 30), torch.arange(30, 42) # 20 prompt, 12 answer tokens labels = torch.cat([torch.full((n_vis,), -100), # image positions: no loss torch.full((20,), -100), # system + question: no loss answer_ids]) # answer: loss print("sequence length:", labels.numel(), "tokens with loss:", (labels != -100).sum().item()) # 2) BLIP-2 Q-Former: one self-attention mask per objective (True = may attend) def qformer_mask(nq, nt, kind): n = nq + nt; m = torch.zeros(n, n, dtype=torch.bool) m[:nq, :nq] = True # queries always see each other if kind == "ITC": m[nq:, nq:] = True # text sees text; no crossing if kind == "ITM": m[:, :] = True # everything sees everything if kind == "ITG": # text sees all queries + earlier text m[nq:, :nq] = True m[nq:, nq:] = torch.tril(torch.ones(nt, nt, dtype=torch.bool)) return m for k in ["ITC", "ITM", "ITG"]: print(k, "allowed pairs:", qformer_mask(4, 5, k).sum().item(), "of", 81) # 3) ITC similarity: best query wins, formula (181.5) Z = F.normalize(torch.randn(8, 32, 256), dim=-1) # 8 images x 32 query outputs t = F.normalize(torch.randn(8, 256), dim=-1) # 8 captions ([CLS] embedding) sim = torch.einsum("iqd,jd->ijq", Z, t).amax(-1) / 0.07 # [8 images, 8 texts] y = torch.arange(8) itc = (F.cross_entropy(sim, y) + F.cross_entropy(sim.T, y)) / 2 print("ITC loss on random vectors:", round(itc.item(), 3), " log(8) =", round(torch.log(torch.tensor(8.)).item(), 3)) # 4) Flamingo: gated cross-attention, tanh gate initialised at 0, image-causal mask class GatedXAttn(nn.Module): def __init__(s, d, heads=4): super().__init__() s.attn = nn.MultiheadAttention(d, heads, batch_first=True) s.ffw = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) s.a_attn = nn.Parameter(torch.zeros(1)); s.a_ffw = nn.Parameter(torch.zeros(1)) def forward(s, y, x, mask, has_img): # mask[i, j] = True: text i may NOT see visual j y = y + torch.tanh(s.a_attn) * s.attn(y, x, x, attn_mask=mask)[0] * has_img return y + torch.tanh(s.a_ffw) * s.ffw(y) d = 64 img_of_text = torch.tensor([0, 0, 1, 1, 1, 2, 2, 2]) # last image seen before each text token (0 = none) vis_img = torch.repeat_interleave(torch.tensor([1, 2]), 3) # 2 images x 3 visual tokens mask = img_of_text[:, None] != vis_img[None, :] mask[img_of_text == 0] = False # no image yet: any keys (avoid an all-masked row) ... has_img = (img_of_text > 0).float()[None, :, None] # ... and the result is set to zero blk, ytxt, xvis = GatedXAttn(d), torch.randn(1, 8, d), torch.randn(1, 6, d) out = blk(ytxt, xvis, mask, has_img) print("at init the frozen LM is unchanged:", torch.allclose(out, ytxt)) with torch.no_grad(): blk.a_attn.fill_(0.5) print("tanh(0.5) =", round(torch.tanh(torch.tensor(0.5)).item(), 3), " change:", round((blk(ytxt, xvis, mask, has_img) - ytxt).norm().item(), 3) > 0) # 5) Perceiver resampler (Flamingo, Idefics2): 729 visual tokens -> 64 learned latents class Resampler(nn.Module): def __init__(s, d, n_lat=64, layers=2): super().__init__() s.lat = nn.Parameter(torch.randn(n_lat, d) * 0.02) s.att = nn.ModuleList(nn.MultiheadAttention(d, 4, batch_first=True) for _ in range(layers)) s.ff = nn.ModuleList(nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) for _ in range(layers)) def forward(s, xf): x = s.lat.expand(xf.size(0), -1, -1) for att, ff in zip(s.att, s.ff): kv = torch.cat([xf, x], dim=1) # keys/values: image features AND latents, formula (181.6) x = x + att(x, kv, kv)[0] x = x + ff(x) return x print("resampler:", tuple(Resampler(d)(torch.randn(2, 729, d)).shape)) # (2, 64, 64)
Output (PyTorch 2.x, CPU): projector params: 20979712, sequence length: 608 tokens with loss: 12, allowed pairs 41, 81 and 51 (the same as Figure 5), ITC loss on random vectors: 2.215 (close to \(\log 8 = 2.079\), as expected for an untrained model), at init the frozen LM is unchanged: True, tanh(0.5) = 0.462, and resampler: (2, 64, 64).
Where this is used
| Application | Which idea | Why |
|---|---|---|
| General visual chat assistants (open models) | LLaVA recipe: encoder + MLP + LLM, two stages, instruction data with format prompts | Simple, cheap to reproduce, strong when the LLM is trained (session 186) |
| Image captioning and alt-text at scale | BLIP-2 with a frozen LLM | Small trainable module, good zero-shot captions; the same captioners clean web data (session 187) |
| Video question answering and video captioning | Flamingo-style 1 FPS frames + resampler to a fixed token count; Q-Former in video models such as Video-LLaMA [Likely] | A fixed number of tokens per frame or clip keeps long videos inside the budget |
| Key-moment detection and video retrieval | BLIP-2 / Q-Former ITC embeddings per frame, then rerank with a generative VLM [Likely] | The contrastive head gives cheap per-frame scores; generation is used only on the top candidates |
| Few-shot labeling of new classes (defects, sports actions) | Flamingo / IDEFICS in-context prompts, examples chosen by RICES | No fine-tuning: change the examples in the prompt to change the task |
| Multi-image and Video-RAG answers | Pooled tokens (64 per image, Idefics2) or cross-attention | Retrieved frames and documents fit in the context without thousands of tokens each |
| Document and chart reading | Idefics2 image splitting, native aspect ratio, PDF pretraining | Text in images needs resolution; splitting buys it only when needed |
Common mistakes
- Computing loss on the image tokens or the prompt. Labels at image and instruction positions must be −100. Otherwise the model wastes capacity predicting system prompts and the training signal for the answer is diluted.
- Forgetting the response-format prompt when mixing short-answer VQA data with chat data. The model learns to answer in one word everywhere. LLaVA-1.5 fixed this with prompts such as “Answer the question using a single word or phrase.”
- Initializing new gates or projections so that they disturb a frozen LLM at step 0. Flamingo starts gates at 0; PaliGemma zero-initializes its projection [Likely]. Random large outputs into a frozen LM give a high starting loss and large, harmful gradients.
- Skipping BLIP-2’s stage 1 to save time. The paper shows lower zero-shot VQA and forgetting with OPT. A frozen LLM needs well-formed visual prompts.
- Using the wrong mask for the wrong objective in a shared Q-Former. A bidirectional mask in ITC leaks the text into the image embedding and makes the contrastive task trivial.
- Letting every text token attend to all previous images in a cross-attention model. Cost grows with the number of images, and a test-time prompt with more images than in training goes out of distribution.
- Reading ablation tables across groups. In Idefics2, each table is a separate run; 72.1 in one group and 60.3 in another are not comparable. Compare only within a group, and do not over-read 1-point gaps after 6,000 steps.
- Counting “trainable parameters” as cost. BLIP-2 trains 108M weights but the forward pass still runs the full 12.1B model. Training compute depends on total parameters and tokens, not only trainable ones.
Interview questions
Describe the LLaVA architecture and its two training stages.basic
How did LLaVA create instruction data without a multimodal model?core
What is the Q-Former and why does BLIP-2 need it?core
Explain the three Q-Former losses and their attention masks.advanced
Why are Flamingo’s gates initialized at zero?core
What is image-causal masking in Flamingo and what does it buy?advanced
BLIP-2 beat Flamingo-80B on zero-shot VQAv2 with 54x fewer trainable parameters. Does that mean BLIP-2 is 54x cheaper?advanced
Cross-attention or unified sequence: which is better?core
What did Idefics2 find matters most when building a VLM?core
Why does few-shot in-context learning work in Flamingo but not in LLaVA-1.5?advanced
You must build a VLM that runs on many video frames per question. Which ideas from these four papers would you reuse?advanced
Exercises
- LLaVA (original) uses CLIP ViT-L/14 at 224 pixels and a linear projection to Vicuna-13B (\(d = 5120\)). How many visual tokens per image, and how many weights in the projection (with bias)?
- For the Q-Former masks with 32 queries and 20 text tokens, how many query–text and text–text pairs are allowed in ITC, ITM and ITG? (Count all \(52^2\) cells.)
- Flamingo-80B adds a cross-attention block before layer 0 and then before every 7th layer of an 80-layer LM. How many blocks? What is \(\tanh(\alpha)\) for \(\alpha = 1\)?
- In Figure 6’s prompt with image-causal masking, how many cross-attention scores (per head) does the last token compute if each image has 64 tokens and there are 32 shots? And with “all previous images”?
- Idefics2 with image splitting uses 320 tokens per image. How many image tokens for a 3-image prompt, and how does that compare with LLaVA-1.5?
- BLIP-2 ViT-g FlanT5-XXL trains 108M of 12.1B weights; Flamingo-80B trains about 10.2B of 80B. Compute both fractions.
Answerscheck
Key takeaways
- LLaVA: CLIP + projection (181.1, 181.2) + LLM, trained in two stages with loss only on answers (181.3). The real innovation was the data: 158K GPT-4-written instruction samples. LLaVA-1.5 showed that an MLP, 336 px and well-formatted academic data are enough to lead benchmarks in one day on 8 A100s.
- BLIP-2: freeze encoder and LLM; train a 188M Q-Former whose 32 queries cross-attend to image features (181.4). Stage 1 uses ITC (with the max over queries, 181.5), ITM and ITG with three masks; stage 2 feeds 32 soft-prompt vectors to the frozen LLM. Stage 1 prevents forgetting.
- Flamingo: a Perceiver resampler to 64 tokens (181.6), gated cross-attention starting at zero (181.7), image-causal masking (181.8) and a weighted multi-dataset loss (181.9) over interleaved web data give real few-shot learning.
- Idefics2: the LLM matters most; frozen backbones favor cross-attention, LoRA-trained backbones favor the unified sequence; 64 pooled tokens beat 729; native aspect ratio and image splitting give control over cost.
- Compare designs by what is trained, how many visual tokens enter the context, and what data teaches the connection.
Key formulas
References for this session
- Liu et al. (2023). Visual Instruction Tuning
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Laurençon et al. (2024). What matters when building vision-language models?
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
- Liu et al. (2023). Visual Instruction Tuning
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Laurençon et al. (2024). What matters when building vision-language models?
- Laurençon et al. (2023). OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
- Li et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
- Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation
- Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
- McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
- Dai et al. (2023). InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Fang et al. (2022). EVA: Exploring the Limits of Masked Visual Representation Learning at Scale