LLM & VLM courseBlog
181Track 7: Vision and VLMs · session 19 of 38

Case studies 1: LLaVA, BLIP-2, Flamingo, IDEFICS

Four models shaped almost every open vision-language model that came after them. LLaVA joined CLIP to an LLM with one small projection and trained it on instruction data written by GPT-4. BLIP-2 froze both big models and trained only a small Q-Former between them. Flamingo read images and text mixed in one document and learned new tasks from a few examples in the prompt. Idefics2 then ran careful ablations to find out which of these choices really matter. This session takes each one apart, with the real numbers from the papers.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 166 (CLIP), 173 (ALBEF and BLIP) and 177 (anatomy of a vision-language model); session 20 (self-attention vs cross-attention) helps

What you will learn

  • LLaVA: a CLIP encoder, a linear (later MLP) projector and Vicuna, trained in two stages, and how GPT-4 wrote 158K instruction samples from captions and boxes without seeing the images
  • LLaVA-1.5: the small changes (MLP, 336 px, academic VQA data with response-format prompts) that took it to the top of 11 benchmarks in about one day on 8 A100s
  • BLIP-2: a 188M Q-Former with 32 learned queries between a frozen image encoder and a frozen LLM, its three stage-1 losses and their attention masks, and why stage 1 prevents forgetting
  • Flamingo: the Perceiver resampler, gated cross-attention layers that start at zero, image-causal masking, and few-shot prompting with interleaved images and text
  • IDEFICS and Idefics2: an open Flamingo, then six ablation findings (language model first, frozen vs LoRA, 64 pooled tokens, aspect ratio, image splitting)
  • How to compare designs by trainable parameters, visual tokens in the context and training data, with numbers you can check

Why this matters

Between 2022 and 2024, four papers set the patterns that most open vision-language models still follow. If you can explain these four, you can place almost any new model on the map in a few minutes.

In session 177 we took a VLM apart into three parts: a vision encoder, a connector and a language model. We saw two ways to join them: put image tokens into the LLM’s sequence, or let the text attend to the image through new cross-attention layers. This session is the history behind that map. Each case study answers one big question:

  • LLaVA (2023): how simple can the bridge be? Answer: one linear layer (later a small MLP), if you have good instruction data. LLaVA got that data by asking a text-only GPT-4 to write conversations about images it never saw.
  • BLIP-2 (2023): can we keep both big models frozen and train only a small module in the middle? Answer: yes, with a 188M-parameter Q-Former trained in two stages.
  • Flamingo (2022): can a VLM learn a new task from a few examples in the prompt, like GPT-3? Answer: yes, if it is trained on web pages where images and text are interleaved.
  • IDEFICS and Idefics2 (2023, 2024): an open copy of Flamingo, then a careful study of what matters when you build a VLM, with controlled ablations.

These are not museum pieces. LLaVA’s recipe (encoder, MLP, LLM, two stages) is the starting point of Qwen-VL, InternVL and many others (session 182). The Q-Former and the Perceiver resampler live on as token compressors, which matter a lot for video, where you have many frames (session 183). Flamingo’s gated cross-attention returned in Llama 3.2 Vision [Likely]. And Idefics2’s ablations are the best public evidence for what to spend your compute on.

One frame for four models

Each model makes three choices. Which parts are trained? How many visual tokens does the LLM see per image, and where? What data teaches the connection? We will keep coming back to these three. Figure 1 lets you compare the four models on the first two.

Figure 1. Four designs side by side. Parameters: log scale; for each model the light bar is the total size and the colored bar is what is updated in the main multimodal training stage (the rest is frozen). Visual tokens: how many image tokens enter the LLM’s own sequence (its context and KV cache) for the chosen number of images. Flamingo’s image tokens are read by cross-attention instead, so they do not use context.
How to read Figure 1

In “Parameters”, compare BLIP-2 and Flamingo-80B: both freeze the LLM, but BLIP-2 trains 108M weights (0.9% of 12.1B) and Flamingo trains about 10.2B (13% of 80B), mostly in its new cross-attention layers. LLaVA-1.5 trains a 21M MLP first, then also the whole 6.7B LLM. In “Visual tokens”, move the slider to 5: LLaVA-1.5 puts 2,880 image tokens in the context, Idefics2 320 (or 1,600 with image splitting), BLIP-2 160, Flamingo zero. This is why Flamingo-style and pooled designs were the first to handle many images, and why video models compress tokens.

LLaVA: the simplest bridge, and data written by GPT-4

The architecture

LLaVA (Large Language and Vision Assistant) takes a pretrained CLIP ViT-L/14 image encoder \(g\) and a pretrained LLM (Vicuna, a chat-tuned LLaMA). Between them sits one trainable matrix \(W\). The image \(X_v\) goes through CLIP; the grid of patch features \(Z_v\) is multiplied by \(W\) so that each patch becomes a vector of the same size as a word embedding:

\[ H_v = W \cdot Z_v, \qquad Z_v = g(X_v) \tag{181.1} \]
Read the symbols
\(X_v\)
the input image
\(g\)
the frozen CLIP vision encoder; LLaVA reads the grid features of a late layer (before or after the last one; the second-to-last worked best on ScienceQA)
\(Z_v\)
one 1,024-dimensional vector per image patch
\(W\)
the trainable projection, \(1024 \to d_{\text{LLM}}\)
\(H_v\)
the visual tokens: vectors the LLM reads exactly like word embeddings

That is the whole connector. The visual tokens are placed in the sequence where the image appears, and the LLM does the rest. LLaVA-1.5 replaced the single matrix with a two-layer MLP, inspired by self-supervised learning, where an MLP head beats a linear one:

\[ H_v = W_2 \, \mathrm{GELU}(W_1 Z_v + b_1) + b_2 \tag{181.2} \]

For LLaVA-1.5-7B, \(W_1\) is \(1024 \times 4096\) and \(W_2\) is \(4096 \times 4096\): 20,979,712 parameters, about 21M. The image is 336×336 pixels and the patch is 14 pixels, so CLIP gives \(24 \times 24 = 576\) patch tokens (the CLS token is dropped). The Hugging Face config confirms it: GELU activation, features from layer −2 [Certain].

Figure 2 runs this projector in your browser on a toy image. The image is cut into a grid of patches; each patch has a 2-number CLIP feature (top patches look like sky, bottom patches like ground). A tiny MLP with the shape of (181.2), \(2 \to 3 \to 3\), turns each patch into one visual token. Watch the sequence at the bottom: one patch gives one token, so the token count is set by the grid.

patch / visual tokentext tokencurrent patch
  1. input: image \(X_v\); projector \(W_1, b_1, W_2, b_2\)
  2. \(Z_v \leftarrow g(X_v)\) // one vector \(\mathbf{z}_p\) per patch
  3. for \(p = 1 \dots N_{\text{patch}}\):
  4. \(\mathbf{a} \leftarrow W_1 \mathbf{z}_p + b_1\)
  5. \(\mathbf{u} \leftarrow \mathrm{GELU}(\mathbf{a})\)
  6. \(\mathbf{h}_p \leftarrow W_2 \mathbf{u} + b_2\) // (181.2)
  7. append \(\mathbf{h}_p\) to the LLM sequence
  8. LLM reads [text] + \(\mathbf{h}_1 \dots \mathbf{h}_{N_{\text{patch}}}\) + [text]
Figure 2. The LLaVA-1.5 projector, animated on a toy image. Left: the patch grid with each patch’s 2-number feature (a dot marks patches already projected). Middle: the MLP on the current patch, with the live numbers. Right: the visual-token counter. Bottom: the sequence the LLM reads, filling one visual token per patch. Every number is computed in your browser.
How to read Figure 2

Press Play or Step. For patch 1, \(\mathbf{z} = (1.1, -0.2)\) gives \(\mathbf{a} = (1.080, -0.050, -0.740)\), GELU gives \(\mathbf{u} = (0.929, -0.024, -0.170)\): the large positive value passes almost unchanged, the negative ones shrink toward 0. Then \(\mathbf{h}_1 = (0.713, -0.283, 0.133)\) joins the sequence. With 3 × 3 patches the LLM reads 9 visual tokens + 7 text tokens; switch to 4 × 4 and it reads 16. LLaVA-1.5 has 24 × 24 patches, so 576 tokens per image. The projector never looks at two patches together: it is a lookup from “CLIP words” to “LLM words”, applied to each patch.

The data: GPT-4 writes about images it cannot see

In early 2023 there was almost no data of the form “image, instruction, good long answer”. LLaVA’s key idea was to create it with a text-only GPT-4 (and ChatGPT). For a COCO image, the authors gave GPT-4 a symbolic description: the human-written captions and the object bounding boxes (class name plus coordinates). GPT-4 never saw pixels. From that text it wrote three kinds of responses [Certain]:

  • Conversation: a multi-turn chat with questions a person might ask about the image (what objects, how many, where, what are they doing). 58K samples.
  • Detailed description: a long, complete description, prompted from a list of instructions such as “Describe the following image in detail”. 23K samples.
  • Complex reasoning: questions that need step-by-step reasoning beyond what is visible (“What challenges might these people face?”). 77K samples.

Total: 158K instruction samples, released as LLaVA-Instruct-158K. A few hand-written examples were placed in each prompt as few-shot demonstrations, so GPT-4 knew the style.

Algorithm 1: LLaVA instruction-data generation
  1. input: images with captions \(C\) and boxes \(B\) (COCO), a text-only LLM (GPT-4), a few hand-written seed examples per type
  2. for each image \(X_v\) do
  3.   context \(\leftarrow\) the captions \(C\) and the boxes \(B\) as plain text // e.g. “person: [0.68, 0.24, 0.77, 0.69]”
  4.   type \(\leftarrow\) one of {conversation, detailed description, complex reasoning}
  5.   prompt \(\leftarrow\) system message for that type + seed examples + context
  6.   response \(\leftarrow\) GPT-4(prompt) // GPT-4 never sees the pixels
  7.   store \((X_v, \text{response})\) as one training sample
  8. return the instruction dataset (158K samples)

Two-stage training and the loss

Every training sample is turned into one token sequence. LLaVA uses a chat format: a system message, then turns of “Human: … <STOP> Assistant: … <STOP>”, with the image tokens placed in the first human turn. The model is trained with the normal next-token loss, but only on the answer tokens. For an answer \(X_a\) of length \(L\):

\[ p(X_a \mid X_v, X_{\text{instruct}}) = \prod_{i=1}^{L} p_\theta\big(x_i \mid X_v,\ X_{\text{instruct}, \lt i},\ X_{a, \lt i}\big) \tag{181.3} \]
Read the symbols
\(x_i\)
the \(i\)-th token of the assistant’s answers (all turns)
\(X_{\text{instruct}, \lt i}\), \(X_{a, \lt i}\)
the instructions and the answers that come before token \(i\): the model can see earlier turns
\(\theta\)
the trainable weights: \(W\) only in stage 1; \(W\) and the LLM in stage 2

The image tokens, the system message and the human turns are part of the input, but they get no loss: their label is set to −100 in PyTorch, so cross-entropy ignores them. Figure 3 shows which tokens are trained.

  • Stage 1, feature alignment. 595K image–caption pairs filtered from CC3M. Each pair becomes a one-turn sample: a short random instruction (“Describe the image briefly”) and the caption as the answer. Encoder and LLM are frozen; only \(W\) is trained. One epoch, learning rate \(2 \times 10^{-3}\), batch 128, under 4 hours on 8 A100s [Certain]. The paper calls this “training a compatible visual tokenizer for the frozen LLM”.
  • Stage 2, end-to-end fine-tuning. The 158K instruction samples. Encoder still frozen; \(W\) and the whole LLM are trained. Three epochs, learning rate \(2 \times 10^{-5}\), batch 32, about 10 hours [Certain].
input only (label −100)image tokens (input only)answer tokens (loss)
Figure 3. One training sequence for each case, drawn as blocks with their token counts (text counts are a typical example; the 576 image tokens are exact for LLaVA-1.5). Block width grows with the square root of the count (and is at least as wide as its label), so that short blocks stay visible. Click a block to see its content.
How to read Figure 3

In stage 1, only the caption and the stop marker (14 tokens) get loss, out of 605 tokens: 2.3%. Most of the sequence is image tokens, which teach nothing directly; they only condition the answer. In the two-turn chat, both assistant answers get loss, and the second answer can see the first question and answer. In the LLaVA-1.5 VQA sample, the instruction ends with “Answer the question using a single word or phrase.” and the answer is one or two tokens. This format prompt is what let LLaVA-1.5 learn from short-answer VQA datasets without forgetting how to write long answers.

Worked example: what one LLaVA-1.5 sample costs

Take the two-turn chat of Figure 3: 576 image tokens + 68 tokens of system message, questions and role markers + 119 tokens of answers and stop markers. Sequence length 763; loss on 119 tokens (15.6%). The 576 image tokens are 75% of the sequence. With five images it would be \(5 \times 576 = 2{,}880\) image tokens, which is exactly the number Idefics2 reports for LLaVA-NeXT-13B per image, because LLaVA-NeXT splits one image into up to four crops plus a downsized full view [Likely]. The projector for the 13B model (\(d = 5120\)) has \(1024 \cdot 5120 + 5120 + 5120 \cdot 5120 + 5120 = 31{,}467{,}520\) weights.

LLaVA-1.5: small changes, large gains

The first LLaVA was a good chatbot but weak on academic benchmarks with short answers, such as VQAv2. LLaVA-1.5 (October 2023) kept the design and changed four things: an MLP projector (formula 181.2), CLIP at 336 pixels, Vicuna v1.5, and a 665K instruction mix that adds academic VQA, OCR and region data, each with a response-format prompt. Its ablation table shows how each step helped (7B unless noted) [Certain]:

StepResolutionGQAMMEMM-Vet
LLaVA (original)224–809.625.5
+ VQAv222447.01197.027.7
+ format prompt22446.81323.826.3
+ MLP connector22447.31355.227.8
+ OKVQA, OCR data22450.01377.629.6
+ region-level VQA22450.31426.530.8
+ resolution 33633651.4145030.3
+ GQA data33662.01469.230.7
+ ShareGPT (text-only chat)33662.01510.731.1
+ 13B LLM33663.31531.336.1

The GQA jump from 51.4 to 62.0 comes from training on GQA itself, so it is not zero-shot; the paper marks such numbers [Certain]. The final model used 558K pretraining pairs and 665K instruction samples, about 1.2M in total, and the 13B model trained in about one day on one 8-A100 node [Certain]. The abstract reports state-of-the-art results on 11 benchmarks at the time [Certain]. The lesson that spread across the field: data and format matter more than a clever connector. LLaVA-1.5 also changed hyper-parameters very little: the stage-1 learning rate was halved to \(10^{-3}\) because the MLP is deeper.

BLIP-2: freeze both, train the Q-Former

The idea

Training a large VLM end to end is expensive, and updating the LLM can make it forget its language skills (catastrophic forgetting). BLIP-2 keeps both the image encoder and the LLM frozen. The only trained part is a Querying Transformer (Q-Former) in the middle. Its job is to read the frozen image features and produce a short, fixed set of vectors that the frozen LLM can understand.

The Q-Former is a BERT-base-sized transformer (initialized from BERT-base) with one extra ingredient: 32 learned query vectors of dimension 768. These queries are parameters, not inputs. They pass through self-attention layers (where they can also see text tokens, depending on the task) and through cross-attention layers into the frozen image features, inserted in every other block. The cross-attention layers start from random weights. In total, 188M parameters [Certain].

\[ Q^{(\ell+1)} = Q^{(\ell)} + \softmax\!\left(\frac{(Q^{(\ell)} W_Q)(F W_K)^\top}{\sqrt{d}}\right) F W_V \tag{181.4} \]
Read the symbols
\(Q^{(\ell)}\)
the 32 query vectors after layer \(\ell\), a \(32 \times 768\) matrix (before layer 1: the learned queries)
\(F\)
the frozen image features, for example \(257 \times 1024\) for CLIP ViT-L/14 (256 patches plus CLS); BLIP-2 uses the second-to-last layer of the ViT
\(W_Q, W_K, W_V\)
the cross-attention weights, trained

The output \(Z\) is only \(32 \times 768 = 24{,}576\) numbers, versus \(257 \times 1024 = 263{,}168\) for the ViT-L features: about 11 times fewer. The paper calls this a bottleneck: it forces the queries to keep only the visual information that is useful for text.

Figure 4 runs one Q-Former cross-attention layer, formula (181.4), on the same kind of toy image. Each learned query computes a score with every image feature, turns the scores into weights with a softmax, and adds the weighted sum of the features to itself. To keep the numbers readable, \(W_Q = W_K = W_V = I\) and \(d = 2\). A last linear layer \(P\) (\(2 \to 3\)) maps each query to the LLM’s size. Change the patch grid: the output stays at one token per query.

scores (keys)softmax weights (values)queries and output tokens
  1. input: frozen features \(F\) (\(N_{\text{patch}} \times d\)), learned queries \(Q\) (\(n_q \times d\)), projection \(P\)
  2. for each query \(\mathbf{q}_i\):
  3. \(s_{ij} \leftarrow \mathbf{q}_i \cdot \mathbf{f}_j / \sqrt{d}\) for every feature \(j\)
  4. \(\mathbf{w}_i \leftarrow \softmax(\mathbf{s}_i)\)
  5. \(\mathbf{q}_i \leftarrow \mathbf{q}_i + \sum_j w_{ij} \mathbf{f}_j\) // (181.4)
  6. return \(P \mathbf{q}_1, \dots, P \mathbf{q}_{n_q}\): \(n_q\) tokens, prepended to the text
Figure 4. One Q-Former cross-attention layer, animated. Left: the image features; during a query’s turn the grid shows its scores (orange), then its softmax weights (green). Middle: the learned queries; a prime (′) marks a query that has been updated. Right: the current query’s weighted sum, its new value and its LLM token. Bottom: what the frozen LLM reads. All numbers are computed in your browser.
How to read Figure 4

With 3 × 3 patches and 2 queries: \(Q_1 = (2, 0)\) “looks for sky”. Its largest score is 1.980 on patch 3, whose feature is \((1.4, -0.4)\); after the softmax that patch gets weight 0.258 and each bottom-row (ground) patch gets less than 0.05. Q1 reads \(\mathbf{o} = (0.986, -0.018)\) and becomes \(Q_1' = (2.986, -0.018)\). \(Q_2 = (0, 2)\) looks for ground and puts its largest weight, 0.243, on patch 7. Now switch to 4 × 4: 16 features go in, still 2 tokens come out. BLIP-2 does this with 32 queries over 257 features, in several layers, with trained \(W_Q, W_K, W_V\).

Stage 1: three losses, three attention masks

In stage 1 the Q-Former is trained with the frozen image encoder but without the LLM, using three losses inherited from BLIP (session 173). The same network handles all three; only the self-attention mask between queries and text changes:

  • Image–text contrastive (ITC). Pull matching image and text embeddings together, push others apart. Queries and text must not see each other (a unimodal mask), or the task would leak. The image has 32 output vectors \(z_1, \dots, z_{32}\) and the text has one, the [CLS] output \(t\). The similarity is the best match:
    \[ s(I, T) = \max_{q = 1, \dots, 32} \; z_q^\top t \tag{181.5} \]
    Because the encoder is frozen, BLIP-2 fits more samples per GPU and uses in-batch negatives instead of BLIP’s momentum queue [Certain].
  • Image–text matching (ITM). A binary classifier: does this text describe this image? Here everything sees everything (a bidirectional mask). Each of the 32 query outputs goes through a two-class linear layer, and the 32 logits are averaged. Hard negatives are picked as in ALBEF [Likely].
  • Image-grounded text generation (ITG). Generate the caption. Text tokens see all queries and the earlier text tokens; queries do not see the text (a multimodal causal mask, as in UniLM). The text cannot see the image directly, so all visual information must flow through the queries. The [CLS] token is replaced by a [DEC] token.

The stage-1 loss is the sum \(\mathcal{L}_1 = \mathcal{L}_{\text{ITC}} + \mathcal{L}_{\text{ITM}} + \mathcal{L}_{\text{ITG}}\). Figure 5 shows the three masks.

query \(\to\) queryquery \(\leftrightarrow\) texttext \(\to\) textblocked
Figure 5. The Q-Former’s self-attention mask for a toy case with 4 queries (the real model has 32) and 5 text tokens. Row = the token that looks; column = the token it may look at. Click a row label to highlight what that token can see.
How to read Figure 5

ITC allows 41 of 81 pairs: the two blocks on the diagonal only, so the image summary and the text summary are computed separately and can be compared by a dot product. ITM allows all 81, so the queries can check the text word by word. ITG allows 51: the text rows see all queries and a lower triangle of text, while the query rows still see only queries. Click the “[DEC]” row in ITG: it sees the 4 queries and itself, so the first word is generated from the image alone.

Worked example: the max in ITC

Take 4 query outputs \(z_1 = (1, 0)\), \(z_2 = (0, 1)\), \(z_3 = (0.6, 0.8)\), \(z_4 = (-1, 0)\) and a text embedding \(t = (0.6, 0.8)\), all unit length. The dot products are \(0.6, 0.8, 1.0, -0.6\). Formula (181.5) takes the maximum, 1.0: one query that found the right content is enough. The mean would be 0.45 and would punish the image for queries that look at other things, which is exactly what we want queries to be free to do.

Stage 2: plug in the frozen LLM

Now the Q-Former, still attached to the frozen image encoder, is connected to a frozen LLM. A single fully connected layer maps the 32 query outputs from 768 to the LLM’s embedding size (2,560 for OPT-2.7B, so \(768 \cdot 2560 + 2560 = 1{,}968{,}640\) weights). The 32 projected vectors are prepended to the text embeddings as a soft visual prompt. For a decoder-only LLM (OPT), the loss is plain language modeling of the text. For an encoder–decoder LLM (FlanT5), the text is split: the first part goes with the visual prompt into the encoder, the rest is the target for the decoder.

Why not skip stage 1? The paper tested it. Without stage-1 representation learning, zero-shot VQA was much lower for both LLM types, and with OPT the performance fell as training went on: catastrophic forgetting [Certain]. Stage 1 gives the LLM visual prompts that already carry text-relevant information, so the frozen LLM does not have to adapt to raw features.

Algorithm 2: BLIP-2 two-stage pretraining
  1. input: frozen image encoder \(E\) (ViT-L/14 from CLIP or ViT-g/14 from EVA-CLIP), frozen LLM, 129M images (BLIP’s data, with CapFilt captions)
  2. initialize the Q-Former from BERT-base; cross-attention and the 32 queries random
  3. Stage 1, 250k steps: for each batch \((I, T)\): \(F \leftarrow E(I)\) // no gradient into E
  4.   \(\mathcal{L}_1 \leftarrow \mathcal{L}_{\text{ITC}}\)(unimodal mask) + \(\mathcal{L}_{\text{ITM}}\)(bidirectional mask) + \(\mathcal{L}_{\text{ITG}}\)(causal mask); update the Q-Former
  5. Stage 2, 80k steps: add a linear layer \(P: 768 \to d_{\text{LLM}}\)
  6.   for each batch: prompt \(\leftarrow P(\text{QFormer}(E(I)))\), 32 vectors; \(\mathcal{L}_2 \leftarrow\) LM loss of the frozen LLM on \(T\) given the prompt
  7.   update the Q-Former and \(P\) only
  8. return Q-Former, \(P\)

The cost was modest for 2023: the largest model (ViT-g with FlanT5-XXL) took under 6 days for stage 1 and under 3 days for stage 2 on one machine with 16 A100 (40 GB) GPUs, with frozen weights stored in FP16 (BF16 for FlanT5) [Certain]. Images are 224×224.

The results, and the famous comparison

On zero-shot VQAv2 (test-dev), BLIP-2 with ViT-g and FlanT5-XXL scored 65.0, against 56.3 for Flamingo-80B: 8.7 points higher, with 188M trainable parameters against Flamingo’s 10.2B, which is 54 times fewer [Certain]. On OK-VQA, which needs world knowledge, Flamingo-80B stayed ahead (50.6 vs 45.9); the authors credit Flamingo’s 70B language model [Certain]. Two lessons: a good bottleneck can replace a lot of trainable capacity, and knowledge lives in the LLM, so a bigger LLM still wins on knowledge tasks.

BLIP-2’s weakness is that its frozen LLM was never taught to follow visual instructions, and 32 tokens are a tight budget for reading text in images. InstructBLIP added instruction tuning and made the Q-Former also read the instruction; LLaVA-style designs with more tokens later won on OCR-heavy tasks [Likely].

Flamingo: interleaved images and text, and few-shot learning

The goal

GPT-3 showed that a large LM can learn a task from a few examples in the prompt (in-context learning, session 143). Flamingo (DeepMind, 2022) asked for the same with images: give the model “(image, caption), (image, caption), image, ?” and it should continue in the same style. To do that, the model must read sequences of images and text mixed together, and it must keep the knowledge of a large frozen LM. Flamingo-80B is built on the frozen 70B Chinchilla LM and a frozen NFNet-F6 image encoder (pretrained contrastively, like CLIP).

The Perceiver resampler

The vision encoder gives a variable number of features (more for larger images, many more for video). The Perceiver resampler turns any number of features into exactly 64 visual tokens. Like the Q-Former, it has learned latent queries; each layer lets the latents attend to the image features and to themselves:

\[ X \leftarrow X + \mathrm{Attn}\big(q = X,\ kv = [X_f;\, X]\big), \qquad X \leftarrow X + \mathrm{FFW}(X) \tag{181.6} \]
Read the symbols
\(X\)
the 64 latent vectors (learned at the start, updated by each layer)
\(X_f\)
the flattened visual features; for video, frames are sampled at 1 frame per second, each frame is encoded, and a learned time embedding is added before flattening
\([X_f; X]\)
features and latents stacked: the keys and values

Flamingo uses 6 resampler layers of width 1,536 with 16 heads, 194M parameters, the same for all model sizes [Certain]. In its ablation, the Perceiver resampler beat an MLP and a plain transformer of similar budget, and was faster.

Gated cross-attention: start at zero

The visual tokens do not enter the LM’s sequence. Instead, new GATED XATTN-DENSE blocks are inserted between the frozen LM layers. In each one, the text hidden states \(y\) attend to the visual tokens \(x\) (queries from text, keys and values from the image), then pass through a new feed-forward layer. Both results are multiplied by a tanh gate with a learned scalar \(\alpha\):

\[ \begin{aligned} y &\leftarrow y + \tanh(\alpha_{\text{xattn}}) \cdot \mathrm{Attn}(q = y,\ kv = x) \\ y &\leftarrow y + \tanh(\alpha_{\text{dense}}) \cdot \mathrm{FFW}(y) \\ y &\leftarrow \text{frozen LM block}(y) \end{aligned} \tag{181.7} \]

Both \(\alpha\) start at 0, and \(\tanh(0) = 0\). So at the first step, every new block is an identity: the model is exactly the frozen LM, and its text skills are intact. As training goes on, the gates open where the image helps. The paper plots \(|\tanh \alpha|\) over training: the gates grow quickly from 0 in all layers, and more in deeper layers [Certain]. Without the gating the overall ablation score fell from 70.7 to 66.5 [Certain].

Cross-attention blocks are expensive, so Flamingo adds one before every layer in the 3B model, every 4th layer in the 9B model, and every 7th layer in the 80B model (12 blocks in 80 layers) [Certain]. These blocks are most of the trainable weights: 10B of the 80B model. Adding them every 4th layer instead of every layer made training 66% faster and lowered the overall score by only 1.9% [Certain].

Image-causal masking

A web page can have several images. Which image should a text token look at? Flamingo’s answer: only the last image before it. Formally, the model is trained to maximize

\[ p(y \mid x) = \prod_{\ell = 1}^{L} p\big(y_\ell \mid y_{\lt \ell},\ x_{\le \ell}\big) \tag{181.8} \]
Read the symbols
\(y_\ell\)
the \(\ell\)-th text token
\(x_{\le \ell}\)
the images and videos that come before token \(\ell\) in the interleaved sequence

In practice, the cross-attention mask lets token \(\ell\) see only the 64 visual tokens of the immediately preceding image. Earlier images still influence it indirectly, because the LM’s own self-attention sees all earlier text, and that text has already looked at its own images. Text before the first image sees no image. This makes the number of cross-attention scores per text token constant (64) no matter how many images the prompt has, and it let Flamingo train on up to 5 images per sequence and then run with 32 shots at test time. Try it in Figure 6, which also runs the gated update (181.7) on toy numbers.

visible visual token (darker = more weight)visual tokens read now (bigger = more weight)message \(\mathbf{a}\) (diamond)\(\mathbf{y}\) before\(\tanh(\alpha)\, \mathbf{a}\)\(\mathbf{y}\) after
  1. for each text token \(\ell\) with hidden state \(\mathbf{y}\):
  2. \(m \leftarrow\) the last image before \(\ell\) // image-causal mask (181.8)
  3. if there is none: \(\mathbf{a} \leftarrow \mathbf{0}\)
  4. else \(s_j \leftarrow \mathbf{y} \cdot \mathbf{x}_j / \sqrt{d}\) over the visual tokens of \(m\); \(\mathbf{w} \leftarrow \softmax(\mathbf{s})\); \(\mathbf{a} \leftarrow \sum_j w_j \mathbf{x}_j\)
  5. \(\mathbf{y} \leftarrow \mathbf{y} + \tanh(\alpha_{\text{xattn}})\, \mathbf{a}\) // (181.7)
  6. \(\mathbf{y} \leftarrow\) frozen LM block\((\mathbf{y})\) // not simulated
Figure 6. Flamingo’s gated cross-attention, animated on a 2-shot captioning prompt: “<image 1> A cat on a sofa. <image 2> A dog in snow. <image 3> A”. Left: the cross-attention mask; columns are visual tokens (4 per image here, 64 in Flamingo), rows are text tokens, and the current row shows its attention weights. Right: the gate and the current token’s 2-number hidden state before and after the update. Two steps per token: attend, then gated add. Click a text token to jump to it. The visual tokens and hidden states are toy numbers; every result is computed in your browser.
How to read Figure 6

Step to the token “dog” (or click it). With image-causal masking it sees only image 2; its weights are 0.33, 0.26, 0.25, 0.16 and the message is \(\mathbf{a} = (-0.094, 0.922)\), close to the “dog” image tokens. At \(\alpha = 0.5\) the gate is 0.462, so \(\mathbf{y}\) moves from \((-0.1, 0.9)\) to \((-0.143, 1.326)\). Switch to “all previous images”: the same row now reads 8 tokens, the cat image pulls the message to \((0.268, 0.644)\), and the final “A” reads 12 tokens instead of 4. With 32 shots of 64 tokens each it would read 2,112 (32 support images plus the query image). Now move the slider to 0: the gate is 0, the orange step disappears and the hidden state is exactly what the frozen LM computes. At \(\alpha = 2\) the gate is 0.964, nearly fully open. A negative \(\alpha\) gives a negative gate, which subtracts the visual message; training decides the sign. In every case the LM sequence holds 11 text tokens and no visual tokens.

Data and the training loss

Few-shot skill comes from the data. Flamingo was trained on a mixture of four datasets [Certain]:

  • M3W (MultiModal MassiveWeb): about 43 million web pages with 185M images and 182 GB of text, with <image> tags inserted where the images appear in the page layout. For each page, a random 256-token span with up to the first 5 images is taken.
  • ALIGN: 1.8 billion images with alt-text (noisy).
  • LTIP (Long Text and Image Pairs): 312M image–text pairs with longer, better descriptions.
  • VTP (Video and Text Pairs): 27M short videos (about 22 seconds on average) with descriptions.

The loss is a weighted sum of the per-dataset negative log-likelihoods:

\[ \mathcal{L} = \sum_{m=1}^{M} \lambda_m \, \E_{(x, y) \sim \mathcal{D}_m} \left[ -\sum_{\ell=1}^{L} \log p\big(y_\ell \mid y_{\lt \ell},\ x_{\le \ell}\big) \right] \tag{181.9} \]

The weights \(\lambda_m\) were tuned and are “key to performance”. Gradients are accumulated over all datasets in every step, which beat a round-robin schedule (one dataset per step): 70.7 vs 62.9 on the ablation score [Certain]. Removing M3W was the single most damaging change: the score fell from 70.7 to 53.4 [Certain]. The largest run used 1,536 TPU chips for 15 days [Certain].

Few-shot results

At test time there is no training. You build a prompt of \(k\) support examples and the query, and the model continues the text. Flamingo-80B on VQAv2 (test-dev) and COCO captions (CIDEr) [Certain]:

ShotsVQAv2COCO CIDEr
056.384.3
463.1103.2
3267.6113.8
Fine-tuned state of the art (at the time)80.2143.3

More shots help steadily. Across 16 benchmarks, Flamingo set a new few-shot state of the art, and on 6 of them it beat the fine-tuned state of the art using only 32 examples [Certain]. For classification with many classes, it chose the support examples by visual similarity to the query (RICES, retrieval-based in-context example selection) [Certain]. Flamingo was never released; that gap is what IDEFICS filled.

IDEFICS and Idefics2: what matters when building a VLM

IDEFICS: an open Flamingo

Hugging Face first built OBELICS, an open dataset of interleaved web documents: 141 million pages from Common Crawl, 353 million images and 115 billion text tokens [Certain]. On it they trained IDEFICS (9B and 80B), which follows Flamingo’s architecture closely: frozen LLaMA, frozen OpenCLIP encoder, a Perceiver resampler and new cross-attention blocks. In IDEFICS-80B, 14B of the 80B weights are trained: 13.9B in cross-attention, 126M in the resampler [Certain]. It showed that Flamingo’s recipe can be reproduced with open data.

Idefics2: six findings from controlled ablations

For the next model, the authors asked a sharper question: with everything else fixed, which choices actually matter? Each ablation trains for 6,000 steps and reports the average 4-shot score over VQAv2, TextVQA, OK-VQA and COCO [Certain]. The six findings, with their numbers:

  1. The language model matters more than the vision encoder. Replacing LLaMA-1-7B with Mistral-7B gave +5.1 (62.5 → 67.6). Replacing CLIP-ViT-H with SigLIP-SO400M gave +3.3 (57.4 → 60.7). EVA-CLIP-5B, ten times bigger than SigLIP, scored about the same (60.2), which suggests it is under-trained [Certain].
  2. Frozen backbones favor cross-attention; trainable backbones favor the unified sequence. With frozen backbones: cross-attention 66.7, fully autoregressive (image tokens in the sequence) 60.3. With LoRA on the backbones: fully autoregressive 69.5, cross-attention 67.3, even though cross-attention has 1.3B more trainable parameters [Certain].
  3. Unfreezing everything in the fully autoregressive design diverged. LoRA (session 57) both stabilized training and added enough freedom [Certain]. Note that Prismatic VLMs had found that unfreezing the encoder hurts; Idefics2 suggests parameter-efficient training is the difference.
  4. Learned pooling to 64 tokens helps. A Perceiver resampler that pools SigLIP’s 729 tokens to 64 raised the score by 8.5 points (51.8 → 60.3) and cut the tokens 11 times. 128 tokens was not better than 64 (71.2 vs 71.7) [Certain].
  5. Keeping the aspect ratio costs little and saves compute. Passing images at their own size and aspect ratio (378 to 768 pixels) scored 72.1 versus 73.1 for 768-pixel squares, while saving memory and time [Certain] (session 180).
  6. Image splitting trades compute for accuracy. Splitting into 4 crops plus the full image (5 × 64 = 320 tokens) helps most on text reading; it is used only in the instruction stage, and you can turn it on or off at inference [Certain].
Figure 7. Real ablation numbers. Flamingo: overall score (each of 5 benchmarks divided by its state of the art, averaged) for Flamingo-3B, baseline 70.7. Idefics2: average 4-shot score on 4 benchmarks; each group is its own experiment, so compare bars only within a group. Click a bar.
How to read Figure 7

In the Flamingo view, the longest drops are removing M3W (−17.3), training the LM from random weights (−12.9) and a single cross-attention block in the middle (−10.9). Unfreezing the pretrained LM also hurt (−8.0): this is catastrophic forgetting measured directly. In the Idefics2 view, the architecture group is measured against the frozen fully autoregressive model (60.3): LoRA on its backbones adds +9.2, and removing the perceiver pooling costs −8.5. Small differences (around 1 point) after only 6,000 steps should not be over-read.

The Idefics2 recipe

The final Idefics2-8B: SigLIP-SO400M encoder, a modality projection plus a Perceiver resampler to 64 tokens per image (or 320 with splitting), Mistral-7B-v0.1, fully autoregressive, images up to 980×980 at their native aspect ratio. Pretraining has two stages on OBELICS, image–text pairs with synthetic captions (LAION COCO) and, in the second stage, PDF documents for OCR; the whole model is trained, with LoRA on the pretrained backbones, at learning rate \(10^{-4}\) for about 2 epochs: about 1.5 billion images and 225 billion text tokens [Certain]. Synthetic captions beat alt-text in their ablation (52.9 vs 49.8) [Certain]. Instruction tuning uses The Cauldron, 50 vision–language datasets in one question–answer format, plus text-only instruction data, with DoRA and loss only on the answers [Certain].

Results with only 64 tokens per image: MMMU 43.5 (val), MathVista 51.6, TextVQA 70.4, MMBench 76.8; with 320 tokens TextVQA rises to 73.0 [Certain]. For comparison in the same table, LLaVA-NeXT-13B uses 2,880 tokens per image and scores 36.2 on MMMU and 67.1 on TextVQA [Certain]. As a base model with 8 in-context examples, Idefics2 scored 70.3 on VQAv2, against 58.0 for Flamingo-9B and 56.4 for IDEFICS-9B [Certain].

The four designs compared

LLaVA-1.5BLIP-2FlamingoIdefics2
Vision encoderCLIP ViT-L/14, 336 px, frozenEVA-CLIP ViT-g/14 (or CLIP ViT-L), 224 px, frozenNFNet-F6, frozenSigLIP-SO400M, LoRA
Connector2-layer MLP, 21M (7B)Q-Former 188M + linearPerceiver resampler 194MMLP + Perceiver resampler
Language modelVicuna-7B/13B, trained in stage 2OPT or FlanT5, frozenChinchilla 1.4B/7B/70B, frozenMistral-7B, LoRA
How the LLM sees the image576 tokens in the sequence32 tokens in the sequence64 tokens via gated cross-attention64 (or 320) tokens in the sequence
Key data558K captions + 665K instructions129M images, CapFilt captionsM3W, ALIGN, LTIP, VTPOBELICS, LAION COCO, PDFs, The Cauldron
Best at (in its time)chat, cheap to reproducezero-shot VQA, captioningfew-shot, many images, videoefficiency per token, OCR, open data

The connector types are studied in depth in session 178 and the two families in session 179. The broad trend after 2024: the unified sequence with an MLP won for general models, because when the LLM can be trained, it is simpler and as good or better (Idefics2 finding 2; MM1 found the connector type matters little). Compression modules (Q-Former, resampler, pixel shuffle) survive where tokens are expensive: many images, video and phones.

Worked example: the token bill for a 5-image prompt

A prompt with 5 images and 256 text tokens. LLaVA-1.5: \(5 \times 576 + 256 = 3{,}136\) tokens in the sequence, so self-attention computes about \(3{,}136^2 \approx 9.8\) million scores per head per layer at prefill. Idefics2 (64 per image): \(5 \times 64 + 256 = 576\) tokens, about 0.33 million scores, 30 times fewer. Flamingo: 256 tokens in the sequence; each cross-attention block scores \(256 \times 64 = 16{,}384\) pairs, the same for 1 or 32 images thanks to image-causal masking (it still encodes and resamples every image). With 32 images LLaVA-1.5 would need \(32 \times 576 + 256 = 18{,}688\) tokens, more than Vicuna v1.5’s 4,096-token context.

Code: the four key mechanisms in PyTorch

The file below runs on a CPU in about a second and is deterministic. It builds the LLaVA projector and label mask, the three Q-Former masks, the ITC loss with the max over queries, a Flamingo gated cross-attention block with image-causal masking, and a Perceiver resampler.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) LLaVA-1.5: MLP projector + loss only on the answer, formulas (181.2) and (181.3)
d_vis, d_llm, n_vis = 1024, 4096, 576
proj = nn.Sequential(nn.Linear(d_vis, d_llm), nn.GELU(), nn.Linear(d_llm, d_llm))
print("projector params:", sum(p.numel() for p in proj.parameters()))     # 20,979,712
Zv = torch.randn(1, n_vis, d_vis)                  # CLIP layer -2 features, CLS dropped
Hv = proj(Zv)                                      # 576 visual tokens in the LLM's space
prompt_ids, answer_ids = torch.arange(10, 30), torch.arange(30, 42)   # 20 prompt, 12 answer tokens
labels = torch.cat([torch.full((n_vis,), -100),    # image positions: no loss
                    torch.full((20,), -100),       # system + question: no loss
                    answer_ids])                   # answer: loss
print("sequence length:", labels.numel(), "tokens with loss:", (labels != -100).sum().item())

# 2) BLIP-2 Q-Former: one self-attention mask per objective (True = may attend)
def qformer_mask(nq, nt, kind):
    n = nq + nt; m = torch.zeros(n, n, dtype=torch.bool)
    m[:nq, :nq] = True                             # queries always see each other
    if kind == "ITC":   m[nq:, nq:] = True         # text sees text; no crossing
    if kind == "ITM":   m[:, :] = True             # everything sees everything
    if kind == "ITG":                              # text sees all queries + earlier text
        m[nq:, :nq] = True
        m[nq:, nq:] = torch.tril(torch.ones(nt, nt, dtype=torch.bool))
    return m
for k in ["ITC", "ITM", "ITG"]:
    print(k, "allowed pairs:", qformer_mask(4, 5, k).sum().item(), "of", 81)

# 3) ITC similarity: best query wins, formula (181.5)
Z = F.normalize(torch.randn(8, 32, 256), dim=-1)  # 8 images x 32 query outputs
t = F.normalize(torch.randn(8, 256), dim=-1)      # 8 captions ([CLS] embedding)
sim = torch.einsum("iqd,jd->ijq", Z, t).amax(-1) / 0.07    # [8 images, 8 texts]
y = torch.arange(8)
itc = (F.cross_entropy(sim, y) + F.cross_entropy(sim.T, y)) / 2
print("ITC loss on random vectors:", round(itc.item(), 3), " log(8) =", round(torch.log(torch.tensor(8.)).item(), 3))

# 4) Flamingo: gated cross-attention, tanh gate initialised at 0, image-causal mask
class GatedXAttn(nn.Module):
    def __init__(s, d, heads=4):
        super().__init__()
        s.attn = nn.MultiheadAttention(d, heads, batch_first=True)
        s.ffw = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
        s.a_attn = nn.Parameter(torch.zeros(1)); s.a_ffw = nn.Parameter(torch.zeros(1))
    def forward(s, y, x, mask, has_img):           # mask[i, j] = True: text i may NOT see visual j
        y = y + torch.tanh(s.a_attn) * s.attn(y, x, x, attn_mask=mask)[0] * has_img
        return y + torch.tanh(s.a_ffw) * s.ffw(y)
d = 64
img_of_text = torch.tensor([0, 0, 1, 1, 1, 2, 2, 2])   # last image seen before each text token (0 = none)
vis_img = torch.repeat_interleave(torch.tensor([1, 2]), 3)  # 2 images x 3 visual tokens
mask = img_of_text[:, None] != vis_img[None, :]
mask[img_of_text == 0] = False                     # no image yet: any keys (avoid an all-masked row) ...
has_img = (img_of_text > 0).float()[None, :, None]  # ... and the result is set to zero
blk, ytxt, xvis = GatedXAttn(d), torch.randn(1, 8, d), torch.randn(1, 6, d)
out = blk(ytxt, xvis, mask, has_img)
print("at init the frozen LM is unchanged:", torch.allclose(out, ytxt))
with torch.no_grad(): blk.a_attn.fill_(0.5)
print("tanh(0.5) =", round(torch.tanh(torch.tensor(0.5)).item(), 3), " change:", round((blk(ytxt, xvis, mask, has_img) - ytxt).norm().item(), 3) > 0)

# 5) Perceiver resampler (Flamingo, Idefics2): 729 visual tokens -> 64 learned latents
class Resampler(nn.Module):
    def __init__(s, d, n_lat=64, layers=2):
        super().__init__()
        s.lat = nn.Parameter(torch.randn(n_lat, d) * 0.02)
        s.att = nn.ModuleList(nn.MultiheadAttention(d, 4, batch_first=True) for _ in range(layers))
        s.ff = nn.ModuleList(nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) for _ in range(layers))
    def forward(s, xf):
        x = s.lat.expand(xf.size(0), -1, -1)
        for att, ff in zip(s.att, s.ff):
            kv = torch.cat([xf, x], dim=1)         # keys/values: image features AND latents, formula (181.6)
            x = x + att(x, kv, kv)[0]
            x = x + ff(x)
        return x
print("resampler:", tuple(Resampler(d)(torch.randn(2, 729, d)).shape))  # (2, 64, 64)

Output (PyTorch 2.x, CPU): projector params: 20979712, sequence length: 608 tokens with loss: 12, allowed pairs 41, 81 and 51 (the same as Figure 5), ITC loss on random vectors: 2.215 (close to \(\log 8 = 2.079\), as expected for an untrained model), at init the frozen LM is unchanged: True, tanh(0.5) = 0.462, and resampler: (2, 64, 64).

Where this is used

ApplicationWhich ideaWhy
General visual chat assistants (open models)LLaVA recipe: encoder + MLP + LLM, two stages, instruction data with format promptsSimple, cheap to reproduce, strong when the LLM is trained (session 186)
Image captioning and alt-text at scaleBLIP-2 with a frozen LLMSmall trainable module, good zero-shot captions; the same captioners clean web data (session 187)
Video question answering and video captioningFlamingo-style 1 FPS frames + resampler to a fixed token count; Q-Former in video models such as Video-LLaMA [Likely]A fixed number of tokens per frame or clip keeps long videos inside the budget
Key-moment detection and video retrievalBLIP-2 / Q-Former ITC embeddings per frame, then rerank with a generative VLM [Likely]The contrastive head gives cheap per-frame scores; generation is used only on the top candidates
Few-shot labeling of new classes (defects, sports actions)Flamingo / IDEFICS in-context prompts, examples chosen by RICESNo fine-tuning: change the examples in the prompt to change the task
Multi-image and Video-RAG answersPooled tokens (64 per image, Idefics2) or cross-attentionRetrieved frames and documents fit in the context without thousands of tokens each
Document and chart readingIdefics2 image splitting, native aspect ratio, PDF pretrainingText in images needs resolution; splitting buys it only when needed

Common mistakes

  • Computing loss on the image tokens or the prompt. Labels at image and instruction positions must be −100. Otherwise the model wastes capacity predicting system prompts and the training signal for the answer is diluted.
  • Forgetting the response-format prompt when mixing short-answer VQA data with chat data. The model learns to answer in one word everywhere. LLaVA-1.5 fixed this with prompts such as “Answer the question using a single word or phrase.”
  • Initializing new gates or projections so that they disturb a frozen LLM at step 0. Flamingo starts gates at 0; PaliGemma zero-initializes its projection [Likely]. Random large outputs into a frozen LM give a high starting loss and large, harmful gradients.
  • Skipping BLIP-2’s stage 1 to save time. The paper shows lower zero-shot VQA and forgetting with OPT. A frozen LLM needs well-formed visual prompts.
  • Using the wrong mask for the wrong objective in a shared Q-Former. A bidirectional mask in ITC leaks the text into the image embedding and makes the contrastive task trivial.
  • Letting every text token attend to all previous images in a cross-attention model. Cost grows with the number of images, and a test-time prompt with more images than in training goes out of distribution.
  • Reading ablation tables across groups. In Idefics2, each table is a separate run; 72.1 in one group and 60.3 in another are not comparable. Compare only within a group, and do not over-read 1-point gaps after 6,000 steps.
  • Counting “trainable parameters” as cost. BLIP-2 trains 108M weights but the forward pass still runs the full 12.1B model. Training compute depends on total parameters and tokens, not only trainable ones.

Interview questions

Describe the LLaVA architecture and its two training stages.basic
A frozen CLIP ViT-L/14 encodes the image into patch features; a projection (linear in LLaVA, a 2-layer MLP with GELU in LLaVA-1.5) maps each patch feature to the LLM embedding size; these visual tokens are placed in the sequence with the text and Vicuna generates the answer. Stage 1 trains only the projection on 595K (LLaVA) or 558K (LLaVA-1.5) image–caption pairs, aligning visual features with word embeddings. Stage 2 trains projection and LLM on instruction data (158K, then 665K); the encoder stays frozen. Loss is next-token prediction on answer tokens only.
How did LLaVA create instruction data without a multimodal model?core
It gave a text-only GPT-4 a symbolic view of each COCO image: the human captions and the object boxes with coordinates. With a few hand-written examples in the prompt, GPT-4 wrote three types of samples: conversations (58K), detailed descriptions (23K) and complex reasoning (77K), 158K in total. The risk is that answers include details that are plausible from the text but not in the image, which can teach hallucination.
What is the Q-Former and why does BLIP-2 need it?core
A BERT-base-initialized transformer with 32 learned query vectors (768-d) that cross-attend to frozen image features every other block; 188M parameters. It outputs 32 vectors, a bottleneck about 11 times smaller than ViT-L features. Because both the image encoder and the LLM are frozen, something trainable must convert image features into a form the LLM understands; the Q-Former learns to extract text-relevant visual information, first with ITC/ITM/ITG losses, then as a soft prompt for the frozen LLM through one linear layer.
Explain the three Q-Former losses and their attention masks.advanced
ITC (contrastive): unimodal mask, queries and text cannot see each other; similarity is the max over the 32 query outputs of the dot product with the text [CLS]; in-batch negatives. ITM (matching): bidirectional mask, everything sees everything; a 2-class linear head on each query output, logits averaged; hard negatives. ITG (generation): multimodal causal mask, text sees all queries and earlier text, queries do not see text; the caption must be generated through the queries, which forces them to capture what the text needs. One network, three masks, summed losses.
Why are Flamingo’s gates initialized at zero?core
Each new GATED XATTN-DENSE block adds \(\tanh(\alpha) \cdot \text{(cross-attention)}\) and \(\tanh(\alpha) \cdot \text{(FFW)}\) to the residual stream. With \(\alpha = 0\) the gate is \(\tanh(0) = 0\), so the new blocks are identities and the model at step 0 is exactly the frozen pretrained LM: no damage to its language ability and a stable start. Training opens the gates where vision helps. Removing the gating cost about 4 points (70.7 → 66.5) in the ablation.
What is image-causal masking in Flamingo and what does it buy?advanced
In cross-attention, each text token may attend only to the visual tokens of the last image before it (none if no image yet). Earlier images still influence it through the LM’s self-attention over earlier text. Benefits: constant cross-attention cost per token (64 visual tokens) regardless of the number of images, a natural way to bind each caption to its image in few-shot prompts, and generalization from 5 images in training to 32 shots at test time.
BLIP-2 beat Flamingo-80B on zero-shot VQAv2 with 54x fewer trainable parameters. Does that mean BLIP-2 is 54x cheaper?advanced
No. 188M vs 10.2B is the trainable count. Forward compute depends on the total model (BLIP-2 with FlanT5-XXL is 12.1B, Flamingo 80B), and training compute also depends on tokens and steps. Frozen weights still cost forward passes and backward passes for activations up to the first trainable layer. The comparison also differs in data, encoder and LLM. BLIP-2 is clearly cheaper, but not by the ratio of trainable weights. And Flamingo still won on OK-VQA (50.6 vs 45.9), a knowledge task, thanks to its 70B LM.
Cross-attention or unified sequence: which is better?core
It depends on whether you train the LLM. Idefics2: with frozen backbones, cross-attention 66.7 vs fully autoregressive 60.3; with LoRA on the backbones, fully autoregressive 69.5 vs cross-attention 67.3, despite cross-attention having 1.3B more trainable parameters. Cross-attention protects a frozen LLM and keeps image tokens out of the context (good for many images); the unified sequence is simpler and better when the LLM can adapt. Most 2024+ open models use the unified sequence.
What did Idefics2 find matters most when building a VLM?core
(1) The LLM backbone matters more than the vision backbone at fixed size (+5.1 from Mistral-7B vs +3.3 from SigLIP). (2) The architecture choice interacts with freezing. (3) Full unfreezing diverged; LoRA stabilizes. (4) Pooling to 64 tokens with a perceiver improved the score by 8.5 and cut tokens 11x; more than 64 did not help. (5) Native aspect ratio costs little and saves compute. (6) Image splitting trades compute for accuracy, mostly on OCR.
Why does few-shot in-context learning work in Flamingo but not in LLaVA-1.5?advanced
Flamingo was pretrained on 43M interleaved web pages (M3W), so it saw sequences with several images, each followed by related text, and learned to use earlier image–text pairs as context. Removing M3W dropped its ablation score from 70.7 to 53.4. LLaVA-1.5 was trained on single-image samples, and with 576 tokens per image a 32-shot prompt would not fit in a 4K context. Interleaved pretraining data and compact image tokens are both needed.
You must build a VLM that runs on many video frames per question. Which ideas from these four papers would you reuse?advanced
A strong pretrained LLM first (Idefics2 finding 1). A token compressor per frame: a perceiver resampler or Q-Former to 32–64 tokens, since raw 576 tokens per frame explodes the context. Sample frames at a low rate with time embeddings (Flamingo uses 1 FPS and learned temporal embeddings). Train on interleaved and video–text data so the model handles many visual inputs. Zero-initialized gates or projections if you add modules to a pretrained LLM. Train the backbones with LoRA if you use the unified sequence. Evaluate with a fixed token budget.

Exercises

  1. LLaVA (original) uses CLIP ViT-L/14 at 224 pixels and a linear projection to Vicuna-13B (\(d = 5120\)). How many visual tokens per image, and how many weights in the projection (with bias)?
  2. For the Q-Former masks with 32 queries and 20 text tokens, how many query–text and text–text pairs are allowed in ITC, ITM and ITG? (Count all \(52^2\) cells.)
  3. Flamingo-80B adds a cross-attention block before layer 0 and then before every 7th layer of an 80-layer LM. How many blocks? What is \(\tanh(\alpha)\) for \(\alpha = 1\)?
  4. In Figure 6’s prompt with image-causal masking, how many cross-attention scores (per head) does the last token compute if each image has 64 tokens and there are 32 shots? And with “all previous images”?
  5. Idefics2 with image splitting uses 320 tokens per image. How many image tokens for a 3-image prompt, and how does that compare with LLaVA-1.5?
  6. BLIP-2 ViT-g FlanT5-XXL trains 108M of 12.1B weights; Flamingo-80B trains about 10.2B of 80B. Compute both fractions.
Answerscheck
1. \((224/14)^2 = 256\) tokens; \(1024 \times 5120 + 5120 = 5{,}248{,}000\) weights. 2. Total cells \(52^2 = 2{,}704\). Query–query is always \(32^2 = 1{,}024\). ITC: plus text–text \(20^2 = 400\), total 1,424. ITM: all 2,704. ITG: plus text→query \(20 \times 32 = 640\) and the text lower triangle \(20 \cdot 21/2 = 210\), total 1,874. 3. Layers 0, 7, …, 77: 12 blocks; \(\tanh(1) = 0.762\). 4. 64 scores (only the query image); with all previous images, \(33 \times 64 = 2{,}112\) (32 support images plus the query image). 5. \(3 \times 320 = 960\) tokens vs \(3 \times 576 = 1{,}728\) for LLaVA-1.5; without splitting Idefics2 needs only 192. 6. \(108/12{,}100 = 0.89\%\); \(10.2/80 = 12.7\%\).

Key takeaways

  • LLaVA: CLIP + projection (181.1, 181.2) + LLM, trained in two stages with loss only on answers (181.3). The real innovation was the data: 158K GPT-4-written instruction samples. LLaVA-1.5 showed that an MLP, 336 px and well-formatted academic data are enough to lead benchmarks in one day on 8 A100s.
  • BLIP-2: freeze encoder and LLM; train a 188M Q-Former whose 32 queries cross-attend to image features (181.4). Stage 1 uses ITC (with the max over queries, 181.5), ITM and ITG with three masks; stage 2 feeds 32 soft-prompt vectors to the frozen LLM. Stage 1 prevents forgetting.
  • Flamingo: a Perceiver resampler to 64 tokens (181.6), gated cross-attention starting at zero (181.7), image-causal masking (181.8) and a weighted multi-dataset loss (181.9) over interleaved web data give real few-shot learning.
  • Idefics2: the LLM matters most; frozen backbones favor cross-attention, LoRA-trained backbones favor the unified sequence; 64 pooled tokens beat 729; native aspect ratio and image splitting give control over cost.
  • Compare designs by what is trained, how many visual tokens enter the context, and what data teaches the connection.

Key formulas

\[ H_v = W_2\,\mathrm{GELU}(W_1 Z_v), \quad s(I, T) = \max_q z_q^\top t, \quad y \leftarrow y + \tanh(\alpha)\,\mathrm{Attn}(y, x), \quad p(y \mid x) = \prod_\ell p(y_\ell \mid y_{\lt \ell}, x_{\le \ell}) \]

References for this session

  1. Liu et al. (2023). Visual Instruction Tuning
  2. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  3. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  4. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
  5. Laurençon et al. (2024). What matters when building vision-language models?
  6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
  7. Liu et al. (2023). Visual Instruction Tuning
  8. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  9. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  10. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
  11. Laurençon et al. (2024). What matters when building vision-language models?
  12. Laurençon et al. (2023). OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
  13. Li et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  14. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  15. Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
  16. Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
  17. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  18. Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation
  19. Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  20. McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  21. Dai et al. (2023). InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  22. Fang et al. (2022). EVA: Exploring the Limits of Masked Visual Representation Learning at Scale