LLM & VLM courseBlog
173Track 7: Vision and VLMs · session 11 of 38

ALBEF and BLIP: contrastive, matching and captioning losses together

CLIP aligns images and text with one dot product, but it cannot check fine details and it cannot write. ALBEF and BLIP keep the contrastive loss, add a fusion encoder trained to say “match or not” on hard negatives, and (in BLIP) a decoder trained to write captions. BLIP then uses its own captioner and matcher to clean its noisy web training data. This session builds all three losses by hand, shows how they share one network, and ends with CapFilt.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 3 (cross-entropy and KL), 166 (CLIP) and 172 (two-stream and single-stream VLMs)

What you will learn

  • Why “align before fuse”: a contrastive loss on the unimodal encoders makes the later fusion encoder easier to train and gives a fast retrieval stage
  • The image–text contrastive loss (ITC) with a momentum encoder, a 65,536-entry queue and soft targets from momentum distillation, with a worked example
  • Image–text matching (ITM): a binary classifier on the fused [CLS] token, trained on hard negatives sampled from the contrastive similarities
  • Masked language modeling in ALBEF and the captioning loss (LM) in BLIP, and how BLIP’s encoder and decoder share every layer except self-attention
  • Retrieval in two stages: rank everything with ITC, rerank the top k with ITM, and why that is 30 times cheaper on COCO
  • CapFilt: generate synthetic captions with nucleus sampling, filter web and synthetic captions with ITM, and train a new model on the result
  • The real numbers: model sizes, recipes, ablations and results from the ALBEF and BLIP papers

Why this matters

In session 166 we saw a dual encoder: an image tower and a text tower that meet only in one dot product. It is fast and it is great for search, but it has two holes. It is weak at fine details (which object is red, which one is on the left), because one pooled vector per side must hold everything. And it cannot write: it can pick a caption, not produce one. In session 172 we saw the other family: fusion models such as UNITER that read image and text together and reason well, but are slow, need an object detector, and must run once for every (image, text) pair.

ALBEF (Li et al., 2021, “Align before Fuse”) and BLIP (Li et al., 2022, “Bootstrapping Language-Image Pre-training”), both from Salesforce Research, join the two families into one network trained with three losses at once:

  • Image–text contrastive loss (ITC): the CLIP loss on the two unimodal encoders. It aligns the two spaces and gives a fast retrieval stage.
  • Image–text matching loss (ITM): a fusion encoder reads the image and the text together and answers “matched or not?”. It is trained on hard negatives, captions that are almost right, so it learns fine details.
  • A language loss: masked language modeling (MLM) in ALBEF, and a captioning loss (LM, next-token prediction) in BLIP, so the model can write.

BLIP adds one more idea that matters a lot in practice: CapFilt. Web captions are noisy. BLIP uses its own captioner to write new captions for web images and its own matcher to throw away captions that do not fit, then trains a new model on the cleaner data. This “use a model to clean the data for the next model” pattern is now everywhere: synthetic re-captioning is a standard step when building VLM training sets (session 187). And BLIP-2 reuses exactly these three losses to train its Q-Former, the connector between a frozen image encoder and a frozen LLM (session 178, session 181).

Align before fuse: one network, three paths

Older fusion models (LXMERT, UNITER, OSCAR) fed region features from a pretrained object detector into a joint transformer. The ALBEF paper names three problems with that design [Certain]: the region features and the word embeddings live in unrelated spaces, so the fusion encoder must learn the alignment from scratch; the detector needs box annotations to train and high-resolution input (for example 600 × 1000) at test time; and the web captions are noisy, which MLM can overfit to.

ALBEF’s answer is in its name: first align the two unimodal representations with a contrastive loss, then fuse them. The model has three parts [Certain]:

  • An image encoder: a 12-layer ViT-B/16 initialized from DeiT weights trained on ImageNet-1k (85.8 million parameters). It returns the patch embeddings and a [CLS] embedding \(\mathbf{v}_{\text{cls}}\). No object detector.
  • A text encoder: the first 6 layers of BERT-base, bidirectional self-attention. It returns token embeddings and \(\mathbf{w}_{\text{cls}}\).
  • A multimodal (fusion) encoder: the last 6 layers of BERT-base, with a cross-attention layer added in each block. The text tokens are the queries; the image patch embeddings are the keys and values (session 20).

The paper gives 123.7 million parameters for the BERT-base text side [Certain]. That matches BERT-base (about 109.5M) plus six cross-attention layers of about 2.36M each, so the count includes the cross-attention [Likely]. ALBEF is about 209.5 million parameters in total. BLIP keeps the ViT image encoder (ViT-B/16 or ViT-L/16) but rearranges the text side into a multimodal mixture of encoder–decoder (MED), one text transformer that can run in three modes [Certain]:

  1. Unimodal text encoder: BERT with a [CLS] token at the start; no image. Used by ITC.
  2. Image-grounded text encoder: the same blocks plus a cross-attention layer between self-attention and the feed-forward network (FFN). A task token [Encode] replaces [CLS]; its output is the joint representation. Used by ITM.
  3. Image-grounded text decoder: the bidirectional self-attention is replaced by causal self-attention (each token sees only earlier tokens, session 18). A [Decode] token starts the sequence and an end-of-sequence token ends it. Used by LM.

The encoder and the decoder share all parameters except the self-attention layers [Certain]. The reason the authors give: the embedding, cross-attention and FFN layers do similar work in both modes, while the self-attention is where encoding (look at everything) and decoding (look only back) really differ. One image goes through the heavy ViT once per training step; the text transformer runs three times, once per mode [Certain]. Figure 1 shows which parts each loss uses.

image sidetext sideactive in this lossnot used by this loss
Figure 1. One network, four ways to use it. The left part shows the modules; the boxes of the chosen loss are highlighted. The right part shows the self-attention mask of the text tokens in that mode (row = the token that looks, column = the token it may see). Same color in a block means shared weights in BLIP.
How to read Figure 1

Click ITC: only the two unimodal encoders run, the text mask is full (bidirectional), and the output is two 256-d vectors compared by a dot product. Click ITM: the text goes through the image-grounded encoder, whose cross-attention reads the image patches; the mask is still full; the output is one [Encode] vector and a 2-way head. Click LM: the same cross-attention and FFN weights are used, but the self-attention is the separate causal one, so the mask is lower-triangular. Click MLM to see ALBEF’s version: 15% of the words are hidden and the fusion encoder must recover them from the image and the rest of the text.

Loss 1: image–text contrastive (ITC) with a momentum queue

ITC is the CLIP loss of session 166, formula (166.5), applied to the unimodal encoders. Two linear maps \(g_v\) and \(g_w\) project the [CLS] embeddings to 256 dimensions and L2-normalize them [Certain]. One difference from CLIP is important. CLIP used a batch of 32,768 pairs; ALBEF trains with a batch of 512 on 8 GPUs [Certain]. A batch of 512 gives only 511 negatives per positive. To get many more negatives cheaply, ALBEF borrows two ideas from MoCo (He et al., 2019):

  • A momentum encoder: a second copy of the encoders whose weights are a slow moving average of the trained weights. It gets no gradient.
  • Two queues that store the last \(M = 65{,}536\) image and text features produced by the momentum encoders [Certain]. Every step, the new batch is pushed in and the oldest batch is dropped.

The image of a pair is compared against all \(M\) text features in the queue (which include the momentum features of the current batch), and the other way around:

\[ p^{\text{i2t}}_m(I) = \frac{\exp\!\big(s(I, T_m)/\tau\big)}{\sum_{m'=1}^{M} \exp\!\big(s(I, T_{m'})/\tau\big)}, \qquad s(I, T_m) = g_v(\mathbf{v}_{\text{cls}})^\top\, g'_w(\mathbf{w}'_{\text{cls}, m}) \tag{173.1} \]
\[ \mathcal{L}_{\text{itc}} = \tfrac{1}{2}\, \E_{(I,T)}\Big[ H\big(\mathbf{y}^{\text{i2t}}, \mathbf{p}^{\text{i2t}}(I)\big) + H\big(\mathbf{y}^{\text{t2i}}, \mathbf{p}^{\text{t2i}}(T)\big) \Big] \tag{173.2} \]
Read the symbols
\(g_v, g_w\)
linear projections of the image and text [CLS] embeddings to unit vectors in \(\R^{256}\)
\(g'_w(\mathbf{w}'_{\text{cls}, m})\)
the text feature of entry \(m\) of the queue, made by the momentum text encoder (the prime marks “momentum”)
\(M\)
the number of candidates: the queue length, 65,536 in ALBEF
\(\tau\)
a learned temperature, initialized at 0.07; the BLIP code clamps it to \([0.001, 0.5]\) [Certain]
\(\mathbf{y}^{\text{i2t}}\)
the target distribution: 1 on the true text, 0 elsewhere (a one-hot vector)
\(H(\mathbf{y}, \mathbf{p})\)
cross-entropy \(-\sum_m y_m \log p_m\); \(\mathbf{p}^{\text{t2i}}\) is the same as (173.1) with the roles of image and text swapped

The momentum weights \(\theta'\) follow the trained weights \(\theta\) slowly:

\[ \theta' \leftarrow \mu\, \theta' + (1 - \mu)\, \theta, \qquad \mu = 0.995 \tag{173.3} \]

Why not just store old features from the trained encoder? Because the trained encoder changes every step, and features computed 100 steps ago would come from a different network, so the comparison would be inconsistent. A slowly moving encoder keeps the queue consistent: with \(\mu = 0.995\), the momentum weights are an average over roughly the last \(1/(1-\mu) = 200\) steps. The queue is cheap: \(65{,}536 \times 256\) float32 numbers is 67.1 MB per queue, 134 MB for both. The ceiling on what the contrastive loss can measure, \(\ln M\) from (166.8), rises from \(\ln 512 = 6.24\) to \(\ln 65{,}536 = 11.09\) nats.

Momentum distillation: soft targets for noisy pairs

A one-hot target says: “only your own caption is right; every other caption is equally wrong”. With web data that is often false. Another caption in the queue may describe the image just as well (two photos of dogs on grass), and the image’s own caption may mention things not in the picture. The one-hot loss pushes such a false negative away hard. ALBEF’s fix is momentum distillation (MoD): the momentum model, a running average of the model itself, acts as a teacher (session 59). Its similarities \(s'\) (both sides from momentum encoders) give soft targets \(\mathbf{q}\), computed like (173.1) with \(s'\) in place of \(s\):

\[ \mathcal{L}^{\text{mod}}_{\text{itc}} = (1-\alpha)\, \mathcal{L}_{\text{itc}} + \frac{\alpha}{2}\, \E_{(I,T)}\Big[ \KL\big(\mathbf{q}^{\text{i2t}}(I) \,\Vert\, \mathbf{p}^{\text{i2t}}(I)\big) + \KL\big(\mathbf{q}^{\text{t2i}}(T) \,\Vert\, \mathbf{p}^{\text{t2i}}(T)\big) \Big] \tag{173.4} \]
Read the symbols
\(\mathbf{q}^{\text{i2t}}(I)\)
the momentum model’s softmax over the \(M\) texts: a “second opinion” on which texts fit the image
\(\alpha\)
the weight of the soft targets: 0.4 in ALBEF and BLIP, ramped up from 0 during the first epoch in ALBEF [Certain]; the paper found 0.3, 0.4 and 0.5 similar [Certain]
\(\KL(\mathbf{q} \Vert \mathbf{p})\)
the KL divergence from session 3; it is zero when the model agrees with the teacher

A useful identity: since \(\KL(\mathbf{q} \Vert \mathbf{p}) = H(\mathbf{q}, \mathbf{p}) - H(\mathbf{q})\) and \(H(\mathbf{q})\) does not depend on the model, (173.4) has the same gradient as one cross-entropy against a mixed target \(\mathbf{t} = (1-\alpha)\,\mathbf{y} + \alpha\, \mathbf{q}\). That is exactly how the BLIP code implements it [Certain]. The gradient on each logit is then simply \(p_m - t_m\) (divided by \(\tau\) for the gradient on the similarity).

Worked example: a false negative with and without soft targets

Image: a dog on grass. Four candidate texts: T1 its own caption, T2 “a brown dog running on a lawn” (from another image, but it fits), T3 “a red car”, T4 “a bowl of soup”. Online similarities \(s = (0.42, 0.38, 0.05, 0.02)\), momentum similarities \(s' = (0.40, 0.39, 0.06, 0.03)\), \(\tau = 0.07\).

Logits \(s/\tau = (6.00, 5.43, 0.71, 0.29)\). Model softmax \(\mathbf{p} = (0.636, 0.359, 0.003, 0.002)\). Teacher \(\mathbf{q} = (0.532, 0.461, 0.004, 0.003)\): it also thinks T2 fits. Mixed target with \(\alpha = 0.4\): \(\mathbf{t} = 0.6\,(1,0,0,0) + 0.4\,\mathbf{q} = (0.813, 0.184, 0.002, 0.001)\).

One-hot: loss \(-\ln 0.636 = 0.453\); gradient on the logits \(\mathbf{p} - \mathbf{y} = (-0.364, +0.359, +0.003, +0.002)\). T2 is pushed away almost as strongly as T1 is pulled in. Soft: gradient \(\mathbf{p} - \mathbf{t} = (-0.177, +0.175, +0.002, +0.001)\). The push on the false negative is halved (0.359 to 0.175). The value of (173.4) is \(0.6 \times 0.453 + 0.4 \times 0.022 = 0.281\), where 0.022 is \(\KL(\mathbf{q} \Vert \mathbf{p})\).

model pteacher qmixed target tgradient p − t
Figure 2. Momentum distillation on the worked example: one image row of the ITC loss with four texts, of which T2 is a false negative. Left: the model softmax \(\mathbf{p}\), the teacher softmax \(\mathbf{q}\) and the mixed target \(\mathbf{t} = (1-\alpha)\mathbf{y} + \alpha \mathbf{q}\). Right: the gradient \(p_m - t_m\) on each logit (negative = pulled closer, positive = pushed away).
How to read Figure 2

At \(\alpha = 0.4\) and \(\tau = 0.07\) the readout shows the numbers of the worked example: the push on T2 is \(+0.175\) instead of \(+0.359\). Slide \(\alpha\) to 0: you get the plain one-hot loss and the full push on T2. Slide \(\alpha\) to 1: the target is the teacher alone, the gradient on T2 turns negative (\(-0.102\)): T2 is pulled closer, because the teacher gives it more probability than the model does. Raise \(\tau\) to 0.5: every softmax becomes flat, the easy negatives T3 and T4 get real probability, and the loss stops caring about the difference between T1 and T2.

Loss 2: image–text matching (ITM) with hard negatives

ITC compares two separate vectors. It cannot check “the red cube is left of the blue ball” versus “the blue cube is left of the red ball”: with mostly the same words and the same objects, the two pooled vectors are very close. ITM looks at the pair together. The fusion encoder (BLIP: the image-grounded text encoder) reads the text, cross-attends to every image patch, and a linear head on its first output token ([CLS] in ALBEF, [Encode] in BLIP) predicts two classes, matched or not [Certain]:

\[ \mathcal{L}_{\text{itm}} = \E_{(I,T)}\, H\big(\mathbf{y}^{\text{itm}}, \mathbf{p}^{\text{itm}}(I, T)\big), \qquad \mathbf{p}^{\text{itm}}(I, T) = \softmax\!\big(W_{\text{itm}}\, \mathbf{h}_{\text{[CLS]}}(I, T)\big) \tag{173.5} \]
Read the symbols
\(\mathbf{h}_{\text{[CLS]}}(I, T)\)
the fused output vector of the first text token, after self-attention over the text and cross-attention to the image in every layer
\(W_{\text{itm}}\)
a linear layer to 2 logits (the “ITM head”)
\(\mathbf{y}^{\text{itm}}\)
one-hot label: matched or not matched

The key question is which negatives. A random caption from the batch (“a bowl of soup” for a dog photo) is too easy: the head learns to say “no” after one glance and stops learning. ALBEF samples negatives that are hard (similar meaning, wrong detail), using the ITC similarities it has already computed, so the mining costs nothing extra [Certain]. For each image \(I_i\) of the batch, one negative text is drawn with probability

\[ P(j \mid i) = \frac{\exp\!\big(s(I_i, T_j)/\tau\big)}{\sum_{j' \ne i} \exp\!\big(s(I_i, T_{j'})/\tau\big)}, \qquad j \ne i \tag{173.6} \]

and likewise one negative image for each text. A batch of \(B\) pairs therefore gives \(B\) positive pairs and \(2B\) negative pairs for ITM; the label vector in the BLIP code is \(B\) ones followed by \(2B\) zeros [Certain]. The code also adds \(10^{-4}\) to every weight before zeroing the diagonal, so every negative keeps a small chance [Certain]. Note that the mining uses only the current batch, not the queue: the queue holds features, not images, and ITM needs the actual image and text.

Why sample instead of always taking the most similar caption? The most similar caption is sometimes a true match (two captions of near-identical photos). Always taking it would teach the model that correct captions are wrong. Sampling in proportion to similarity favours hard negatives but spreads the risk.

true pairselected image rowprobability of being drawncount in 1,000 draws
Figure 3. Hard negative mining (173.6) on a batch of six pairs with fixed cosine similarities. Click a row of the matrix to pick the image. The bars on the right give the probability that each other caption becomes this image’s ITM negative; “Draw” samples 1,000 negatives with a seeded random generator.
How to read Figure 3

Row 1 (dog on grass) at \(\tau = 0.07\): the puppy-on-a-beach caption is drawn with probability 0.945, so almost every ITM negative of this image is the hard one; the mean cosine of the drawn negative is 0.364, against 0.136 for uniform sampling. Press Draw: about 944 of 1,000 draws are the puppy. Raise \(\tau\) to 1: sampling becomes nearly uniform (0.253 for the puppy) and the negatives become easy. Pick row 5 (red car): its hard negative is the city bus (0.916).

The ALBEF ablation shows each piece helps [Certain, their Table 1, 4M images]. Starting from MLM + ITM, adding ITC raised the Flickr30K text-retrieval average recall from 93.96 to 96.55 and VQA test-dev from 71.40 to 73.29. Hard negatives for ITM raised them further to 97.01 and 73.81. Momentum distillation on ITC and MLM, and then on the downstream tasks too, gave 97.83 and 74.54 for the full model. Fine-tuning without hard negatives lowered Flickr30K text-retrieval average recall from 98.57 to 98.22 (their Table 6).

Loss 3: masked language modeling (ALBEF) and captioning (BLIP)

ALBEF uses BERT’s masked language modeling, but conditioned on the image: 15% of the text tokens are selected (80% of those become [MASK], 10% a random token, 10% unchanged), and the fusion encoder must predict them from the image and the visible words [Certain] (session 31):

\[ \mathcal{L}_{\text{mlm}} = \E_{(I, \hat{T})}\, H\big(\mathbf{y}^{\text{msk}}, \mathbf{p}^{\text{msk}}(I, \hat{T})\big), \qquad \mathcal{L}_{\text{ALBEF}} = \mathcal{L}_{\text{itc}} + \mathcal{L}_{\text{mlm}} + \mathcal{L}_{\text{itm}} \tag{173.7} \]
Read the symbols
\(\hat{T}\)
the text with some tokens masked
\(\mathbf{p}^{\text{msk}}\)
the predicted distribution over the vocabulary at a masked position
\(\mathcal{L}_{\text{ALBEF}}\)
the full pretraining loss, an unweighted sum; with MoD, ITC and MLM each get the soft-target form (173.4)

MLM teaches understanding, but a model trained with it still cannot write a caption from nothing. BLIP replaces MLM by a language-modeling loss on the image-grounded decoder: predict each caption token from the image and the earlier tokens, exactly the next-token loss of session 6, with label smoothing 0.1 [Certain]:

\[ \mathcal{L}_{\text{lm}} = -\frac{1}{L} \sum_{t=1}^{L} \sum_{w \in \mathcal{V}} \tilde{y}_{t,w}\, \log p_\theta\big(w \mid I, T_{\lt t}\big), \qquad \tilde{y}_{t,w} = (1-\varepsilon)\, [w = T_t] + \frac{\varepsilon}{|\mathcal{V}|} \tag{173.8} \]
\[ \mathcal{L}_{\text{BLIP}} = \mathcal{L}_{\text{itc}} + \mathcal{L}_{\text{itm}} + \mathcal{L}_{\text{lm}} \tag{173.9} \]
Read the symbols
\(T_{\lt t}\)
the caption tokens before position \(t\), starting with [Decode]; the causal mask enforces this
\(L\)
the number of caption tokens (padding is ignored)
\(\varepsilon\)
label smoothing, 0.1: the target keeps 0.9 on the true token and spreads 0.1 evenly over the vocabulary
\(|\mathcal{V}|\)
vocabulary size: BERT’s 30,522 WordPiece tokens plus BLIP’s added special tokens (30,524 in the Hugging Face config [Certain])

Label smoothing puts a floor under the loss: even a perfect model cannot go below the entropy of the smoothed target, which is 1.358 nats for \(\varepsilon = 0.1\) and \(|\mathcal{V}| = 30{,}524\). So a BLIP captioning loss of 1.5 is not “bad”; compare it with this floor, not with zero.

Does sharing weights between encoder and decoder hurt? BLIP tested it [Certain, their Table 3, 14M images, web captions]:

Encoder and decoder shareParametersCOCO TR@1 (fine-tuned)COCO caption CIDErNoCaps CIDEr (zero-shot)
All layers224M77.3125.9100.9
All except cross-attention252M77.5126.1101.2
All except self-attention (BLIP)252M78.4127.8102.2
Nothing361M78.3127.4101.8

Sharing everything except self-attention is as good as (here slightly better than) separate networks, with 109M fewer parameters, 70% of the size. Sharing the self-attention too hurts: one attention cannot be good at both looking everywhere and looking only back. TR@1 is “text retrieval recall at 1”: the fraction of images whose top-ranked caption is correct. CIDEr is a captioning metric based on n-gram agreement with human captions (session 149).

Algorithm 1: one BLIP pretraining step (ALBEF differs in step 9)
  1. input: a batch of \(B\) pairs \((I_i, T_i)\); model \(\theta\); momentum model \(\theta'\); queues \(Q_I, Q_T\) of length \(M\)
  2. \(V \leftarrow \text{ViT}_\theta(I)\) // patches + [CLS], one pass of the heavy encoder
  3. \(\mathbf{u}_i, \mathbf{v}_i \leftarrow\) normalized 256-d projections of the image and text [CLS] // text mode 1: unimodal
  4. \(\theta' \leftarrow \mu\theta' + (1-\mu)\theta\); compute \(\mathbf{u}'_i, \mathbf{v}'_i\) with \(\theta'\) // (173.3), no gradient
  5. \(\mathcal{L}_{\text{itc}} \leftarrow\) (173.2) against \([\mathbf{v}'; Q_T]\) and \([\mathbf{u}'; Q_I]\), targets \((1-\alpha)\mathbf{y} + \alpha\mathbf{q}\) // (173.4)
  6. push \(\mathbf{u}', \mathbf{v}'\) into the queues, drop the oldest \(B\)
  7. for each \(i\): draw \(j \sim P(\cdot \mid i)\) by (173.6) for a negative text, and a negative image the same way
  8. \(\mathcal{L}_{\text{itm}} \leftarrow\) cross-entropy of the ITM head on \(B\) positive and \(2B\) negative pairs // text mode 2: [Encode] + cross-attention
  9. \(\mathcal{L}_{\text{lm}} \leftarrow\) next-token loss (173.8) with causal self-attention // text mode 3; ALBEF: MLM on the fusion encoder instead
  10. \(\theta \leftarrow\) AdamW step on \(\mathcal{L}_{\text{itc}} + \mathcal{L}_{\text{itm}} + \mathcal{L}_{\text{lm}}\) // (173.9)

Using the three heads: retrieval, VQA, captioning

Each loss leaves behind a useful tool. Captioning uses the decoder directly; BLIP fine-tunes on COCO, generates with beam search (beam 3, at most 20 tokens) and starts each caption with the prompt “a picture of” [Certain]. VQA is treated as answer generation: the image-grounded encoder reads the image and the question, and a decoder writes the answer (at test time it ranks a fixed list of candidate answers) [Certain] (session 191). Retrieval uses ITC and ITM together, in two stages [Certain]:

\[ \mathcal{C}_k(T) = \operatorname{top-}k_{\,I}\; s_{\text{itc}}(I, T), \qquad \hat{I}(T) = \argmax_{I \in \mathcal{C}_k(T)}\; p^{\text{itm}}_{\text{match}}(I, T) \tag{173.10} \]
Read the symbols
\(s_{\text{itc}}\)
the cheap dot product of the unimodal embeddings; image embeddings can be computed once and indexed
\(\mathcal{C}_k(T)\)
the \(k\) images with the highest ITC score for query \(T\)
\(p^{\text{itm}}_{\text{match}}\)
the ITM probability of “matched”: one fusion pass per candidate pair

BLIP uses \(k = 256\) on COCO and \(k = 128\) on Flickr30K [Certain]. ALBEF found the result almost insensitive to \(k\): Flickr30K text-retrieval average recall was 98.60, 98.57 and 98.57 for \(k = 16, 128, 256\) [Certain].

Worked example: the cost of reranking on COCO

The COCO 5K test set has 5,000 images and 25,000 captions (5 per image). Scoring every pair with ITM in both directions needs \(25{,}000 \times 5{,}000 + 5{,}000 \times 25{,}000 = 250{,}000{,}000\) fusion passes. With ITC first and ITM on the top \(k = 256\): \(25{,}000 \times 256 + 5{,}000 \times 256 = 7{,}680{,}000\) passes, 32.6 times fewer. With \(k = 16\) it is 480,000 passes, 521 times fewer. The ITC stage itself is only 30,000 encoder passes plus a matrix product, which is negligible.

Figure 4. The cost of retrieval (173.10) in both directions, counted in fusion-encoder passes. The gallery goes from 1,000 to 10 million images; \(k\) goes from 1 to 1,024 in powers of 2.
How to read Figure 4

The default is the COCO 5K test set with \(k = 256\): 250 million passes for exhaustive ITM against 7.68 million for reranking, 32.6 times fewer. Move the gallery to 10 million images: exhaustive ITM needs 1,000 trillion passes, which is impossible, while reranking grows only linearly with the number of queries. This is why every real system ranks with a dual encoder first and uses a fusion model only on a short list (session 169).

CapFilt: let the model clean its own data

ALBEF pretrains on 4.0 million images (5.1 million pairs) from COCO, Visual Genome, Conceptual Captions and SBU, and on 14.1 million images when the much noisier Conceptual 12M is added [Certain]. The web part dominates, and web alt-text often does not describe the image: “from bridge near my house”, “the current castle was built in 1180, replacing a 9th century wooden castle” (examples from the BLIP paper) [Certain]. Momentum distillation softens the damage; BLIP attacks the cause.

CapFilt (Captioning and Filtering) has two modules, both initialized from the same pretrained MED and then fine-tuned separately on COCO, a small human-annotated set [Certain]:

  • The captioner is the image-grounded decoder, fine-tuned with LM. For every web image \(I_w\) it writes one synthetic caption \(T_s\).
  • The filter is the image-grounded encoder, fine-tuned with ITC and ITM. A text is removed if the ITM head predicts “unmatched”. It filters both the original web texts \(T_w\) and the synthetic texts \(T_s\).

The new dataset is the filtered web pairs, the filtered synthetic pairs and the human-annotated pairs:

\[ \mathcal{D}' = \big\{(I_w, T_w) : f(I_w, T_w)\big\} \,\cup\, \big\{(I_w, T_s) : f(I_w, T_s)\big\} \,\cup\, \{(I_h, T_h)\} \tag{173.11} \]
Read the symbols
\((I_w, T_w)\)
web image and its original alt-text
\(T_s\)
synthetic caption of \(I_w\) from the captioner (nucleus sampling)
\(f(I, T)\)
the filter’s decision: true if ITM predicts “matched”
\((I_h, T_h)\)
human-annotated pairs (COCO, Visual Genome), always kept

Then a new model is pretrained from scratch on \(\mathcal{D}'\). On the 14M set the number of texts grew from 15.3M to 24.7M [Certain].

Algorithm 2: CapFilt dataset bootstrapping
  1. input: web pairs \(\{(I_w, T_w)\}\), human pairs \(\{(I_h, T_h)\}\)
  2. pretrain a MED \(\theta_0\) on all pairs with \(\mathcal{L}_{\text{itc}} + \mathcal{L}_{\text{itm}} + \mathcal{L}_{\text{lm}}\) // Algorithm 1
  3. captioner \(\leftarrow\) copy of \(\theta_0\), fine-tune with LM on COCO
  4. filter \(\leftarrow\) another copy of \(\theta_0\), fine-tune with ITC + ITM on COCO // separate weights: no confirmation bias
  5. for each web image \(I_w\): \(T_s \leftarrow\) captioner\((I_w)\) with nucleus sampling, \(p = 0.9\) // diverse, not the safest caption
  6.   keep \((I_w, T_w)\) if filter says matched; keep \((I_w, T_s)\) if filter says matched
  7. \(\mathcal{D}' \leftarrow\) kept web pairs \(\cup\) kept synthetic pairs \(\cup\) human pairs // (173.11)
  8. pretrain a new model on \(\mathcal{D}'\) // not continued training of \(\theta_0\)

Figure 5 is a toy simulation of the filter. The scores are synthetic; only the mechanism is real.

clean captionnoisy captionthreshold
Figure 5. A toy CapFilt on 1,000 web images. Each image has one web caption (noisy with the chosen rate) and one synthetic caption (noisy 20% of the time). The filter gives each caption a match probability; clean captions tend to score high, noisy ones low. Captions below the threshold are removed. In “shares weights” mode the filter is biased towards its twin’s captions: noisy synthetic captions score higher.
How to read Figure 5

Default (45% noisy web captions, threshold 0.5, separate filter): before filtering, 2,000 captions of which 1,348 are clean (67.4%). After filtering, 1,313 captions are kept and 96.4% of them are clean; the filter rejects 22.3% of the synthetic captions. Switch to “shares weights”: the filter now rejects only 9.5% of the synthetic captions and the clean share drops to 87.9%. This is the confirmation bias that BLIP measured (noise ratio 8% with sharing, 25% without). Push the threshold to 0.9: almost everything kept is clean, but you keep only 380 captions and throw away 968 clean ones.

What BLIP measured [Certain, their Tables 1, 2, 4, 12 and 13; ViT-B/16, 14M images unless noted]:

SettingCOCO TR@1 / IR@1 (fine-tuned)COCO CIDErNoCaps CIDEr (zero-shot)
Original web texts78.4 / 60.7127.8102.2
Captioner only, or filter only79.1 to 79.7 / 61.5 to 62.0128.2 to 128.9102.7 to 103.4
Captioner + filter (CapFilt)80.6 / 63.1129.7105.1
Beam search captions instead of nucleus (19% filtered)79.6 / 61.9128.9103.5
Captioner and filter share weights (8% filtered)79.8 / 62.2129.0103.5
Web texts repeated to 24.7M samples (longer training only)78.3 / 60.5127.7102.1
CapFilt data, but continue training the old model80.6 / 63.0129.9104.5
129M images (adds 115M LAION), CapFilt81.9 / 64.3131.4106.3
129M images, ViT-L/16, CapFilt with ViT-L82.4 / 65.1136.7113.2

Three lessons, each a favourite interview point:

  • Diversity beats safety. Nucleus sampling (sample from the smallest set of tokens whose probabilities add up to \(p = 0.9\), session 79) produced noisier captions than beam search (25% versus 19% rejected by the filter), yet the final model was better. The authors’ explanation: beam search writes safe, common captions that add little new information [Certain].
  • Captioner and filter must not share weights. When they did, the filter accepted the captioner’s mistakes (only 8% rejected) and results dropped. This is confirmation bias: a model judging its own output agrees with itself.
  • Train a new model on the new data. Repeating the old data does not help, and continuing to train the old model gives less gain, as in knowledge distillation where the student is not initialized from the teacher [Certain].

The recipes and the results

ALBEFBLIP
Image encoderViT-B/16 (DeiT, ImageNet-1k)ViT-B/16 or ViT-L/16 (ImageNet)
Text sideBERT-base split 6 + 6 layers (text, fusion)BERT-base as MED: 12 layers, 3 modes
LossesITC + ITM (hard negatives) + MLM, momentum distillationITC + ITM (hard negatives) + LM, ITC with momentum soft labels
Data4M or 14M images14M or 129M images (with LAION), bootstrapped by CapFilt
Batch, epochs512, 30 epochs, 8 A100 GPUs2,880 (ViT-B) or 2,400 (ViT-L), 20 epochs, two 16-GPU nodes
OptimizerAdamW, weight decay 0.02, warm-up to 1e-4 in 1,000 steps, cosine to 1e-5AdamW, weight decay 0.05, warm-up to 3e-4 (B) or 2e-4 (L), decayed with a rate of 0.85
Resolution256 pretraining, 384 fine-tuning224 pretraining, 384 fine-tuning (480 for VQA)

All entries are from the two papers [Certain]. Headline results, same sources [Certain]:

  • Fine-tuned retrieval, COCO 5K, text R@1: ALIGN (1.8B images) 77.0, ALBEF (14M) 77.6, BLIP (14M) 80.6, BLIP ViT-L (129M) 82.4. ALBEF with 14M images beat models trained on 100 times more data.
  • Zero-shot retrieval, Flickr30K, text R@1: CLIP 88.0, ALIGN 88.6, ALBEF (14M) 94.1, BLIP ViT-L 96.7 (the models were fine-tuned on COCO, then tested on Flickr30K).
  • VQA v2 test-dev: ALBEF (14M) 75.84, BLIP (14M) 77.54, BLIP (129M) 78.24, above SimVLM-base (77.87), which used 1.8B images.
  • Captioning, COCO Karpathy test, CIDEr: BLIP (14M) 129.7, BLIP ViT-L (129M) 136.7; NoCaps validation 113.2.
  • Zero-shot video: BLIP’s COCO retrieval model, with 8 frames per video concatenated and no temporal modeling, reaches 43.3 R@1 on MSRVTT text-to-video retrieval, above the fine-tuned VideoCLIP (30.9).

These results also explain the next step. Training all of this end to end is expensive, and the text side is only BERT-base. BLIP-2 (Li et al., 2023) freezes a strong image encoder and a large LLM and trains only a small Q-Former between them, with 32 learned query tokens [Certain] and the same three losses (contrastive, matching, image-grounded generation) controlled by different attention masks. CoCa (Yu et al., 2022) took a parallel path: one contrastive loss and one captioning loss in a single encoder–decoder. You will compare CLIP, BLIP and BLIP-2 yourself in session 176.

The PyTorch version

A complete, deterministic mini BLIP on synthetic data. Each “image” is 4 patches: two hold a coloured object (the colour is mixed into the object feature non-linearly, so colour and object are bound together), two are noise; the patches are shuffled. The caption names the two objects with their colours, e.g. [red, cube, blue, ball]. With 6 colours and 10 objects, the hard negative of an image is often the same two objects with the colours swapped: exactly what a pooled ITC vector finds hard. The model has a one-layer ViT, a two-block text transformer with separate encoder and decoder self-attention and shared cross-attention and FFN, a momentum copy, a queue, MoD soft targets, hard-negative ITM, and LM with label smoothing. It runs on a CPU in about a minute.

import copy, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) synthetic pairs: an image holds two coloured objects; the caption names them in order
NC, NO, DP, D = 6, 10, 16, 64            # colours, object types, patch feature size, model width
BOS, ENC = 0, 1                          # special tokens: [Decode]/BOS and [Encode]
COL, OBJ = 2, 2 + NC                     # word ids: colours 2..7, objects 8..17
V = 2 + NC + NO                          # vocabulary size 18
col_vec, obj_vec = torch.randn(NC, DP), torch.randn(NO, DP)
def make_pairs(n):
    c = torch.randint(0, NC, (n, 2)); o = torch.stack([torch.randperm(NO)[:2].sort().values for _ in range(n)])  # objects named in id order
    patch = torch.tanh(col_vec[c] * 2.0 + obj_vec[o]) + 0.3 * torch.randn(n, 2, DP)  # colour bound to object
    img = torch.cat([patch, 0.3 * torch.randn(n, 2, DP)], 1)[:, torch.randperm(4)]   # 4 patches, shuffled
    txt = torch.stack([COL + c[:, 0], OBJ + o[:, 0], COL + c[:, 1], OBJ + o[:, 1]], 1)
    return img, txt
img_tr, txt_tr = make_pairs(8192)
img_te, txt_te = make_pairs(500)

# 2) one text block: self-attention (bi or causal), optional cross-attention, FFN
class Block(nn.Module):
    def __init__(s):
        super().__init__()
        s.sa_enc = nn.MultiheadAttention(D, 4, batch_first=True)   # encoder self-attention
        s.sa_dec = nn.MultiheadAttention(D, 4, batch_first=True)   # decoder self-attention (not shared)
        s.ca = nn.MultiheadAttention(D, 4, batch_first=True)       # cross-attention (shared)
        s.ffn = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))  # shared
        s.n1, s.n2, s.n3 = nn.LayerNorm(D), nn.LayerNorm(D), nn.LayerNorm(D)
    def forward(s, x, img=None, causal=False):
        T = x.shape[1]
        mask = torch.triu(torch.ones(T, T, dtype=torch.bool), 1) if causal else None
        sa = s.sa_dec if causal else s.sa_enc
        h = s.n1(x); x = x + sa(h, h, h, attn_mask=mask)[0]
        if img is not None:
            h = s.n2(x); x = x + s.ca(h, img, img)[0]
        return x + s.ffn(s.n3(x))

class MED(nn.Module):                     # BLIP's mixture of encoder-decoder, tiny
    def __init__(s):
        super().__init__()
        s.patch = nn.Linear(DP, D); s.cls = nn.Parameter(torch.zeros(1, 1, D))
        s.vit = nn.TransformerEncoderLayer(D, 4, 4 * D, batch_first=True, norm_first=True)
        s.tok = nn.Embedding(V, D); s.pos = nn.Parameter(0.02 * torch.randn(1, 5, D))
        s.blocks = nn.ModuleList([Block() for _ in range(2)])
        s.proj_i, s.proj_t = nn.Linear(D, 32), nn.Linear(D, 32)   # ITC projections
        s.itm = nn.Linear(D, 2); s.lm = nn.Linear(D, V)
    def image(s, img):
        x = torch.cat([s.cls.expand(img.shape[0], -1, -1), s.patch(img)], 1)
        return s.vit(x)                                           # (B, 5, D); index 0 = [CLS]
    def text(s, ids, img=None, causal=False):
        x = s.tok(ids) + s.pos[:, :ids.shape[1]]
        for b in s.blocks: x = b(x, img, causal)
        return x
    def feats(s, img, txt):                                       # unit ITC embeddings
        cls_tok = torch.zeros(txt.shape[0], 1, dtype=torch.long) + ENC
        return (F.normalize(s.proj_i(s.image(img)[:, 0]), dim=-1),
                F.normalize(s.proj_t(s.text(torch.cat([cls_tok, txt], 1))[:, 0]), dim=-1))
    def match(s, img_tokens, txt):                                # ITM logits from the [Encode] token
        ids = torch.cat([torch.full((txt.shape[0], 1), ENC), txt], 1)
        return s.itm(s.text(ids, img_tokens)[:, 0])

model = MED(); teacher = copy.deepcopy(model).requires_grad_(False)  # momentum model
temp = nn.Parameter(torch.tensor(0.07))
opt = torch.optim.AdamW(list(model.parameters()) + [temp], lr=1e-3, weight_decay=0.05)
Q = 1024                                                         # queue length
q_img = F.normalize(torch.randn(Q, 32), dim=-1); q_txt = F.normalize(torch.randn(Q, 32), dim=-1)
alpha, mom, B = 0.4, 0.995, 64

for step in range(1200):
    idx = torch.randint(0, 8192, (B,)); img, txt = img_tr[idx], txt_tr[idx]
    t = temp.clamp(0.001, 0.5)
    # ---- ITC with momentum features, a queue and soft targets (173.1)-(173.4)
    u, v = model.feats(img, txt)
    with torch.no_grad():
        for p, pm in zip(model.parameters(), teacher.parameters()): pm.mul_(mom).add_((1 - mom) * p)
        um, vm = teacher.feats(img, txt)
        all_i, all_t = torch.cat([um, q_img]), torch.cat([vm, q_txt])
        onehot = torch.zeros(B, B + Q); onehot[:, :B] = torch.eye(B)
        tgt_i2t = alpha * F.softmax(um @ all_t.T / t, 1) + (1 - alpha) * onehot
        tgt_t2i = alpha * F.softmax(vm @ all_i.T / t, 1) + (1 - alpha) * onehot
    s_i2t, s_t2i = u @ all_t.T / t, v @ all_i.T / t
    l_itc = -0.5 * ((F.log_softmax(s_i2t, 1) * tgt_i2t).sum(1).mean()
                    + (F.log_softmax(s_t2i, 1) * tgt_t2i).sum(1).mean())
    q_img, q_txt = torch.cat([um, q_img])[:Q], torch.cat([vm, q_txt])[:Q]   # enqueue newest, drop oldest
    # ---- ITM with hard negatives sampled from in-batch similarity (173.5)-(173.6)
    img_tok = model.image(img)
    with torch.no_grad():
        w_i2t = F.softmax(u @ v.T / t, 1) + 1e-4; w_i2t.fill_diagonal_(0)
        w_t2i = F.softmax(v @ u.T / t, 1) + 1e-4; w_t2i.fill_diagonal_(0)
        neg_txt = torch.multinomial(w_i2t, 1).squeeze(1)       # one hard negative text per image
        neg_img = torch.multinomial(w_t2i, 1).squeeze(1)       # one hard negative image per text
    itm_logits = torch.cat([model.match(img_tok, txt), model.match(img_tok, txt[neg_txt]),
                            model.match(img_tok[neg_img], txt)])
    itm_labels = torch.cat([torch.ones(B), torch.zeros(2 * B)]).long()
    l_itm = F.cross_entropy(itm_logits, itm_labels)
    # ---- LM: causal decoder, cross-attends to the image, label smoothing 0.1 (173.7)
    dec_in = torch.cat([torch.full((B, 1), BOS), txt[:, :-1]], 1)
    l_lm = F.cross_entropy(model.lm(model.text(dec_in, img_tok, causal=True)).reshape(-1, V),
                           txt.reshape(-1), label_smoothing=0.1)
    loss = l_itc + l_itm + l_lm
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 400 == 399:
        print(step + 1, "itc %.3f itm %.3f lm %.3f temp %.3f" % (l_itc.item(), l_itm.item(), l_lm.item(), t.item()))

# 3) evaluation: ITC retrieval, ITC top-k + ITM rerank, greedy captioning
model.eval()
with torch.no_grad():
    u, v = model.feats(img_te, txt_te); sim = u @ v.T            # (500 images, 500 texts)
    same = (txt_te[:, None] == txt_te[None]).all(-1)            # duplicates count as correct
    print("ITC image->text R@1:", same[torch.arange(500), sim.argmax(1)].float().mean().item())
    k, hits = 16, 0
    tok = model.image(img_te)
    for i in range(500):
        cand = sim[i].topk(k).indices                            # cheap first stage
        s_itm = F.softmax(model.match(tok[i:i + 1].expand(k, -1, -1), txt_te[cand]), 1)[:, 1]
        hits += int(same[i, cand[s_itm.argmax()]])                   # rerank with the fusion model
    print("ITC top-%d + ITM rerank R@1:" % k, hits / 500)
    ids = torch.full((500, 1), BOS)
    for _ in range(4):
        nxt = model.lm(model.text(ids, tok, causal=True))[:, -1].argmax(-1, keepdim=True)
        ids = torch.cat([ids, nxt], 1)
    print("exact caption match:", (ids[:, 1:] == txt_te).all(1).float().mean().item())

Output with PyTorch 2.8 on CPU (two runs gave identical numbers): after 400, 800 and 1,200 steps the losses are ITC 3.011, 2.404, 1.949; ITM 0.346, 0.231, 0.101; LM 0.817, 0.760, 0.702, and the learned temperature settles near 0.026. The LM floor for \(\varepsilon = 0.1\) and a vocabulary of 18 is 0.580, so the decoder is close to its best. On 500 test pairs, ITC alone ranks the right caption first for 75.0% of the images; reranking the top 16 with ITM raises this to 90.4%. The fusion encoder fixes the colour–object binding mistakes that the pooled vectors make. Greedy captioning gets all four words right for 77.4% of the test images. Exact values can differ slightly across PyTorch versions [Likely].

Where this is used

ApplicationWhich head does the work
Image–text search with high precisionITC embeddings in a vector index for the first stage, ITM reranking of the top \(k\) (173.10) (session 151)
Text-to-video retrievalFrame features from the image encoder, ITC for candidates, ITM for reranking; BLIP even works zero-shot on MSRVTT with 8 concatenated frames (session 208)
Key-moment detection and Video-RAGScore sampled frames or clips against a text query with ITC, confirm the best ones with ITM, and caption the kept clips so an LLM can read them (session 210)
Data filtering and re-captioningCapFilt-style pipelines: caption web images, score (image, text) pairs with ITM, keep the good ones; used widely to build VLM training data [Likely] (session 187)
Captions for text-to-image trainingBLIP captions were widely used to (re)caption image sets for diffusion fine-tuning [Likely] (session 228)
VQA and video QAImage-grounded encoder plus answer decoder; zero-shot transfer to MSRVTT-QA (19.2) and MSVD-QA (35.2) [Certain]
Weakly supervised groundingGrad-CAM on ALBEF’s cross-attention maps locates the words in the image without box labels (RefCOCO+) [Certain] (session 174)
Connectors of modern VLMsBLIP-2’s Q-Former is trained with ITC, ITM and image-grounded generation before it is connected to an LLM (session 178)
Video fingerprinting and near-duplicatesITC embeddings of frames as a compact signature; ITM to verify a candidate match when two clips look alike (session 209)

Common mistakes

  • Mining ITM negatives from the queue. The queue stores feature vectors; ITM needs the real image and text to run the fusion encoder. Mine hard negatives from the current batch.
  • Taking the single hardest negative. The most similar caption is often a true match. Sample in proportion to similarity, as (173.6) does, and keep the diagonal at zero.
  • Letting gradient flow through the momentum model or the sampling weights. The momentum encoder is updated only by (173.3), and the hard-negative weights are computed under no_grad.
  • Forgetting that the ITM batch is 1 : 2. With \(B\) positives and \(2B\) negatives, a head that always says “no” is right 67% of the time, and a head that always outputs 1/3 for “matched” already has loss 0.637. Report balanced accuracy or the loss, not plain accuracy.
  • Sharing self-attention between encoder and decoder. BLIP’s ablation shows it hurts (77.3 versus 78.4 TR@1). Share embeddings, cross-attention and FFN; keep separate self-attention layers with the right masks.
  • Using the same network as captioner and filter. Confirmation bias: the filter accepts its own mistakes. Fine-tune two separate copies.
  • Generating synthetic captions with beam search. They look cleaner, but they are less diverse and the final model is worse. Use nucleus sampling and let the filter remove the bad ones.
  • Comparing the LM loss with zero. With label smoothing 0.1 and a 30,524-token vocabulary, the floor is 1.358 nats.

Interview questions

What are ITC, ITM and LM, and why train with all three?basic
ITC is a symmetric contrastive loss on the unimodal image and text embeddings: it aligns the two spaces and gives fast retrieval. ITM is a binary classifier on the fused representation (text cross-attending to image patches): it learns fine-grained matching. LM is next-token prediction of the caption conditioned on the image: it lets the model generate. Each covers what the others miss: ITC is fast but coarse, ITM is precise but needs one forward pass per pair, and neither can write. ALBEF’s ablation shows adding ITC to MLM + ITM improved every downstream task (for example VQA 71.40 to 73.29), and hard negatives for ITM added more.
What does “align before fuse” mean, and why does it help?core
The contrastive loss is applied to the unimodal encoders before the fusion encoder. The fusion encoder then receives image and text features that already live in a shared, aligned space, so cross-attention can find related tokens more easily than with raw detector features. It also trains the unimodal encoders to be good on their own (useful for retrieval), and the ITC similarities give a free way to mine hard negatives for ITM.
How are hard negatives for ITM chosen? Why sample instead of taking the top one?core
For each image in the batch, one negative text is sampled with probability proportional to \(\exp(s(I_i, T_j)/\tau)\) over \(j \ne i\), using the ITC similarities already computed; likewise one negative image per text. So the batch gives \(B\) positives and \(2B\) negatives at no extra encoder cost. Sampling rather than argmax is used because the most similar caption may actually be a correct description (web data has near-duplicates); always choosing it would train the model on wrong labels. Sampling still concentrates on hard cases: in Figure 3, a near-duplicate gets 0.945 of the probability at τ = 0.07.
Why does ALBEF use a momentum encoder and a queue? What do μ and M control?core
The batch is only 512, so in-batch negatives are few. A queue of the last \(M = 65{,}536\) features gives 128 times more negatives at the cost of storing vectors (67 MB per queue in float32 at 256-d). The features must come from a consistent encoder, so they are produced by a momentum copy, an exponential moving average of the weights with \(\mu = 0.995\) (an average over roughly 200 steps). Larger \(\mu\) gives a more consistent but more stale queue; larger \(M\) gives more negatives but older entries. The momentum model also serves as the teacher for momentum distillation.
Explain momentum distillation. Write the loss and its gradient.advanced
The momentum model produces soft targets \(\mathbf{q}\) (its softmax over candidates). The loss is \((1-\alpha)\,H(\mathbf{y}, \mathbf{p}) + \alpha\, \KL(\mathbf{q} \Vert \mathbf{p})\), which has the same gradient as cross-entropy against \(\mathbf{t} = (1-\alpha)\mathbf{y} + \alpha\mathbf{q}\); the gradient on logit \(m\) is \(p_m - t_m\). It helps with noisy web pairs: a caption that also fits the image (a false negative) gets some target mass, so it is pushed away less. In the worked example the push on such a caption drops from 0.359 to 0.175 at \(\alpha = 0.4\). ALBEF applies it to ITC and MLM, and also during fine-tuning.
What exactly is shared between BLIP’s text encoder and decoder, and why?core
Everything except the self-attention layers: the embeddings, the cross-attention layers and the FFNs are shared; the encoder has bidirectional self-attention and the decoder has its own causal self-attention. The authors argue the difference between encoding and decoding lives in self-attention. Their ablation: sharing all layers 224M parameters and 77.3 TR@1; all except self-attention 252M and 78.4; no sharing 361M and 78.3. So sharing saves 109M parameters at no cost, while sharing self-attention creates a conflict between the two tasks.
How do you do image–text retrieval with BLIP on a large gallery?core
Two stages. Embed every image once with the image encoder and ITC projection and store the vectors in an ANN index. For a text query, compute its ITC embedding, get the top \(k\) images by cosine, then run the image-grounded encoder on each of the \(k\) pairs and rerank by ITM match probability. BLIP uses \(k = 256\) on COCO; ALBEF showed the result barely changes from \(k = 16\) to 256. On COCO 5K this is 7.68M fusion passes instead of 250M for exhaustive ITM.
Explain CapFilt. Why must the captioner and the filter be separate?basic
Pretrain a MED, then fine-tune two copies on COCO: a captioner (decoder, LM loss) and a filter (encoder, ITC + ITM). The captioner writes one synthetic caption per web image with nucleus sampling; the filter removes any web or synthetic caption that ITM judges unmatched. Human-annotated pairs are added and a new model is pretrained on the result. If captioner and filter share weights, the filter tends to accept the captioner’s mistakes (confirmation bias): BLIP measured only 8% of synthetic captions filtered with sharing versus 25% without, and worse downstream results (COCO TR@1 79.8 versus 80.6).
Why did nucleus sampling beat beam search for generating synthetic captions?advanced
Beam search finds high-probability captions, which tend to be generic and similar to common training captions, so they add little information. Nucleus sampling (\(p = 0.9\)) gives more diverse and surprising captions that carry new details. They are noisier (the filter rejected 25% versus 19%), but the filter removes the bad ones, and the remaining diversity helps: COCO TR@1 80.6 versus 79.6, NoCaps CIDEr 105.1 versus 103.5. The general lesson for synthetic data: diversity plus a good filter beats safe, repetitive generation.
Is the CapFilt gain just from more training samples or longer training?advanced
No. BLIP repeated the original web texts so that each epoch had the same 24.7M samples as the bootstrapped set: results did not improve (78.3 versus 78.4 TR@1), while CapFilt gave 80.6. They also tried continuing to train the old model on the new data, which gave less gain than pretraining a new model (NoCaps CIDEr 104.5 versus 105.1), consistent with distillation practice where the student is not initialized from the teacher.
ALBEF reaches better retrieval than CLIP with 14M instead of 400M images. How?core
Several reasons, and the comparison is not one-to-one. ALBEF is fine-tuned on COCO before the “zero-shot” Flickr30K test and uses ITM reranking, a fusion model that checks each candidate pair in detail; CLIP is a pure dual encoder. ALBEF also starts from pretrained DeiT and BERT weights and trains on cleaner, more caption-like data (COCO and Visual Genome are human-written). So: Flickr30K zero-shot TR@1 94.1 versus 88.0, but CLIP’s advantages remain, such as zero-shot classification over any label set and much cheaper inference.

Exercises

  1. ALBEF’s momentum coefficient is 0.995. After 1,000 steps, how much weight do the trained weights from more than 1,000 steps ago still have in the momentum model?
  2. Recompute the worked MoD example with \(\alpha = 1\). What is the gradient on the logit of T2, and what does its sign mean?
  3. A batch has \(B = 512\) pairs. How many pairs does the ITM head see per step? What is the loss of a head that always outputs probability 1/3 for “matched”?
  4. For the Flickr30K 1K test set (1,000 images, 5,000 captions) and \(k = 128\), how many fusion passes does two-stage retrieval in both directions need, compared with exhaustive ITM?
  5. With label smoothing \(\varepsilon = 0.1\) over a vocabulary of 30,524 tokens, what probability does the target give to the true token, and to each other token?
  6. In BLIP’s Table 3, how many parameters does the decoder’s separate self-attention cost, compared with sharing everything?
Answerscheck
1. \(0.995^{1000} = 0.0067\): less than 1% of the momentum weights still come from before those 1,000 steps (the initial value), so the teacher is effectively an average over the last few hundred steps. 2. With \(\alpha = 1\), \(\mathbf{t} = \mathbf{q} = (0.532, 0.461, 0.004, 0.003)\), so the gradient on T2 is \(0.359 - 0.461 = -0.102\). Negative means gradient descent raises the T2 logit: the model is pulled to agree with the teacher that T2 fits. 3. \(512 + 2 \times 512 = 1{,}536\) pairs. Loss \(= -(\tfrac13 \ln \tfrac13 + \tfrac23 \ln \tfrac23) = 0.637\) nats, the entropy of the 1 : 2 label mix. 4. Two-stage: \(5{,}000 \times 128 + 1{,}000 \times 128 = 768{,}000\). Exhaustive: \(2 \times 5{,}000 \times 1{,}000 = 10{,}000{,}000\), 13.0 times more. 5. True token \(0.9 + 0.1/30{,}524 = 0.900003\); every other token \(0.1/30{,}524 = 3.28 \times 10^{-6}\). 6. 252M versus 224M: 28M parameters, the cost of a second set of self-attention layers.

Key takeaways

  • ALBEF and BLIP combine a dual encoder and a fusion encoder in one network: ITC on the unimodal encoders (173.1, 173.2), ITM on the fused [CLS] (173.5), and a language loss, MLM in ALBEF (173.7) or captioning in BLIP (173.8).
  • With a batch of only 512, ALBEF gets 65,536 negatives from a queue filled by a momentum encoder (173.3); the same momentum model provides soft targets that soften false negatives (173.4).
  • ITM learns fine details because its negatives are hard: sampled in proportion to the ITC similarity, one text per image and one image per text (173.6).
  • BLIP’s MED is one text transformer in three modes; encoder and decoder share everything except self-attention, saving 109M parameters at no loss.
  • Retrieval runs in two stages: ITC top-\(k\), then ITM rerank (173.10), about 30 times cheaper on COCO than exhaustive ITM and nearly as accurate.
  • CapFilt (173.11) re-captions web images with nucleus sampling and filters web and synthetic captions with a separate ITM model; then a new model is trained. Diversity, separate weights and a fresh model all matter.

Key formulas

\[ \mathcal{L}_{\text{itc}} = \tfrac12\, \E\big[H(\mathbf{y}^{\text{i2t}}, \mathbf{p}^{\text{i2t}}) + H(\mathbf{y}^{\text{t2i}}, \mathbf{p}^{\text{t2i}})\big], \quad \theta' \leftarrow \mu\theta' + (1-\mu)\theta \]
\[ \mathbf{t} = (1-\alpha)\,\mathbf{y} + \alpha\,\mathbf{q}, \qquad P(j \mid i) \propto e^{s(I_i, T_j)/\tau},\ j \ne i \]
\[ \mathcal{L}_{\text{BLIP}} = \mathcal{L}_{\text{itc}} + \mathcal{L}_{\text{itm}} + \mathcal{L}_{\text{lm}}, \qquad \hat{I}(T) = \argmax_{I \in \text{top-}k(s_{\text{itc}})} p^{\text{itm}}_{\text{match}}(I, T) \]

References for this session

  1. Li et al. (2021). Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
  2. Li et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  3. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  5. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 4: Training Data and Preprocessing for VLMs
  6. Li et al. (2021). Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
  7. Li et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  8. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  9. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  10. Jia et al. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  11. He et al. (2019). Momentum Contrast for Unsupervised Visual Representation Learning
  12. Chen et al. (2019). UNITER: UNiversal Image-TExt Representation Learning
  13. Kim, Son & Kim (2021). ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
  14. Wang, Yu, Yu, Dai, Tsvetkov & Cao (2021). SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  15. Yu, Wang, Vasudevan, Yeung, Seyedhosseini & Wu (2022). CoCa: Contrastive Captioners are Image-Text Foundation Models
  16. Changpinyo, Sharma, Ding & Soricut (2021). Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
  17. Holtzman et al. (2019). The Curious Case of Neural Text Degeneration
  18. Hinton et al. (2015). Distilling the Knowledge in a Neural Network
  19. Touvron et al. (2020). Training data-efficient image transformers & distillation through attention
  20. Schuhmann et al. (2022). LAION-5B: An open large-scale dataset for training next generation image-text models