Two-stream and single-stream VLMs: ViLBERT, LXMERT, UNITER
Before CLIP and before LLaVA, vision-language models were BERT-style encoders that read object-detector regions and words together. ViLBERT and LXMERT keep two streams that talk through co-attention; UNITER puts everything in one stream. This session shows how regions become tokens, what the pretraining losses (masked words, masked regions, image–text matching) compute, what the papers measured, and why this design faded.
What you will learn
- How a Faster R-CNN detector turns an image into 10 to 36 region tokens, each a 2048-d feature plus a 5-d or 7-d box vector
- The co-attention layer of ViLBERT and the cross-modality layer of LXMERT, and how they differ from one joint stream (UNITER)
- Why a two-stream layer pair and a single-stream layer compute the same number of attention scores, (T + K)² = T² + K² + 2TK
- The pretraining losses: masked language modeling with the image, three kinds of masked region modeling, image–text matching, image QA and word–region alignment
- What the ablations measured: conditional masking, more data and in-domain data, and the ViLBERT single-stream baseline
- Why these models faded: about 900 ms per image for region features, under 10M training pairs, one forward pass per image–text pair for retrieval, and no text generation
- Which ideas survive in ALBEF, BLIP, BLIP-2 and modern VLM rerankers
Why this matters
In 2018, BERT showed that one large encoder, pretrained on masked words and then fine-tuned with a small head, beats task-specific models on almost every text task. In 2019, several groups asked the obvious next question: can we do the same for images and text together? The answers were ViLBERT, LXMERT, VisualBERT, VL-BERT and, a few months later, UNITER. They are the first generation of pretrained vision-language models.
They all share one recipe. An object detector finds 10 to 100 regions in the image (a person, a horse, a hat). Each region becomes one visual token, just like a word is one text token. A transformer reads the words and the regions and learns, from image–caption pairs, which words go with which regions. Then the same model is fine-tuned for visual question answering (VQA), visual commonsense reasoning (VCR), referring expressions and image–text retrieval.
The models differ in one main design choice: where the two modalities meet. ViLBERT and LXMERT use two streams, one transformer per modality, that exchange information through special co-attention layers. UNITER (and VisualBERT, VL-BERT) use one stream: the words and regions are joined into one sequence and every layer is ordinary self-attention.
Why study models from 2019 in a course about modern VLMs? Three reasons. First, their pretraining losses are still used: ALBEF and BLIP (session 173) keep masked language modeling and image–text matching, and BLIP-2’s Q-Former uses image–text matching too. Second, the two-stream versus one-stream question did not go away: Flamingo-style cross-attention versus LLaVA-style “just put image tokens in the sequence” is the same question at a larger scale (session 179). Third, the reasons these models faded (a slow detector, small paired data, no text generation) explain the design of everything that came after: CLIP’s huge web data (session 166), ViT patches instead of regions (session 164), and decoder LLMs that write answers.
Keep one difference from CLIP in mind for the whole session. CLIP is a dual encoder: image and text are encoded separately and compared only at the end with a dot product. The models here are fusion encoders: image and text tokens interact inside the network, many layers deep. Fusion gives finer reasoning; separate encoders give speed. Both facts matter below.
From pixels to visual tokens: region features
Words come in a natural unit: the token. Images do not. In 2019, the best unit available was the object region, because of one very successful earlier paper: “bottom-up attention” (Anderson et al., 2018). It trained a Faster R-CNN detector (a two-stage object detector: a region proposal network suggests boxes, then a head classifies and refines each box) with a ResNet-101 backbone on Visual Genome, a dataset with about 1,600 object classes and attributes per box [Certain]. For every detected box, the detector also gives a 2048-dimensional feature vector: the average of the backbone’s feature map inside the box (RoI pooling). These features had already pushed VQA and captioning forward, so ViLBERT, LXMERT and UNITER all reused them.
So the image becomes a set of regions \(\{(\mathbf{f}_i, \mathbf{b}_i)\}_{i=1}^{K}\): a feature \(\mathbf{f}_i \in \R^{2048}\) and a box \(\mathbf{b}_i = (x_1, y_1, x_2, y_2)\) in pixels. How many regions? ViLBERT keeps boxes whose class confidence is above a threshold, but never fewer than 10 or more than 36 [Certain]. LXMERT always keeps exactly 36, so every image has the same length and no padding is wasted [Certain]. UNITER also uses Faster R-CNN features pretrained on Visual Genome, with a variable number of boxes [Certain].
Where is the region? The location vector
Words have an order, so BERT adds a position embedding. Regions have no order; what matters is where they are. So each model builds a small location vector from the box. ViLBERT uses 5 numbers: the box corners divided by the image size, and the fraction of the image the box covers [Certain]:
UNITER uses 7 numbers: the 4 normalized corners, the normalized width and height, and their product (the area) [Certain]:
- Read the symbols
- \(x_1, y_1, x_2, y_2\)
- the top-left and bottom-right corners of the box, in pixels
- \(W, H\)
- the image width and height in pixels
- \(\hat x_1, \hat y_1, \dots\)
- the same corners divided by \(W\) or \(H\), so they lie between 0 and 1
- \(\hat w, \hat h\)
- box width and height as a fraction of the image width and height
One embedding per region
The feature and the location live in different spaces (2048-d and 5-d or 7-d), so each gets its own linear layer into the model width \(d\), and the results are added. UNITER then applies layer normalization, and adds a modality embedding (like BERT’s segment embedding) so the model can tell an image token from a word [Certain]:
- Read the symbols
- \(\mathbf{f}_i\)
- the 2048-d detector feature of region \(i\)
- \(W_f, W_p\)
- learned matrices of size \(d \times 2048\) and \(d \times 7\) (or \(d \times 5\))
- \(\mathbf{e}_{\text{img}}, \mathbf{e}_{\text{txt}}\)
- two learned vectors that mark “this is a region” or “this is a word”
- \(E_{\text{word}}, E_{\text{pos}}\)
- BERT’s word-piece and position embedding tables
- \(\operatorname{LN}\)
- layer normalization
LXMERT does almost the same, but normalizes the two parts separately and averages them: \(\mathbf{v}_i = \tfrac12\big(\operatorname{LN}(W_F\mathbf{f}_i + \mathbf{b}_F) + \operatorname{LN}(W_P\mathbf{p}_i + \mathbf{b}_P)\big)\) [Certain]. ViLBERT also adds a special IMG token at the start of the region sequence: the mean of all region features, with the location of the whole image [Certain]. It plays the role that [CLS] plays for text.
An image is \(640 \times 480\) pixels. The detector finds a horse with box \((160, 120, 480, 360)\). Normalized corners: \(160/640 = 0.25\), \(120/480 = 0.25\), \(480/640 = 0.75\), \(360/480 = 0.75\). The box is \(320 \times 240\) pixels, so it covers \(76{,}800 / 307{,}200 = 0.25\) of the image.
ViLBERT, formula (172.1): \((0.25, 0.25, 0.75, 0.75, 0.25)\). UNITER, formula (172.2): \(\hat w = 0.5\), \(\hat h = 0.5\), so \((0.25, 0.25, 0.75, 0.75, 0.5, 0.5, 0.25)\). Note that the area is the same in both, and that two boxes with the same size but different places differ only in the first four numbers.
The caption “a man rides a horse on the beach” has 8 words. With 6 regions, a single-stream input has \(1\ (\text{IMG}) + 6 + 1\ (\text{CLS}) + 8 + 1\ (\text{SEP}) = 17\) tokens. Check it in Figure 1.
How to read Figure 1
Move the threshold up: weak boxes (bird, saddle, wave) disappear first, and the sequence gets shorter. The real models use the same rule with bigger limits: ViLBERT keeps 10 to 36 boxes, LXMERT exactly 36. Now notice what the model can never see: anything the detector did not box. If the question is “what color is the water?” and no box covers the water, the answer must come from other boxes. This is the first weakness of region features. Click a dropped box: its location vector is still computed, but it is not part of the input.
Two streams: ViLBERT’s co-attention
ViLBERT (Lu et al., 2019) starts from a simple argument: words and regions are different kinds of data. A word embedding is low-level and needs many layers of context. A region feature is already the output of a 101-layer CNN and a detector head, so it is high-level and needs less processing [Certain]. So ViLBERT keeps two transformers: a text stream initialized from BERT-base (12 layers, width 768, 12 heads) and a new visual stream (width 1024, 8 heads) [Certain]. The text stream runs several ordinary layers first; then the two streams meet in co-attentional transformer layers (Co-TRM), each followed by ordinary layers (TRM) in both streams. The best depth was 6 such Co-TRM→TRM blocks [Certain].
A co-attention layer is ordinary multi-head attention with one change: each stream takes its queries from itself but its keys and values from the other stream.
- Read the symbols
- \(H_W \in \R^{T \times d_w}\)
- the hidden states of the \(T\) text tokens
- \(H_V \in \R^{K \times d_v}\)
- the hidden states of the \(K\) region tokens
- \(W^Q, W^K, W^V\)
- learned projection matrices; each stream has its own set
- \(\operatorname{MHA}\)
- multi-head attention with an output projection; after it come layer normalization and a feed-forward block, as in any transformer layer
Read the first line as: “every word asks a question, and only the regions answer”. The result for word \(j\) is a mixture of region values: image-conditioned text. The second line is the reverse: text-conditioned regions. After a co-attention layer, each stream keeps its own length and its own width; only the content has been mixed.
LXMERT: three encoders
LXMERT (Tan and Bansal, 2019) uses the same idea with a cleaner layout: a language encoder of 9 self-attention layers, an object-relationship encoder of 5 self-attention layers over the regions, and a cross-modality encoder of 5 layers, all with width 768 [Certain]. Inside each cross-modality layer, two cross-attention sub-layers run first (language to vision and vision to language), then a self-attention sub-layer in each stream, then a feed-forward sub-layer in each stream [Certain]. The authors count a single-modality layer as half a cross-modality layer: \((9 + 5)/2 + 5 = 12\), the depth of BERT-base [Certain]. Unlike ViLBERT, LXMERT trains everything from scratch; loading BERT weights made results worse in their tests [Certain].
One stream: UNITER
The single-stream answer is simpler. Embed the words with (172.4), embed the regions with (172.3), concatenate them into one sequence of length \(T + K\), and run a standard BERT encoder. Every layer is plain self-attention, so from the first layer every word can attend to every region and every region to every word. VisualBERT and VL-BERT did this in 2019 [Certain]; UNITER (Chen et al., 2020) did it with more data and better losses, in two sizes: UNITER-base (12 layers, width 768, 12 heads, 86M parameters) and UNITER-large (24 layers, width 1024, 16 heads, 303M parameters) [Certain].
Watch the tokens talk: one stream versus two streams
The two layouts differ in one place: which keys each query may read in each layer. Everything else is the same attention formula. To see what that changes, take a tiny example you can follow by hand: 4 words (man, rides, horse, beach) and 3 regions (man, horse, sand), each a 5-number vector. The first four numbers mean roughly “person, animal, place, action”. The fifth is the modality embedding: \(+m\) for words, \(-m\) for regions, like \(\mathbf{e}_{\text{txt}}\) and \(\mathbf{e}_{\text{img}}\) in (172.3) and (172.4). To keep the numbers readable, the toy uses no learned projections (\(W^Q = W^K = W^V = I\)), no feed-forward block, and averages the residual with \(\tfrac12\) instead of layer normalization. One toy layer is then:
The set \(J(i)\) is the whole difference. Single-stream layer: \(J(i)\) is all \(T + K\) tokens. Co-TRM: \(J(i)\) is the other modality only. TRM: \(J(i)\) is the same modality only. To measure how much image information a token carries, follow its image share \(\rho_i\): it starts at 0 for a word and 1 for a region, and mixes with the same weights:
- Read the symbols
- \(\mathbf{h}_i\)
- the state of token \(i\) before the layer; \(\mathbf{h}_i'\) after it. All tokens are updated at the same time from the old states
- \(J(i)\)
- the keys token \(i\) may attend to in this layer
- \(a_{ij}\)
- attention weight from query \(i\) to key \(j\); the weights over \(J(i)\) sum to 1
- \(d\)
- the vector size, 5 here, so the scores are divided by \(\sqrt{5} \approx 2.236\)
- \(\rho_i\)
- the fraction of token \(i\)’s state that came from region inputs
How to read Figure 3
Press Play in single-stream mode with \(m = 1\). The query “horse” reads all 7 tokens, but its weights to the regions are small (0.035, 0.148, 0.027), because the modality numbers \(+1 \cdot (-1)\) lower every word–region score. After 2 layers the words carry only about 16–23% image information. Move \(m\) to 0: the shares grow to about 26–33%. Move it to 2: they fall to about 1–2%. In a single stream, how much the modalities mix is something the model must learn. Now switch to two streams. In Co-TRM the word “horse” may read only regions, so its weights (0.169, 0.705, 0.127) put most mass on the horse region, and every word’s image share becomes exactly 0.5 after the first Co-TRM layer, whatever \(m\) is: the constant \(-m^2\) shift cancels inside the softmax. Co-attention forces the exchange; TRM layers then mix inside each stream. Watch the purple “new” bars: no token changes until line 7, because all tokens read the old states.
Is two-stream cheaper? Count the attention scores
A common belief is that two streams save compute. Count the attention scores (the entries of \(QK^\top\)) per layer. A single-stream layer over \(T\) words and \(K\) regions computes \((T + K)^2\) scores. A ViLBERT Co-TRM layer computes \(TK\) word-to-region scores plus \(KT\) region-to-word scores. The TRM layer after it computes \(T^2\) word-to-word plus \(K^2\) region-to-region scores. Add them:
So one Co-TRM→TRM pair computes exactly as many scores as one single-stream layer; the two-stream model just splits the big attention matrix into four blocks and computes them in two steps. The same holds for one LXMERT cross-modality layer. The real differences are elsewhere: two streams have separate weights for each modality (so more parameters: ViLBERT-style models are larger), they can use different widths and depths per modality, and the modalities meet only at chosen layers instead of in every layer.
How to read Figure 4
With \(T = 20\) and \(K = 36\), a single-stream layer fills the whole square: 3,136 scores. Co-TRM fills only the two off-diagonal blocks (1,440); TRM fills only the two diagonal blocks (1,696). Together: 3,136 again. LXMERT’s cross layer does both inside one layer. Now set \(K = 100\) (Oscar and VinVL used up to 50 or more regions): the region-to-region block dominates in every design, because attention cost grows with the square of the sequence length.
What did the experiments say? ViLBERT’s own paper trained a single-stream baseline on the same data and features, and it lost on every task, for example 68.85 versus 70.55 on VQA test-dev [Certain]. But UNITER-base, a single-stream model with more and cleaner data and better losses, reached 72.70 on VQA test-dev [Certain]. A later controlled study (Bugliarello et al., 2021) re-trained several of these models with the same data and hyper-parameters and found that most of the reported differences came from training data and hyper-parameters, not from one-stream versus two-stream [Certain]. The honest summary: the fusion layout matters less than the data and the losses.
Pretraining tasks
All three models are pretrained on image–caption pairs with a mix of self-supervised tasks, then fine-tuned. The tasks are BERT’s tasks, extended to two modalities.
Masked language modeling with the image (MLM)
Mask about 15% of the words (80% become [MASK], 10% a random word, 10% unchanged, as in BERT) and predict them from the other words and all regions [Certain]:
- Read the symbols
- \((w, v)\)
- one caption (words) and its image (regions)
- \(\mathcal{M}\)
- the set of masked word positions
- \(w_{\setminus \mathcal{M}}\)
- the words that are not masked
If the caption is “a man rides a [MASK] on the beach”, the text alone allows horse, bike or camel. The region of a horse makes “horse” likely. This is how the model is forced to look at the image.
Masked region modeling (MRM)
Mask about 15% of the regions by setting their features to zero, and ask the model to recover them from the other regions and the words [Certain]. A region is a continuous 2048-d vector, not a word from a vocabulary, so “recover” needs a definition. UNITER tested three [Certain]:
- Read the symbols
- \(\mathcal{M}_v\)
- the set of masked regions
- \(\mathbf{h}_i\)
- the model’s output vector at masked region \(i\)
- \(g_\theta\)
- a linear layer back to 2048 dimensions; MRFR regresses the original feature \(\mathbf{f}_i\)
- \(\tilde{\mathbf{c}}_i\)
- the detector’s class distribution for that box (for example 1,600 Visual Genome classes)
- \(q_\theta\)
- the model’s predicted class distribution (a linear layer and softmax)
- \(c_i^{*}\)
- the detector’s top class, used as a hard label in MRC
ViLBERT uses only the KL version (it zeroes masked region features 90% of the time and leaves them unchanged 10%) and argues that regression would be hard to balance against the text loss [Certain]. LXMERT uses both regression and detected-label classification [Certain]. Note what all three variants do: the target comes from the detector. The model learns to agree with Faster R-CNN, not with the real image.
The detector is unsure about a masked box: \(\tilde{\mathbf{c}} = (0.7, 0.2, 0.1)\) over (horse, cow, dog). The model predicts \(q = (0.5, 0.3, 0.2)\).
MRC uses the hard label “horse”: loss \(-\ln 0.5 = 0.693\). MRC-kl compares full distributions: \(\KL(\tilde{\mathbf{c}} \,\|\, q) = 0.7\ln\tfrac{0.7}{0.5} + 0.2\ln\tfrac{0.2}{0.3} + 0.1\ln\tfrac{0.1}{0.2} = 0.085\). The KL is small because the model already has the right shape; it would be 0 only if \(q = \tilde{\mathbf{c}}\). MRC would keep pushing \(q(\text{horse})\) toward 1, even though the detector itself gives “cow” 20%. In UNITER’s ablation, MRC-kl gave a higher total score than MRC (397.09 vs 393.97) [Certain].
Why does MRC keep pushing? Look at the gradient. For a softmax output \(q = \softmax(\mathbf{z})\), the gradient of the cross-entropy against any target distribution \(\mathbf{t}\) with respect to the logits \(\mathbf{z}\) is \(q - \mathbf{t}\). MRC uses a one-hot target, MRC-kl the detector’s distribution (the KL and the cross-entropy differ only by a constant, the entropy of \(\tilde{\mathbf{c}}\)):
- Read the symbols
- \(\mathbf{z}\)
- the logits of the region-classification head at one masked region
- \(\mathbf{e}_{c^*}\)
- the one-hot vector of the detector’s top class
- \(\eta\)
- the step size (learning rate)
The MRC gradient is zero only when \(q\) is one-hot, which needs infinite logits, so it never stops. The MRC-kl gradient is zero exactly when \(q = \tilde{\mathbf{c}}\). Figure 5 runs both losses on the worked example, step by step.
How to read Figure 5
Step through MRC: the first gradient is \((-0.5, 0.3, 0.2)\), so the horse logit goes up and the others go down. With \(\eta = 1\), after 20 steps \(q(\text{horse})\) is about 0.97, and the distance to the detector’s belief, KL\((\tilde{\mathbf{c}} \,\|\, q)\), first drops to 0.006 at \(t = 1\) (\(q\) passes close to \(\tilde{\mathbf{c}}\) on its way), then grows to about 0.45: the model became more certain than the detector. Switch to MRC-kl: the first gradient is \((-0.2, 0.1, 0.1)\), smaller, and \(q\) settles at \((0.70, 0.20, 0.10)\) with KL near 0. Lower the detector’s confidence to 0.45: MRC still drives \(q(\text{horse})\) toward 1.
Image–text matching (ITM)
Give the model a caption and an image, and ask one yes/no question: do they belong together? Half the time (LXMERT: probability 0.5) the caption or the image is replaced with one from another pair [Certain]. UNITER puts a linear layer and a sigmoid on the [CLS] output [Certain]; ViLBERT multiplies the IMG and CLS outputs element by element first [Certain]:
- Read the symbols
- \(\mathbf{h}_{\text{CLS}}\)
- the output at the [CLS] token, a summary of the whole pair (ViLBERT uses \(\mathbf{h}_{\text{IMG}} \odot \mathbf{h}_{\text{CLS}}\))
- \(\mathbf{u}\)
- a learned vector; \(\sigma\) is the sigmoid
- \(y\)
- 1 if the pair is real, 0 if one side was swapped
This is a binary cross-entropy, like SigLIP’s loss (session 167), with one big difference: here the score comes out of a deep fusion network that sees both inputs together, so it must be recomputed for every pair. Random negatives are easy (a beach caption with a kitchen image), so the task becomes easy quickly. ALBEF later fixed this with hard negatives chosen by a contrastive model (session 173).
Extra tasks: image QA and word–region alignment
LXMERT adds image question answering to pretraining: about one third of its sentences are questions from VQA v2, GQA and Visual Genome QA, and the model predicts the answer from a table of 9,500 candidates [Certain]. UNITER adds word–region alignment (WRA) with optimal transport: it finds a soft matching \(\mathbf{T}\) between the \(T\) words and \(K\) regions that moves all “mass” from words to regions at the lowest total cosine distance, and uses that cost as a loss [Certain]:
Here \(\Pi(\mathbf{a}, \mathbf{b})\) is the set of non-negative \(T \times K\) matrices whose rows sum to the word weights \(\mathbf{a}\) and columns to the region weights \(\mathbf{b}\). UNITER solves it approximately with a few matrix-vector iterations (the IPOT algorithm) [Certain]. In words: each word should have some region close to it, and each region some word.
Conditional masking
ViLBERT and LXMERT mask words and regions at the same time. That creates a problem. If “horse” is masked and the horse region is also masked, there is no information left to recover either one; the model is asked to guess. UNITER masks one modality at a time: an MLM batch masks only words, an MRM batch masks only regions [Certain]. It also samples one task per mini-batch, so each update uses one loss [Certain]. Turning conditional masking off lowered UNITER’s total score from 399.97 to 396.51 [Certain]. Figure 6 shows how often the clash happens.
How to read Figure 6
At \(p = 0.15\), each linked pair clashes with probability \(p^2 = 0.0225\), and at least one of the three pairs clashes with probability \(1 - (1 - 0.0225)^3 = 0.066\): about one caption in 15. Real captions link more words to regions, so the rate is higher. Raise \(p\) to 0.5 and more than half of the samples have a clash. Switch to conditional masking: the rate is exactly 0, at the cost of never training “both sides missing”, which never happens at test time anyway.
Which tasks helped? The UNITER ablation
UNITER pretrained a base model with many task mixes and fine-tuned each one on four benchmarks: VQA, Flickr30K image retrieval (IR) and text retrieval (TR), each the average of R@1, R@5 and R@10, NLVR2 (does a sentence describe a pair of images?) and RefCOCO+ with detected boxes (find the box a phrase refers to). The sum of these five scores is called Meta-Sum [Certain]. Figure 7 shows the table.
How to read Figure 7
Row 1 (no pretraining) to row 5 (MLM on image–text pairs) is the biggest jump: +71.76 Meta-Sum. Text-only pretraining (row 2) barely helps NLVR2: the model must learn to use images. MRC-kl (row 9) beats plain MRC (row 7); adding WRA (row 11) adds 0.96. Removing conditional masking (row 12) costs 3.46. And data: out-of-domain web data alone (row 13, CC and SBU) is worse than in-domain COCO and Visual Genome (row 11), but using both (row 14) is best.
The data and the recipes
| ViLBERT | LXMERT | UNITER | |
|---|---|---|---|
| Layout | two streams, co-attention | two streams, 9 + 5 + 5 layers | one stream, 12 or 24 layers |
| Text init | BERT-base | from scratch | BERT |
| Regions | 10–36, 5-d location | 36, box coordinates | variable, 7-d location |
| Pretraining data | Conceptual Captions, about 3.1M pairs (3.3M minus broken links) | 9.18M image–sentence pairs on 180K images (COCO, Visual Genome, VQA v2, GQA, VG-QA) | COCO + VG: 5.6M pairs; with CC and SBU: about 9.6M pairs on about 4.2M images |
| Tasks | MLM, MRC-kl, ITM | MLM, RoI regression, detected-label classification, ITM, image QA | MLM, MRFR, MRC-kl, ITM, WRA (conditional masking) |
| Compute | 8 Titan X, batch 512, 10 epochs | 4 Titan Xp, 10 days, batch 256, 20 epochs | 882 (base) and 3,645 (large) V100 GPU hours |
| VQA v2 test-dev | 70.55 | 72.42 | 72.70 (base), 73.82 (large) |
All entries come from the three papers [Certain]. Two results stood out at the time. LXMERT raised NLVR2 test accuracy from 53.5 (the previous best) to 76.2 [Certain], a task where earlier models were close to chance. And ViLBERT showed that more pretraining data helps steadily: with 0%, 25%, 50% and 100% of Conceptual Captions, VQA test-dev went 68.93, 69.82, 70.30, 70.55, and zero-shot Flickr30K image retrieval R@1 went 0, 20.40, 26.76, 31.86 [Certain].
The PyTorch version of the core pieces, with toy sizes. It builds the embeddings (172.3) and (172.4), a ViLBERT co-attention layer (172.5), a UNITER-style single stream, and the losses (172.7), (172.8), (172.10) and (172.11) with conditional masking and one task per mini-batch.
- input: image, caption, frozen detector (Faster R-CNN, Visual Genome classes), \(K_{\min}, K_{\max}\), threshold \(\tau\)
- run the detector once; get boxes \(\mathbf{b}_i\), class distributions \(\tilde{\mathbf{c}}_i\), RoI features \(\mathbf{f}_i \in \R^{2048}\) // about 0.8 s per image, cached on disk before training
- keep boxes with top confidence \(\ge \tau\); if fewer than \(K_{\min}\), take the top \(K_{\min}\); if more than \(K_{\max}\), keep the top \(K_{\max}\)
- \(\mathbf{p}_i \leftarrow\) location vector, formula (172.1) or (172.2)
- \(\mathbf{v}_i \leftarrow \operatorname{LN}(W_f\mathbf{f}_i + W_p\mathbf{p}_i + \mathbf{e}_{\text{img}})\) // formula (172.3)
- tokenize the caption into word pieces; \(\mathbf{t}_j \leftarrow \operatorname{LN}(E_{\text{word}}[w_j] + E_{\text{pos}}[j] + \mathbf{e}_{\text{txt}})\) // formula (172.4)
- single stream: return \([\mathbf{t}_{\text{CLS}}, \mathbf{t}_1, \dots, \mathbf{t}_T, \mathbf{t}_{\text{SEP}}, \mathbf{v}_1, \dots, \mathbf{v}_K]\); two streams: return the two lists separately, with IMG first in the region list
- input: batch of (caption, regions, detector distributions), task list {MLM, MRFR, MRC-kl, ITM, WRA}
- task \(\leftarrow\) sample one task for this mini-batch
- if MLM: mask 15% of words only; loss \(\leftarrow\) cross-entropy at masked words, formula (172.7)
- if MRFR or MRC-kl: zero 15% of region features only; loss \(\leftarrow\) formula (172.8) or (172.10) at masked regions
- if ITM: with probability 0.5 replace the caption (or image) by one from another pair; loss \(\leftarrow\) binary cross-entropy on [CLS], formula (172.11)
- if WRA: compute the transport plan between word and region outputs (IPOT); loss \(\leftarrow\) formula (172.12)
- backward, AdamW step // one objective per update
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, C, FEAT, D, K, T = 100, 10, 64, 32, 6, 8 # vocab, detector classes, RoI dim, width, regions, words MASK, CLS, SEP = 1, 2, 3 class Embed(nn.Module): # formulas (172.3) and (172.4) def __init__(s): super().__init__() s.feat, s.loc = nn.Linear(FEAT, D), nn.Linear(7, D) s.word, s.pos, s.mod = nn.Embedding(V, D), nn.Embedding(64, D), nn.Embedding(2, D) s.ln_v, s.ln_w = nn.LayerNorm(D), nn.LayerNorm(D) def forward(s, f, loc, w): v = s.ln_v(s.feat(f) + s.loc(loc) + s.mod.weight[1]) # regions: no order, only location pos = torch.arange(w.shape[1]) t = s.ln_w(s.word(w) + s.pos(pos) + s.mod.weight[0]) # words: position embedding return t, v class CoAttention(nn.Module): # ViLBERT co-TRM, formula (172.5) def __init__(s): super().__init__() s.a_w = nn.MultiheadAttention(D, 4, batch_first=True) s.a_v = nn.MultiheadAttention(D, 4, batch_first=True) def forward(s, t, v): t2 = t + s.a_w(t, v, v, need_weights=False)[0] # words ask, regions answer v2 = v + s.a_v(v, t, t, need_weights=False)[0] # regions ask, words answer return t2, v2 class SingleStream(nn.Module): # UNITER-style: one sequence, one encoder def __init__(s): super().__init__() s.emb = Embed() layer = nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0.0, batch_first=True) s.enc = nn.TransformerEncoder(layer, 2) s.mlm, s.mrfr, s.mrc, s.itm = nn.Linear(D, V), nn.Linear(D, FEAT), nn.Linear(D, C), nn.Linear(D, 1) def forward(s, f, loc, w): t, v = s.emb(f, loc, w) h = s.enc(torch.cat([t, v], dim=1)) # (B, T + K, D) return h[:, :w.shape[1]], h[:, w.shape[1]:] def batch(B=4): # fake detector output + captions f = torch.randn(B, K, FEAT).relu() xy = torch.rand(B, K, 2) * 0.5; wh = torch.rand(B, K, 2) * 0.5 loc = torch.cat([xy, xy + wh, wh, wh[..., :1] * wh[..., 1:]], -1) # 7-d, formula (172.2) det = F.softmax(torch.randn(B, K, C) * 2, -1) # detector class distribution w = torch.randint(4, V, (B, T)); w[:, 0] = CLS; w[:, -1] = SEP return f, loc, det, w def step(model, task, f, loc, det, w, p=0.15): B = w.shape[0] if task == "mlm": # mask words only (conditional masking) m = (torch.rand(B, T) < p); m[:, [0, -1]] = False; m[0, 1] = True ht, _ = model(f, loc, w.masked_fill(m, MASK)) return F.cross_entropy(model.mlm(ht[m]), w[m]) # formula (172.7) if task in ("mrfr", "mrc_kl"): # mask regions only m = (torch.rand(B, K) < p); m[0, 0] = True _, hv = model(f.masked_fill(m[..., None], 0.0), loc, w) if task == "mrfr": return ((model.mrfr(hv[m]) - f[m]) ** 2).sum(-1).mean() # formula (172.8) return F.kl_div(F.log_softmax(model.mrc(hv[m]), -1), det[m], reduction="batchmean") # (172.10) y = (torch.rand(B) < 0.5).float() # ITM: half the captions are swapped w_in = torch.where(y[:, None].bool(), w, w.roll(1, dims=0)) ht, _ = model(f, loc, w_in) return F.binary_cross_entropy_with_logits(model.itm(ht[:, 0]).squeeze(-1), y) # formula (172.11) model = SingleStream() opt = torch.optim.AdamW(model.parameters(), lr=1e-3) tasks = ["mlm", "mrfr", "mrc_kl", "itm"] for it in range(8): # one task per mini-batch, as in UNITER task = tasks[it % 4] loss = step(model, task, *batch()) opt.zero_grad(); loss.backward(); opt.step() print(f"step {it} {task:7s} loss {loss.item():.3f}") co = CoAttention() t, v = model.emb(*batch()[:2], batch()[3]) t2, v2 = co(t, v) print(tuple(t2.shape), tuple(v2.shape)) # (4, 8, 32) (4, 6, 32): each stream keeps its length
The script is deterministic (same output on every run) and runs in about a second on a CPU. The losses are on random data, so they do not go down in 8 steps; the point is the shapes and which positions each loss reads. Two details to notice: the MLM branch never masks regions and the region branches never mask words (conditional masking), and the ITM negatives come from rolling the batch, so each wrong caption belongs to another image in the same batch.
Why they faded
By 2022, almost no new model used this recipe. Four reasons, each with numbers.
1. The detector is slow. ViLT (Kim et al., 2021) timed the visual input step on one GPU (a P40) with 800 × 1,333 images: the ResNet-101 backbone takes about 75 ms, and the region steps (proposals, RoI Align, non-maximum suppression and RoI heads) about 810 ms, so about 900 ms per image before the transformer even starts; the transformer itself takes about 15 ms [Certain]. Per-class non-maximum suppression over 1,600 classes alone takes more than 500 ms [Certain]. During pretraining you can cache the features on disk, but at inference every new image pays the full cost. ViLT replaced the detector with a linear patch embedding (about 0.4 ms) and reached similar accuracy at about 15 ms in total [Certain].
2. The data is small and hand-made. The biggest pretraining sets here have about 9–10M pairs, mostly from COCO and Visual Genome, whose images also appear in the test benchmarks. CLIP trained on 400M web pairs [Certain], about 42 times UNITER’s full set. Region features also tie you to the detector’s 1,600 classes: things outside that list are hard to represent.
3. Retrieval is quadratic. A fusion encoder needs one full forward pass for each (image, text) pair:
On the Flickr30K test set (1,000 images, 5,000 captions), that is 5,000,000 passes against 6,000 for a CLIP-style model: about 833 times more. On a web-scale index it is impossible. ViLBERT already cached the text stream before the first co-attention layer to save time [Certain], but each pair still needs its own fusion layers.
4. No generation. These are bidirectional encoders. They classify (pick an answer from 3,129 or 9,500 candidates), score, and point at boxes, but they do not write a sentence. Modern VLMs answer open questions in free text with a decoder LLM (session 177).
How to read Figure 8
In “time per image”, the orange part is the detector’s region work: about 90% of the total for a region model. Slide the number of images to \(10^6\): a region model needs about 250 GPU hours just to look at the images once, a patch model about 4 hours. In “pretraining pairs”, each grid line is 10 times more data; CLIP is more than one grid line beyond UNITER, and LAION-5B (5.85B pairs [Certain]) another grid line beyond CLIP.
The ideas that survived: MLM and ITM as losses, ITM as a reranker on top of a fast dual encoder (ALBEF, BLIP), cross-attention as a fusion layer (Flamingo, the BLIP-2 Q-Former), and the lesson that in-domain data and careful masking matter. The ideas that did not: frozen detector features as the image input. A middle step worth knowing: Oscar (2020) added the detector’s object tags as extra words, and VinVL (2021) trained a much better detector; VinVL-base reached 75.95 on VQA test-dev [Certain], the peak of the region era, just as CLIP and ViLT appeared.
Where this is used
| Task | How a fusion encoder solves it | Today |
|---|---|---|
| Visual question answering (VQA v2, GQA) | [CLS] or \(\mathbf{h}_{\text{IMG}} \odot \mathbf{h}_{\text{CLS}}\) → classifier over the 3,129 most frequent answers [Certain] | Decoder VLMs generate the answer as text (session 191) |
| Visual commonsense reasoning (VCR) | Score each of 4 answers with the image; softmax over the 4 scores | Multiple choice prompts to a VLM |
| Referring expressions (RefCOCO+) | Score every region output against the phrase; pick the best box | Grounded detectors and VLMs that output box coordinates (session 174) |
| Image–text retrieval | ITM score for every pair, formula (172.13) | Dual encoder for recall, then a fusion ITM head to rerank the top 100 or so (ALBEF, BLIP) |
| NLVR2, visual entailment | Two images or an image plus a hypothesis, binary or 3-way classifier | Benchmarks for VLM reasoning |
| Video retrieval and video QA | Same recipe with clip or frame features instead of regions: VideoBERT, HERO (by the UNITER team) and others [Likely] | Video-language models and Video-RAG (session 175) |
| Key-moment detection in video | A cross-encoder scores (query, clip) pairs to find the moment that matches a sentence [Likely] | Dual-encoder search over clips, then fusion reranking of the top few |
Common mistakes
- Calling CLIP a “two-stream model like ViLBERT”. CLIP’s streams never interact before the final dot product (a dual encoder). ViLBERT’s streams interact in 6 co-attention layers (a fusion encoder). The costs and abilities are very different.
- Thinking two streams save attention compute. One Co-TRM→TRM pair computes \(T^2 + K^2 + 2TK = (T + K)^2\) scores, the same as one single-stream layer (172.6). Two streams cost more parameters, not fewer scores.
- Ignoring the detector in latency estimates. The transformer is about 15 ms; the region step is about 900 ms. Caching features hides this during training, not in a product.
- Masking a word and its region together and then wondering why the masked-region loss is noisy. Use conditional masking: one modality per batch.
- Regressing raw 2048-d features with an unscaled L2 loss next to a cross-entropy loss. The regression term can be orders of magnitude larger (in the toy code above it is around 50 while the others are around 1); weight or normalize it, or use the KL target.
- Easy ITM negatives only. A random caption is almost always obviously wrong, so the ITM loss falls quickly and teaches little. Mine hard negatives from a contrastive model (ALBEF).
- Comparing papers that used different data. LXMERT, UNITER and ViLBERT were trained on different sets; a controlled re-training showed most gaps come from data and hyper-parameters (Bugliarello et al.).
- Test images inside pretraining data. COCO and Visual Genome images appear in many benchmarks. UNITER removed downstream validation and test images (and Flickr30K overlaps) from pretraining; do the same.
Interview questions
What is the difference between a single-stream and a two-stream vision-language model?basic
How does an image become a sequence of tokens in ViLBERT or UNITER?basic
Write the co-attention layer. How does it differ from self-attention and from a decoder’s cross-attention?core
Is a two-stream model cheaper than a single-stream one?core
Explain the three masked region modeling variants. Why not just regress the feature?core
What is image–text matching, how are negatives made, and what is its weakness?core
What is conditional masking and why does it help?core
Why can CLIP search a billion images but UNITER cannot?core
Give four reasons region-based VLP models faded.basic
ViLBERT’s paper found two streams better than one. UNITER, a single stream, beat ViLBERT. How do you reconcile this?advanced
Why did LXMERT not initialize its language encoder from BERT, while ViLBERT did?advanced
Which ideas from this era are still in modern VLMs?advanced
Exercises
- An image is 800 × 600. A box is \((200, 150, 600, 450)\). Write its ViLBERT (5-d) and UNITER (7-d) location vectors.
- A batch has captions of \(T = 30\) tokens and \(K = 50\) regions. How many attention scores does one single-stream layer compute? A Co-TRM layer? A TRM layer?
- With joint masking at \(p = 0.15\) and a caption with 5 independent word–region links, what is the probability that at least one link is fully masked?
- The detector gives \(\tilde{\mathbf{c}} = (0.6, 0.4)\) and the model predicts \(q = (0.6, 0.4)\). What are the MRC and MRC-kl losses? Which one is zero, and why does that matter?
- You must rank 10,000 product images for each of 100 queries. How many forward passes does a fusion encoder need? A dual encoder? A two-stage system that reranks the dual encoder’s top 50 per query with the fusion encoder?
- Using ViLT’s numbers, how many images per second can one GPU embed with region features, and with linear patches (ignore the transformer)?
Answerscheck
Key takeaways
- First-generation VLMs turn an image into 10–36 detector regions: a 2048-d RoI feature plus a 5-d or 7-d location vector (172.1, 172.2), embedded like words (172.3, 172.4).
- Two streams (ViLBERT, LXMERT) meet through co-attention (172.5); one stream (UNITER) concatenates everything. Both compute the same attention scores per layer pair (172.6); the layout matters less than data and losses.
- Pretraining: masked words with the image (172.7), masked regions with detector targets (172.8–172.10), image–text matching (172.11), plus image QA (LXMERT) and word–region alignment (172.12). Conditional masking avoids hiding a word and its region together.
- They faded because the detector costs about 900 ms per image, the data was under 10M pairs, fusion retrieval needs \(N_I N_T\) passes (172.13), and encoders cannot generate text.
- MLM, ITM reranking and cross-attention fusion live on in ALBEF, BLIP, BLIP-2 and Flamingo-style VLMs.
Key formulas
References for this session
- Lu et al. (2019). ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Tan & Bansal (2019). LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Chen et al. (2019). UNITER: UNiversal Image-TExt Representation Learning
- Lu et al. (2019). ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Tan & Bansal (2019). LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Chen et al. (2019). UNITER: UNiversal Image-TExt Representation Learning
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Li et al. (2021). Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- Li et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- Ren, He, Girshick & Sun (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Anderson et al. (2018). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Krishna et al. (2016). Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Sharma, Ding, Goodman & Soricut (2018). Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
- Li, Yatskar, Yin, Hsieh & Chang (2019). VisualBERT: A Simple and Performant Baseline for Vision and Language
- Su et al. (2019). VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Li et al. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
- Zhang et al. (2021). VinVL: Revisiting Visual Representations in Vision-Language Models
- Kim, Son & Kim (2021). ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Bugliarello, Cotterell, Okazaki & Elliott (2021). Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs
- Goyal et al. (2016). Making the V in VQA Matter