LLM & VLM courseBlog
176Track 7: Vision and VLMs · session 14 of 38

compare CLIP, BLIP and BLIP-2 on retrieval, captioning and VQA

In this lab you run CLIP, BLIP and BLIP-2 on the same 500 COCO photos and measure them on retrieval, captioning and visual question answering. You build two-stage retrieval with a contrastive shortlist and a matching reranker, score captions with CIDEr and with a VLM judge asked in both orders, and score answers with the VQA metric and a blind baseline. Then you explain every failure by its architecture, its loss or its data.

Time: 3 to 5 hours (reading 1.5 hours; the full runs take about 1 to 2 hours on a laptop GPU, BLIP-2 being the slowest)Level: Intermediate · hands-on labBefore this: Session 166 (CLIP), Session 173 (ALBEF and BLIP) and the zero-shot lab (170); Session 178 or 181 for the Q-Former helps; Python and PyTorch

What you will learn

  • How to run CLIP, BLIP and BLIP-2 from Hugging Face on one shared image set, and what each checkpoint was trained on
  • How two-stage retrieval works: an ITC shortlist over the whole gallery, then ITM reranking of the top k, and how deep to rerank
  • Why CLIP can only rank text, and what that means when you force it to caption or answer questions
  • How CIDEr-D weights rare n-grams, why its value depends on tokenizer, IDF set and decoding, and why a model can beat a human on it
  • How to use a VLM as a pairwise judge without being fooled by position bias
  • How the VQA accuracy gives partial credit with 10 annotators, and why you always need a blind baseline
  • How to sort failures into architecture, loss and data causes, including test-set contamination

Why this matters

You have met three famous model families in this track. Session 166 learns one vector per image and one per sentence. Session 173 adds a matching head that reads image and text together, and a decoder that writes captions. Session 181 and session 178 showed BLIP-2, which plugs a frozen image encoder into a frozen large language model (LLM) through a small Q-Former. In this lab you run all three on the same 500 photos and measure them on three tasks: retrieval (find the right photo for a sentence and the right sentence for a photo), captioning (write a sentence for a photo) and visual question answering (VQA: answer a question about a photo).

The goal is not a leaderboard. The goal is to learn to read results the way a senior engineer does: every number comes with a reason in the architecture (what can talk to what inside the model), the loss (what the model was rewarded for) and the data (what it has already seen). You will see a dual encoder fail at VQA below a blind baseline, two models score above a held-out human caption on CIDEr, a small judge that mostly picks a position instead of a caption, and a VQA score that is high partly because the model was trained on the very questions we test. Every number on this page comes from our own run of the code below, unless it is tagged as coming from a paper.

The short version of what we measured: for retrieval, the matching heads of BLIP and BLIP-2 lift text-to-image recall@1 from 0.591 (CLIP) to about 0.86. For captioning, CLIP can only copy a caption of another photo (CIDEr 61.8), BLIP writes short COCO-style captions (97.0), and BLIP-2, never fine-tuned on COCO, writes longer ones and gets the best score, 140.8. For VQA, CLIP scores 4.2%, a blind “most common answer” rule 32.2%, BLIP-2 zero-shot 55.2% and BLIP, which was fine-tuned on these questions, 83.0%.

Setup: three models, one image set

You need Python 3.10 or newer, about 25 GB of disk (BLIP-2 alone is 15 GB in float32 on the Hub) and either a GPU or an Apple-silicon Mac; we used torch 2.15 and transformers 5.23 on an Apple M5 Pro with the mps backend. Other jobs shared the GPU during our runs, so treat the times below as rough.

python -m venv venv && source venv/bin/activate
pip install torch transformers datasets pillow pycocoevalcap pandas pyarrow

We use public checkpoints from Hugging Face. The parameter counts were read from the loaded models [Certain]:

SystemCheckpointParametersImage inputFine-tuned on COCO?
CLIP ViT-B/32openai/clip-vit-base-patch32151.3 M224 px, 49 patches + 1no (400M web pairs)
BLIP retrievalSalesforce/blip-itm-base-coco223.7 M384 px, 576 patches + 1yes, Karpathy train
BLIP captioningSalesforce/blip-image-captioning-base224.0 M384 pxyes, Karpathy train
BLIP VQASalesforce/blip-vqa-base361.2 M384 px (the paper used 480)yes, on VQAv2 train and val
BLIP-2 retrieval (stage 1)Salesforce/blip2-itm-vit-g1.172 B (ViT-g 986.0 M + Q-Former side 185.7 M)224 px, 256 patches + 1not stated on the card [Likely: stage-1 pretrained only]
BLIP-2 generationSalesforce/blip2-opt-2.7b3.745 B (ViT-g 986.0 M, Q-Former 105.2 M, projection 2.0 M, OPT 2.652 B)224 px → 32 query tokensno (“pre-trained only”)

Three facts in this table decide many results later. First, BLIP’s checkpoints are task-specific: one model per task, each fine-tuned on that task’s data. Second, the BLIP paper says the VQA model was trained on “both training and validation splits” of VQAv2, plus Visual Genome questions [Certain]. Our VQA questions come from the VQAv2 validation split, so BLIP has seen them. Third, in BLIP-2 only the Q-Former and the projection are trained: \(105.2\text{M} + 2.0\text{M} = 107.1\text{M}\) parameters, which matches the 107M “trainable params” in the BLIP-2 paper [Certain]. The Q-Former of the retrieval checkpoint is larger (185.7 M) because it keeps the text embeddings and the ITC and ITM heads; the paper gives 188M for the full Q-Former [Certain].

A. CLIP ViT-B/32 image text ViT-B/32 text tower · one vector each; they meet only in a dot product can rank, cannot write B. BLIP base (ViT-B/16) ViT-B/16 577 image tokens (cross-attention) text encoder → ITC + cross-attn → ITM decoder → caption one image tower, three text heads (shared weights); separate checkpoints fine-tuned for retrieval, captions and VQA (the VQA model: question encoder + answer decoder) C. BLIP-2 (ViT-g/14 + Q-Former + OPT-2.7B) image EVA ViT-g/14frozen, 257 × 1408 Q-Former32 learned queries linear768 → 2560 OPT-2.7Bfrozen LLM text stage 1 heads: ITC (max over 32 queries), ITM dashed border = frozen. The retrieval checkpoint stops at the Q-Former; the generation checkpoint adds the projection and OPT.
Figure 1. The three systems of this lab. CLIP keeps image and text apart until one dot product. BLIP adds a matching head that reads both together (cross-attention) and a decoder that writes. BLIP-2 freezes a large image encoder and a large language model and trains only the small Q-Former and a linear layer between them.

The image set. We take the first 500 images of the COCO Karpathy test split (the standard 5,000-image test split for COCO captioning and retrieval, from Karpathy and Fei-Fei). Each has 5 human captions, so there are 2,500 captions. All these images are in COCO’s val2014 set, which also has VQAv2 questions; we take the first 3 questions of each image, 1,500 in total (563 yes/no, 181 number, 756 other), each with 10 human answers. One image set, three tasks:

# prep.py: 500 COCO Karpathy-test images, their 5 captions, 3 VQAv2 val questions each
import json, os, collections, urllib.request, pandas as pd
# inputs: yerevann/coco-karpathy test + train parquet files, and the official VQAv2 val questions and annotations
kt = pd.read_parquet("data/karpathy_test.parquet").iloc[:500]
qs = json.load(open("data/v2_OpenEnded_mscoco_val2014_questions.json"))["questions"]
an = {a["question_id"]: a for a in json.load(open("data/v2_mscoco_val2014_annotations.json"))["annotations"]}
byimg = collections.defaultdict(list)
for q in qs: byimg[q["image_id"]].append(q)
items, vqa = [], []
for _, r in kt.iterrows():
    cid = int(r["cocoid"])
    items.append({"cocoid": cid, "file": r["filename"], "captions": [s.strip() for s in r["sentences"]][:5]})
    for q in sorted(byimg[cid], key=lambda q: q["question_id"])[:3]:
        a = an[q["question_id"]]
        vqa.append({"qid": q["question_id"], "cocoid": cid, "question": q["question"],
                    "answers": [x["answer"] for x in a["answers"]], "atype": a["answer_type"]})
    p = f"data/img/{r['filename']}"
    if not os.path.exists(p): urllib.request.urlretrieve(r["url"], p)     # images.cocodataset.org
json.dump({"items": items, "vqa": vqa}, open("data/set.json", "w"))
tr = pd.read_parquet("data/karpathy_train.parquet")                 # 82,783 train images, used only by CLIP below
json.dump([ss[0].strip() for ss in tr["sentences"]], open("data/pool.json", "w"))

Step 1: retrieval with ITC, then ITM

All three families can score an image–text pair in a cheap way. Image and text are encoded separately into unit vectors (vectors of length 1), and the score is their dot product, the cosine. This is the image–text contrastive (ITC) score. Because the two encoders never see each other, you can embed a whole gallery once, offline, and search it with one matrix product. CLIP and BLIP use one vector per image. BLIP-2’s Q-Former gives 32 vectors per image (one per learned query) and keeps the best one:

\[ s_{\text{ITC}}(I, T) = \mathbf{u}(I)^\top \mathbf{t}(T) \quad \text{(CLIP, BLIP)}, \qquad s_{\text{ITC}}(I, T) = \max_{q = 1, \dots, 32} \; \mathbf{z}_q(I)^\top \mathbf{t}(T) \quad \text{(BLIP-2)} \tag{176.1} \]
Read the symbols
\(\mathbf{u}(I)\)
unit image vector: CLIP’s projected image embedding (512 numbers), or BLIP’s projected [CLS] token of the image encoder (256 numbers)
\(\mathbf{t}(T)\)
unit text vector from the text encoder’s [CLS] (or end-of-text) token, projected to the same size
\(\mathbf{z}_q(I)\)
the output of query \(q\) of BLIP-2’s Q-Former, projected to 256 numbers and normalized; the BLIP-2 paper takes “the highest one as the image-text similarity” [Certain]

BLIP and BLIP-2 also have an expensive score. The image–text matching (ITM) head reads the text with cross-attention to the image tokens (each text token can look at every image patch) and outputs a binary logit: “is this a match?”. It sees the words and the image together, so it can check details that one pooled vector loses. But it needs a full forward pass for every pair: a gallery of a million images would need a million passes per query. The standard answer, used in both papers, is a two-stage search: ITC picks the top \(k\) candidates, ITM reranks only those [Certain]. In the authors’ evaluation code the final score of a candidate is the ITM logit plus the ITC cosine [Certain]:

\[ \text{score}(I, T) = \begin{cases} \ell_{\text{ITM}}(I, T) + s_{\text{ITC}}(I, T) & \text{if } I \text{ is in the ITC top-}k \text{ for } T \\ \text{ITC order, after all reranked candidates} & \text{otherwise} \end{cases} \tag{176.2} \]
Read the symbols
\(\ell_{\text{ITM}}(I, T)\)
the “match” logit of the ITM head; BLIP reads it from the [CLS] position, BLIP-2 averages it over the 32 query outputs
\(k\)
the rerank depth: how many ITC candidates get the expensive pass; the papers use \(k = 256\) (BLIP on COCO) and \(k = 128\) (BLIP-2); we use 16
Algorithm 1: two-stage retrieval (ITC shortlist, ITM rerank)
  1. offline: for every gallery image \(I\): keep its unit ITC vector(s) and its image tokens // ITM needs the tokens, not only the vector
  2. for each query text \(T\):
  3.   \(\mathbf{t} \leftarrow\) unit text vector; \(s_I \leftarrow s_{\text{ITC}}(I, T)\) for all \(I\) // one matrix product, formula (176.1)
  4.   \(C \leftarrow\) the \(k\) images with the largest \(s_I\)
  5.   for \(I \in C\): \(\text{score}_I \leftarrow \ell_{\text{ITM}}(I, T) + s_I\) // k cross-attention passes, batched
  6.   return \(C\) sorted by score, then the rest in ITC order // formula (176.2)
  7. image → text is the same with the roles swapped (top \(k\) captions per image)

The code below is a compact version of our ret.py (which batches the same calls). Note what each model needs to keep: for BLIP the 577 image tokens of 768 numbers, for BLIP-2 the 257 frozen ViT-g tokens of 1,408 numbers per image. The ITC vector alone is not enough for ITM.

# ret.py (core): ITC for all pairs, ITM rerank of the top-K. 500 images, 2,500 captions; caption c is of image c // 5
import torch, torch.nn.functional as F
from transformers import BlipForImageTextRetrieval, BlipProcessor, Blip2ForImageTextRetrieval, Blip2Processor
torch.set_grad_enabled(False); dev = "mps"                  # or "cuda"
ones = lambda e: torch.ones(e.shape[:2], dtype=torch.long, device=dev)

# ---- BLIP: image tokens once; ITC = projected [CLS] vectors; ITM = text encoder with cross-attention ----
m = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco").to(dev).eval()
p = BlipProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
emb = torch.cat([m.vision_model(pixel_values=p(images=images[i:i + 25], return_tensors="pt")["pixel_values"].to(dev))
                 .last_hidden_state for i in range(0, len(images), 25)])           # (500, 577, 768)
U = F.normalize(m.vision_proj(emb[:, 0]), dim=-1)                                  # (500, 256)
tok = p.tokenizer(caps, padding="max_length", truncation=True, max_length=35, return_tensors="pt").to(dev)
h = m.text_encoder(input_ids=tok["input_ids"], attention_mask=tok["attention_mask"]).last_hidden_state
T = F.normalize(m.text_proj(h[:, 0]), dim=-1)                                      # (2500, 256); batch this on small GPUs
itc = U @ T.T                                                                      # (500, 2500), formula (176.1)
def blip_itm(ii, tt):                                                              # index tensors of images and captions
    e = emb[ii]
    h = m.text_encoder(input_ids=tok["input_ids"][tt], attention_mask=tok["attention_mask"][tt],
                       encoder_hidden_states=e, encoder_attention_mask=ones(e)).last_hidden_state
    return m.itm_head(h[:, 0])[:, 1]                                                # the "match" logit

# ---- BLIP-2 stage 1: frozen ViT-g tokens, 32 Q-Former queries; heads kept in float32 ----
m2 = Blip2ForImageTextRetrieval.from_pretrained("Salesforce/blip2-itm-vit-g", dtype=torch.float16).to(dev).eval()
for mod in (m2.vision_projection, m2.text_projection, m2.itm_head): mod.float()     # fp16 scores create ties
p2 = Blip2Processor.from_pretrained("Salesforce/blip2-itm-vit-g")
emb2 = torch.cat([m2.vision_model(pixel_values=p2(images=images[i:i + 10], return_tensors="pt")["pixel_values"]
                  .to(dev, torch.float16)).last_hidden_state for i in range(0, len(images), 10)])   # (500, 257, 1408)
q = m2.query_tokens.expand(len(images), -1, -1)
z = m2.qformer(query_embeds=q, encoder_hidden_states=emb2, encoder_attention_mask=ones(emb2)).last_hidden_state
Z = F.normalize(m2.vision_projection(z.float()), dim=-1)                          # (500, 32, 256)
tok2 = p2.tokenizer(caps, padding="max_length", truncation=True, max_length=35, return_tensors="pt").to(dev)
h = m2.qformer(query_embeds=m2.embeddings(input_ids=tok2["input_ids"]), query_length=0,
               attention_mask=tok2["attention_mask"]).last_hidden_state
T2 = F.normalize(m2.text_projection(h[:, 0].float()), dim=-1)                     # (2500, 256)
itc2 = torch.einsum("iqd,td->itq", Z, T2).max(-1).values                          # max over the 32 queries
def blip2_itm(ii, tt):
    e = emb2[ii]; qq = m2.query_tokens.expand(len(ii), -1, -1)
    am = torch.cat([ones(qq), tok2["attention_mask"][tt]], 1)                      # queries and text attend to each other
    h = m2.qformer(query_embeds=m2.embeddings(input_ids=tok2["input_ids"][tt], query_embeds=qq), query_length=32,
                   attention_mask=am, encoder_hidden_states=e, encoder_attention_mask=ones(e)).last_hidden_state
    return m2.itm_head(h[:, :32].float())[:, :, 1].mean(1)                         # mean "match" logit over queries

# ---- rerank (176.2), text -> image; image -> text is the same with itc.T ----
def rerank_t2i(itc, itm, K=16):
    top = itc.T.topk(K, dim=1).indices                                            # (2500, K) candidate images
    tt = torch.arange(itc.shape[1], device=dev)[:, None].expand(-1, K)
    s = torch.cat([itm(top.flatten()[i:i + 128], tt.flatten()[i:i + 128]) for i in range(0, top.numel(), 128)])
    s = s.view(-1, K) + itc.T.gather(1, top)                                       # ITM logit + ITC cosine
    return top.gather(1, s.argsort(1, descending=True))                            # reranked top-K per caption

One detail: the original BLIP code replaces the first text token [CLS] by a special [ENC] token for ITM; the Hugging Face forward pass keeps [CLS]. We tried both on the first caption of every image (500 queries, top-16 rerank): R@1 0.850 with [CLS] and 0.848 with [ENC], against 0.798 for ITC alone. No difference here (we took id 30523 for [ENC] from the way the original code extends the BERT vocabulary [Likely]).

CLIP needs no new code: it is the dual encoder from session 170, with get_image_features and get_text_features. Our results, recall@k as in (176.3) below, on 500 images and 2,500 captions:

SystemText → image R@1R@5R@10Image → text R@1R@5R@10
CLIP B/32 (ITC)0.5910.8900.9600.7880.9620.984
BLIP ITC0.8220.9760.9920.9260.9940.998
BLIP ITC + ITM (\(k = 16\))0.8610.9830.9940.9440.9960.998
BLIP-2 ITC0.8160.9740.9900.8760.9901.000
BLIP-2 ITC + ITM (\(k = 16\))0.8570.9780.9940.9440.9961.000

Four things to read from this table. (1) CLIP is far behind: 0.591 against about 0.82 with ITC alone. CLIP B/32 has 49 patches of 32 pixels and was never trained on COCO captions; BLIP was fine-tuned on the Karpathy train split, the same kind of photos and the same caption style. (2) ITM reranking adds about 4 points of text → image R@1 for both BLIP and BLIP-2. (3) BLIP-2, with a frozen 1-billion-parameter image encoder, does not beat the much smaller BLIP here. The BLIP-2 paper’s retrieval numbers come from a model fine-tuned on COCO [Certain]; the public stage-1 checkpoint we use is not that model [Likely]. The size of the vision tower is not the whole story; matching the target data is worth as much. (4) The ITC-only image → text R@1 of BLIP-2 (0.876) is clearly below BLIP’s 0.926, but after ITM both reach 0.944.

\[ \mathrm{R@}k = \frac{1}{|Q|} \sum_{q \in Q} \mathbb{1}\big[\, r_q \le k \,\big] \tag{176.3} \]
Read the symbols
\(Q\)
the queries: 2,500 captions (text → image) or 500 images (image → text)
\(r_q\)
the rank of the first correct item (1 is best); for image → text any of the 5 captions of the image is correct, which makes that direction easier
CLIP B/32BLIPBLIP-2solid: ITC only; dashed: ITC + ITM rerank
Figure 2. Measured recall@k (176.3) on our 500 COCO test images and 2,500 captions, for \(k = 1\) to 16 (the rerank depth). Solid lines: ITC only; dashed: after ITM reranking of the top 16.
How to read Figure 2

At \(k = 1\), text → image, the readout lists all five systems: CLIP 0.591, BLIP ITC 0.822, BLIP with ITM 0.861. Move \(k\) to 5: CLIP reaches 0.890 and all four BLIP and BLIP-2 variants are above 0.97. The gap between CLIP and the others shrinks as \(k\) grows: CLIP usually has the right photo near the top, but not first. That is exactly the case a reranker is built for. Switch to image → text: every curve starts higher, because each image has 5 correct captions.

How deep should you rerank? Each extra candidate costs one more ITM pass per query. Figure 3 shows our measured R@1 for every depth from 1 (no rerank) to 16.

BLIPBLIP-2
Figure 3. Measured R@1 after reranking only the ITC top \(k\) with ITM (176.2), for \(k = 1\) (ITC alone) to 16. The readout also gives the number of ITM passes this depth costs for all 2,500 caption queries.
How to read Figure 3

Start at \(k = 1\): this is ITC alone (BLIP 0.822, BLIP-2 0.816). Move to \(k = 2\): BLIP jumps to 0.852, a big part of the total gain, at only 2 ITM passes per query. By \(k = 4\) BLIP is at 0.858 and the curve is almost flat; \(k = 8\) gives 0.862 and \(k = 16\) gives 0.861. Most of the gain comes from fixing near-misses at ranks 2 to 4. In a real system you would measure this curve on your own data and pick the knee, because the ITM cost grows linearly with \(k\).

Cost, in our run (rough, on a shared GPU): computing all BLIP ITC vectors took 30.7 s; the 48,000 ITM passes (16 per query in both directions) took 954 s, about 50 pairs per second. For BLIP-2, the ITC stage took 62.4 s (we expect most of it in the 1-billion-parameter ViT-g, but did not time the parts), and the ITM passes 366.7 s. This is the asymmetry that shapes every production system: ITC cost grows with the gallery once, offline; ITM cost grows with queries × depth, online.

A word-order probe. Is a single pooled vector a “bag of words”? Yuksekgonul et al. showed that contrastive VLMs often ignore word order [Certain]. We test it directly: for each of the 2,500 captions we make a negative with the same words in a random order (“man on moped helmet a with a a small red A dirt on road”), and count how often the model scores the true caption higher for its own image. Chance is 0.5.

ScoreCLIP ITCBLIP ITCBLIP ITMBLIP-2 ITCBLIP-2 ITM
true caption scores higher than shuffled0.8500.6640.7630.6530.788

Two results stand out. The ITC heads of BLIP and BLIP-2 prefer the true caption only 66.4% and 65.3% of the time: their pooled vectors are close to a bag of words. The ITM heads of the same models, which read the words with cross-attention to the image, reach 76.3% and 78.8%. Word order needs interaction. The surprise is CLIP at 85.0%, better than both ITM heads. We can only guess why [Speculation]: BLIP and BLIP-2 train ITC next to ITM and mine hard negatives for ITM with ITC, so the fine-grained work may move into the ITM head, while CLIP has only its contrastive loss and a very large batch of web text. A shuffled sentence is also not a realistic negative; benchmarks such as ARO swap only the roles of two words, which is much harder.

Step 2: captioning, and how to score a sentence

Now the models must write. Here the architecture decides what is even possible:

  • CLIP has no decoder. It cannot write a word. The best it can do is captioning by retrieval: return the closest caption from a pool. We use one caption of each of the 82,783 COCO Karpathy train images, so the true captions of our test images are never in the pool.
  • BLIP has a text decoder that cross-attends to the image tokens, fine-tuned on COCO captions with the prompt “a picture of” [Certain]. We use the paper’s decoding: beam search with 3 beams and a maximum length of 20 tokens [Certain], then remove the prompt.
  • BLIP-2 turns the image into 32 Q-Former outputs, projects them to OPT’s 2,560-number embeddings, and puts them in front of the text as soft visual prompts (input vectors that are not word embeddings). The frozen OPT-2.7B then continues the text. We give it no prompt, and use 5 beams and at most 30 new tokens.
# gen.py (captioning part): one caption per image from each system
from transformers import BlipForConditionalGeneration, BlipProcessor, Blip2ForConditionalGeneration, Blip2Processor
# CLIP: nearest caption in a pool of 82,783 train captions (U: unit image vectors, te(): unit text vectors)
P = torch.cat([te(pool[i:i + 500]) for i in range(0, len(pool), 500)])
clip_caps = [pool[j] for j in (U @ P.T).argmax(1).tolist()]

# BLIP: prompt "a picture of", 3 beams, max length 20 (the paper's settings)
m = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base").to(dev).eval()
p = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
x = p(images=batch, text=["a picture of"] * len(batch), return_tensors="pt").to(dev)
out = p.batch_decode(m.generate(**x, num_beams=3, max_length=20, min_length=5), skip_special_tokens=True)
blip_caps = [s[len("a picture of"):].strip() for s in out]

# BLIP-2 + OPT-2.7B, pre-trained only: no prompt, 5 beams; decoder-only LLM, so pad on the left
m2 = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", dtype=torch.float16).to(dev).eval()
p2 = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b"); p2.tokenizer.padding_side = "left"
x = p2(images=batch, text=[""] * len(batch), padding=True, return_tensors="pt").to(dev, torch.float16)
o = m2.generate(**x, num_beams=5, max_new_tokens=30, min_new_tokens=3)
blip2_caps = [s.strip() for s in p2.batch_decode(o[:, x["input_ids"].shape[1]:], skip_special_tokens=True)]

Scoring with CIDEr. There is no single correct caption, so the standard metric compares a candidate with the 5 human references of its image. CIDEr (Consensus-based Image Description Evaluation, Vedantam et al.) counts shared word n-grams (runs of 1 to 4 words), but weights each n-gram by how rare it is across all images. “a” and “on a” appear in almost every reference and carry little weight; “dirt road” appears in few and carries a lot. This weighting is TF-IDF (term frequency times inverse document frequency):

\[ g_\omega(s) = h_\omega(s) \cdot \log \frac{N}{\max\big(1,\; \mathrm{df}(\omega)\big)} \tag{176.4} \]

COCO evaluation uses CIDEr-D, which adds two guards against gaming the metric: a candidate n-gram count is clipped at the reference count, and a Gaussian penalty punishes length differences:

\[ \text{CIDEr-D}(c, R) = \frac{10}{4\,|R|} \sum_{n=1}^{4} \sum_{r \in R} \frac{\sum_{\omega} \min\big(g_\omega(c), g_\omega(r)\big)\, g_\omega(r)}{\lVert \mathbf{g}^n(c) \rVert\, \lVert \mathbf{g}^n(r) \rVert}\; e^{-\frac{(\ell_c - \ell_r)^2}{2\sigma^2}} \tag{176.5} \]
Read the symbols
\(\omega\)
an n-gram; \(h_\omega(s)\) is how often it occurs in sentence \(s\)
\(N, \mathrm{df}(\omega)\)
the number of images in the evaluation set, and in how many of them \(\omega\) occurs in at least one reference: the weights depend on the test set itself
\(\mathbf{g}^n(s)\)
the vector of TF-IDF weights of all n-grams of length \(n\) in \(s\)
\(R\)
the references of the image (5 here); \(\ell_c, \ell_r\) are the lengths of candidate and reference, \(\sigma = 6\)
\(10\)
a scale factor from the reference code; papers then multiply by 100 again, so “CIDEr 97.0” means 0.970 from the code

We use the pycocoevalcap CIDEr scorer. Its standard tokenizer needs Java, so we replace it by a simple one (lower case, keep letters, digits and apostrophes). We also wrote our own CIDEr-D in 25 lines to check our understanding: it agrees with pycocoevalcap to 15 digits (0.9697293142528624 against …625 for BLIP).

# metrics.py (CIDEr part)
import re
from pycocoevalcap.cider.cider import Cider
def tok(s): return " ".join(re.findall(r"[a-z0-9]+(?:'[a-z]+)?", s.lower()))   # stand-in for the PTB tokenizer
def cider(cands, refs):                                    # cands: list of str; refs: list of lists of str
    gts = {i: [tok(r) for r in rs] for i, rs in enumerate(refs)}
    res = {i: [tok(c)] for i, c in enumerate(cands)}
    mean, per_image = Cider().compute_score(gts, res)        # CIDEr-D, sigma = 6; IDF from these refs
    return 100 * mean, per_image
Worked example: why rare n-grams count, on our 500 images

Image 1 (COCO id 391895) shows a man on a small motorbike on a dirt road. Among our 500 images, the word “a” occurs in the references of 499 images, so its IDF is \(\log(500/499) = 0.002\): almost nothing. “on a” occurs in 222 images (IDF 0.812), “motorcycle” in 9 (4.017), and “dirt road” in only 1, this one (6.215). BLIP-2’s “a man riding a motorcycle on a dirt road” shares “motorcycle”, “dirt road” and longer runs such as “riding a motorcycle” and “on a dirt road” with the references: per-image CIDEr-D 159.4. BLIP’s “a person riding a motorcycle” has “motorcycle” but no “dirt road”, and it is 5 words long while the references have 9 to 29 (tokenized): 46.4. CLIP’s copied “A motorcycle is parked in front of a washed out road.” gets 35.4: it has the rare word, but the scene is wrong (parked, not ridden). One rare bigram is worth thousands of “a”.

Our results on the 500 images. The second CIDEr column takes the IDF from all 5,000 Karpathy test images instead of our 500; the third column is the mean caption length in words:

SystemCIDEr (IDF on 500)CIDEr (IDF on 5,000)WordsExample (image 1)
CLIP (nearest of 82,783 train captions)61.857.810.9A motorcycle is parked in front of a washed out road.
BLIP base, paper decoding97.092.16.1a person riding a motorcycle
BLIP base, no prompt109.0–7.9a man riding a motorcycle down a dirt road
BLIP base, prompt, min. 10 tokens107.0–7.2a man riding a motorcycle down a dirt road
BLIP-2 OPT-2.7B, pre-trained only140.8134.89.1a man riding a motorcycle on a dirt road
held-out human (reference 1 against the other 4)91.6–10.4 (all refs)A man with a red helmet on a small moped on a dirt road.

Read this table slowly; it has three surprises. First, BLIP-2 wins clearly (140.8) although it was never fine-tuned on COCO. Its captions are longer (9.1 words, against 10.4 for the references) and more specific. Its pretraining data includes COCO’s human captions and synthetic captions written by BLIP-large [Certain], so its default style is already close to COCO’s [Likely]. Second, decoding settings move the score a lot. With the paper’s settings, BLIP writes short, safe captions (6.1 words) and gets 97.0; without the prompt it gets 109.0, and with a minimum length of 10 tokens 107.0. Twelve points from settings alone. Third, two models beat the humans: a held-out human caption, scored against the other 4 references, gets 91.6 (with 4 references instead of 5, so the comparison is rough). CIDEr measures agreement with the consensus wording, not truth. CLIP’s copied captions get 61.8: they describe a similar scene, but rarely this one. Do not compare these values with the paper’s (129.7 to 133.3 for BLIP base on the full test split [Certain]): we score 500 images, with another tokenizer and IDF, and the IDF alone moved BLIP from 97.0 to 92.1.

A second opinion: a VLM as judge. CIDEr only rewards overlap with five sentences. A different check is to ask a stronger model which caption is better. We use Qwen2-VL-2B-Instruct, a small modern VLM that is not related to the three models, show it the image and two captions, and read the probability of the next token “A” against “B”. Judges have a known position bias (a preference for the first or second option, Zheng et al.) [Certain], so we ask twice with the order swapped and count a win only when both orders agree:

\[ \text{win}(x \succ y) = \mathbb{1}\big[\, p_{xy}(A) \gt \tfrac12 \;\text{ and }\; p_{yx}(A) \lt \tfrac12 \,\big] \tag{176.6} \]
Read the symbols
\(p_{xy}(A)\)
the judge’s probability of answering “A” when caption \(x\) is shown as A and \(y\) as B (renormalized over the two letters)
inconsistent
both orders pick the same position: the judge followed the slot, not the caption; we count it as no decision
Algorithm 2: pairwise judging with order swap
  1. for each image \(I\) and caption pair \((x, y)\):
  2.   \(p_1 \leftarrow P(\text{“A”} \mid I, \text{A} = x, \text{B} = y)\) // one forward pass, read two logits
  3.   \(p_2 \leftarrow P(\text{“A”} \mid I, \text{A} = y, \text{B} = x)\) // same question, order swapped
  4.   if \(p_1 \gt 0.5\) and \(p_2 \lt 0.5\): \(x\) wins; elif \(p_1 \lt 0.5\) and \(p_2 \gt 0.5\): \(y\) wins; else: inconsistent
  5. report wins of \(x\), wins of \(y\) and the inconsistent count // a high count means the judge is unreliable here
# judge.py: Qwen2-VL-2B-Instruct compares two captions; both orders
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
mid = "Qwen/Qwen2-VL-2B-Instruct"
j = Qwen2VLForConditionalGeneration.from_pretrained(mid, dtype=torch.float16).to(dev).eval()
jp = AutoProcessor.from_pretrained(mid, min_pixels=448 * 448, max_pixels=448 * 448)
A_id, B_id = (jp.tokenizer.encode(t, add_special_tokens=False)[0] for t in ("A", "B"))
def prefer_a(im, a, b):
    msg = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text":
           f"Which caption describes this image more accurately?\nCaption A: {a}\nCaption B: {b}\nAnswer with one letter, A or B."}]}]
    x = jp(images=[im], text=[jp.apply_chat_template(msg, add_generation_prompt=True)], return_tensors="pt").to(dev)
    lg = j(**x).logits[0, -1].float()
    return float(torch.softmax(lg[[A_id, B_id]], 0)[0])             # P("A") among {"A", "B"}
verdicts = [(prefer_a(im, cx, cy), prefer_a(im, cy, cx)) for im, cx, cy in zip(images, caps_x, caps_y)]
Pair (x vs y)x winsy winsinconsistent
BLIP vs BLIP-226171303 (1 times slot A, 300 times slot B)
held-out human caption vs BLIP13781282 (1 times slot A, 278 times slot B)
held-out human caption vs BLIP-256161283 (0 times slot A, 281 times slot B)

The judge has a strong position bias: in 300 of the 500 BLIP-versus-BLIP-2 images it chose slot B in both orders, so the verdict depended on the slot, not the caption; the other two pairs look the same (278 and 281). Only the decisions that survive the swap count. Among those, BLIP-2 beats BLIP 171 to 26, the held-out human caption beats BLIP 137 to 81, and BLIP-2 beats the human caption 161 to 56. So judge and CIDEr agree on the order BLIP-2 above BLIP, and both rank BLIP-2 above a single human caption. A 2-billion-parameter judge is weak, and Figure 4 shows a consistent verdict that is plainly wrong; before trusting it, compare it with human ratings on a sample, or use a much larger judge [Likely]. Without the swap we would have reported 300 extra “wins” for whichever caption we happened to put second.

Figure 4. Measured captions for 12 of our 500 test images: the 5 human references, the caption of each system with its per-image CIDEr-D (176.5, ×100, IDF on 500 images), and the judge’s two probabilities for BLIP against BLIP-2 (176.6). The images are COCO photos; search for the COCO id to see one.
How to read Figure 4

Image 1 (the motorbike): BLIP-2 has by far the highest CIDEr (159.4 against 46.4 for BLIP), but the judge says “B” in both orders (P(A) = 0.116 and 0.453), so there is no decision. Image 2: the references say the girl blows out candles, and so does BLIP-2 (96.3), but the judge prefers BLIP’s “a girl eating food at a table” (21.5) in both orders: a consistent verdict can still be wrong. Image 4: BLIP and BLIP-2 wrote the same sentence, and the judge still says “B” both times (0.148): pure position bias. Image 9: CLIP’s copied caption (“A group of people ride bikes in a street’s large bike lane.”) has the best CIDEr, 115.6, because a train caption happened to fit well. A per-image CIDEr near 0 does not mean the caption is wrong, only that it shares no weighted n-grams with the references.

Step 3: visual question answering

VQA needs the question and the image to interact: “What color is the train?” asks for one property of one object. Each system gets the question in a different way:

  • CLIP can only score text against an image. We give it 1,000 candidate answers (the most frequent answers in VQAv2 val) and score the text “question: {q} answer: {a}” for each, keeping the best. The question and the image never interact inside the model; they only meet in the final dot product.
  • BLIP (the VQA checkpoint) encodes the question with cross-attention to the image, and a separate decoder writes the answer. The paper ranks 3,128 candidate answers with the decoder [Certain]; the Hugging Face model generates freely, which we use (3 beams, at most 10 tokens).
  • BLIP-2 gets the 32 visual tokens and then the text “Question: {q} Answer:”, the paper’s prompt for OPT, with 5 beams and length penalty −1 to favour short answers [Certain]. We keep the text up to the first line break.
  • A blind baseline never looks at the image: for each question type (VQAv2 labels each question with a type such as “what color is the” or “how many”), answer the most common answer of that type on the other val images.
# gen.py (VQA part)
# CLIP: rank 1,000 candidate answers; U[i] is the unit image vector of the question's image
cands = [a for a, _ in collections.Counter(a["multiple_choice_answer"] for a in ann).most_common(1000)]
clip_ans = [cands[int((U[img_idx[q["cocoid"]]] @ te([f"question: {q['question']} answer: {a}" for a in cands]).T).argmax())]
            for q in vqa]
# BLIP VQA: question encoder + answer decoder
mq = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base").to(dev).eval()
pq = BlipProcessor.from_pretrained("Salesforce/blip-vqa-base")
x = pq(images=ims, text=[q["question"] for q in batch], padding=True, return_tensors="pt").to(dev)
blip_ans = pq.batch_decode(mq.generate(**x, num_beams=3, max_new_tokens=10), skip_special_tokens=True)
# BLIP-2: the paper's zero-shot prompt and decoding for OPT
x = p2(images=ims, text=[f"Question: {q['question']} Answer:" for q in batch], padding=True, return_tensors="pt").to(dev, torch.float16)
o = m2.generate(**x, num_beams=5, max_new_tokens=10, length_penalty=-1.0)
blip2_ans = [s.split("\n")[0].strip() for s in p2.batch_decode(o[:, x["input_ids"].shape[1]:], skip_special_tokens=True)]
# blind baseline: most common answer per question type, counted on the other val images
cnt = collections.defaultdict(collections.Counter)
for a in ann:
    if a["image_id"] not in our_images: cnt[a["question_type"]][a["multiple_choice_answer"]] += 1
prior_ans = [cnt[qtype[q["qid"]]].most_common(1)[0][0] for q in vqa]

The VQA accuracy. Each question has 10 human answers, and people disagree (“8”, “9”, “10” sheep). The VQA metric gives full credit if at least 3 people gave your answer, and partial credit below that. The official script averages this over the 10 ways to leave one annotator out [Certain]:

\[ \mathrm{Acc}(a) = \frac{1}{10} \sum_{j=1}^{10} \min\!\Big(1,\; \frac{\#\{\, i \ne j : a_i = a \,\}}{3}\Big) \tag{176.7} \]
Read the symbols
\(a\)
the model’s answer after normalization: lower case, punctuation removed, number words to digits (“two” → “2”), articles (“a”, “an”, “the”) removed
\(a_i\)
the normalized answer of annotator \(i\); the sum over \(j\) leaves annotator \(j\) out, so each subset has 9 humans
Worked example: “How many sheep are there?”

The 10 humans answered “8” five times, “9” three times and “10” twice. BLIP answered “10”. In the 2 subsets that leave out a “10” annotator, 1 of the other 9 says “10”: credit \(\min(1, 1/3) = 0.333\). In the other 8 subsets, 2 say “10”: credit \(2/3 = 0.667\). So \(\mathrm{Acc} = (2 \times 0.333 + 8 \times 0.667)/10 = 0.600\). The answer “8” would get 1.0 and “9” would get \((3 \times 0.667 + 7 \times 1)/10 = 0.900\). The metric rewards agreeing with the crowd, which is fair for colors and hard for counts of many objects.

# metrics.py (VQA part): a close copy of the official normalization, then (176.7)
manual = {"none": "0", "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4", "five": "5",
          "six": "6", "seven": "7", "eight": "8", "nine": "9", "ten": "10"}
punct = [";", "/", "[", "]", '"', "{", "}", "(", ")", "=", "+", "\\", "_", "-", ">", "<", "@", "`", ",", "?", "!"]
period_strip = re.compile(r"(?!<=\d)(\.)(?!\d)"); comma_strip = re.compile(r"(\d)(,)(\d)")
def norm_answer(s):
    s = s.replace("\n", " ").strip().lower(); out = s
    for p in punct:
        out = out.replace(p, "") if (p + " " in s or " " + p in s or comma_strip.search(s)) else out.replace(p, " ")
    out = period_strip.sub("", out)
    return " ".join(manual.get(w, w) for w in out.split() if w not in {"a", "an", "the"})
def vqa_acc(pred, answers):
    a, g = norm_answer(pred), [norm_answer(x) for x in answers]
    return sum(min(1.0, sum(x == a for x in g[:j] + g[j + 1:]) / 3) for j in range(len(g))) / len(g)

(The official script also expands contractions such as “dont” → “don’t”; our version has a short list of them.) Our results on the 1,500 questions:

SystemAll (1,500)yes/no (563)number (181)other (756)Saw these questions in training?
CLIP B/32 (ranks 1,000 answers)4.2%0.2%0.2%8.2%no
blind prior (no image)32.2%66.2%26.5%8.4%uses answer counts of the other val images
BLIP-2 OPT-2.7B (zero-shot)55.2%71.4%37.8%47.3%no
BLIP VQA base83.0%96.5%67.7%76.7%yes: VQAv2 train and val

Four lessons from this table. (1) CLIP is far below the blind baseline: 4.2% against 32.2%. It never answered “yes” or “no” (0 of 1,500 answers), got 1.2% on “what color” questions and 0.0% on “how many”. Its answers name things in the picture (“suzuki”, “train”, “stove”), because the question cannot change what the image vector describes; the candidate that matches the image best wins, whatever was asked. (2) The blind baseline gets 66.2% on yes/no questions without seeing any image. VQAv2 was built with pairs of similar images and different answers to reduce such priors [Certain], and still a third of the score needs no vision. (3) BLIP-2’s 55.2% is zero-shot and close to the 53.5 its paper reports on the full val split for this model [Certain]. Its errors show the habits of a frozen LLM: it says “yes” to 71.4% of yes/no questions (the human majority says “yes” for 51.9%), answers “Two” or “None” to many counting questions, and writes “An airplane” where annotators wrote “plane”. (4) BLIP’s 83.0% is the best, and the least trustworthy: the model was fine-tuned on VQAv2 val, so it has seen these questions with these answers. On test-dev the paper reports 77.54 for the base model [Certain]. Even so, counting is its weakest type (67.7%).

Figure 5. Top: measured VQA accuracy (176.7) per answer type. Bottom: 12 of the 1,500 questions with the 10 human answers (grouped after normalization), the four systems’ answers and their scores. Click any human answer to see the score that answer would get, computed live with (176.7).
How to read Figure 5

First the bars: switch to yes/no and see CLIP at almost 0 while the blind baseline gets about two thirds, just by answering “yes” or “no” by question type. Switch to number: every system drops. Then step through the questions. On the sheep question, click “8”, “9” and “10”: 1.000, 0.900 and 0.600, the numbers of the worked example. Look at CLIP’s answers: they are names of things in the picture, often unrelated to the question. That is what a model does when the question cannot change how it sees the image.

Step 4: where each one fails, and why

Every failure we measured has a cause you can name. Sort them into three boxes: the architecture (which parts can exchange information), the loss (what the training rewarded) and the data (what the model saw, including the test set itself).

What we sawEvidence in our runMain cause
CLIP cannot write a captioncaptioning by retrieval: CIDEr 61.8; the copied caption fits a similar sceneArchitecture: no decoder; it can only rank existing text
CLIP cannot answer questionsVQA 4.2%, below the blind 32.2%; 0 of its 1,500 answers is “yes” or “no”; 0.0% on “how many” questionsArchitecture: question and image meet only in one dot product. Loss: contrastive on captions, never on question–answer pairs
CLIP has the lowest retrievaltext → image R@1 0.591 against 0.82 (ITC of BLIP and BLIP-2)Data: no COCO fine-tuning. Architecture: one vector from 49 patches of 32 pixels
Pooled vectors are close to a bag of wordsshuffled captions: BLIP ITC 0.664, BLIP-2 ITC 0.653; their ITM heads 0.763 and 0.788Architecture: order needs cross-attention. Loss: ITC is rarely punished for word order
BLIP writes short, generic captions6.1 words against 10.4; CIDEr 97.0, and 109.0 without the promptDecoding: prompt, beam search and length limits. Data: COCO fine-tuning teaches the safe consensus phrase
BLIP-2 says “yes” too often“yes” on 71.4% of yes/no questions, where the human majority says “yes” on 51.9%Architecture and data: a frozen OPT never trained on VQA fills gaps with its language prior [Likely]; only 32 visual tokens from a 224-pixel image
BLIP-2 counts poorly and prefers some answers“how many”: 43.2% (BLIP 72.8%); answered “none” (normalized to “0”) 41 times out of 151; “blue” for 40 of 154 color questions (human majority: 22)Data: no question–answer training; caption data rarely states counts. Architecture: low resolution through a 32-token bottleneck [Likely]
BLIP’s VQA score is too good to be true83.0% on VQAv2 val; the paper reports 77.54 on test-dev for the base model [Certain]Data: the model was fine-tuned on VQAv2 train and val [Certain]; our test questions are in its training set
Exact-match scoring punishes correct answersBLIP-2: “An airplane” → “airplane”, but the humans wrote “plane”: score 0Metric: VQA accuracy is string match after normalization; a zero-shot LLM does not know the house style of the annotators
Everybody fails on look-alike photos35 of our 500 images show a bathroom; “A clean bathroom is seen in this image.” has many correct answers, but only one is labeledEvaluation protocol: one labeled match per caption; part of every recall@1 “error” is a vague query
Figure 6. The whole lab on one chart, all values measured. “Score per task” shows text → image R@1 (best variant of each family), CIDEr-D (×100) and VQA accuracy, with a cross where a family cannot do the task natively. “Word-order probe” shows how often each score prefers the true caption to its shuffled copy. “Judge consistency” shows how many of the 500 judge decisions survive the order swap.
How to read Figure 6

In “Score per task”, CLIP is competitive only in its home task, retrieval, and even there it is behind; its captioning and VQA bars are the ranking work-arounds, marked with a cross. BLIP has the best retrieval and VQA bars, but both rest on in-domain training data (COCO Karpathy train, and the VQAv2 val questions themselves). BLIP-2 has the best captioning bar without any COCO fine-tuning. Switch to “Word-order probe”: the ITM bars are higher than the ITC bars of the same family; that is the value of cross-attention. Switch to “Judge consistency”: the share of inconsistent answers tells you how much to trust a small judge.

Three lessons carry over to every VLM you will evaluate:

  • Check what the model has seen. BLIP’s 83.0% VQA accuracy is not a zero-shot number; on the official test-dev split, which it did not train on, the BLIP paper reports 77.54 for the base model with 14M images [Certain]. BLIP-2’s VQA number is zero-shot. Comparing them as equals would be a mistake. For captions the story is subtler: BLIP was fine-tuned on Karpathy train and BLIP-2 OPT was not, yet BLIP-2 wins. Its pretraining mix contains COCO’s own human captions and synthetic captions written by BLIP-large [Certain], a captioner that BLIP fine-tunes on COCO for CapFilt [Likely], so “pre-trained only” does not mean “never saw COCO-style text”.
  • A metric is a model of “good”, with its own biases. CIDEr rewards the consensus wording of the five references and punishes length differences; in our run it scored both BLIP and BLIP-2 above a held-out human caption. The judge has its own bias (position) and its own blind spots. Report two views and how they disagree.
  • Interaction decides what a model can do. If question and image only meet in a final dot product (CLIP), the model cannot answer “what color is the train?”: it can only say what the image is about. Cross-attention (BLIP’s ITM and VQA encoder, BLIP-2’s Q-Former) lets the text pick which part of the image to read. That is why every modern VLM (session 177) feeds visual tokens into the language model instead of pooling them into one vector.

Where this is used

ApplicationWhat from this lab carries over
Photo and product searchAlgorithm 1: a dual encoder (CLIP, SigLIP or BLIP ITC) over the whole catalogue, then an ITM-style reranker on the top 16 to 100; measure the R@1 against depth curve (Figure 3) before choosing \(k\)
Alt-text and image descriptionsA generative model (BLIP-2 style or a modern VLM); evaluate with CIDEr and a judge with order swap; check that the style matches your users, not COCO
Visual assistants and customer supportVisual tokens fed to an LLM (BLIP-2 and its successors); test zero-shot on data the model has not seen; keep a blind baseline to measure the language prior
Dataset curation and filteringBLIP’s CapFilt idea: a captioner writes synthetic captions and an ITM filter removes bad pairs; BLIP-2’s data used BLIP-large captions ranked by CLIP ViT-L/14 similarity [Certain] (session 187)
Text-to-video retrievalITC on sampled frames or pooled clip vectors for the shortlist, then a cross-attention reranker on the top clips; recall@k per direction (session 208)
Video captioning and key-moment detectionCaption sampled frames with a BLIP-2-style model, then search the captions with text, or score frames against a query with ITM to find the moment; watch for invented details
Video question answering and Video-RAGRetrieve the relevant clips with ITC, then answer with a model whose language side reads the visual tokens (session 210); evaluate with VQA-style accuracy plus a blind baseline
Content moderationITC scores against policy descriptions as a fast filter, ITM or a VLM for the uncertain cases; never trust a score threshold tuned on another model

Common mistakes

  • Comparing a fine-tuned number with a zero-shot number. BLIP’s VQA model trained on VQAv2 val; scoring it on VQAv2 val and putting it next to BLIP-2’s zero-shot score is not a fair comparison. Always write down, per checkpoint, which datasets and splits it was trained on.
  • Comparing your CIDEr with a paper’s CIDEr. The value depends on the tokenizer, on the set the IDF is computed from (ours moved from 97.0 to 92.1 for BLIP when the IDF came from 5,000 images instead of 500) and on post-processing. Compare systems only inside one evaluation run.
  • Leaving the prompt in the caption. BLIP generates “a picture of a person riding a motorcycle”. The authors’ code removes the prompt; if you forget, our BLIP CIDEr goes from 97.0 to 104.9, because the longer caption gets a smaller length penalty. A “better” score from a bug.
  • Running ITM over the whole gallery, or skipping it. ITM costs one cross-attention pass per pair: 50 pairs per second in our run. Use it only to rerank an ITC shortlist, and do not call ITC-only numbers “BLIP retrieval” without saying so.
  • A probe with a hidden second difference. Our first word-order probe kept the final period on the true captions and dropped it from the shuffled ones. That alone moved BLIP-2 ITC from 0.653 to 0.550 and BLIP-2 ITM from 0.788 to 0.844. A controlled test must change exactly one thing.
  • Right-padding a decoder-only LLM in batched generation. BLIP-2 with OPT must pad on the left; with right padding, pad tokens sit between the prompt and the first generated token and the answers change [Likely].
  • Trusting a single-order judge. Ask twice with the order swapped and report the inconsistent cases (176.6); a judge that follows the slot is measuring itself, not your models.
  • No blind baseline for VQA. Without the question-type prior (32.2% here) you cannot tell how much of a VQA score needs the image at all.

Interview questions

What is the difference between ITC and ITM, and why do retrieval systems use both?basic
ITC (image–text contrastive) encodes image and text separately and scores a pair with a cosine (176.1). Gallery vectors are computed once, so a query costs one matrix product over the whole gallery. ITM (image–text matching) reads the text with cross-attention to the image tokens and outputs a match logit; it sees details and word order, but needs one full pass per pair. So systems use ITC to shortlist the top \(k\) and ITM to rerank them (176.2). In our run BLIP went from 0.822 to 0.861 text → image R@1 with \(k = 16\), and BLIP-2 from 0.816 to 0.857, at a cost of 16 ITM passes per query.
Why can CLIP not do captioning or VQA directly? What did it score when forced?basic
CLIP has no decoder, so it cannot produce new text; and its image and text towers only meet in a final dot product, so a question cannot change which part of the image is read. Forced work-arounds: captioning by retrieval from 82,783 train captions gave CIDEr 61.8, because the copied caption is about a similar scene but rarely this one. VQA by ranking 1,000 answers with “question: q answer: a” gave 4.2%, below a blind question-type baseline (32.2%): CLIP picked words that describe the image (“suzuki”, “train”) and almost never “yes” or “no”, which have no visual content.
How does BLIP-2 connect a frozen image encoder to a frozen LLM, and what is trained?core
A Q-Former with 32 learned query vectors cross-attends to the frozen ViT-g features (257 tokens of 1,408 numbers at 224 px) and outputs 32 vectors of 768. A linear layer maps them to OPT’s 2,560-dimensional input space, and they are prepended to the text as soft visual prompts. Stage 1 trains the Q-Former with ITC, ITM and image-grounded generation; stage 2 trains it with the LLM’s language-modeling loss. Only the Q-Former and projection are trained: 107.1 M of 3.745 B parameters in our loaded model (2.9%), matching the paper’s 107M.
Define CIDEr. Why can a model score higher than a human caption?core
CIDEr represents captions as TF-IDF vectors of 1- to 4-grams, where the IDF is computed over the evaluation set’s references (176.4), and averages the cosine with each reference over n and references; CIDEr-D clips counts and applies a Gaussian length penalty (176.5). It rewards the consensus wording. A model fine-tuned on COCO learns that consensus style, while a single human caption is one individual’s wording. In our run a held-out human caption scored 91.6 against the other 4 references, while BLIP scored 97.0 and BLIP-2 140.8 against all 5. A higher CIDEr does not mean a better description.
Your BLIP VQA model scores 83% on VQAv2 val. What do you check before you report it?core
Whether the model trained on that split. The BLIP paper fine-tunes VQA on VQAv2 train and val plus Visual Genome, and reports 77.54 on test-dev for the base model [Certain]; our 83.0% on val is inflated by contamination. Also check: the answer normalization and the 10-annotator formula (176.7), the generation vs candidate-ranking difference with the paper, and a blind baseline (32.2% here) to see how much is language prior. Report per answer type: BLIP gets 96.5% on yes/no but 67.7% on number questions.
How does the VQA accuracy metric work? Give the score of an answer that 2 of 10 annotators gave.basic
Normalize all answers, then for each of the 10 subsets of 9 annotators give \(\min(1, \text{matches}/3)\) and average (176.7). If 2 of 10 gave the answer: in the 2 subsets that drop one of them there is 1 match (1/3); in the other 8 there are 2 matches (2/3). The score is \((2/3 + 16/3)/10 = 0.6\). Three or more matching humans in every subset give 1.0.
Why did BLIP-2 not beat BLIP on retrieval in this lab, despite a 1-billion-parameter vision encoder?advanced
Three reasons, two of them about data. The BLIP checkpoint was fine-tuned on COCO Karpathy train with ITC and ITM, so our test photos and captions are in-domain for it. The public BLIP-2 retrieval checkpoint is (most likely) the stage-1 pretrained Q-Former, not the COCO fine-tuned model of the paper’s table [Likely]. And BLIP-2 sees the image at 224 px through a frozen encoder, while BLIP fine-tunes its ViT at 384 px. Result: ITC R@1 0.822 against 0.816, and 0.861 against 0.857 after ITM. To compare architectures you must hold the fine-tuning data fixed.
Is a CLIP-style embedding a bag of words? How would you test it?advanced
Build a controlled negative: the same words in a random order, and count how often the true caption scores higher for its own image (chance 0.5). In our run (true captions and shuffles both without the final period): CLIP 0.850, BLIP ITC 0.664 and ITM 0.763, BLIP-2 ITC 0.653 and ITM 0.788. The ITC heads of BLIP and BLIP-2 are close to bag-of-words; the ITM heads, which read the text with cross-attention to the image, are far less so. For a fuller test use benchmarks built for it, such as the ARO suite of Yuksekgonul et al., which also proposes training with word-order hard negatives.
How would you use an LLM or VLM as a caption judge without fooling yourself?core
Use pairwise comparisons, read the answer from token probabilities rather than parsing free text, and ask both orders; count a win only when both agree (176.6) and report the inconsistent share. In our run with Qwen2-VL-2B, the judge picked the second slot in both orders for 300 of the 500 BLIP-versus-BLIP-2 images; only 197 decisions survived the swap, and those favoured BLIP-2 171 to 26. Also check the judge against humans on a sample, keep the judge different from the systems being compared (a BLIP-family judge would favour BLIP-style text), and report it next to a reference-based metric such as CIDEr.
Design an evaluation to choose between a BLIP-style and a BLIP-2-style model for product alt-text.advanced
Collect a few hundred of your own product images with 3 to 5 references written in your house style, never used for training. Measure CIDEr (one run, one tokenizer, IDF on your set), a judge with order swap, and a human check of factual errors (invented colors, brands, counts) on a sample. Slice by product type and by text-on-product. Add speed and memory: BLIP base is 224 M parameters, BLIP-2 OPT-2.7B is 3.7 B (about 7.5 GB in fp16). If BLIP-2 wins on the judge but loses on CIDEr, the style differs; a light fine-tune of the Q-Former on your captions can fix style without touching the frozen parts.
What do the three families tell you about modern VLM design?core
CLIP shows that contrastive pretraining gives excellent, cheap, general image features, which is why most VLMs still start from a CLIP- or SigLIP-type encoder. BLIP shows that cross-attention between text and image is needed for matching details and that a decoder is needed to write, and that bootstrapped captions clean noisy web data. BLIP-2 shows that a small trainable bridge can connect frozen experts, and that the LLM brings fluency and knowledge but also its own style and priors. LLaVA-style models keep the frozen-encoder idea but replace the Q-Former by a simple projector and feed many visual tokens to an instruction-tuned LLM (session 178).

Exercises

  1. One annotator out of 10 gave the answer “gravel”; nobody else did. What VQA accuracy does the answer “gravel” get with (176.7)?
  2. A gallery has 1 million images and your service gets 1,000 text queries per second. How many ITM passes per second do you need with \(k = 16\), and how many if you ran ITM on every pair?
  3. In our 500 images, the unigram “a” occurs in the references of 499 images and the bigram “dirt road” in 1. Compute their IDF \(\log(N / \mathrm{df})\) and the ratio. What does this say about which words drive CIDEr?
  4. BLIP-2 OPT-2.7B has 3,744.8 M parameters, of which the Q-Former (105.2 M) and the projection (2.0 M) are trained. What fraction is trained? How many visual tokens does OPT see per image, compared with BLIP’s image encoder output at 384 px with 16-pixel patches?
  5. Our first word-order probe compared captions that end with a period against shuffles without one. Why is that a confound, and how would it bias the result?
  6. Run the VQA step again with BLIP-2 and the prompt “Question: {} Short answer:”. Which answer types change most? Explain the change with the length of the answers.
Answerscheck
1. In the 1 subset that drops that annotator, 0 matches: 0. In the other 9 subsets, 1 match: 1/3. The score is \(9 \times (1/3) / 10 = 0.3\). 2. With \(k = 16\): \(1{,}000 \times 16 = 16{,}000\) passes per second (for example 320 GPUs at our measured 50 pairs per second, or far fewer with a fast GPU and batching). On every pair: \(1{,}000 \times 10^6 = 10^9\) passes per second, impossible. 3. \(\log(500/499) = 0.002\) and \(\log(500/1) = 6.215\); the ratio is about 3,100. CIDEr is driven by rare, content-bearing n-grams; function words count for almost nothing, so a caption that misses the key object or scene word scores near zero even if it is fluent. 4. \(107.1 / 3{,}744.8 = 2.86\%\). OPT sees 32 visual tokens; BLIP’s ViT-B/16 at 384 px outputs \((384/16)^2 + 1 = 577\) tokens, 18 times more. (BLIP-2’s frozen ViT-g gives 257 tokens, which the Q-Former compresses 8 times.) 5. A model can prefer the true caption just because of the period (punctuation is a token), not because of word order; this would inflate every score, more for models whose tokenizer and training data treat the final period as informative. The fix is to strip it from both sides, as in our final probe. 6. Open-ended. Expect the share of one-word answers to change; because VQA accuracy needs an exact match after normalization, longer answers such as “a man on a horse” score 0 even when correct, so the answer style can move the score by several points without any change in what the model sees.

Key takeaways

  • Run every model on the same images with the same metric code; a number is only meaningful next to the numbers it is compared with.
  • Retrieval: ITC is cheap and scales to a gallery (176.1); ITM reranking of the top \(k\) (176.2) added about 4 points of R@1 for BLIP and BLIP-2. Most of the gain came by \(k = 4\).
  • CLIP cannot write and cannot condition on a question: captioning by retrieval gave CIDEr 61.8 and VQA by ranking 4.2%, below a blind 32.2% baseline.
  • CIDEr (176.4, 176.5) rewards the consensus style and depends on tokenizer, IDF set and post-processing; BLIP-2 (140.8) and BLIP (97.0) both scored above a held-out human caption (91.6), and BLIP’s decoding settings alone moved it by 12 points. Pair it with a judge that is asked in both orders (176.6).
  • VQA accuracy (176.7) gives partial credit for answers that fewer than 3 humans gave; always report per answer type and a blind baseline.
  • Check training data before comparing: BLIP’s 83.0% VQA is on questions it trained on; BLIP-2’s 55.2% is zero-shot.
  • Cross-attention is what lets text read the image: on shuffled captions the ITM heads beat the ITC heads of the same family by a wide margin.

Key formulas

\[ s_{\text{ITC}} = \mathbf{u}^\top \mathbf{t}, \qquad s^{\text{BLIP-2}}_{\text{ITC}} = \max_q \mathbf{z}_q^\top \mathbf{t}, \qquad \text{score} = \ell_{\text{ITM}} + s_{\text{ITC}} \;\; (\text{top-}k \text{ only}) \]
\[ \mathrm{R@}k = \tfrac{1}{|Q|} \textstyle\sum_q \mathbb{1}[r_q \le k], \qquad g_\omega(s) = h_\omega(s) \log \frac{N}{\max(1, \mathrm{df}(\omega))} \]
\[ \text{CIDEr-D} = \frac{10}{4|R|} \sum_{n, r} \frac{\sum_\omega \min(g_\omega(c), g_\omega(r))\, g_\omega(r)}{\lVert \mathbf{g}^n(c) \rVert \lVert \mathbf{g}^n(r) \rVert} e^{-\frac{(\ell_c - \ell_r)^2}{2\sigma^2}}, \qquad \mathrm{Acc}(a) = \frac{1}{10} \sum_j \min\!\Big(1, \frac{\#\{i \ne j : a_i = a\}}{3}\Big) \]

References for this session

  1. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  2. Li et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  3. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  4. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  5. Li et al. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  6. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  7. Li et al. (2021). Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
  8. Goyal et al. (2016). Making the V in VQA Matter
  9. Antol et al. (2015). VQA: Visual Question Answering
  10. Vedantam, Zitnick & Parikh (2015). CIDEr: Consensus-based Image Description Evaluation
  11. Chen et al. (2015). Microsoft COCO Captions: Data Collection and Evaluation Server
  12. Karpathy & Fei-Fei (2015). Deep Visual-Semantic Alignments for Generating Image Descriptions
  13. Yuksekgonul et al. (2023). When and why vision-language models behave like bags-of-words, and what to do about it?
  14. Zhang et al. (2022). OPT: Open Pre-trained Transformer Language Models
  15. Fang et al. (2022). EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
  16. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  17. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  18. Hugging Face (docs). Transformers documentation: BLIP and BLIP-2 model pages