LLM & VLM courseBlog
170Track 7: Vision and VLMs · session 8 of 38

zero-shot classification and text-to-image search

In this lab you run two real dual encoders, CLIP ViT-B/32 and SigLIP ViT-B/16, on a laptop CPU. You build a zero-shot classifier for CIFAR-10 from class names and prompt templates, and a text-to-image search engine for Flickr8k with a FAISS index. Then you measure them honestly: accuracy with a paired significance test, recall@k, the recall cost of an approximate index, and four controlled probes that show where both models break (counting, small text, small objects and words written on the image).

Time: 3 to 4 hours (mostly hands-on; the embedding runs take 10 to 20 minutes on a CPU)Level: Beginner · hands-on labBefore this: Session 166 (CLIP) and Session 167 (SigLIP); Session 169 helps; Python and basic PyTorch

What you will learn

  • How to turn class names and templates into a zero-shot classifier matrix, and run it with CLIP and SigLIP from Hugging Face
  • Why CLIP gives softmax probabilities and SigLIP gives independent sigmoids with a learned bias, and why their scores and thresholds are not interchangeable
  • How to tell a real prompt improvement from noise with per-image results and McNemar’s paired test
  • How to build text-to-image search with a FAISS flat index, and why you embed and index in separate processes
  • How to compute recall@k and MRR in both directions, and what the protocol does to the number
  • How an IVF index trades recall for speed, and why the default nprobe = 1 is a trap
  • How to design controlled probes for counting, small text, small objects and typographic attacks, and what they can and cannot prove

Why this matters

In session 166 and session 167 you saw how a dual encoder puts images and sentences into one vector space, and in session 169 you saw what people build on top of it. In this lab you run two real models, measure them on real data, and find out where they break. You will build two systems that many companies run in production: a zero-shot classifier (a classifier made only from class names, with no training images) and a text-to-image search engine (type a sentence, get the matching photos).

Everything here runs on a laptop CPU. The models are OpenAI’s CLIP ViT-B/32 and Google’s SigLIP ViT-B/16 at 224 pixels, both from Hugging Face. The data is a 1,000-image subset of the CIFAR-10 test set for classification and the 1,000-image Flickr8k test split, with 5,000 human captions, for search. Every number on this page comes from our own run of the code below, unless it is tagged as coming from a paper. The page also shows how to decide whether a difference is real: many prompt “improvements” on small test sets are noise.

What you will measure, in short: SigLIP beats CLIP on both tasks (93.4% versus 89.5% zero-shot accuracy, 72.8% versus 53.7% text-to-image recall@1). Prompt templates move CLIP by up to 4 points and SigLIP by much less. And both models fail in the same places the papers warn about: counting, very small text, very small objects, and words written on the image, which flip CLIP’s answer 92% of the time.

Setup

You need Python 3.10 or newer and about 2 GB of disk for the two models and the two datasets. We used torch 2.15, transformers 5.23 and faiss-cpu 1.16 on an Apple-silicon laptop; any recent versions should work.

python -m venv venv && source venv/bin/activate
pip install torch transformers datasets faiss-cpu pillow sentencepiece protobuf

The two models, with facts read from the loaded checkpoints [Certain]:

CLIP ViT-B/32SigLIP ViT-B/16
Hugging Face idopenai/clip-vit-base-patch32google/siglip-base-patch16-224
Parameters (both towers)151.3 million203.2 million
Patches per 224 × 224 image\((224/32)^2 = 49\)\((224/16)^2 = 196\)
Embedding size \(d\)512768
Learned logit scale \(s\)100.0 (the cap)117.33
Learned bias \(b\)none−12.93
Training losssoftmax over the batchone sigmoid per pair
Text paddingpad to the longest text in the batchalways pad to 64 tokens

SigLIP has 4 times more patches, so its image tower does about 4 times more work per image. In our run it embedded the 1,000 CIFAR images 3.9 times slower than CLIP (on a busy machine, so treat the absolute times as rough). The last row matters: the Hugging Face documentation says to pass padding="max_length" to SigLIP “because that is how the model was trained” [Certain]. If you pad SigLIP texts like CLIP texts, the embeddings change and the accuracy drops.

A. Zero-shot classification (CIFAR-10) 10 class names ×M templates text tower average, normalizeW: 10 × d 1,000 test images image tower U: 1000 × d U Wᵀargmax accuracy,confusion B. Text-to-image search (Flickr8k test) 1,000 gallery images image tower FAISS indexbuilt once, offline “two dogs playin the snow” text tower top-k images recall@k,failures
Figure 1. The two systems of this lab. Both use the same two towers. In A, the text tower runs only once per class, so the classifier is a fixed \(10 \times d\) matrix. In B, the image tower runs once per gallery image, offline, and only the query text is embedded at search time.

The code is split into one shared file and four steps. common.py loads both models and gives two functions that return unit vectors (each row divided by its length), so a dot product is a cosine similarity:

# common.py: load the two models and embed images or texts (unit vectors)
import torch, torch.nn.functional as F
from transformers import AutoModel, AutoProcessor
torch.manual_seed(0); torch.set_grad_enabled(False)
MODELS = {"clip": "openai/clip-vit-base-patch32", "siglip": "google/siglip-base-patch16-224"}
M = {k: AutoModel.from_pretrained(v).eval() for k, v in MODELS.items()}
P = {k: AutoProcessor.from_pretrained(v) for k, v in MODELS.items()}

def _vec(out):                       # some versions return a tensor, others an output object
    return out if isinstance(out, torch.Tensor) else out.pooler_output

def embed_images(k, images, bs=64):
    out = []
    for i in range(0, len(images), bs):
        x = P[k](images=images[i:i + bs], return_tensors="pt")
        out.append(F.normalize(_vec(M[k].get_image_features(pixel_values=x["pixel_values"])), dim=-1))
    return torch.cat(out)                                    # (n, d), every row has length 1

def embed_texts(k, texts, bs=256):
    out = []
    for i in range(0, len(texts), bs):
        if k == "siglip":   # SigLIP was trained with texts padded to 64 tokens: pad the same way
            x = P[k](text=texts[i:i + bs], padding="max_length", max_length=64, truncation=True, return_tensors="pt")
        else:
            x = P[k](text=texts[i:i + bs], padding=True, truncation=True, return_tensors="pt")
        feats = M[k].get_text_features(**{n: x[n] for n in ("input_ids", "attention_mask") if n in x})
        out.append(F.normalize(_vec(feats), dim=-1))
    return torch.cat(out)

The processor does the image preprocessing for you: resize, center crop to 224, and normalization with each model’s own mean and standard deviation (session 163). Never feed one model’s preprocessed tensors to the other.

Step 1: a classifier made from class names

A normal classifier has a weight matrix that was learned from labeled images. A zero-shot classifier gets its weight matrix from the text tower instead. For each class \(k\) we fill \(M\) sentence templates (patterns such as “a photo of a {}.”) with the class name, embed the \(M\) sentences, average them, and normalize. Stacking the \(K\) rows gives the matrix \(W\). An image is assigned to the row with the largest cosine:

\[ \mathbf{w}_k = \frac{\sum_{m=1}^{M} \mathbf{t}_{k,m}}{\big\lVert \sum_{m=1}^{M} \mathbf{t}_{k,m} \big\rVert}, \qquad \hat{y}(x) = \argmax_{k \in \{1,\dots,K\}} \; \mathbf{w}_k^\top \mathbf{u}(x) \tag{170.1} \]
Read the symbols
\(\mathbf{t}_{k,m}\)
unit text embedding of template \(m\) filled with the name of class \(k\)
\(\mathbf{w}_k\)
the classifier row of class \(k\): the normalized average of its template embeddings
\(\mathbf{u}(x)\)
unit image embedding of the test image \(x\)
\(K, M\)
number of classes (10 here) and number of templates (1 or 18 here)
\(\hat{y}(x)\)
the predicted class; the scale and bias of the model do not change the argmax, so we can ignore them here
Algorithm 1: zero-shot evaluation with several prompt choices
  1. input: labeled test images \((x_i, y_i)_{i=1..n}\), class names \(c_{1..K}\), prompt sets \(\mathcal{T}_1, \dots\), a dual encoder
  2. \(U \leftarrow\) image embeddings of all \(x_i\), shape \((n, d)\) // once per model; save them to disk
  3. for each prompt set \(\mathcal{T}\):
  4.   for \(k = 1..K\): \(\mathbf{w}_k \leftarrow\) normalize(mean of text embeddings of \(t(c_k)\), \(t \in \mathcal{T}\)) // formula (170.1)
  5.   \(\hat{y} \leftarrow \argmax\) over the rows of \(U W^\top\) // (n, K) cosines
  6.   store the per-image results \(\mathbb{1}[\hat{y}_i = y_i]\), not only the mean // needed for the paired test in Step 3
  7. report accuracy and the \(K \times K\) confusion matrix for every prompt set

We take the first 100 test images of each CIFAR-10 class (1,000 images; CIFAR images are only 32 × 32 pixels, so the processor scales them up to 224) and try five prompt choices: the bare class name, “a photo of a {}.”, SigLIP’s documented default “This is a photo of {}.”, a deliberately mismatched template “a drawing of a {}.”, and the ensemble of the 18 templates that the CLIP authors published for CIFAR-10 [Certain]:

# step1_zeroshot.py: zero-shot CIFAR-10 with five prompt choices, two models
import json, torch, torch.nn.functional as F
from datasets import load_dataset
from common import M, embed_images, embed_texts

ds = load_dataset("uoft-cs/cifar10", split="test")
names = ds.features["label"].names               # airplane, automobile, bird, cat, ...
keep, seen = [], [0] * 10
for i, y in enumerate(ds["label"]):               # the first 100 test images of every class
    if seen[y] < 100: keep.append(i); seen[y] += 1
ds = ds.select(keep)
y = torch.tensor(ds["label"]); images = [im.convert("RGB") for im in ds["img"]]

T18 = ["a photo of a {}.", "a blurry photo of a {}.", "a black and white photo of a {}.",
       "a low contrast photo of a {}.", "a high contrast photo of a {}.", "a bad photo of a {}.",
       "a good photo of a {}.", "a photo of a small {}.", "a photo of a big {}.", "a photo of the {}.",
       "a blurry photo of the {}.", "a black and white photo of the {}.", "a low contrast photo of the {}.",
       "a high contrast photo of the {}.", "a bad photo of the {}.", "a good photo of the {}.",
       "a photo of the small {}.", "a photo of the big {}."]       # CLIP's 18 CIFAR-10 templates
PROMPTS = {"bare": ["{}"], "photo": ["a photo of a {}."], "siglip": ["This is a photo of {}."],
           "drawing": ["a drawing of a {}."], "ens18": T18}

def classifier(k, templates):                     # formula (170.1): one unit row per class
    rows = [F.normalize(embed_texts(k, [t.format(c) for t in templates]).mean(0), dim=-1) for c in names]
    return torch.stack(rows)                      # (10, d)

results = {}
for k in M:
    u = embed_images(k, images)                   # (1000, d), computed once per model
    torch.save(u, f"cifar_{k}.pt")
    for p, templates in PROMPTS.items():
        pred = (u @ classifier(k, templates).T).argmax(1)
        results[f"{k}/{p}"] = (pred == y).int().tolist()     # keep per-image results for paired tests
        print(f"{k:7s} {p:8s} accuracy {(pred == y).float().mean():.3f}")
json.dump({"y": y.tolist(), "names": names, "correct": results}, open("step1.json", "w"))

Our output (accuracy on the 1,000 images):

PromptCLIP B/32SigLIP B/16
bare name “{}”0.8740.915
“a photo of a {}.”0.8950.934
“This is a photo of {}.”0.9070.928
“a drawing of a {}.” (mismatched on purpose)0.9100.923
ensemble of 18 CLIP templates0.9160.929

Three things stand out. The bare name is the worst choice for both models. The ensemble is the best for CLIP. And the “wrong” template, “a drawing of a {}.”, beats “a photo of a {}.” for CLIP. One plausible reason [Speculation]: a 32-pixel image scaled up to 224 is blurry and flat, and looks less like a typical web photo than like a simple picture. This is exactly why you test templates on your own data instead of copying them. But is a difference of 1.5 points real, or luck of the 1,000 images? Step 3 answers that. Figure 2 shows where the errors are.

Figure 2. Measured confusion matrices (rows: true class, columns: predicted class, 100 images per row) for both models and all five prompt choices. Click a cell to see how many images of the row class were predicted as the column class. The bars on the right are the per-class accuracies, with the change against “a photo of a {}.”
How to read Figure 2

Start with CLIP and “a photo of a {}.”: accuracy 0.895, and the weakest classes are deer (72 of 100) and frog (75). Click the cell (deer, horse) or look along the deer row to see where the deer go. Now switch to the 18-template ensemble: deer rises to 80 and airplane from 90 to 95, giving 0.916 overall. Switch to the bare name: airplane drops to 75. Then pick SigLIP: the matrix is much cleaner (0.934 with the photo template), and the template choice changes it far less. Big, systematic changes in one row are more informative than small changes in the total.

You can also change the class names. CIFAR-10 calls its second class “automobile”. Renaming it to “car” changed the overall accuracy only slightly in our run (CLIP 0.895 to 0.893, SigLIP 0.934 to 0.937), well within the noise you will compute in Step 3. Names matter much more when a word has two meanings (“crane”, “boxer”) or when the dataset’s name is unusual; fix those first.

Step 2: from scores to probabilities, softmax or sigmoid

The argmax only needs cosines. But products often need a confidence: “show the label only if the model is sure”. The two models turn scores into probabilities in different ways, because they were trained with different losses. CLIP was trained with a softmax over the batch, so its natural probability is a softmax over the \(K\) classes:

\[ p_{\text{CLIP}}(k \mid x) = \frac{\exp\!\big(s\, \mathbf{w}_k^\top \mathbf{u}\big)}{\sum_{k'=1}^{K} \exp\!\big(s\, \mathbf{w}_{k'}^\top \mathbf{u}\big)} \tag{170.2} \]

SigLIP was trained with one independent sigmoid per image–text pair (session 167): “is this text a match for this image, yes or no?”. Its natural probability is therefore a separate yes/no probability for every class, with the learned scale and bias:

\[ p_{\text{SigLIP}}(\text{match}_k \mid x) = \sigma\!\big(s\, \mathbf{w}_k^\top \mathbf{u} + b\big) = \frac{1}{1 + \exp\!\big(-(s\, \mathbf{w}_k^\top \mathbf{u} + b)\big)} \tag{170.3} \]
Read the symbols
\(s\)
the learned logit scale: 100.0 for CLIP B/32, 117.33 for SigLIP B/16 (read from the checkpoints)
\(b\)
SigLIP’s learned bias, −12.93; the SigLIP paper starts it at −10 so that, at the start of training, almost every pair is predicted as “no match”, which is true for all but one pair in each row [Certain]
\(\sigma\)
the logistic sigmoid; the \(K\) values of (170.3) do not have to add up to 1
Worked example: the first test image (a cat), both models

SigLIP. The cosine of the image with “a photo of a cat.” is 0.0842, and with “a photo of a dog.” 0.0495. The logits are \(117.33 \times 0.0842 - 12.93 = -3.06\) and \(117.33 \times 0.0495 - 12.93 = -7.13\). The sigmoids are \(\sigma(-3.06) = 0.045\) and \(\sigma(-7.13) = 0.0008\). The 10 sigmoids add up to only 0.046. So SigLIP says “cat” is by far the best class, but also that it is only 4.5% sure the sentence “a photo of a cat.” matches this blurry picture. A sigmoid probability reaches 0.5 only when \(s \cos + b = 0\), at cosine \(12.93 / 117.33 = 0.110\). If you apply a softmax to the same SigLIP logits instead, the bias cancels and “cat” gets 0.980.

CLIP. The cosines are 0.2100 (cat), 0.2042 (dog), 0.1974 (ship) and 0.1950 (bird). With \(s = 100\) the softmax over all 10 classes gives cat 0.417, dog 0.233, ship 0.119 and bird 0.093. CLIP is right, but not confident. And a sigmoid on CLIP logits is meaningless: with no bias, \(\sigma(100 \times 0.21) = \sigma(21) \approx 1.0\) for every class.

The lesson: CLIP cosines (around 0.2 to 0.3 for matches) and SigLIP cosines (around 0.1) live on different scales, and the two probability formulas answer different questions. A threshold tuned for one model is wrong for the other.

cosinesoftmax (170.2)sigmoid (170.3)true class
Figure 3. Measured cosines of three CIFAR-10 test images with the 10 prompts “a photo of a {class}.”, for both models. The softmax and sigmoid bars are computed live from these cosines with the scale and bias of the sliders.
How to read Figure 3

With CLIP and image 0 the softmax gives cat 0.417, the value of the worked example, and the sigmoid bars are all at 1.0 because CLIP has no bias. Switch to SigLIP: the sliders jump to \(s = 117.33\) and \(b = -12.93\); the softmax gives cat 0.980, but the sigmoid only 0.045. Now image 3, the airplane: CLIP is right with 0.507 (bird 0.416), SigLIP is wrong (bird 0.705, airplane 0.085). Slide the SigLIP bias towards 0 and watch every sigmoid rise together: the bias sets the threshold, not the ranking. Slide the scale down to 1: every softmax becomes almost uniform (about 0.1), because cosine differences of a few hundredths mean nothing without the scale.

Step 3: is the difference real?

An accuracy measured on \(n\) images is itself a random number. If you drew another 1,000 CIFAR images, you would get a slightly different value. Its standard error (the typical size of that random change) is

\[ \mathrm{SE}(\hat{a}) = \sqrt{\frac{\hat{a}\,(1 - \hat{a})}{n}} \tag{170.4} \]

For \(\hat{a} = 0.895\) and \(n = 1{,}000\) this is 0.0097, so a 95% interval is about \(\pm 1.9\) points. By this rough rule, most of the template differences in the table are “inside the noise”. But this rule is too pessimistic for comparing two prompts, because both prompts are tested on the same images. Most images are right (or wrong) with both prompts; only the images where the two prompts disagree carry information. That is the idea of McNemar’s test (a paired test for two classifiers on the same examples):

\[ n_{10} = \#\{i : A \text{ right}, B \text{ wrong}\}, \quad n_{01} = \#\{i : A \text{ wrong}, B \text{ right}\}, \qquad p = 2 \sum_{j=0}^{\min(n_{10}, n_{01})} \binom{n_{10} + n_{01}}{j}\, 2^{-(n_{10} + n_{01})} \tag{170.5} \]
Read the symbols
\(n_{10}, n_{01}\)
the two kinds of disagreement; images where A and B agree are ignored
\(p\)
the exact two-sided p-value: how likely a split at least this uneven would be if A and B were equally good (then each disagreement is a fair coin flip); capped at 1

Our paired results, always against “a photo of a {}.” on the same 1,000 images:

ModelOther promptAccuracy change\(n_{10}\) (photo only)\(n_{01}\) (other only)pVerdict
CLIPbare name−2.135140.004real: worse
CLIP“This is a photo of {}.”+1.214260.081not shown
CLIP“a drawing of a {}.”+1.57220.008real: better
CLIP18-template ensemble+2.13240.00005real: better
SigLIPbare name−1.92560.0009real: worse
SigLIP“This is a photo of {}.”−0.626200.46no difference
SigLIP“a drawing of a {}.”−1.122110.080not shown
SigLIP18-template ensemble−0.51050.30no difference

Look at the CLIP ensemble row: +2.1 points sounds small, but the ensemble fixes 24 images and breaks only 3. That split is very unlikely by chance. The SigLIP ensemble row shows the opposite: a change of −0.5 points from a 10-versus-5 split is noise. Comparing the two models with the photo template: CLIP alone is right on 39 images, SigLIP alone on 78, \(p = 0.0004\). SigLIP is really better on this data. In short: for CLIP, prompts matter; for SigLIP on this data, they mostly do not, except that a bare name is always worse.

# paired test (170.5) on the per-image results saved by step 1
import json, math
r = json.load(open("step1.json"))["correct"]
def mcnemar(a, b):
    n10 = sum(x == 1 and y == 0 for x, y in zip(a, b)); n01 = sum(x == 0 and y == 1 for x, y in zip(a, b))
    n = n10 + n01
    p = min(1.0, 2 * sum(math.comb(n, j) for j in range(min(n10, n01) + 1)) / 2 ** n) if n else 1.0
    return n10, n01, p
for k in ["clip", "siglip"]:
    for p in ["bare", "siglip", "drawing", "ens18"]:
        print(k, p, mcnemar(r[f"{k}/photo"], r[f"{k}/{p}"]))
print("clip vs siglip", mcnemar(r["clip/photo"], r["siglip/photo"]))

Step 4: text-to-image search with a FAISS index

Now the second system. The gallery is the Flickr8k test split: 1,000 photos of everyday scenes, each with 5 captions written by people (5,000 captions in total) [Certain]. We use each caption as a query and ask: is its own photo among the top results? The design is the dual-encoder design from session 166: embed every gallery image once, offline, store the vectors in an index (a data structure for fast nearest-neighbour search), and at query time embed only the query text.

FAISS (Facebook AI Similarity Search, Johnson et al.) is the standard library for this (session 151). Its simplest index, IndexFlatIP, stores the raw vectors and compares the query with all of them by inner product: exact search, the reference every faster index is measured against. Because our vectors have length 1, the inner product is the cosine. We checked: all stored norms were between 0.9999998 and 1.0000001.

Algorithm 2: offline indexing and online text-to-image search
  1. offline: \(U \leftarrow\) unit embeddings of all \(N\) gallery images, shape \((N, d)\), float32 // the expensive part, done once
  2. index \(\leftarrow\) IndexFlatIP\((d)\); index.add\((U)\); keep a table from row number to image id // FAISS returns row numbers
  3. online, per query \(q\): \(\mathbf{v} \leftarrow\) unit text embedding of \(q\) // same model, same normalization
  4. scores, ids \(\leftarrow\) index.search\((\mathbf{v}, k)\) // top-k by cosine, sorted
  5. return the images ids; optionally rerank the top \(k\) with a stronger, slower model // see Where this is used

Two practical points. First, FAISS and PyTorch each ship their own copy of the OpenMP runtime (the library that runs loops on many CPU cores). On our Mac, importing both in one Python process crashed with “OMP: Error #15”, in either import order. The clean fix, and good design anyway, is to embed in one process, save the vectors to .npy files, and index in another. (The error message suggests the environment variable KMP_DUPLICATE_LIB_OK=TRUE; it calls this itself “unsafe” and says it may give wrong results, so do not use it.) Second, FAISS wants contiguous float32 numpy arrays.

# step2_embed_flickr.py: embed 1,000 Flickr8k test images and their 5,000 captions, save to disk
import json, numpy as np
from datasets import load_dataset
from common import M, embed_images, embed_texts

ds = load_dataset("jxie/flickr8k", split="test")
images = [im.convert("RGB") for im in ds["image"]]
captions = [ds[f"caption_{j}"][i].strip() for i in range(len(ds)) for j in range(5)]  # caption c is of image c // 5
json.dump(captions, open("captions.json", "w"))
for k in M:
    np.save(f"img_{k}.npy", embed_images(k, images).numpy().astype("float32"))
    np.save(f"txt_{k}.npy", embed_texts(k, captions).numpy().astype("float32"))
    print(k, "saved", len(images), "images and", len(captions), "captions")
# step3_search.py: FAISS index, text-to-image search, recall@k. Run in its own process (no torch import).
import json, numpy as np, faiss
captions = json.load(open("captions.json"))
gt = np.arange(len(captions)) // 5                       # the image that each caption describes

def recall_at(ranks, k): return float((ranks < k).mean())

for k in ["clip", "siglip"]:
    img, txt = np.load(f"img_{k}.npy"), np.load(f"txt_{k}.npy")    # already unit length
    index = faiss.IndexFlatIP(img.shape[1])              # exact search, score = inner product = cosine
    index.add(img)
    scores, ids = index.search(txt, img.shape[0])        # every caption ranks all 1,000 images
    rank = np.argmax(ids == gt[:, None], axis=1)         # 0 means the true image came first
    print(f"{k}: text->image R@1 {recall_at(rank, 1):.3f}  R@5 {recall_at(rank, 5):.3f}  R@10 {recall_at(rank, 10):.3f}"
          f"  MRR {np.mean(1 / (rank + 1)):.3f}")

    cap_index = faiss.IndexFlatIP(txt.shape[1]); cap_index.add(txt)
    _, cids = cap_index.search(img, txt.shape[0])        # every image ranks all 5,000 captions
    rank_i = np.argmax((cids // 5) == np.arange(len(img))[:, None], axis=1)   # first correct caption
    print(f"{k}: image->text R@1 {recall_at(rank_i, 1):.3f}  R@5 {recall_at(rank_i, 5):.3f}  R@10 {recall_at(rank_i, 10):.3f}")

    # approximate search: an IVF index over the captions, 64 lists, vary nprobe
    ivf = faiss.IndexIVFFlat(faiss.IndexFlatIP(txt.shape[1]), txt.shape[1], 64, faiss.METRIC_INNER_PRODUCT)
    ivf.train(txt); ivf.add(txt)
    exact = cids[:, :10]
    for nprobe in [1, 4, 16, 64]:
        ivf.nprobe = nprobe
        _, approx = ivf.search(img, 10)
        overlap = np.mean([len(set(a) & set(b)) / 10 for a, b in zip(approx, exact)])
        print(f"{k}: IVF nprobe {nprobe:2d}: {overlap:.3f} of the exact top-10 found")

    # failure slice: queries that contain a count word
    has_count = np.array([any(w in c.lower().split() for w in ["two", "three", "four", "five", "several", "many"])
                          for c in captions])
    print(f"{k}: R@1 with a count word {recall_at(rank[has_count], 1):.3f} ({has_count.sum()} queries),"
          f" without {recall_at(rank[~has_count], 1):.3f}")

Searching all 5,000 captions against 1,000 images took a few hundredths of a second: at this size exact search is the right choice. With 768-dimensional float32 vectors, the SigLIP gallery is \(1{,}000 \times 768 \times 4 = 3.07\) MB. A gallery of 100 million images would be 307 GB, and every query would read all of it. That is when you need Step 6.

Step 5: measure recall@k

For one query, let \(r_q\) be the position of its correct item in the ranked list (1 is best). Recall@k is the fraction of queries whose correct item is in the top \(k\). The mean reciprocal rank (MRR) rewards being near the top more smoothly:

\[ \mathrm{R@}k = \frac{1}{|Q|} \sum_{q \in Q} \mathbb{1}\big[r_q \le k\big], \qquad \mathrm{MRR} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{r_q} \tag{170.6} \]
Read the symbols
\(Q\)
the set of queries: 5,000 captions for text → image, 1,000 images for image → text
\(r_q\)
the rank of the first correct item for query \(q\); for image → text, any of the 5 captions of the image counts as correct
\(\mathbb{1}[\cdot]\)
1 if the condition holds, else 0

Our output:

ModelText → image R@1R@5R@10MRRImage → text R@1R@5R@10
CLIP B/320.5370.8070.8890.6590.7110.9190.956
SigLIP B/160.7280.9210.9660.8150.8480.9760.992

Two remarks. Image → text is “easier” in this protocol because each image has five correct captions and only the first one has to be found, while each caption has exactly one correct image. Always say which direction and which protocol you report. And these are numbers on Flickr8k with 1,000 images; with a gallery of a million images, recall@1 would be much lower, because more look-alike images compete for the top place. Recall@k depends on the gallery size.

CLIP B/32SigLIP B/16random ranking (\(k / 1{,}000\) for text → image)
Figure 4. Measured recall@k curves on the Flickr8k test split (1,000 images, 5,000 captions), exact FAISS search. Each point is the fraction of queries whose correct item is in the top k.
How to read Figure 4

At \(k = 1\), text → image, CLIP finds the right photo for 53.7% of the captions and SigLIP for 72.8%. Slide \(k\) to 5: 80.7% and 92.1%. Slide to 10: 88.9% and 96.6%. The gap between the models shrinks as k grows, because both models usually have the right photo “nearby”: the hard part is putting it first. That is why a two-stage system (fast search for the top 50, then a slow, precise reranker) works so well. Switch to image → text: SigLIP reaches 99.2% at \(k = 10\).

Look at one query where both models fail. The very first caption, “The dogs are in the snow in front of a fence.”, puts its true photo at rank 15 with CLIP and rank 9 with SigLIP. Both models return the same photo first. Many Flickr8k photos show dogs in snow, and the caption does not say anything that separates them. Part of what recall@1 calls an “error” is a correct answer for a vague query: the benchmark has only one labeled match per caption. Keep this in mind when you study failures.

Step 6: approximate search with an IVF index

An inverted file index (IVF) first runs k-means on the stored vectors to find \(n_{\text{list}}\) centroids, and puts each vector into the list of its nearest centroid. A query is compared with all centroids, and then only with the vectors in the \(n_{\text{probe}}\) closest lists. The cost of one query, counted in vector comparisons, is about

\[ \text{cost} \approx n_{\text{list}} + \frac{n_{\text{probe}}}{n_{\text{list}}}\, N \qquad \text{(versus } N \text{ for a flat index)} \tag{170.7} \]
Read the symbols
\(N\)
number of stored vectors
\(n_{\text{list}}\)
number of k-means cells (lists); the first term is the comparison with every centroid
\(n_{\text{probe}}\)
how many of the closest cells are scanned; FAISS’s default is 1
\(n_{\text{probe}} / n_{\text{list}}\)
the fraction of the data scanned, if the lists have about equal size

The price is recall against exact search: a true neighbour that sits in a cell you did not probe is lost. We built an IVF index with 64 lists over the 5,000 caption embeddings and searched with the 1,000 images. The fraction of the exact top-10 that the IVF index also returned:

\(n_{\text{probe}}\) (of 64)1248163264
CLIP0.4430.6240.7730.8930.9630.9961.000
SigLIP0.4930.6630.8050.9050.9680.9951.000

With the default \(n_{\text{probe}} = 1\) the index finds less than half of the true top 10. This is a classic production bug: “we switched to IVF and search got worse”. With 16 of 64 lists (a quarter of the data) it finds 96 to 97%, and with all 64 it equals exact search, as it must. On 5,000 vectors IVF saves nothing worth having; Figure 5 lets you see the mechanism, and (170.7) tells you when it pays: for \(N = 10^8\), \(n_{\text{list}} = 65{,}536\) and \(n_{\text{probe}} = 32\), a query makes about 114,000 comparisons instead of 100 million, 874 times fewer.

querytrue top-10, foundtrue top-10, missedpoints in probed cells
Figure 5. A toy IVF index computed live in your browser: 2,000 seeded random points in 2-D (12 clusters), k-means with \(n_{\text{list}}\) cells (10 iterations), and 200 seeded queries. The readout averages over all 200 queries: the recall of the exact top-10 and the fraction of points scanned. The picture shows one query.
How to read Figure 5

Start with 16 lists and 1 probe: over the 200 queries the recall is 0.928 while only 6.9% of the points are scanned. The example query (the ring) scans only the points of its own cell (highlighted); 2 of its true neighbours fall just across a cell border and are missed (drawn in the other colour). Raise \(n_{\text{probe}}\) to 3: recall 0.999 at 20.2% scanned. Move to 64 lists: each cell is smaller, so one probe scans only 1.8% but the recall falls to 0.790; 4 probes give 0.999 at 7.0% scanned, cheaper than 16 lists for the same recall. Set \(n_{\text{probe}} = n_{\text{list}}\): recall is exactly 1, the same as flat search. In 2-D a few probes are enough; in 512 or 768 dimensions many more neighbours sit near cell borders, which is why the real Flickr index above needed 16 of 64 lists for 96%.

FAISS has many other index types: IndexHNSWFlat (a graph of neighbours, fast and memory-hungry), and IndexIVFPQ (IVF plus product quantization, which compresses each vector to a few bytes). They are compared in session 151. The measurement method stays the same: recall against a flat index on your own queries, plus latency and memory.

Step 7: study the failures

Average metrics hide what a model cannot do. The CLIP paper lists counting, small or unusual text and fine detail among CLIP’s weak points [Certain] (session 166). We first tried to see counting in the retrieval results: split the Flickr8k queries into those with a count word (“two”, “three”, “several”, …) and the rest. The result was the opposite of what we expected: queries with a count word had a higher recall@1 (CLIP 0.584 versus 0.526, SigLIP 0.768 versus 0.719; 935 of the 5,000 queries have a count word). A count word usually comes with a more detailed caption, and the detail helps more than the count hurts. Slicing a benchmark by keywords mixes many effects.

The retrieval failures themselves are still instructive. Some top-1 mistakes with a count word in the query (the second column shows the first caption of the image the model put first):

ModelQueryWhat the top-1 image shows (its first caption)Rank of true image
SigLIPTwo white and brown dogs playing on a beach.The three dogs are standing in the sand.2
SigLIPThree children playing in sand at beachA little girl and boy play in the sand on the beach.2
CLIPTwo men and a woman pose for a picture.A man and two women standing by a wood pole in front of steps.2
CLIPTwo dogs playing in the snow.A big white dog and a small black dog sit in the snow.6

The third row is a binding error: the same words (“two”, “man”, “woman”) attached to different nouns. A single pooled vector struggles to keep “which number goes with which noun”. To measure such skills cleanly, build a controlled probe: a small synthetic test set where only one factor changes. We built four:

  1. Counting: 1 to 6 red circles on white, 20 images per count, prompts “a photo of {n} red circles.”.
  2. Reading small text: one of 10 words printed in black at a random place, font size from 48 down to 6 pixels, prompts “a sign that says "{word}".”.
  3. Small objects: each of the 1,000 CIFAR images shrunk to a side of \(S\) pixels and pasted at a random place on a grey 224 × 224 canvas.
  4. Written words (typographic attack): each CIFAR image with the name of a different, wrong class printed on a white strip at the bottom.
# step4_probes.py: controlled failure probes (counting, small text, small objects, written words)
import random, torch
from PIL import Image, ImageDraw, ImageFont
from datasets import load_dataset
from common import M, embed_images, embed_texts

def zero_shot_acc(k, images, prompts, y):
    return ((embed_images(k, images) @ embed_texts(k, prompts).T).argmax(1) == torch.tensor(y)).float().mean().item()

# 1) counting: n red circles (n = 1..6), 20 images per count
rng = random.Random(0)
def circles(n):
    im = Image.new("RGB", (224, 224), "white"); d = ImageDraw.Draw(im); pts = []
    while len(pts) < n:
        x, y = rng.randint(30, 194), rng.randint(30, 194)
        if all((x - a) ** 2 + (y - b) ** 2 > 50 ** 2 for a, b in pts): pts.append((x, y))
    for x, y in pts: d.ellipse([x - 20, y - 20, x + 20, y + 20], fill=(220, 40, 40))
    return im
WORDS = ["one", "two", "three", "four", "five", "six"]
count_imgs = [circles(n) for n in range(1, 7) for _ in range(20)]
count_y = [n for n in range(6) for _ in range(20)]
count_prompts = ["a photo of one red circle."] + [f"a photo of {w} red circles." for w in WORDS[1:]]

# 2) reading: one of 10 words, printed at a given font size, at a random place
TW = ["apple", "river", "guitar", "pencil", "window", "rocket", "turtle", "basket", "candle", "planet"]
def word_img(w, size, seed):
    r = random.Random(seed); im = Image.new("RGB", (224, 224), "white"); d = ImageDraw.Draw(im)
    f = ImageFont.load_default(size=size); x0, y0, x1, y1 = d.textbbox((0, 0), w, font=f)
    x, y = r.randint(0, max(0, 224 - (x1 - x0))), r.randint(0, max(0, 224 - (y1 - y0)))
    d.text((x - x0, y - y0), w, fill="black", font=f); return im

# 3) small objects: a CIFAR-10 image shrunk to side s and pasted on a grey 224 x 224 canvas
ds = load_dataset("uoft-cs/cifar10", split="test"); names = ds.features["label"].names
keep, seen = [], [0] * 10
for i, c in enumerate(ds["label"]):
    if seen[c] < 100: keep.append(i); seen[c] += 1
ds = ds.select(keep); cy = ds["label"]; cims = [im.convert("RGB") for im in ds["img"]]
def pasted(s, seed=1):
    r = random.Random(seed); out = []
    for im in cims:
        c = Image.new("RGB", (224, 224), (128, 128, 128))
        c.paste(im.resize((s, s), Image.BICUBIC), (r.randint(0, 224 - s), r.randint(0, 224 - s))); out.append(c)
    return out
photo = [f"a photo of a {c}." for c in names]

# 4) typographic attack: write the name of a different class on the image
def labelled():
    out, written = [], []
    for i, im in enumerate(cims):
        c = im.resize((224, 224), Image.BICUBIC); d = ImageDraw.Draw(c); t = (cy[i] + 1 + i % 9) % 10
        d.rectangle([10, 180, 214, 214], fill="white")
        d.text((16, 182), names[t], fill="black", font=ImageFont.load_default(size=28)); out.append(c); written.append(t)
    return out, written

for k in M:
    print(k, "counting accuracy", round(zero_shot_acc(k, count_imgs, count_prompts, count_y), 3))
    for size in [48, 24, 12, 8, 6]:
        ims = [word_img(w, size, 100 * i + j) for i, w in enumerate(TW) for j in range(10)]
        print(k, f"reading, font {size:2d} px", round(zero_shot_acc(k, ims, [f'a sign that says "{w}".' for w in TW],
                                                                   [i for i in range(10) for _ in range(10)]), 3))
    for s in [224, 112, 64, 32, 16]:
        print(k, f"object side {s:3d} px", round(zero_shot_acc(k, pasted(s), photo, cy), 3))
    ims, written = labelled()
    print(k, "with a written wrong label: accuracy", round(zero_shot_acc(k, ims, photo, cy), 3),
          " follows the written word", round(zero_shot_acc(k, ims, photo, written), 3))
Figure 6. Measured results of the four probes. Counting shows the \(6 \times 6\) confusion matrix of the selected model (rows: true count, columns: predicted count; click a cell). The other probes show both models, with the selected one in front. Chance level is 1/6 for counting and 1/10 for the others.
How to read Figure 6

Counting with CLIP: 0.500 accuracy. One circle is always right, but two circles are called “three” 19 times out of 20, and four or five circles are mostly called “six”. SigLIP reaches 0.775, but six circles are always “five”. Small text: both read 12-pixel words perfectly; at 6 pixels CLIP falls to 0.33 and SigLIP keeps 0.91. Small objects: at full size the accuracy is the normal 0.895 and 0.934; at 32 pixels (the native CIFAR size, now a small patch of the canvas) it falls to 0.454 and 0.724, at 16 pixels to 0.236 and 0.303. Written words: CLIP’s accuracy falls from 0.895 to 0.075, and 92.4% of its answers follow the written word; SigLIP falls to 0.273 and follows the word 72.2% of the time.

What the probes tell you, and what they do not:

  • Counting is weak but not absent. Both models are fine for one or two objects (SigLIP) and fall apart beyond that. Captions on the web rarely state exact counts, so the loss rarely needs them [Likely].
  • Resolution limits small things. The image is one 224 × 224 input. A 6-pixel word or a 16-pixel object covers a small part of one patch (32 pixels for CLIP B/32, 16 for SigLIP B/16). SigLIP’s finer patches are one plausible reason it reads smaller text [Likely]; its training data also differs, so this probe alone cannot separate the two causes. VLMs fix this with higher input resolution and tiling (session 180).
  • Text in the image can win over the object. Both models learned to read, and for CLIP the written word outweighs a whole photo almost every time. This is the typographic attack reported for CLIP by Goh et al. [Certain]. Any system that classifies user images (moderation, product categories) must expect it.
  • Limits of our probes. They are small (100 to 1,000 images), synthetic and use one font and one prompt each. They show that a weakness exists and its rough size, not a benchmark score. Change one factor at a time when you build your own.

Where this is used

ApplicationWhat from this lab carries over
Photo-library and stock-image searchAlgorithm 2: embed the library once, FAISS index, text query; recall@k on a labeled query set; rerank the top 50 with a stronger model
E-commerce product categoriesZero-shot classifier from category names (170.1); add a category by writing a sentence; McNemar before shipping a new prompt set; watch for text printed on products
Content moderation and filteringSigmoid scores (170.3) as independent yes/no flags per policy; thresholds tuned per model; typographic attacks as a known risk
Text-to-video retrievalEmbed sampled frames, pool them over time, index one vector per clip (or per frame) and search with text, as CLIP4Clip does (session 208)
Video classification and action recognitionZero-shot labels “a video of a person {action}.” against averaged frame embeddings; works for scenes and objects, weaker for actions that need motion [Likely]
Key-moment detectionScore every frame against a text query (“a goal is scored”), keep the peaks; the frame index is a FAISS index
Video-RAGFrame or clip embeddings as the retrieval index for a VLM that answers questions about long videos (session 210)
Near-duplicate detection and fingerprintingThe same index with image queries; the recall-versus-nprobe trade-off of Step 6 decides cost (session 169, session 209)
Dataset curationSearch a raw image pool with text to find rare cases (“a truck at night in snow”) for labeling, or filter pairs with low image–text cosine

Common mistakes

  • Forgetting to normalize. IndexFlatIP computes inner products, not cosines. If the stored vectors are not unit length, long vectors win every query. Normalize before index.add and normalize every query.
  • Wrong SigLIP text padding. SigLIP was trained with texts padded to 64 tokens. Calling its processor with padding=True (the CLIP habit) changes the text embeddings; always pass padding="max_length".
  • Using one model’s thresholds or probabilities for another. CLIP matches have cosines around 0.2 to 0.3, SigLIP matches around 0.1; CLIP gives softmax probabilities over the label set, SigLIP gives independent sigmoids. Re-tune thresholds whenever you change the model.
  • Mixing embeddings from different models or versions in one index. Vectors from two models live in different spaces; a query from model B against an index built with model A returns garbage without an error. Store the model id with the index.
  • Leaving IVF at the default nprobe = 1. In our run it found only 44 to 49% of the exact top-10. Always measure recall against a flat index and tune nprobe.
  • Calling a 1-point gain an improvement. On 1,000 images the standard error is about 1 point. Use a paired test (McNemar) on per-image results, choose prompts on a validation split, and report the test set once.
  • Reporting recall@k without the protocol. Direction (text → image or image → text), gallery size, and how many correct items each query has all change the number. Our image → text R@1 is 0.848 and text → image 0.728 for the same SigLIP model.
  • Importing torch and faiss in one process on macOS and “fixing” the OpenMP crash with KMP_DUPLICATE_LIB_OK=TRUE. Split embedding and indexing into separate processes instead.

Interview questions

How do you build a zero-shot image classifier from a CLIP-style model?basic
For each class, fill one or more templates (“a photo of a {}.”) with the class name, embed the sentences with the text tower, average the unit embeddings and normalize again. Stack the \(K\) rows into a matrix \(W\). For an image, compute the unit image embedding \(\mathbf{u}\) and predict \(\argmax_k \mathbf{w}_k^\top \mathbf{u}\). The text side is computed once, so each image costs one image-tower pass and a \(K \times d\) product. In our lab this gave 89.5% (CLIP B/32) and 93.4% (SigLIP B/16) on 1,000 CIFAR-10 test images, with no training images.
Why do prompt templates matter, and how do you choose them?core
The text tower was trained on captions, not single words, so a bare class name is unusual input; a template also adds context and can remove ambiguity. Averaging several templates in embedding space cancels the quirks of each one. Choose them on a validation split with a paired test, because the effects are small: in our run the 18-template ensemble beat the single photo template for CLIP by 2.1 points (24 images fixed, 3 broken, \(p = 0.00005\)), but for SigLIP no template beat the photo template significantly. Surprises happen: “a drawing of a {}.” beat “a photo of a {}.” for CLIP on blurry upscaled CIFAR images.
CLIP gives softmax probabilities and SigLIP gives sigmoid probabilities. What is the practical difference?core
A softmax over the \(K\) classes always sums to 1, so it says which class is most likely among the ones you listed, even if none is present. SigLIP’s sigmoid \(\sigma(s \cos + b)\) is an independent “does this text match?” probability per class, so all can be low (none present) or several high (multi-label). For our first CIFAR image SigLIP gave “cat” only 0.045 by sigmoid but 0.980 by softmax over the same logits. Use sigmoids for multi-label and “none of these” decisions, but calibrate them for your data; with CLIP, sigmoids are meaningless because it has no bias (every \(\sigma(100 \cdot 0.2)\) is about 1).
Design text-to-image search for 100 million photos.core
Offline: embed every photo with the image tower (batch jobs on GPUs), L2-normalize, store with the model id. Index: flat float32 at 768 dimensions would be 307 GB and read fully per query, so use an approximate index: IVF with product quantization (compressed codes) or HNSW, sharded across machines. Online: embed the query text, search the top few hundred, optionally rerank with a cross-encoder or VLM. Measure recall@k against exact search on a sample, plus latency and memory, and re-tune nprobe or the graph parameters. Handle new photos with incremental adds, and re-embed everything when the model changes.
What does nprobe do in an IVF index, and what happens if you forget it?basic
IVF clusters the vectors into \(n_{\text{list}}\) cells with k-means; a query scans only the \(n_{\text{probe}}\) cells whose centroids are closest. The cost is about \(n_{\text{list}} + (n_{\text{probe}}/n_{\text{list}}) N\) comparisons. FAISS’s default is 1. With 64 lists over 5,000 caption embeddings, nprobe = 1 found only 44% (CLIP) and 49% (SigLIP) of the exact top-10; nprobe = 16 found 96 to 97%; nprobe = 64 equals exact search. Forgetting to set it is a common cause of “the new index made search worse”.
Define recall@k and MRR. Why is image → text recall higher than text → image on Flickr?basic
Recall@k is the fraction of queries whose correct item appears in the top \(k\); MRR is the mean of \(1/\text{rank}\) of the first correct item. On Flickr each image has 5 correct captions and only the first has to be found, while each caption has exactly one correct image among 1,000. So the image → text task is easier by construction: SigLIP gave R@1 0.848 for image → text and 0.728 for text → image. Always state the direction, the gallery size and the number of correct items per query.
You change the prompt and accuracy goes from 89.5% to 90.7% on 1,000 images. Ship it?core
Not yet. The standard error of each accuracy is about \(\sqrt{0.9 \cdot 0.1 / 1000} \approx 1\) point, but the right test is paired, because both prompts see the same images. Count the disagreements: in our run (“This is a photo of {}.” versus “a photo of a {}.” for CLIP) 26 images were fixed and 14 broken, McNemar \(p = 0.08\), not convincing. Collect more data or test on a separate split. Also check that the template was not chosen on the same test set, and look at per-class changes in the confusion matrix.
Where do CLIP-style models fail, and how would you test for it?core
Known weak points: counting, binding attributes and numbers to the right noun, word order, small text, small objects, and written words in the image that override the object. Average benchmark numbers hide them, and keyword slices can mislead (in our run, queries with count words had higher recall@1 because they were more detailed). Build controlled probes that change one factor: our circle-counting probe gave CLIP 50% and SigLIP 77.5%; at 6-pixel text, 33% and 91%; with a wrong class name written on the image, CLIP followed the written word 92.4% of the time.
Why does SigLIP read small text better than CLIP B/32 in this lab? Can you be sure of the reason?advanced
SigLIP B/16 cuts the 224-pixel image into 196 patches of 16 pixels; CLIP B/32 into 49 patches of 32 pixels. A 6-pixel word covers a smaller fraction of a 32-pixel patch, so finer patches plausibly help. But the two models also differ in training data (WebLI versus WIT), loss, embedding size and text tower, so this experiment cannot isolate patch size. To test it, compare models that differ only in patch size, such as CLIP B/32 against CLIP B/16 from the same training run, on the same probe.
How would you defend a product classifier against typographic attacks?advanced
Measure first: our probe printed a wrong class name on each image and CLIP’s accuracy fell from 89.5% to 7.5%. Mitigations: detect text with an OCR model and treat images with large text regions separately; mask or blur detected text regions before embedding; add training (or fine-tuning) data with misleading text and the correct label; ensemble with a vision-only model trained on labels; and for high-stakes decisions, send uncertain cases to a VLM or a human. Re-run the probe after each change.
Why embed in one process and index in another, and what must you store with the index?advanced
Practically, on macOS torch and faiss-cpu each load their own OpenMP runtime and crashed together in our run. Architecturally, embedding is a batch GPU job and indexing or serving is a CPU or memory-bound service, so they scale differently anyway. Store with the index: the model id and version, the preprocessing and normalization, the text padding rule, the embedding size, the index type and its parameters (nlist, nprobe), and a table from FAISS row numbers to item ids. A query embedded with a different model silently returns wrong results.

Exercises

  1. SigLIP B/16 has \(s = 117.33\) and \(b = -12.93\). At what cosine does its sigmoid probability reach 0.5? What probability does it give at cosine 0.0842?
  2. What is the standard error of an accuracy of 0.934 measured on 1,000 images, and the approximate 95% interval?
  3. Two prompts are compared on the same images: prompt A alone is right on 3 images, prompt B alone on 24. Compute the McNemar statistic with continuity correction, \((|n_{10} - n_{01}| - 1)^2 / (n_{10} + n_{01})\). Is it above the 5% critical value 3.84?
  4. Five queries have their correct item at ranks 1, 3, 1, 12 and 2. Compute R@1, R@5, R@10 and MRR.
  5. A flat index stores 100 million 768-dimensional float32 vectors. How much memory is that? With IVF, \(n_{\text{list}} = 65{,}536\) and \(n_{\text{probe}} = 32\), about how many comparisons does one query need?
  6. Run Step 1 with your own dataset of 5 classes and 3 new templates. Which result tells you more: the total accuracy or one row of the confusion matrix? Write down one change you would make to the class names after looking at it.
Answerscheck
1. The probability is 0.5 when \(117.33 \cos - 12.93 = 0\), so \(\cos = 12.93 / 117.33 = 0.110\). At 0.0842 the logit is \(-3.06\) and \(\sigma(-3.06) = 0.045\), the cat of the worked example. 2. \(\sqrt{0.934 \times 0.066 / 1000} = 0.0079\); the 95% interval is about \(0.934 \pm 0.015\), so roughly 91.9% to 94.9%. 3. \((|3 - 24| - 1)^2 / 27 = 400 / 27 = 14.8\), far above 3.84: the difference is real (the exact test in Step 3 gave \(p = 0.00005\)). 4. \(\text{R@}1 = 2/5 = 0.4\); \(\text{R@}5 = 4/5 = 0.8\) (ranks 1, 3, 1, 2); \(\text{R@}10 = 0.8\) (rank 12 is still missing); \(\text{MRR} =\) \((1 + 1/3 + 1 + 1/12 + 1/2)/5 = 2.917/5 = 0.583\). 5. \(10^8 \times 768 \times 4\) bytes = 307.2 GB. IVF: \(65{,}536 + (32/65{,}536) \times 10^8 = 65{,}536 + 48{,}828 \approx 114{,}000\) comparisons, about 874 times fewer than 100 million. 6. Open-ended. Usually one row tells you more: a class that is mostly confused with one other class points to an ambiguous or unusual name (like “automobile” or “crane”) or to a visual similarity you can fix with a more specific template.

Key takeaways

  • A zero-shot classifier is a matrix of normalized, template-averaged text embeddings (170.1). On 1,000 CIFAR-10 test images it gave 89.5% (CLIP B/32) and 93.4% (SigLIP B/16), with no training images.
  • Prompt effects are real but small: for CLIP the 18-template ensemble added 2.1 points (24 fixed, 3 broken); for SigLIP no template beat “a photo of a {}.” significantly; a bare name was worse for both. Decide with a paired test (170.5), not the raw difference.
  • CLIP’s natural confidence is a softmax over the label set (170.2); SigLIP’s is an independent sigmoid with scale 117.33 and bias −12.93 (170.3). Their cosines and probabilities are on different scales.
  • Text-to-image search is Algorithm 2: embed the gallery once, a FAISS index, embed the query. On Flickr8k, text → image R@1 was 0.537 (CLIP) and 0.728 (SigLIP); recall@k (170.6) depends on direction and gallery size.
  • IVF trades recall for speed (170.7). With the default nprobe = 1 it found under half of the exact top-10; always measure against a flat index.
  • Controlled probes reveal what averages hide: counting (50% and 77.5%), 6-pixel text (33% and 91%), 16-pixel objects (24% and 30%), and written words, which CLIP followed 92.4% of the time.

Key formulas

\[ \mathbf{w}_k = \frac{\sum_m \mathbf{t}_{k,m}}{\lVert \sum_m \mathbf{t}_{k,m} \rVert}, \qquad \hat{y} = \argmax_k \mathbf{w}_k^\top \mathbf{u} \]
\[ p_{\text{CLIP}} = \softmax_k\big(s\, \mathbf{w}_k^\top \mathbf{u}\big), \qquad p_{\text{SigLIP}} = \sigma\big(s\, \mathbf{w}_k^\top \mathbf{u} + b\big) \]
\[ \mathrm{R@}k = \tfrac{1}{|Q|} \textstyle\sum_q \mathbb{1}[r_q \le k], \qquad \mathrm{SE} = \sqrt{\hat{a}(1 - \hat{a})/n}, \qquad \text{cost}_{\text{IVF}} \approx n_{\text{list}} + \tfrac{n_{\text{probe}}}{n_{\text{list}}} N \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  2. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  3. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  4. Meta (code). FAISS library
  5. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  6. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  7. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  8. Meta (code). FAISS library
  9. Johnson et al. (2017). Billion-scale similarity search with GPUs
  10. A. Krizhevsky (2009). Learning Multiple Layers of Features from Tiny Images (CIFAR-10 technical report)
  11. Hodosh, Young & Hockenmaier (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Journal of Artificial Intelligence Research)
  12. OpenAI (code). CLIP repository: prompt templates per dataset (data/prompts.md)
  13. Hugging Face (docs). Transformers documentation: SigLIP
  14. Goh et al. (2021). Multimodal Neurons in Artificial Neural Networks (Distill)
  15. Q. McNemar (1947). Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages (Psychometrika)
  16. Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
  17. Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
  18. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval