LLM & VLM courseBlog
171Track 7: Vision and VLMs · session 9 of 38

compare vision features with linear probes

In this lab you run three frozen image encoders, DINOv2, CLIP and SigLIP 2, on the same CIFAR-10 images and measure what their features know. You train linear probes the right way, compare accuracy from 1 to 500 labels per class and against zero-shot, test whether the gaps are real, measure the speed, and probe the patch tokens on a simple localization task. Then you decide which encoder you would put in a VLM.

Time: 3 to 4 hours (reading 1.5 hours; the full script runs in about 10 minutes on a laptop GPU)Level: Intermediate · hands-on labBefore this: Sessions 164 to 167 (ViT, self-supervised vision, CLIP, SigLIP) and the zero-shot lab (170); Python and PyTorch

What you will learn

  • Which output vector to take from each encoder: DINOv2’s CLS, CLIP’s CLS before the projection, SigLIP’s attention-pooled output
  • How to extract and cache frozen features once, with each model’s own normalization and no center crop for dense tasks
  • How to train a linear probe properly: standardize, sweep the L2 strength on a validation split, test once
  • How accuracy grows from 1 to 500 labels per class, and how many labels zero-shot is worth
  • How to tell a real gap from noise with the paired bootstrap on the same test images
  • How patch size sets the token count and the compute per image (about 46 against 35 GFLOPs)
  • How to probe dense patch tokens on a localization task, and why DINOv2 leads there by a wide margin
  • How to argue for an encoder choice in a VLM: SigLIP 2 by default, fused with DINOv2 when spatial detail matters

Why this matters

Every vision-language model starts with one decision: which image encoder do we use? The encoder turns pixels into vectors, and the language model can only reason about what those vectors contain. In this lab you take three famous frozen encoders, DINOv2, CLIP and SigLIP 2, run them on the same images, and measure what their features know with the simplest possible test: a linear probe.

A linear probe is one linear layer trained on top of a frozen encoder. The encoder does not change. If a single matrix can read the class of the image from the features, the information is there and it is easy to get out. Because the probe is so small, it is cheap (it trains in well under a second here), it has almost no hyper-parameters, and it compares encoders fairly: all three get exactly the same reader. This is the standard way papers report feature quality; both the CLIP paper and the DINOv2 paper do it [Certain].

You met the three encoders in the last sessions. DINOv2 learned from images alone, by matching a teacher network across crops and masked patches (session 165). CLIP learned to match images with their captions using a softmax contrastive loss (session 166). SigLIP 2 uses a sigmoid loss instead, plus captioning and self-distillation losses to improve its patch features (session 167). In the previous lab you used CLIP and SigLIP without any labels at all (session 170). Here you add a few labels and see how far each encoder gets.

You will:

  1. Look inside each encoder: how many tokens it makes, and which vector you should take as “the image feature” (Figure 1).
  2. Extract features once for 5,000 training and 2,000 test images of CIFAR-10, and cache them.
  3. Train linear probes the right way: standardize, sweep the regularization on a validation split, and test once (Figure 2).
  4. Compare accuracy with 1 to 500 labels per class, against zero-shot, and test whether the differences are real (Figure 3).
  5. Measure speed: compute per image, images per second, and probe training time.
  6. Probe the patch tokens on a simple localization task: which part of the image shows which object? (Figure 4.)
  7. Decide which encoder you would put in a VLM, and why (Figure 5).

We ran every script on this page ourselves, on an Apple M5 Pro laptop (PyTorch MPS backend), and the numbers in the tables are our measurements. Your numbers on a CUDA GPU will match closely for accuracy and will differ for speed.

Setup

Hardware. Any machine with 16 GB of memory. One NVIDIA GPU should make the feature extraction take about a minute per encoder [Likely]; on an Apple laptop it took us a few minutes per encoder; on a CPU it takes longer but works. The probes themselves run on the CPU in seconds.

pip install torch transformers datasets numpy

We used PyTorch 2.15 and transformers 5.23. The CLIP checkpoint on the Hub has only a pytorch_model.bin file (no safetensors), which recent transformers versions still load [Certain].

The three encoders. We take the base size of each (ViT-B: 12 layers, width 768), so the comparison is about the training method, not the model size. We counted the parameters of each vision tower by loading it.

Encoder (Hub id)Trained withPatch, tokens at 224 pxVision parametersGlobal feature we probe
DINOv2 ViT-B/14
facebook/dinov2-base
Images only: self-distillation (DINO + iBOT), 142M curated images14 px, \(16 \times 16 = 256\) patches + 1 CLS86.58MCLS token after the final LayerNorm (768)
CLIP ViT-B/16
openai/clip-vit-base-patch16
400M image–text pairs, softmax contrastive loss16 px, \(14 \times 14 = 196\) patches + 1 CLS85.80M (+0.39M projection)CLS after post_layernorm (768), before the projection to 512
SigLIP 2 ViT-B/16
google/siglip2-base-patch16-224
Image–text pairs, sigmoid loss + captioning + self-distillation and masked prediction16 px, \(14 \times 14 = 196\) patches, no CLS92.88M (7.09M of it is the pooling head)Output of the attention-pooling head (768)

DINOv2’s official recipe trains on images of 518 px, but the model works at any size that is a multiple of 14; at 224 px it makes 256 patches [Certain]. SigLIP 2 adds the captioning decoder (LocCa) and, for the last 20% of training, self-distillation and masked-prediction losses; the paper says these improve localization and dense features [Certain].

The data. CIFAR-10 (uoft-cs/cifar10 on the Hub): 50,000 training and 10,000 test images of 32 × 32 pixels in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). We draw a balanced subset: 500 images per class for training (5,000) and 200 per class for testing (2,000), with fixed seeds. CIFAR-10 is small and easy, which is the point: it runs on a laptop, and the differences that remain are still informative. Every image is resized to 224 × 224 with bicubic interpolation before it enters an encoder.

The table you will fill. Our results are in Step 4; copy the empty version into your notes and fill it with your own run.

EncoderProbe, 500 labels per classProbe, 16 labels per classZero-shotPatch accuracy (localization)GFLOPs per imageImages per second
DINOv2 B/14not possible (no text tower)
CLIP B/16
SigLIP 2 B/16

Step 1: what is “the feature” of an image?

A ViT cuts the image into square patches, turns each patch into a token, and runs a transformer over all of them (session 164). At the output you have one vector per token. The number of patch tokens is

\[ N = \left(\frac{H}{p}\right)^{2}, \qquad N_{224,\,p=14} = 16^2 = 256, \qquad N_{224,\,p=16} = 14^2 = 196 \tag{171.1} \]
Read the symbols
\(H\)
the image side in pixels (224 here)
\(p\)
the patch side in pixels: 14 for DINOv2, 16 for CLIP and SigLIP 2
\(N\)
the number of patch tokens; DINOv2 and CLIP add one extra CLS token

For classification we need one vector per image. Call the output tokens \(\mathbf{h}_1, \dots, \mathbf{h}_N \in \R^{d}\) with \(d = 768\). There are three common ways to pool them:

\[ \mathbf{z}_{\text{cls}} = \mathbf{h}_{\text{CLS}}, \qquad \mathbf{z}_{\text{mean}} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{h}_i, \qquad \mathbf{z}_{\text{map}} = \operatorname{MLP}\!\Big(\sum_{i=1}^{N} \alpha_i\, W_V \mathbf{h}_i\Big),\ \ \alpha = \softmax\!\Big(\frac{(W_Q\mathbf{q})^\top W_K \mathbf{h}_i}{\sqrt{d}}\Big)_{i} \tag{171.2} \]
Read the symbols
\(\mathbf{h}_{\text{CLS}}\)
the output of the extra “class” token that DINOv2 and CLIP were trained to use as the image summary
\(\mathbf{z}_{\text{mean}}\)
the plain average of the patch tokens
\(\mathbf{q}\)
one learned query vector (a “probe” token) in SigLIP’s multihead attention pooling (MAP) head; it attends to all patches
\(W_Q, W_K, W_V\)
the head’s projections (the real head has 12 attention heads; one is shown); its output goes through an MLP

Use the pooling the model was trained with. CLIP’s loss only ever saw the CLS token (after a LayerNorm and a projection), so the CLS is where its knowledge is collected. SigLIP 2 has no CLS token at all; its loss saw the MAP output. DINOv2 trains both its CLS token (the DINO loss) and its patch tokens (the iBOT loss), and its paper reports linear probes on the CLS token, optionally concatenated with the mean of the patch tokens [Certain]. For CLIP, the CLIP paper probes the features before the final projection to 512 [Certain]. Figure 1 shows the three paths.

patch tokensCLS token or pooling querythe global feature we probe
Figure 1. The same 224 × 224 image in three encoders. Left: the patch grid (formula 171.1). Middle: the output tokens. Right: how one global vector is made (formula 171.2). The compute numbers come from formula (171.8) in Step 5.
How to read Figure 1

Switch between the encoders. DINOv2 uses smaller patches (14 px), so at the same image size it makes 256 patch tokens instead of 196: a finer grid for dense tasks, and about 32% more compute. CLIP has a CLS token that collects the summary; SigLIP 2 has none and uses a small attention-pooling head instead. All three give a 768-number global vector, so the probes in Step 3 have the same size: \(768 \times 10 + 10 = 7{,}690\) weights.

Step 2: extract the features once

The encoders are frozen, so their outputs never change. Compute them once, store them on disk, and run every probe from the cache. A probe then costs a few small matrix products instead of a pass through an 86M-parameter network. This is also how the DINOv2 authors make their probe sweep cheap: they run the backbone once and feed the output to many linear classifiers [Certain].

Algorithm 1: extract and cache frozen features
  1. input: images \(x_1, \dots, x_n\), encoder \(f\) with its own mean and standard deviation for normalization
  2. for each batch of images do
  3.   resize to \(224 \times 224\) (bicubic), scale to \([0, 1]\), subtract the model’s mean \(\mu\), divide by its std \(\sigma\) // each model has its own values
  4.   \(\mathbf{h} \leftarrow f(x)\) with gradients off and the model in eval() mode
  5.   keep the global feature (formula 171.2) in FP32 and the patch tokens in FP16
  6. end for
  7. save the arrays together with the labels and the exact image order

Three details matter more than they look.

  • Normalization differs. DINOv2 uses the ImageNet mean and std (0.485, 0.456, 0.406 / 0.229, 0.224, 0.225), CLIP its own (0.481, 0.458, 0.408 / 0.269, 0.261, 0.276), and SigLIP 2 maps pixels to \([-1, 1]\) (mean and std 0.5). Feeding one model’s normalization to another costs accuracy quietly, with no error message.
  • Resize, then do not crop. The DINOv2 image processor on the Hub resizes the short side to 256 and then crops the center 224 [Certain]. For classification that is fine. For the localization task in Step 6 a crop would cut off the borders of the image, so we resize directly to 224 for all three models and skip the processors.
  • Know which layer you read. In Hugging Face, DINOv2’s last_hidden_state already has the final LayerNorm applied, CLIP’s does not (its post_layernorm is applied only to the pooled CLS), and SigLIP’s does. We apply CLIP’s post_layernorm to its patch tokens ourselves so all three are treated the same.
Worked example: how big is the cache?

Global features: \(7{,}000 \times 768 \times 4\ \text{bytes} = 21.5\ \text{MB}\) per encoder. Patch tokens for the 1,300 localization images of Step 6: DINOv2 has \(1{,}300 \times 256 \times 768 = 255.6\) million numbers, 511 MB in FP16; CLIP and SigLIP 2 have \(196\) patches, so 391 MB each. Patch features are large: for a real dataset you store them only when you need them, or you reduce them first.

Step 3: the linear probe, done properly

The probe is multinomial logistic regression: a linear layer followed by softmax, trained with cross-entropy (session 3). First standardize every feature dimension with statistics from the training set only:

\[ \tilde{\mathbf{z}} = \frac{\mathbf{z} - \mu_{\text{train}}}{\sigma_{\text{train}}} \tag{171.3} \]

Then fit the weights \(W \in \R^{d \times C}\) and bias \(\mathbf{b} \in \R^{C}\) by minimizing the regularized loss

\[ \mathcal{L}(W, \mathbf{b}) = -\frac{1}{n}\sum_{j=1}^{n} \log \softmax\!\big(W^\top \tilde{\mathbf{z}}_j + \mathbf{b}\big)_{y_j} \;+\; \lambda \lVert W \rVert_F^2 \tag{171.4} \]
Read the symbols
\(\tilde{\mathbf{z}}_j\)
the standardized feature of training image \(j\) (formula 171.3)
\(y_j\)
its class, one of \(C = 10\)
\(\lambda\)
the L2 regularization strength: larger values keep the weights small and the probe simple
\(\lVert W \rVert_F^2\)
the sum of all squared weights

The loss (171.4) is convex: it has a single minimum and no bad local ones, so the result does not depend on the random seed or the starting point. We start from zero weights and use L-BFGS, a quasi-Newton optimizer that reaches the minimum in a few hundred steps on the full batch. The CLIP paper does the same with scikit-learn’s L-BFGS [Certain].

The only hyper-parameter is \(\lambda\). Never choose it on the test set. Hold out part of the training data as a validation split, try a grid of values, and keep the best:

\[ \lambda^\star = \argmax_{\lambda \in \{10^{-5}, 10^{-4}, \dots, 1\}} \ \operatorname{acc}_{\text{val}}\big(W_\lambda, \mathbf{b}_\lambda\big) \tag{171.5} \]

Then refit on all training data with \(\lambda^\star\) and evaluate on the test set once. The CLIP paper sweeps 96 values between \(10^{-6}\) and \(10^{6}\) with a binary search [Certain]; six values, one per decade, are enough for us. The vectors \(\mu_{\text{train}}\) and \(\sigma_{\text{train}}\) in (171.3) hold one mean and one standard deviation per feature dimension.

Algorithm 2: a linear probe with a validation sweep
  1. input: cached features \(Z_{\text{tr}}, Z_{\text{te}}\), labels, grid of \(\lambda\) values
  2. split the training rows into fit (80%) and validation (20%) // fixed seed
  3. for each \(\lambda\) in the grid do
  4.   standardize with the fit rows’ mean and std (171.3); minimize (171.4) on the fit rows with L-BFGS
  5.   record the validation accuracy
  6. end for
  7. \(\lambda^\star \leftarrow\) the best value (171.5); refit on all training rows, with new statistics
  8. return the test accuracy, computed one time

Why standardize? The 768 dimensions of these features have very different scales; for example, ViTs often have a few dimensions with huge values. Without standardization the same \(\lambda\) punishes some dimensions far more than others, and the optimizer converges slowly. With it, one \(\lambda\) grid works for all three encoders. Figure 2 lets you feel what \(\lambda\) and the number of labels do.

Figure 2. A linear probe on toy 2D “features” of three classes. Filled dots are the labeled training points, small rings are 300 test points. The colored regions are the probe’s decisions. The probe is trained live with formula (171.4), 300 full-batch gradient steps.
How to read Figure 2

With “Well separated”, even 1 label per class gives a high test accuracy: good features make probes easy. Switch to “Overlapping” and set 5 labels per class: with \(\lambda = 10^{-5}\) the boundaries chase the few points and the test accuracy is 73.7%; with \(\lambda = 10^{-1}\) it is 77.7%. Now set 60 labels: every \(\lambda\) from \(10^{-5}\) to \(10^{-1}\) gives 80.0%, so the choice hardly matters when labels are many. “Not linear” puts one class in a ring around another: no straight line separates them, so even 60 labels leave a big error. A linear probe measures what is linearly readable; a feature that needs a curved boundary scores low even if the information is there.

Step 4: the results, and are the differences real?

We probed several global features for each encoder, each time with 5,000 training labels and the \(\lambda\) chosen on validation. Our measurements on the 2,000 test images:

EncoderFeatureTest accuracy\(\lambda^\star\)
DINOv2 B/14CLS98.5%\(10^{-5}\)
CLS and patch mean, concatenated (1,536)98.35%\(10^{-3}\)
CLIP B/16CLS after post_layernorm (768)95.75%\(10^{-2}\)
Projected embedding (512)95.25%\(10^{-2}\)
SigLIP 2 B/16MAP head output96.5%\(10^{-1}\)
Mean of the patch tokens96.4%\(10^{-5}\)

The order is DINOv2 first, then SigLIP 2, then CLIP. These numbers agree with the papers. The DINOv2 paper reports 98.7% for its ViT-B/14 on the full CIFAR-10 (50,000 training images) [Certain], and the CLIP paper reports 96.2% for CLIP ViT-B/16 [Certain]. We used only 10% of the training labels and get within about half a point of both. For CLIP, the feature before the projection is better than the 512-number embedding, as the CLIP paper suggests: the projection was trained only to match text, and it throws some visual detail away.

How much noise is in a test accuracy?

A test set of \(n\) images gives an accuracy \(\hat{p}\) with a standard error of

\[ \operatorname{SE}(\hat{p}) = \sqrt{\frac{\hat{p}\,(1 - \hat{p})}{n}} \tag{171.6} \]

But to compare two encoders, use the fact that they saw the same test images. Only the images where they disagree carry information. Let \(r^{A}_j, r^{B}_j \in \{0, 1\}\) say whether each encoder got image \(j\) right. The paired difference and its bootstrap interval are

\[ \hat{\Delta} = \frac{1}{n}\sum_{j=1}^{n} \big(r^{A}_j - r^{B}_j\big), \qquad \text{CI}_{95} = \text{2.5th to 97.5th percentile of } \hat{\Delta} \text{ over resampled test sets} \tag{171.7} \]
Read the symbols
\(\hat{p}\)
the measured test accuracy
\(n\)
the number of test images (2,000)
\(r^{A}_j\)
1 if encoder A classified image \(j\) correctly, else 0
resampled test sets
draw \(n\) images with replacement, 2,000 times, and recompute \(\hat{\Delta}\) on each draw (the paired bootstrap)
Worked example: which gaps are real?

One accuracy: DINOv2 has \(\hat{p} = 0.985\), so \(\operatorname{SE} = \sqrt{0.985 \times 0.015 / 2000} = 0.0027\), or \(\pm 1.96 \times 0.0027 = \pm 0.53\) points for a 95% interval. CLIP has \(\operatorname{SE} = 0.0045\) (\(\pm 0.88\) points).

DINOv2 against CLIP: DINOv2 alone is right on 72 images and CLIP alone on 17. The difference is \((72 - 17) / 2000 = 2.75\) points, with a bootstrap interval of 1.85 to 3.65 points. It is real. DINOv2 against SigLIP 2: 58 against 18 images, \(2\) points, interval 1.20 to 2.90: also real.

SigLIP 2 against CLIP: 54 against 39 images, \(0.75\) points, interval −0.10 to 1.75. The interval contains zero, so on this test set we cannot say that SigLIP 2’s probe beats CLIP’s. Do not write “SigLIP 2 is better” in a report from this run alone.

Few labels, and zero labels

In practice you often have only a handful of labels per class. We repeated the probe with 1, 2, 4, …, 128 labels per class (five random draws each, from a pool of 4,000 training images; \(\lambda\) still chosen on the 1,000 validation images) and 400 per class (the whole pool). We also ran the zero-shot classifier of the last lab, with the single prompt “a photo of a {label}.”: CLIP reaches 89.9% and SigLIP 2 95.05%, with no labels at all. The CLIP paper reports 91.6% zero-shot for B/16 on the full test set with an ensemble of prompts [Certain]; our single prompt is a little lower.

DINOv2 B/14CLIP B/16SigLIP 2 B/16dashed: zero-shot
Figure 3. Our measured test accuracy (2,000 CIFAR-10 test images) of a linear probe on each frozen encoder, against the number of labeled training images per class (log scale). Each point is the mean of five random draws (one draw at 400); the thin bars show ± one standard deviation. Dashed lines: zero-shot with one prompt.
How to read Figure 3

Move the slider. With 1 label per class SigLIP 2 is ahead; its features were trained to line up with words, so one example per class already lands near the right group. From about 4 labels per class DINOv2 catches up and then leads, by about 1.3 to 2 points over SigLIP 2 at 16 to 128 labels. CLIP’s zero-shot accuracy equals a probe with somewhere between 4 and 8 labels per class; SigLIP 2’s zero-shot equals a probe with about 16. If your classes can be named in words, zero-shot is a strong free baseline; if they cannot (a defect type, a specific product), only the probe helps.

Step 5: speed

Accuracy is half the story. In a VLM, the encoder runs on every image, and in video on every frame. The cost of a ViT forward pass is mostly the linear layers, plus attention, which grows with the square of the number of tokens. Per layer and per token, the attention projections cost \(2 \cdot 4d^2\) FLOPs and the MLP (width \(4d\)) costs \(2 \cdot 8d^2\), so

\[ \text{FLOPs per image} \approx L\,\big(24\,N d^{2} + 4\,N^{2} d\big) \tag{171.8} \]
Read the symbols
\(L\)
the number of layers (12)
\(N\)
the number of tokens, including the CLS token if there is one
\(d\)
the width (768)
\(4N^2 d\)
the two attention matrix products \(QK^\top\) and \(AV\)
Worked example: why DINOv2 costs about a third more

CLIP, \(N = 197\): linear part \(12 \times 24 \times 197 \times 768^2 = 33.46\) GFLOPs, attention \(12 \times 4 \times 197^2 \times 768 = 1.43\) GFLOPs. With the patch embedding (0.23 GFLOPs) the total is 35.1 GFLOPs.

DINOv2 at the same 224 px, \(N = 257\): \(43.66 + 2.43 + 0.23 =\) 46.3 GFLOPs, 1.32 times CLIP. The model has the same size; the cost comes only from the smaller patch (14 instead of 16 px). SigLIP 2 has 196 tokens and a pooling head of about 0.47 GFLOPs: 35.4 GFLOPs.

Our measured throughput (batch 64, FP32, best of three runs of 10 batches, Apple M5 Pro GPU through PyTorch MPS, while other jobs shared the machine):

EncoderTokensGFLOPs per image (171.8)Images per second (measured)Probe fit, \(5{,}000 \times 768\) (seconds, CPU)
DINOv2 B/1425746.3980.20
CLIP B/1619735.11300.18
SigLIP 2 B/1619635.41250.08

Treat our absolute speeds as rough: the machine was shared, and an NVIDIA GPU in BF16 is much faster. The ratio is the useful part. The probe itself is nearly free: under half a second, thousands of times cheaper than extracting the features. That is why you can afford a full \(\lambda\) sweep, many few-shot draws and many encoders.

Step 6: dense features on a simple localization task

A VLM does not read only the global vector. It reads the patch tokens, one per region of the image (session 177). So we must also test whether each patch token knows what is at its own position, not just what the whole image shows.

The task: build a 224 × 224 mosaic from four CIFAR-10 images of four different classes, each resized to 112 × 112, one per quadrant. Then train a linear probe on single patch tokens to predict the class of the quadrant the patch lies in. Both grids align with the quadrants: \(112 / 14 = 8\) and \(112 / 16 = 7\) patches per side. We train on 1,000 mosaics built from training images and test on 300 mosaics built from test images. One probe sees every patch of every mosaic, so DINOv2’s probe has 256,000 training rows and the other two 196,000.

\[ \hat{y}_i = \argmax_{c}\ \big(W^\top \tilde{\mathbf{h}}_i + \mathbf{b}\big)_{c}, \qquad \text{patch acc} = \frac{1}{M N}\sum_{m=1}^{M}\sum_{i=1}^{N} \mathbb{1}\big[\hat{y}_{m,i} = y_{m,i}\big], \qquad \text{IoU}_c = \frac{|\hat{S}_c \cap S_c|}{|\hat{S}_c \cup S_c|} \tag{171.9} \]
Read the symbols
\(\tilde{\mathbf{h}}_i\)
the standardized output token of patch \(i\) (no pooling)
\(y_{m,i}\)
the class of the quadrant that contains patch \(i\) of mosaic \(m\)
\(M, N\)
the number of test mosaics (300) and of patches per mosaic
\(\hat{S}_c, S_c\)
the sets of patches predicted as class \(c\), and truly of class \(c\), over all test mosaics; the mean of \(\text{IoU}_c\) over the 10 classes is the mIoU
Algorithm 3: a patch probe for localization
  1. input: mosaics with quadrant labels, a frozen encoder
  2. run the encoder; keep all \(N\) patch tokens of each mosaic // no pooling
  3. give every patch the label of its quadrant; flatten to one row per patch
  4. standardize (171.3), choose \(\lambda\) on held-out mosaics, fit (171.4) // the same probe as before
  5. predict every test patch; report patch accuracy, accuracy near the quadrant borders, and mIoU (171.9)

Our results on the 300 test mosaics. “Border” patches are the two rows and two columns of patches that touch the lines between quadrants; “interior” patches are all the others.

EncoderGridPatch accuracyInterior patchesBorder patchesmIoU
DINOv2 B/1416 × 1680.7%83.6%71.1%68.0%
CLIP B/1614 × 1466.7%70.3%56.7%50.4%
SigLIP 2 B/1614 × 1470.1%73.3%61.0%54.3%

Now the gap is large: DINOv2 is about 11 points ahead of SigLIP 2 and 14 points ahead of CLIP, compared with 2 to 3 points on whole-image classification. This matches the published picture. In the DINOv2 paper, a linear layer on frozen DINOv2 ViT-g/14 patch tokens reaches 49.0 mIoU on ADE20k, against 39.3 for the much larger OpenCLIP ViT-G/14 [Certain]. CLIP’s loss only rewards the CLS token, so its patch tokens are free to mix information from the whole image. SigLIP 2’s extra local losses help: it beats CLIP here, as its paper reports for segmentation (77.1 against 74.5 mIoU on PASCAL for its So/14 model against CLIP L/14) [Certain]. Every encoder is worst at the borders, where a patch’s attention mixes two neighbors. Figure 4 shows real test mosaics with the real predictions of each probe.

Figure 4. Six of the 300 test mosaics. Left: the image the encoders saw (four CIFAR-10 test images). Right: for each encoder, the class its patch probe predicts for every patch, with wrong patches crossed out. The PCA view instead colors each patch by its first three principal components (computed over all test patches of that encoder), with no labels involved.
How to read Figure 4

Step through the mosaics. DINOv2 has the highest patch accuracy on 5 of these 6 mosaics (on mosaic 1, SigLIP 2 is slightly ahead). Its errors sit mostly near the quadrant borders and in the hardest quadrant, often a look-alike pair such as cat and dog. CLIP’s maps are speckled: single patches inside a quadrant take the class of another quadrant, because CLIP’s patch tokens carry a lot of information from the whole image. In the PCA view, DINOv2 and SigLIP 2 already show the four quadrants as different colors, with no labels at all. CLIP’s map is mostly one color with a few very bright patches: a few tokens with extreme values stretch the color range. Such outlier tokens are typical of large ViTs trained without registers [Likely] (session 165).

Step 7: which encoder would you put in a VLM?

Put the measurements together.

  • Global classification with labels: DINOv2 leads (98.5% against 96.5% and 95.75%). The lead over SigLIP 2 is real but small.
  • Very few labels, or none: SigLIP 2 leads at 1 label per class, and only CLIP and SigLIP 2 can do zero-shot at all. Their features already live in a space shared with text.
  • Dense, local information: DINOv2 leads by a wide margin (80.7% patch accuracy against 70.1% and 66.7%).
  • Speed at the same resolution: CLIP and SigLIP 2 are about 1.3 times cheaper than DINOv2 because of the larger patch.

So which one goes into a VLM? The connector of a VLM (a small MLP in LLaVA, session 178) has to map visual tokens into the LLM’s word space. Features that are already aligned with language make that much easier, and in practice text-trained encoders win as the single encoder. The Prismatic VLMs study found that vision-language contrastive backbones (CLIP, SigLIP) are significantly better than the alternatives as a VLM’s only encoder, and that fusing DINOv2 with SigLIP features gives significant gains across the board; its final models use the fused DINOv2 + SigLIP features [Certain]. The “Eyes Wide Shut” study traced many VLM failures on simple visual questions to CLIP’s blind spots and improved them by mixing in DINOv2 features [Certain]. LLaVA uses a CLIP ViT-L/14, and PaliGemma uses a SigLIP So400m encoder [Certain].

Our answer, which you should defend in your own words: a SigLIP 2 encoder as the default, because its features are aligned with text (good for the connector and for few-shot and zero-shot use), it beats CLIP on dense features, and it is cheaper per token than DINOv2. Add DINOv2 features (concatenated per patch) when the product needs grounding, counting, segmentation or fine spatial detail, and you can pay for a second encoder. Pick DINOv2 alone for vision-only systems with labels: classification heads, retrieval of near-duplicates, segmentation and tracking. These are design judgments based on a small experiment and on the studies above, not laws [Likely]. Figure 5 lets you change the priorities.

Figure 5. A weighted score from our measurements. Each criterion is rescaled so the best encoder gets 1 and the worst gets 0 (DINOv2 gets 0 for zero-shot because it has no text tower). The score is the weighted average. The weights are your priorities, not facts.
How to read Figure 5

The default weights describe a general-purpose VLM: text alignment matters most, dense features next. Set “Zero-shot, text alignment” to 0 and “Dense patch features” to 5: DINOv2 wins, which is the vision-only case. Set the dense weight to 0 and compute to 5: SigLIP 2 wins clearly. CLIP wins only if compute is nearly all you care about, and then only by a hair (35.1 against 35.4 GFLOPs): SigLIP 2 is at least as good on every other measured axis.

The full program

This is the script that produced every number on this page (sections 1 to 5 in one file, about 10 minutes on our laptop). It is deterministic up to tiny floating-point differences between GPU types: fixed seeds for the subsets, mosaics, splits and bootstrap, and a convex probe that starts from zero.

# lab.py: compare frozen DINOv2, CLIP and SigLIP 2 features with linear probes (CIFAR-10).
import json, time, numpy as np, torch, torch.nn.functional as F
from datasets import load_dataset
from transformers import AutoModel, AutoProcessor

torch.manual_seed(0)
dev = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
ds = load_dataset("uoft-cs/cifar10")
CLASSES = ds["train"].features["label"].names

# ---------- 1. data: balanced subsets and 2x2 mosaics ----------
def subset(split, per_class, seed):
    y = np.array(ds[split]["label"]); g = np.random.default_rng(seed)
    idx = g.permutation(np.concatenate([g.permutation(np.where(y == c)[0])[:per_class] for c in range(10)]))
    x = np.stack([np.array(ds[split][int(i)]["img"].convert("RGB")) for i in idx])
    return torch.from_numpy(x).permute(0, 3, 1, 2).contiguous(), torch.from_numpy(y[idx])

def mosaics(X, y, n, seed):                       # four images of four different classes
    g = np.random.default_rng(seed); by = [np.where(y.numpy() == c)[0] for c in range(10)]
    cls = np.stack([g.choice(10, 4, replace=False) for _ in range(n)])
    tiles = torch.stack([torch.stack([X[g.choice(by[c])] for c in row]) for row in cls])
    return tiles, torch.from_numpy(cls)           # (n,4,3,32,32), (n,4): TL, TR, BL, BR

Xtr, ytr = subset("train", 500, 0)                # 5,000 labeled training images
Xte, yte = subset("test", 200, 1)                 # 2,000 test images
Mtr, Mtr_y = mosaics(Xtr, ytr, 1000, 2)
Mte, Mte_y = mosaics(Xte, yte, 300, 3)

def pixels(x, mean, std, size=224):               # our own resize: no center crop
    x = F.interpolate(x.float() / 255, size=(size, size), mode="bicubic", align_corners=False).clamp(0, 1)
    return (x - torch.tensor(mean).view(1, 3, 1, 1)) / torch.tensor(std).view(1, 3, 1, 1)

def mosaic_pixels(M, mean, std):
    t = pixels(M.flatten(0, 1), mean, std, 112).view(-1, 2, 2, 3, 112, 112)
    return torch.cat([torch.cat([t[:, r, 0], t[:, r, 1]], -1) for r in range(2)], -2)

# ---------- 2. three frozen encoders, one feature function ----------
MODELS = {"dinov2": ("facebook/dinov2-base", [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
          "clip": ("openai/clip-vit-base-patch16", [0.48145466, 0.4578275, 0.40821073], [0.26862954, 0.26130258, 0.27577711]),
          "siglip2": ("google/siglip2-base-patch16-224", [0.5] * 3, [0.5] * 3)}

@torch.no_grad()
def encode(name, m, pix):
    if name == "dinov2":
        h = m(pixel_values=pix).last_hidden_state                    # final LayerNorm applied
        return {"global": h[:, 0], "cls+mean": torch.cat([h[:, 0], h[:, 1:].mean(1)], 1), "patch": h[:, 1:]}
    o = m.vision_model(pixel_values=pix)
    if name == "clip":
        h = m.vision_model.post_layernorm(o.last_hidden_state)
        return {"global": o.pooler_output, "proj512": m.visual_projection(o.pooler_output), "patch": h[:, 1:]}
    return {"global": o.pooler_output, "mean": o.last_hidden_state.mean(1), "patch": o.last_hidden_state}

def run_all(name, m, X, mean, std, keys, bs=100):
    out = {}
    for i in range(0, len(X), bs):
        f = encode(name, m, pixels(X[i:i + bs], mean, std).to(dev))
        for k in keys: out.setdefault(k, []).append(f[k].float().cpu())
    return {k: torch.cat(v) for k, v in out.items()}

def sync():
    if dev == "cuda": torch.cuda.synchronize()
    if dev == "mps": torch.mps.synchronize()

R = {"device": dev}; FEAT = {}
for name, (hub, mean, std) in MODELS.items():
    m = AutoModel.from_pretrained(hub).to(dev).eval()
    vis = m if name == "dinov2" else m.vision_model
    keys = [k for k in encode(name, m, pixels(Xtr[:2], mean, std).to(dev)) if k != "patch"]
    FEAT[name] = {"tr": run_all(name, m, Xtr, mean, std, keys), "te": run_all(name, m, Xte, mean, std, keys)}
    FEAT[name]["mtr"] = torch.cat([encode(name, m, mosaic_pixels(Mtr[i:i + 50], mean, std).to(dev))["patch"].half().cpu()
                                   for i in range(0, len(Mtr), 50)])
    FEAT[name]["mte"] = torch.cat([encode(name, m, mosaic_pixels(Mte[i:i + 50], mean, std).to(dev))["patch"].half().cpu()
                                   for i in range(0, len(Mte), 50)])
    pix = pixels(Xtr[:64], mean, std).to(dev)                   # speed: best of 3 runs of 10 batches
    for _ in range(3): encode(name, m, pix)
    best = 1e9
    for _ in range(3):
        sync(); t0 = time.perf_counter()
        for _ in range(10): encode(name, m, pix)
        sync(); best = min(best, time.perf_counter() - t0)
    R[name] = {"params_vision": sum(p.numel() for p in vis.parameters()), "img_per_s": 640 / best,
               "tokens": FEAT[name]["mtr"].shape[1]}
    if name != "dinov2":                                         # zero-shot with one prompt
        proc = AutoProcessor.from_pretrained(hub)
        kw = dict(padding="max_length", max_length=64) if name == "siglip2" else dict(padding=True)
        tok = proc(text=[f"a photo of a {c}." for c in CLASSES], return_tensors="pt", **kw).to(dev)
        with torch.no_grad():
            T = m.get_text_features(**tok)
        T = F.normalize(getattr(T, "pooler_output", T).float().cpu(), dim=-1)
        I = F.normalize(FEAT[name]["te"]["proj512" if name == "clip" else "global"], dim=-1)
        R[name]["zeroshot"] = ((I @ T.T).argmax(1) == yte).float().mean().item()
    print(name, R[name], flush=True)
    del m

# ---------- 3. the linear probe ----------
def fit_probe(X, y, lam, C=10, iters=200):
    W = torch.zeros(X.shape[1], C, requires_grad=True); b = torch.zeros(C, requires_grad=True)
    opt = torch.optim.LBFGS([W, b], lr=1, max_iter=iters, history_size=20, line_search_fn="strong_wolfe")
    def closure():
        opt.zero_grad()
        loss = F.cross_entropy(X @ W + b, y) + lam * (W * W).sum()
        loss.backward(); return loss
    opt.step(closure)
    return W.detach(), b.detach()

def stdz(tr, *rest):                              # statistics from the training split only
    mu, sd = tr.mean(0), tr.std(0) + 1e-6
    return [(t - mu) / sd for t in (tr, *rest)]

LAMS = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1.0]
perm = torch.randperm(5000, generator=torch.Generator().manual_seed(0))
va, pool = perm[:1000], perm[1000:]               # 1,000 for choosing lambda, 4,000 for few-shot draws

def choose_lambda(Ftr, tr_idx):                 # sweep lambda, score on the validation images
    A, V = stdz(Ftr[tr_idx], Ftr[va])
    scores = {}
    for l in LAMS:
        W, b = fit_probe(A, ytr[tr_idx], l)
        scores[l] = ((V @ W + b).argmax(1) == ytr[va]).float().mean().item()
    return max(LAMS, key=scores.get)

for name in MODELS:
    tr_f, te_f = FEAT[name]["tr"], FEAT[name]["te"]
    R[name]["probe"] = {}
    for k in tr_f:                                # every global feature variant, all 5,000 labels
        lam = choose_lambda(tr_f[k], pool)
        A, T = stdz(tr_f[k], te_f[k])
        t0 = time.perf_counter(); W, b = fit_probe(A, ytr, lam); secs = time.perf_counter() - t0
        right = ((T @ W + b).argmax(1) == yte)
        R[name]["probe"][k] = {"acc": right.float().mean().item(), "lam": lam, "fit_s": secs}
        if k == "global": R[name]["right"] = right.int().numpy()
    R[name]["fewshot"] = {}
    for k in [1, 2, 4, 8, 16, 32, 64, 128, 400]:  # labels per class
        accs = []
        for s in range(5 if k < 400 else 1):
            g = np.random.default_rng(100 + s); yp = ytr[pool].numpy()
            idx = pool[torch.from_numpy(np.concatenate([g.permutation(np.where(yp == c)[0])[:k] for c in range(10)]))]
            lam = choose_lambda(tr_f["global"], idx)
            A, T = stdz(tr_f["global"][idx], te_f["global"])
            W, b = fit_probe(A, ytr[idx], lam)
            accs.append(((T @ W + b).argmax(1) == yte).float().mean().item())
        R[name]["fewshot"][k] = [float(np.mean(accs)), float(np.std(accs))]
    print(name, R[name]["probe"], R[name]["fewshot"], flush=True)

# ---------- 4. is the difference real? paired bootstrap ----------
g = np.random.default_rng(0); R["paired"] = {}
for a, b in [("dinov2", "clip"), ("dinov2", "siglip2"), ("siglip2", "clip")]:
    d = R[a]["right"] - R[b]["right"]
    bs = [d[g.integers(0, len(d), len(d))].mean() for _ in range(2000)]
    R["paired"][f"{a}-{b}"] = {"diff": float(d.mean()), "ci95": [float(np.percentile(bs, 2.5)), float(np.percentile(bs, 97.5))],
                               "only_a": int((d == 1).sum()), "only_b": int((d == -1).sum())}
print(R["paired"], flush=True)

# ---------- 5. dense features: label every patch of a mosaic ----------
def quadrant_labels(Y, grid):
    r = torch.arange(grid)
    q = (r[:, None] >= grid // 2).long() * 2 + (r[None, :] >= grid // 2).long()
    return Y[:, q.flatten()]

for name in MODELS:
    Ptr, Pte = FEAT[name]["mtr"].float(), FEAT[name]["mte"].float()
    n, P, d = Ptr.shape; grid = int(P ** 0.5)
    Ltr, Lte = quadrant_labels(Mtr_y, grid).reshape(-1), quadrant_labels(Mte_y, grid)
    Xp, Xq = stdz(Ptr.reshape(-1, d), Pte.reshape(-1, d))
    cut = 800 * P                                 # lambda chosen on the last 200 training mosaics
    scores = {}
    for l in [1e-5, 1e-4, 1e-3]:
        W, b = fit_probe(Xp[:cut], Ltr[:cut], l, iters=150)
        scores[l] = ((Xp[cut:] @ W + b).argmax(1) == Ltr[cut:]).float().mean().item()
    lam = max(scores, key=scores.get)
    W, b = fit_probe(Xp, Ltr, lam, iters=150)
    pred = (Xq @ W + b).argmax(1).view(-1, grid, grid); gt = Lte.view(-1, grid, grid)
    ok = (pred == gt).float(); r = torch.arange(grid)
    border = ((r[:, None] - (grid / 2 - 0.5)).abs() <= 1) | ((r[None, :] - (grid / 2 - 0.5)).abs() <= 1)
    iou = [((pred == c) & (gt == c)).sum().item() / max(1, ((pred == c) | (gt == c)).sum().item()) for c in range(10)]
    R[name]["dense"] = {"grid": grid, "lam": lam, "patch_acc": ok.mean().item(), "border_acc": ok[:, border].mean().item(),
                        "interior_acc": ok[:, ~border].mean().item(), "miou": float(np.mean(iou)), "examples": pred[:6].tolist()}
    # 3-component PCA of the patch tokens, for a picture
    Z = Pte[:6].reshape(-1, d); Z = Z - Pte.reshape(-1, d).mean(0)
    U, S, V = torch.pca_lowrank(Pte.reshape(-1, d) - Pte.reshape(-1, d).mean(0), q=3, center=False)
    pc = Z @ V; pc = (pc - pc.min(0).values) / (pc.max(0).values - pc.min(0).values)
    R[name]["dense"]["pca"] = (pc * 255).round().int().view(6, grid, grid, 3).tolist()
    print(name, {k: v for k, v in R[name]["dense"].items() if k not in ("examples", "pca")}, flush=True)

R["mosaic_truth"] = Mte_y[:6].tolist(); R["mosaic_tiles"] = Mte[:6].tolist()
for name in MODELS: R[name].pop("right")
json.dump(R, open("results.json", "w"))

What to try next: probe the 4th-to-last layer as well as the last one (the DINOv2 protocol also tests the last four layers [Certain]); repeat at 448 px; swap in the SigLIP 2 So400m or DINOv3 checkpoints; replace CIFAR-10 by a fine-grained dataset (flowers, cars, aircraft), where the gaps between encoders are usually larger.

Where this is used

SettingWhat is probed or reusedTypical choice
Choosing the vision tower of a VLMGlobal and patch probes, then a small VLM training run per candidateSigLIP or SigLIP 2 as default; DINOv2 + SigLIP fusion for spatial tasks (Prismatic, Cambrian-1 style studies)
Cheap image classifiers in productionA linear head on cached frozen featuresDINOv2 with labels; CLIP or SigLIP zero-shot as a fallback when labels are missing
Video action recognitionA linear probe on frame features averaged over timeDINOv2’s paper probes 8 frames per clip on UCF-101, Kinetics-400 and Something-Something v2 [Certain]
Video retrieval and Video-RAGText-aligned frame embeddings in a vector indexCLIP or SigLIP frame embeddings; a probe or fine-tune on top for domain labels
Key-moment detection and shot classificationA per-frame linear probe, smoothed over timeWhichever encoder wins the probe on your frames; test it like this lab
Near-duplicate detection and video fingerprintingGlobal features compared by cosine similarityDINOv2 global features (strong instance-level retrieval in its paper)
Segmentation and object tracking in videoPatch tokens with a linear or small decoderDINOv2 or DINOv3 patch features

Common mistakes

  • Choosing \(\lambda\) (or the feature, or the layer) on the test set. With a few configurations you overfit the test set by a point or more. Use a validation split, then test once.
  • Skipping standardization. A few feature dimensions with huge values dominate, L-BFGS converges slowly, and one \(\lambda\) grid no longer fits all encoders.
  • Using the wrong normalization constants. Each encoder has its own mean and std; a mismatch silently costs accuracy.
  • Center-cropping for dense tasks. A crop moves the content against your labels and cuts off image borders. Resize instead, or crop the labels too.
  • Probing the projected CLIP embedding when you want visual quality. The 512-number projection was trained for text matching; the 768-number CLS before it scores higher.
  • Comparing with unequal labels or images. All encoders must see the same training images and the same test images, in the same order, so you can use paired tests.
  • Reading small gaps as wins. With 2,000 test images, a gap under about 1 point between two strong encoders is often noise; check the paired interval (171.7).
  • Judging an encoder only by its global probe. A VLM reads the patch tokens; an encoder can be close on classification and far behind on dense tasks, as in Step 6.

Interview questions

What is a linear probe, and why is it used to compare encoders?basic
A linear probe is a single linear layer (logistic regression) trained on frozen features. The encoder is not changed. It measures how much task information is linearly readable from the features. It is cheap, almost free of hyper-parameters (only the regularization strength), convex so the result is reproducible, and it gives every encoder the same reader, so differences come from the features and not from the head. Papers such as CLIP and DINOv2 report linear probes for this reason.
Why do we standardize features before the probe, and where do the statistics come from?basic
Feature dimensions of a ViT have very different scales, and a few can be huge. Standardizing (subtract the mean, divide by the standard deviation per dimension) makes the L2 penalty treat all dimensions equally and makes optimization fast. The mean and standard deviation must come from the training split only; computing them on the test set leaks test information.
How should you choose the regularization strength of a probe?basic
Sweep a log grid (for example \(10^{-5}\) to 1), fit on part of the training set, and score on a held-out validation split. Keep the best value, refit on all training data, and evaluate on the test set once. With few labels the best value is usually larger, because the probe must not chase a handful of points.
Which output of CLIP, DINOv2 and SigLIP should you probe, and why?core
The one each model was trained to use as its summary. CLIP: the CLS token after the final LayerNorm, before the projection to the shared space (the projection was trained only for text matching). DINOv2: the CLS token, optionally concatenated with the mean of the patch tokens, as in its paper. SigLIP: the output of the attention-pooling (MAP) head, since it has no CLS token. For dense tasks, take the patch tokens of all three.
DINOv2 beats SigLIP 2 by 2 points on a linear probe. Should a VLM use DINOv2?core
Not by that number alone. A VLM’s connector must map visual tokens into the LLM’s word space, and text-aligned features (CLIP, SigLIP) make that easier; studies such as Prismatic found contrastive vision-language backbones better than alternatives as a single encoder, and fused DINOv2 + SigLIP features better still. The probe tests label-supervised classification, not language alignment. Also check dense features, cost per token, resolution support, and finally run a small VLM training with each candidate.
Two probes reach 96.5% and 95.75% on 2,000 test images. Is the first better?core
Not necessarily. The standard error of each is about 0.4 points, but the right test is paired: count images only one model gets right. In our run that was 54 against 39; the paired bootstrap interval for the 0.75-point difference ran from −0.10 to 1.75 points, which includes zero. So the data do not show a real difference. A larger test set or several datasets are needed.
Why are CLIP’s patch tokens weaker for localization than DINOv2’s?core
CLIP’s loss is applied only to the pooled CLS token, so nothing forces each patch token to describe its own region; patch tokens can carry global information mixed through attention. DINOv2 adds a patch-level loss (iBOT: predict the teacher’s tokens for masked patches), which makes each patch token describe its local content. SigLIP 2 adds local losses (captioning with location, self-distillation and masked prediction) and lands in between, as our mosaic test shows.
Why does DINOv2 cost about a third more compute than CLIP B/16 at 224 px, with the same model size?core
Its patches are 14 px instead of 16 px, so it makes \(16^2 = 256\) patches instead of \(14^2 = 196\). Compute is about \(L(24Nd^2 + 4N^2d)\): the linear part grows with \(N\) and attention with \(N^2\). That gives about 46.3 against 35.1 GFLOPs, 1.32 times. In a VLM the extra tokens also cost LLM compute if they are passed on.
When does a zero-shot classifier beat a linear probe?advanced
When labels are very few and the classes can be described well in words. In our run CLIP’s zero-shot accuracy (89.9%) equals a probe with between 4 and 8 labels per class, and SigLIP 2’s (95.05%) equals about 16. With more labels the probe wins, and for classes that are hard to name (defects, specific products, fine-grained species) it wins early. A good compromise is to initialize the probe weights from the text embeddings and then train on the few labels.
A linear probe gives low accuracy on some features. Can you conclude the information is not there?advanced
No. A linear probe tests linear readability only. The information may be present but need a curved boundary (Figure 2, “Not linear”), may sit in a different layer, or may need a different pooling (mean versus CLS versus attention pooling). A small MLP probe or k-nearest-neighbors gives an upper view, but more powerful probes can also learn the task themselves, so always compare against a control such as a random-weight encoder.
How would you run this comparison for a video VLM?advanced
Extract frame features at the frame rate the VLM will use (for example 8 frames per clip), cache them, and probe both global and temporal tasks: a linear probe on time-averaged features for action classes (as in the DINOv2 paper on UCF-101, Kinetics-400 and Something-Something v2), per-frame probes for key moments, and patch probes for tracking or segmentation. Report cost per frame and per second of video, since frames multiply the encoder cost, and test whether differences are real with paired statistics over clips.

Exercises

  1. How many trainable parameters does the probe have for DINOv2 CLS + mean features (1,536 numbers) and 10 classes?
  2. Using formula (171.1), how many patch tokens do DINOv2 B/14 and SigLIP 2 B/16 make at 384 × 384? Which input sizes give an equal number?
  3. Compute the 95% half-width of the accuracy of a probe with 92% accuracy on 500 test images (formula 171.6). Would a gap of 2 points between two such probes be convincing without a paired test?
  4. In Figure 2, choose “Overlapping” and 2 labels per class. Find the \(\lambda\) with the best test accuracy. Then set 60 labels per class. How much does the best \(\lambda\) matter now?
  5. In the mosaic task, a model that always predicts the class of the whole image (if each mosaic had one dominant class) would be wrong on three quadrants. What patch accuracy would a probe reach that always predicts the class of the top-left quadrant, everywhere?
  6. Change the code to probe the mean of DINOv2’s patch tokens alone. Predict first: higher or lower than the CLS? (We measured 97.85% in an earlier run with the same splits.)
Answerscheck
1. \(1{,}536 \times 10 + 10 = 15{,}370\). 2. DINOv2: \(384/14 = 27.4\) is not an integer. The patch convolution drops the remainder and makes \(27^2 = 729\) patches, wasting 6 pixel rows and columns; 378 px gives the same 729 with no waste, and 392 px gives \(28^2 = 784\). SigLIP 2 B/16 at 384: \(24^2 = 576\). Equal counts when \(H/p\) is equal: DINOv2 at 14k px and SigLIP at 16k px, for example 224 (DINOv2) and 256 (SigLIP) both give 256 patches. 3. \(\sqrt{0.92 \times 0.08 / 500} = 0.0121\), so ±2.4 points. A 2-point gap is inside the noise of each single number; only a paired test on the same images could show it is real. 4. With 2 labels per class the test accuracy is 65.3% for \(\lambda \le 10^{-3}\) and 67.7% for \(\lambda = 10^{-1}\): the larger penalty helps a little. With 60 labels it is 80.0% for every \(\lambda\) up to \(10^{-1}\) and 79.7% above: the choice no longer matters. 5. Exactly 25%: one quadrant of four is right. 6. Lower: the CLS is the trained summary; we measured 97.85% for the patch mean against 98.5% for the CLS.

Key takeaways

  • A linear probe (171.4) on frozen, cached, standardized (171.3) features is the cheap, fair and reproducible way to compare encoders; choose \(\lambda\) on validation (171.5) and test once.
  • Probe the vector each model was trained to output: DINOv2’s CLS, CLIP’s CLS before the projection, SigLIP’s attention-pooled output (171.2).
  • On CIFAR-10 with 500 labels per class we measured DINOv2 98.5%, SigLIP 2 96.5%, CLIP 95.75%. Only gaps whose paired interval (171.7) excludes zero are real; SigLIP 2 against CLIP was not.
  • With 1 label per class SigLIP 2 led; zero-shot CLIP and SigLIP 2 equal probes with about 4 to 16 labels per class.
  • On patch-level localization DINOv2 led by 11 to 14 points, the biggest gap in the lab; patch size (14 against 16 px) also costs it about 1.32 times the compute (171.8).
  • For a VLM: a text-aligned encoder such as SigLIP 2 by default, fused with DINOv2 when spatial detail matters.

Key formulas

\[ \tilde{\mathbf{z}} = \frac{\mathbf{z}-\mu}{\sigma}, \quad \mathcal{L} = -\frac{1}{n}\sum_j \log\softmax(W^\top\tilde{\mathbf{z}}_j+\mathbf{b})_{y_j} + \lambda\lVert W\rVert_F^2, \quad \operatorname{SE} = \sqrt{\frac{\hat p(1-\hat p)}{n}}, \quad \text{FLOPs} \approx L(24Nd^2 + 4N^2 d) \]

References for this session

  1. Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision
  2. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  3. Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
  4. Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision
  5. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  6. Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
  7. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  8. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  9. Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  10. Tong et al. (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  11. Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
  12. Krizhevsky (2009). Learning Multiple Layers of Features from Tiny Images
  13. Wolf et al. (2020). Transformers: State-of-the-Art Natural Language Processing