LLM & VLM courseBlog
165Track 7: Vision and VLMs · session 3 of 38

Self-supervised vision: MAE, DINO, DINOv2, DINOv3

Labels are expensive, but images are free. This session shows how a vision transformer learns from images alone: MAE hides most patches and rebuilds the pixels, DINO teaches a student to match a slowly moving teacher on different crops, and DINOv2 and DINOv3 scale this up to give clean features for every patch. You will also learn when to use one global vector and when to use the patch tokens, for VLMs and for video.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Session 164 (the Vision Transformer: patches and tokens) and session 59 (knowledge distillation: soft targets)

What you will learn

  • How MAE masks 75% of patches, why the encoder skips mask tokens, and why this makes training about 3× cheaper
  • Why MAE features are excellent after fine-tuning but weaker when frozen
  • How DINO trains a student against an EMA teacher with multi-crop, and how centering and sharpening prevent collapse
  • What DINOv2 adds: the iBOT patch loss, KoLeo, Sinkhorn-Knopp centering, curated data, distillation, and registers
  • Why dense features degrade in long training, and how DINOv3’s Gram anchoring fixes it
  • When to use global features (retrieval, near-duplicates) and when to use dense patch tokens (segmentation, tracking, VLM input)

Why this matters

Labels are expensive. ImageNet has about 1.3 million labeled training images, but the web has billions of images with no labels at all. Self-supervised learning (SSL) trains a vision encoder on images alone: the image itself gives the training signal. The result is a backbone whose features work for classification, retrieval, segmentation, depth and tracking, often with the backbone frozen.

In session 164 you saw how a ViT turns an image into patch tokens. A ViT trained only with class labels learns what it needs to tell 1,000 classes apart, and little more. This session covers the two main ways to train a ViT without labels, and the four models that every vision engineer is expected to know:

  • MAE (masked autoencoder, 2021): hide most of the patches and rebuild their pixels. Like BERT for images.
  • DINO (2021): a student network learns to give the same output as a teacher network on different views of the same image. The teacher is an average of past students.
  • DINOv2 (2023) and DINOv3 (2025): DINO plus a patch-level loss, curated data at scale, and (in v3) a new loss called Gram anchoring that keeps the per-patch features clean. These are the strongest general-purpose frozen image features today for dense tasks. [Likely]

Why should an LLM or VLM engineer care? Because the vision encoder decides what the language model can see. CLIP-style encoders (next session, session 166) are trained to match captions, and they miss details that captions do not mention. Several VLM studies add DINOv2 features next to CLIP or SigLIP features to improve grounding and spatial questions [Likely]. In video, DINO patch features can track an object across frames with no training at all.

Learning from images alone

A self-supervised method invents a task whose answer is already in the data. This task is called a pretext task. We do not care about the task itself; we care about the features the network must learn to solve it. Two families matter today:

  1. Reconstruct what is missing (masked image modeling). Remove part of the input and predict it. MAE predicts pixels. BEiT and iBOT predict tokens or features instead.
  2. Agree across views (joint embedding). Make two random crops of the same image, with different colors and blur, and ask the network for the same output on both. The network must learn what stays the same: the object, not the crop position or the color. DINO belongs here, as do SimCLR, MoCo and BYOL.

The second family has a famous danger: collapse. If the network outputs the same constant vector for every image, all views agree perfectly and the loss is minimal, but the features are useless. Contrastive methods (SimCLR, MoCo, CLIP) avoid this with negatives: they also push apart views of different images. DINO avoids it without negatives, with two simple tricks we will study in detail.

MAE: hide most patches, rebuild the pixels

The masked autoencoder (He et al., 2021) is BERT’s masked-word idea applied to image patches, with two important changes. First, it hides far more: 75% of the patches by default. Second, the big encoder never sees the hidden patches at all.

The steps, for a 224×224 image and 16×16 patches (so \(14 \times 14 = 196\) patches):

  1. Patchify the image into 196 patches, each with \(16 \cdot 16 \cdot 3 = 768\) pixel values.
  2. Mask: choose 75% of the patches uniformly at random and remove them. 49 visible patches remain.
  3. Encode: a large ViT runs on the 49 visible patches only (each with its position embedding). No mask tokens enter the encoder.
  4. Decode: put the 49 encoded tokens back in their places, fill the 147 empty places with a shared, learned mask token, add position embeddings to all 196, and run a small transformer decoder.
  5. Predict pixels: a linear layer maps each decoder output to 768 numbers, the pixels of that patch. The loss is the mean squared error, computed on the masked patches only.
input, 75% removed 4 visible Encoderlarge ViTvisible tokens only + masktokens all 16 places Decodersmall8 blocks, 512-d predicted pixels Loss: mean squared error on the 12 masked patches only (outlined). After pretraining, the decoder is thrown away.
Figure 1. The MAE design, drawn with 16 patches instead of 196. The encoder is large but sees only the visible 25%. The decoder sees all places but is small. Only the encoder is kept for downstream tasks.

The loss, for one image with the set \(\mathcal{M}\) of masked patch indices, is:

\[ \mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \frac{1}{D_p} \big\lVert \hat{\mathbf{x}}_i - \tilde{\mathbf{x}}_i \big\rVert_2^2 \tag{165.1} \]

The paper also tests a normalized pixel target: each patch is normalized by its own mean and standard deviation before it is used as the target. This makes the model care about local contrast and edges, not the overall brightness of the patch, and it improves the features. [Certain]

\[ \tilde{\mathbf{x}}_i = \frac{\mathbf{x}_i - \mu_i}{\sqrt{\sigma_i^2 + \epsilon}} \tag{165.2} \]
Read the symbols
\(\mathbf{x}_i\)
the true pixels of patch \(i\), flattened to a vector of \(D_p = 16 \cdot 16 \cdot 3 = 768\) numbers
\(\tilde{\mathbf{x}}_i\)
the target: the same pixels, normalized inside the patch (or the raw pixels, in the simpler version)
\(\hat{\mathbf{x}}_i\)
the decoder’s prediction for patch \(i\)
\(\mu_i, \sigma_i^2\)
mean and variance of the 768 pixel values of patch \(i\)
\(\mathcal{M}\)
the masked (hidden) patches; visible patches give no loss
\(\epsilon\)
a small number (for example \(10^{-6}\)) that avoids division by zero on flat patches

Why 75%, when BERT masks only 15%?

Words carry a lot of meaning each; a sentence with 15% missing words is already a hard puzzle. Images are very redundant: a missing patch of sky or grass can be guessed from its neighbours by simple interpolation, with no understanding of the scene. With 75% removed at random, most missing patches have no visible neighbour nearby, and the model must use knowledge of objects and scenes. The MAE paper found that 75% works well for both fine-tuning and linear probing. [Certain] Try it in Figure 2.

Figure 2 runs one MAE training step on a toy image, one line of the algorithm at a time. There is no trained decoder in your browser, so a stand-in plays its part: it fills each hidden patch with the average color of the nearest visible patches. That is the best a model with no understanding of the scene can do, so its loss tells you how hard the task is.

picked for masking (step 2)removed patchrow being scoredhidden patch, outline width = its error \(\ell_i\)
  1. patchify: \(\mathbf{x}_1, \dots, \mathbf{x}_N\), each with \(D_p\) numbers
  2. \(\pi \leftarrow\) random shuffle of the indices; \(\mathcal{M} \leftarrow\) first \(M = rN\) of \(\pi\)
  3. visible \(\leftarrow\) the other \(V = N - M\) patches; drop \(\mathcal{M}\)
  4. \(\mathbf{z} \leftarrow E(\text{visible})\) // cost by (165.3)
  5. put a mask token at every \(i \in \mathcal{M}\); \(\hat{\mathbf{x}} \leftarrow D(\text{all } N)\)
  6. for \(i \in \mathcal{M}\): \(\ell_i \leftarrow \frac{1}{D_p}\lVert \hat{\mathbf{x}}_i - \mathbf{x}_i \rVert^2\); \(S \leftarrow S + \ell_i\)
  7. \(\mathcal{L}_{\text{MAE}} \leftarrow S / |\mathcal{M}|\) // (165.1)
Figure 2. One MAE step, computed in your browser on a 224×224 toy image with \(N = 196\) patches. In this toy each patch has 4×4 color cells, so \(D_p = 48\), and the target is the raw pixels (the simple version of (165.2)). Panel 1: the image and its patches. Panel 2: what the encoder receives. Panel 3: the stand-in decoder output; the loss sweeps over the hidden patches one row at a time. The error is also shown as a fraction of the image variance (1.0 would be no better than a flat gray guess).
How to read Figure 2

Press Play or Step. Step 2 shuffles the patch indices and marks the first \(M\) in red: at 75% with the first mask, \(\pi\) starts with patches #26, #31, #137, #148 (patch #\(i\) sits in row \(\lfloor i/14 \rfloor + 1\) and column \((i \bmod 14) + 1\)). Step 3 removes them, and step 4 puts the visible count into the encoder cost (165.3): 49 tokens cost 24.4% of a full pass. Steps 6 to 19 score one row each: row 1 has 11 hidden patches with \(S = 0.2337\), and the worst patch of the image is in row 5, column 12, at the lower edge of the sun, which the stand-in fills with pure sun yellow (\(\ell = 0.1694\)). The last step gives \(\mathcal{L} = 3.7188 / 147 = 0.0253\), which is 0.432 of the image variance.

Now move the slider. At 25% masking the stand-in error is only 0.197 of the variance: the task is too easy and teaches little. At 90% it is 0.654: now you need to know that a roof sits on a house and grass sits below the sky. Switch to Grid at 75%: the error drops to 0.257, because every hidden patch has a visible neighbour. Block masking (large square holes) goes the other way, to 0.854, because whole objects disappear. The MAE paper saw the same thing: grid masking is an easier task and gave worse features than random masking (in its ablation, linear probing 66.0% vs 73.5%). [Certain] Changing a control keeps the current step, so you can stop on a row and compare.

Why the encoder skips the mask tokens

BERT feeds a [MASK] token into the encoder. MAE does not, for two reasons. First, at test time there are no mask tokens, so an encoder trained on 75% mask tokens would see very different inputs in training and at test time. Second, compute: a transformer block costs about \(24 n d^2\) FLOPs for its linear layers and \(4 n^2 d\) for attention, for \(n\) tokens of width \(d\). With a fraction \(r\) of the \(N\) patches removed:

\[ C_{\text{enc}}(r) \approx L_{\text{enc}} \Big( 24 (1-r) N d^2 + 4 (1-r)^2 N^2 d \Big) \tag{165.3} \]

The linear part shrinks by \(1 - r\) and the attention part by \((1-r)^2\). The decoder is small: 8 blocks of width 512, which is about 9% of the per-token FLOPs of a ViT-L encoder (24 blocks of width 1024). [Certain]

Worked example: the cost of MAE with ViT-L

\(N = 196\), \(r = 0.75\), so the encoder sees \(n = 49\) tokens. For one ViT-L block (\(d = 1024\)): linear layers \(24 \cdot 49 \cdot 1024^2 \approx 1.23 \times 10^9\) FLOPs instead of \(4.93 \times 10^9\); attention \(4 \cdot 49^2 \cdot 1024 \approx 9.8 \times 10^6\) instead of \(1.57 \times 10^8\). The encoder does 24.4% of the work of a full pass.

Decoder per token vs encoder per token, linear parts only: \(\frac{8 \cdot 512^2}{24 \cdot 1024^2} = \frac{1}{12} \approx 8.3\%\), close to the paper’s 9%.

Total pretraining FLOPs (encoder + decoder) compared with an encoder that also processes 147 mask tokens: \(\frac{24\,C_{\text{blk}}(196, 1024) + 8\,C_{\text{blk}}(196, 512)}{24\,C_{\text{blk}}(49, 1024) + 8\,C_{\text{blk}}(196, 512)} \approx 3.29\). The paper reports 3.3× fewer FLOPs and a 2.8× wall-clock speedup for this setting. [Certain]

What MAE features are good at, and what not

MAE features shine after fine-tuning. With only ImageNet-1K images, a ViT-H MAE fine-tuned at 448×448 reached 87.8% top-1 on ImageNet. [Certain] But as frozen features they are weaker: a ViT-L MAE gets 75.8% with a linear probe (a linear classifier trained on frozen features), and in the DINOv2 paper a ViT-H/14 MAE gets only 49.4% with a k-NN classifier, against 83.5% for DINOv2 ViT-g/14. [Certain] The reason: predicting pixels forces the network to keep low-level detail, and the classes are not linearly separated in its last layer. The MAE paper shows that fine-tuning only the last block already lifts accuracy from 73.5% to 81.0%. [Certain]

DINO: a student matches a teacher

DINO (Caron et al., 2021) stands for self-distillation with no labels. In ordinary knowledge distillation (session 59), a big trained teacher gives soft targets to a small student. In DINO there is no trained teacher. Both networks have the same architecture (a ViT backbone plus a small MLP “projection head”), and the teacher is built from the student itself.

One training step:

  1. From one image, make several random crops with color jitter, blur and flips: 2 global crops (resized to 224×224, each covering a large part of the image) and several local crops (96×96, covering small parts). This is called multi-crop.
  2. The teacher sees only the 2 global crops. The student sees all crops.
  3. Each network outputs \(K\) numbers (the paper uses \(K = 65{,}536\)), turned into a probability distribution with a softmax. Think of the \(K\) outputs as soft “pseudo-classes” or prototypes that the model invents.
  4. The loss asks the student’s distribution on every crop to match the teacher’s distribution on a different global crop. A local crop of a dog’s ear must produce the same pseudo-class as the whole dog: “local-to-global” correspondence.
  5. Only the student gets gradients. The teacher’s weights are then updated as an exponential moving average (EMA) of the student’s weights.
imageno label global 1224² global 2224² local 96² Teacherglobal crops only Studentall crops, gets gradients EMA of studentweights, no gradient subtract center csoftmax, τt = 0.04stop-gradient softmax, τs = 0.1K outputs cross-entropyH(Pt, Ps) Pt: sharp targetPs: student guess The loss pairs each teacher global crop with every other crop seen by the student.
Figure 3. DINO self-distillation. Two networks with the same architecture; only the student is trained by gradient descent. The teacher follows the student slowly (EMA), and its output is centered and sharpened before it is used as the target.

The math

Let \(g_{\theta_s}\) be the student and \(g_{\theta_t}\) the teacher. For an input crop \(x\), the student’s probability for prototype \(k\) is a softmax with temperature \(\tau_s\):

\[ P_s(x)^{(k)} = \frac{\exp\big(g_{\theta_s}(x)^{(k)} / \tau_s\big)}{\sum_{j=1}^{K} \exp\big(g_{\theta_s}(x)^{(j)} / \tau_s\big)} \tag{165.4} \]

The teacher’s output first has a center vector \(\mathbf{c}\) subtracted, then goes through a softmax with a much lower temperature \(\tau_t\):

\[ P_t(x) = \softmax\!\left( \frac{g_{\theta_t}(x) - \mathbf{c}}{\tau_t} \right) \tag{165.5} \]

With the set \(V\) of all crops and the two global crops \(x^g_1, x^g_2\), the loss is the cross-entropy \(H(a, b) = -\sum_k a^{(k)} \log b^{(k)}\), summed over teacher–student pairs of different crops:

\[ \mathcal{L}_{\text{DINO}} = \sum_{x \in \{x^g_1, x^g_2\}} \ \sum_{\substack{x' \in V \\ x' \neq x}} H\big(P_t(x),\, P_s(x')\big) \tag{165.6} \]
Read the symbols
\(K\)
number of output dimensions (prototypes); 65,536 in DINO
\(\tau_s, \tau_t\)
student and teacher temperatures; DINO uses \(\tau_s = 0.1\) and warms \(\tau_t\) up from 0.04 to 0.07 over the first 30 epochs [Certain]. A lower temperature gives a sharper (more peaked) distribution
\(\mathbf{c}\)
the center: a running mean of the teacher outputs over batches (formula 165.8)
\(V\)
all crops of one image: 2 global and several local (for example 6 or 10)
\(H(P_t, P_s)\)
cross-entropy; the teacher side is treated as a constant (stop-gradient)

The teacher is an exponential moving average of the student. After every optimizer step on the student:

\[ \theta_t \leftarrow \lambda\, \theta_t + (1 - \lambda)\, \theta_s \tag{165.7} \]

DINO increases \(\lambda\) from 0.996 to 1 with a cosine schedule. [Certain] With \(\lambda = 0.996\), the teacher averages roughly the last \(1/(1-\lambda) = 250\) student versions. The paper notes that this averaging acts like an ensemble of past models, and the teacher is better than the student during training, so it gives good targets. [Certain]

Avoiding collapse: centering and sharpening

There are two ways the system can cheat and output useless features:

  • Collapse to one dimension: the teacher puts all mass on the same prototype for every image. Student and teacher agree perfectly, and the features carry no information.
  • Collapse to uniform: the teacher outputs a flat distribution for every image. Again agreement is easy and useless.

DINO uses two opposite forces. Centering subtracts the running mean \(\mathbf{c}\), so no prototype can win for every image; it prevents one-dimension collapse but pushes toward uniform. Sharpening (a low \(\tau_t\)) makes each teacher output peaked; it prevents uniform collapse but pushes toward one dimension. Together, with the EMA teacher, they balance. [Certain] The center is updated with its own EMA over the batch of \(B\) teacher outputs:

\[ \mathbf{c} \leftarrow m\, \mathbf{c} + (1 - m)\, \frac{1}{B} \sum_{i=1}^{B} g_{\theta_t}(x_i) \tag{165.8} \]

Here \(m\) is a rate close to 1 (0.9 in the official code [Likely]). The paper explains collapse with a split of the cross-entropy:

\[ H(P_t, P_s) = h(P_t) + \KL(P_t \,\Vert\, P_s) \tag{165.9} \]

where \(h(P_t)\) is the entropy of the teacher output. A KL that drops to 0 means student and teacher output a constant: collapse. Without centering, \(h(P_t)\) goes to 0 (one dimension); without sharpening, it goes to \(\log K\) (uniform). [Certain]

Figure 4. A batch of 6 images and a teacher with \(K = 8\) prototypes. Every image’s logits contain a shared bias toward prototype 1 (gray, +0.5) and a weaker signal for its own prototype (+0.3). The figure shows the teacher probabilities \(P_t\) of formula (165.5).
How to read Figure 4

Start at the default (\(\tau_t = 0.04\), centering on): each image is confident (mean entropy 0.097 nats) and the six images use six different prototypes, so the batch-average entropy is 1.834, close to \(\ln 6 = 1.792\) plus a little spread. Turn centering off: the shared bias wins everywhere, all six images pick the gray prototype: one-dimension collapse. Turn centering back on and raise \(\tau_t\) to 1.0: each distribution becomes almost flat (entropy 2.074, the maximum is \(\ln 8 = 2.079\)): uniform collapse. Only both tricks together keep the outputs informative.

Worked example: one DINO target with \(K = 3\)

Teacher logits for a global crop: \(g_t = (0.6, 0.5, -0.2)\). The center (running mean over images) is \(\mathbf{c} = (0.3, 0.0, -0.1)\): prototype 1 is high for every image. Centered: \((0.3, 0.5, -0.1)\). Divide by \(\tau_t = 0.04\): \((7.5, 12.5, -2.5)\). Softmax: \(P_t = (0.0067, 0.9933, 0.0000)\). Without centering, the target would be \((0.9241, 0.0759, 0.0000)\): prototype 1 again, as for every other image.

Student logits for a local crop: \(g_s = (0.2, 0.6, 0.0)\). Divide by \(\tau_s = 0.1\) and softmax: \(P_s = (0.0179, 0.9796, 0.0024)\). Cross-entropy: \(H = -(0.0067 \ln 0.0179 + 0.9933 \ln 0.9796) = 0.0474\). The teacher entropy is \(h(P_t) = 0.0402\), so \(\KL(P_t \Vert P_s) = 0.0072\): the student agrees, but not perfectly, and the gradient pushes its second logit up.

The whole loop, running

Figure 4 froze one moment. Figure 5 runs the full DINO loop of Algorithm 2 on a toy: 6 images, \(K = 6\) prototypes, and a student that is a single linear layer \(g_{\theta}(\mathbf{x}) = \mathbf{W}\mathbf{x} + \mathbf{b}\) on 6-dimensional inputs. Each image \(b\) is a fixed unit vector \(\mathbf{e}_b\); a “view” is \(\mathbf{e}_b\) plus Gaussian noise of size \(\sigma = 0.2\), which plays the role of a random crop. The student starts with a shared bias \(b^{(1)} = 1\) toward prototype 1, the same trap as in Figure 4. The teacher starts as a copy of the student, and the center starts at \(\mathbf{c} = \mathbf{0}\).

Only one more formula is needed. For a softmax with temperature \(\tau_s\), the gradient of the cross-entropy with respect to the student logits is \(\partial H / \partial \mathbf{z}_s = (P_s - P_t)/\tau_s\): the student is pushed up where the teacher puts more mass than the student does. The teacher gets no gradient; it only moves through the EMA.

prototype 1 (has the shared bias)23456tracked image (click a group)
  1. for each image: views \(\mathbf{x}^1, \mathbf{x}^2 \leftarrow \mathbf{e}_b + \sigma\varepsilon\)
  2. teacher logits \(\mathbf{z}_t \leftarrow \mathbf{W}_t\mathbf{x} + \mathbf{b}_t\) // no gradient
  3. \(P_t \leftarrow \softmax((\mathbf{z}_t - \mathbf{c})/\tau_t)\) // (165.5)
  4. \(\mathbf{z}_s \leftarrow \mathbf{W}_s\mathbf{x} + \mathbf{b}_s\); \(P_s \leftarrow \softmax(\mathbf{z}_s/\tau_s)\) // (165.4)
  5. \(\mathcal{L} \leftarrow\) mean of \(H(P_t(\mathbf{x}^1), P_s(\mathbf{x}^2))\) and the swap // (165.6)
  6. \(\theta_s \leftarrow \theta_s - \eta\nabla\mathcal{L}\), with \(\partial H/\partial\mathbf{z}_s = (P_s - P_t)/\tau_s\)
  7. \(\theta_t \leftarrow \lambda\theta_t + (1-\lambda)\theta_s\) // (165.7)
  8. \(\mathbf{c} \leftarrow m\mathbf{c} + (1-m)\,\text{mean}(\mathbf{z}_t)\) // (165.8)
Figure 5. A tiny DINO trained live: 6 images, \(K = 6\) prototypes, a linear student and its EMA teacher, \(\tau_s = 0.1\), learning rate \(\eta = 0.1\), center rate \(m = 0.9\). First the teacher \(P_t\) on view 1 and the student \(P_s\) on view 2 for every image, colored by prototype; then the center \(\mathbf{c}\) and a chart of the batch loss and two entropies over the iterations. All numbers are computed in your browser with a fixed seed.
How to read Figure 5

Press Step to walk through iteration 1, one line of the algorithm at a time; the live panel puts the numbers of the tracked image into each formula. In iteration 1 the center is still zero, so the shared bias wins: all six teacher outputs put almost all mass on prototype 1, and the batch loss is 0.4695. Line 7 moves the teacher bias only a little: \(b_t^{(1)} = 0.95 \cdot 1 + 0.05 \cdot 1.0147 = 1.0007\). Line 8 starts to learn the bias into the center: \(c^{(1)} = 0.091\).

Now press “Train +50 iterations” several times and watch the chart. With the defaults the center soaks up the shared bias, the images spread over the prototypes, and by iteration 300 all six images pick six different prototypes: mean teacher entropy 0.033 nats, entropy of the batch average 1.791 (close to the maximum \(\ln 6 = 1.792\)), loss 0.057. Press Reset and try the failures. Centering off: every image stays on prototype 1 and the loss falls to 0.001 by iteration 100 and to 0 after that: one-dimension collapse, the perfect but useless agreement of (165.9). \(\tau_t = 1\): the teacher output is flat for every image and the loss sits at \(\ln 6 = 1.792\): uniform collapse. \(\lambda = 0\) (the teacher is a copy of the student): the targets move as fast as the student and after 300 iterations the run is back on one prototype. \(\lambda = 0.99\): the teacher is very slow, and after 300 iterations only 4 prototypes are in use. This is a toy, not the real model, but the forces are the same.

What DINO learned without labels

A ViT-B/8 trained with DINO reached 80.1% ImageNet top-1 with a linear probe and 77.4% with a simple k-NN classifier on frozen features. [Certain] More surprising: the self-attention of the CLS token in the last layer forms clean object masks, though the model never saw a segmentation label. The paper calls this an “emerging property”. [Certain] The same features also work well for image retrieval and copy detection, and DINO features were already used to propagate segmentation masks through DAVIS videos. [Certain]

DINOv2: DINO plus patch-level learning, at scale

DINO features are good for a whole image. Detection, segmentation, depth and tracking need good features for every patch. DINOv2 (Oquab et al., 2023) combines several known ideas and scales them up. [Certain]

  • Image-level loss: the DINO loss on the CLS token, as above.
  • Patch-level loss (iBOT): mask some patches of the student’s input (not the teacher’s). For each masked patch, the student must predict the teacher’s prototype distribution for that same patch, which the teacher computes from the visible image. This is masked image modeling, but in feature space, not pixel space.
  • Separate heads for the DINO and iBOT losses (untied weights), which worked better at scale.
  • Sinkhorn-Knopp centering (from SwAV) instead of the running-mean center: 3 iterations that make the teacher’s outputs over the batch use all prototypes about equally.
  • KoLeo regularizer: spreads the features of a batch evenly (formula 165.11).
  • Short high-resolution phase at 518×518 at the end of training, because small objects vanish at low resolution.
  • Curated data: LVD-142M, 142 million images selected automatically from a pool of about 1.2 billion web images, by retrieving images that are close to images in curated datasets, after deduplication.
  • Big model, then distillation: a ViT-g/14 with 1.1B parameters (embedding size 1536, 24 heads) is trained first; ViT-S, B and L are distilled from it.

The iBOT loss, for the set \(\mathcal{M}\) of patches masked in the student’s input:

\[ \mathcal{L}_{\text{iBOT}} = -\sum_{i \in \mathcal{M}} \sum_{k=1}^{K} p_{t,i}^{(k)} \log p_{s,i}^{(k)} \tag{165.10} \]
Read the symbols
\(p_{t,i}\)
teacher’s centered, sharpened distribution for patch \(i\), computed from the unmasked image
\(p_{s,i}\)
student’s distribution for patch \(i\), computed from the input where patch \(i\) was replaced by a mask token

The KoLeo regularizer looks at a batch of \(n\) L2-normalized features \(\mathbf{x}_1, \dots, \mathbf{x}_n\) and rewards each point for being far from its nearest neighbour:

\[ \mathcal{L}_{\text{KoLeo}} = -\frac{1}{n} \sum_{i=1}^{n} \log d_{n,i}, \qquad d_{n,i} = \min_{j \neq i} \lVert \mathbf{x}_i - \mathbf{x}_j \rVert_2 \tag{165.11} \]

The full DINOv2 objective is a weighted sum; the KoLeo weight is 0.1 [Certain]:

\[ \mathcal{L}_{\text{DINOv2}} = \mathcal{L}_{\text{DINO}} + \mathcal{L}_{\text{iBOT}} + 0.1\, \mathcal{L}_{\text{KoLeo}} \tag{165.12} \]
Worked example: KoLeo on four points

Four unit vectors in 2-D: \((1, 0)\), \((0.8, 0.6)\), \((0, 1)\), \((-1, 0)\). Nearest-neighbour distances: the first two are 0.6325 apart; the third is 0.8944 from the second; the fourth is \(\sqrt{2} = 1.4142\) from the third. \(\mathcal{L}_{\text{KoLeo}} = -\tfrac14(\ln 0.6325 + \ln 0.6325 + \ln 0.8944 + \ln 1.4142) = 0.1703\). Moving the second point away from the first lowers the loss. The DINOv2 ablation shows KoLeo improves retrieval (Oxford-M) from 55.6 to 63.9 mAP while classification stays about the same. [Certain]

The ablations show which part does what: removing the iBOT masked-modeling term costs almost 3 mIoU on ADE20k segmentation (47.1 vs 44.2), so the patch-level loss is what makes the dense features good. [Certain] With a frozen backbone and a linear probe, DINOv2 ViT-g/14 reaches 86.5% on ImageNet, close to OpenCLIP ViT-G/14 (86.2%), which was trained with text. [Certain]

Registers: a side story about artifacts

Large DINOv2 models show a few high-norm tokens in their patch feature maps, mostly in boring background areas. Darcet et al. (2023) found that the model re-uses these low-information patches as scratch space for global computation; they are about 2% of the tokens and their norm is about 10 times higher than normal. The fix is simple: add a few extra learnable tokens, called registers, that are not image patches. The model then uses the registers as scratch space, and the patch maps become clean. With 4 registers the extra FLOPs are below 2%. [Certain] DINOv3 uses 4 registers by default. [Certain]

DINOv3: 7B parameters and Gram anchoring

DINOv3 (Siméoni et al., 2025) scales the DINOv2 recipe and fixes a problem that appears at scale. [Certain]

Teacher modelDINOv2DINOv3
Backbone, parametersViT-g, 1.1BViT-7B, 6.7B
Blocks, width, heads40, 1536, 2440, 4096, 32
Patch size1416
Position embeddingslearnedRoPE (axial, with box jittering)
Registers44
Training imagesLVD-142MLVD-1689M (1,689 million curated images)
Schedulescosineconstant learning rate, weight decay and EMA (0.999), after warmup; 1M iterations

Batch: 4,096 images, each with 2 global crops of 256×256 and 8 local crops of 112×112. With 16-pixel patches, that is \(2 \cdot 16^2 + 8 \cdot 7^2 = 904\) patch tokens per image, the same as DINOv2 with 224/98 crops and 14-pixel patches, and about 3.7 million patch tokens per batch. [Certain]

The problem: dense features decay during long training

Training DINOv2-style models longer keeps improving ImageNet classification, but after about 200k iterations the segmentation quality of the patch features goes down, and for the 7B model it falls below its early level. [Certain] The cosine similarity maps between one patch and all others, clean at 200k iterations, become noisy by 600k: unrelated patches look similar. The paper also sees the cosine similarity between the CLS token and the patch tokens slowly grow: every patch starts to look like the global image summary, and locality is lost. [Certain]

The fix: Gram anchoring

The Gram matrix of an image is the matrix of all pairwise dot products of its \(P\) patch features. If the features are L2-normalized, entry \((i, j)\) is the cosine similarity of patch \(i\) and patch \(j\). Gram anchoring pulls the student’s Gram matrix toward that of an earlier checkpoint (the Gram teacher), whose patch features were still clean:

\[ \mathcal{L}_{\text{Gram}} = \big\lVert \mathbf{X}_S \mathbf{X}_S^\top - \mathbf{X}_G \mathbf{X}_G^\top \big\rVert_F^2 \tag{165.13} \]
Read the symbols
\(\mathbf{X}_S \in \R^{P \times d}\)
the student’s L2-normalized patch features for one global crop: \(P\) patches, width \(d\)
\(\mathbf{X}_G \in \R^{P \times d}\)
the Gram teacher’s patch features for the same crop (an early EMA teacher)
\(\lVert \cdot \rVert_F\)
the Frobenius norm: the square root of the sum of all squared entries

Why the Gram matrix and not the features themselves? Because it only fixes how patches relate to each other. The features may move freely, for example rotate, as the global loss keeps improving, as long as the pattern of similarities stays the same. Any rotation \(\mathbf{Q}\) (\(\mathbf{Q}\mathbf{Q}^\top = \mathbf{I}\)) leaves the loss unchanged: \((\mathbf{X}\mathbf{Q})(\mathbf{X}\mathbf{Q})^\top = \mathbf{X}\mathbf{X}^\top\).

In DINOv3, Gram anchoring starts after 1M iterations, as a refinement step; the Gram teacher is refreshed to the current EMA teacher every 10k iterations, at most three times, and the loss weight is \(w_{\text{Gram}} = 2\). [Certain] The refinement objective is:

\[ \mathcal{L}_{\text{Ref}} = w_D\, \mathcal{L}_{\text{DINO}} + \mathcal{L}_{\text{iBOT}} + w_{DK}\, \mathcal{L}_{\text{DKoleo}} + w_{\text{Gram}}\, \mathcal{L}_{\text{Gram}} \tag{165.14} \]

(\(\mathcal{L}_{\text{DKoleo}}\) is a distributed version of KoLeo.) Even applied late, the Gram loss “repairs” badly degraded patch features. A further improvement feeds images at twice the resolution to the Gram teacher and down-samples its feature map by 2, which gives smoother targets; this adds about 2 mIoU on ADE20k. [Certain] Explore the idea in Figure 6.

Figure 6. Dense features as a similarity map. Each of the 144 patches has a 16-dimensional toy feature: the embedding of its region plus noise. The drift slider mixes each patch feature with one shared global direction and extra noise, imitating what DINOv3 observed in long training. The readout gives the same kinds of measures the paper uses: similarity contrast, a nearest-neighbour “segmentation” score, and the Gram loss (165.13) against the early features.
How to read Figure 6

Click a patch on the left. At drift 0 the dog patch is similar to the other dog patches (mean cosine 0.837) and not to other regions (−0.022), and every patch’s nearest neighbour is in its own region (100%). Raise the drift to 0.5: other regions climb to 0.659 and the patches move toward the global direction (mean cosine 0.847). At 0.8 the gap is almost gone (0.871 vs 0.857) and the nearest-neighbour score falls to 45.8%: a linear segmentation head would struggle, but a CLS-based classifier might not notice. Now tick Gram anchoring: the features are far from the Gram teacher (mean distance 1.448), yet the Gram loss is exactly 0 and the similarity map is the clean one again. This is a toy, not the real model.

After training: high resolution, a model family, text

DINOv3 adds a short high-resolution adaptation phase (global crops up to 768, still with Gram anchoring), then distills the 7B teacher into a family: ViT-S (21M), ViT-S+ (29M), ViT-B (86M), ViT-L (300M), ViT-H+ (840M), and ConvNeXt models from 29M to 198M parameters. [Certain] With a frozen backbone, the 7B model reaches 88.4% ImageNet linear-probe accuracy and 83.3 J&F on DAVIS video segmentation tracking at high resolution, 6.7 points above DINOv2. [Certain] A text encoder is also trained on top of a frozen DINOv3 ViT-L (the dino.txt recipe) for zero-shot use, and a separate 7B model is trained on 493 million satellite images. [Certain]

Global vs dense features: which to use for VLMs and video

A ViT encoder gives two kinds of output. A global feature is one vector for the whole image: the CLS token, or the mean of the patch tokens. Dense features are the patch tokens themselves: one vector per patch, \(14 \times 14 = 196\) of them for 224×224 with 16-pixel patches, 1,024 at 512×512.

NeedUseWhy
Image or video retrieval, near-duplicate detection, clustering, k-NN classificationglobal (CLS or mean-pooled), L2-normalizedOne vector per image fits a vector index; DINO global features are strong for copy detection [Certain]
Semantic segmentation, depth, keypoint matchingdense patch tokens, often from DINOv2 or DINOv3, plus a light headThe answer differs per pixel; the patch-level losses (iBOT, Gram) make these maps good
Video object tracking (mask propagation)dense tokens of each frame, compared across framesA patch in frame \(t+1\) takes the label of its most similar patches in frame \(t\); no training needed
VLM vision inputdense tokens (usually all patches, after a projector)The LLM must read details and positions; one global vector would throw them away (session 177)
Choosing the encoder for a VLMCLIP or SigLIP for text-aligned semantics; add DINOv2 for spatial detailThe “Eyes Wide Shut” study found image pairs that CLIP sees as similar but DINOv2 does not, and mixing in DINOv2 features improved visual grounding [Certain]

A useful way to remember it: CLIP-style training asks “does this image match this caption?”, so it learns what captions talk about. DINO-style training asks “is this the same image, seen from another crop?”, so it learns shape, parts, position and texture, including things no caption mentions. VLM papers such as Prismatic VLMs and Cambrian-1 compared many encoders and found that combining a language-supervised encoder with DINOv2 helps on spatial and localization tasks. [Likely]

Putting it together: pseudo-code and PyTorch

Algorithm 1: one MAE pretraining step
  1. input: batch of images, encoder \(E\) (large ViT), decoder \(D\) (small), mask ratio \(r = 0.75\)
  2. patchify each image into \(N\) patches; embed each patch and add its position embedding
  3. draw a random number per patch; sort; keep the first \((1-r)N\) patches // uniform random masking
  4. \(\mathbf{z} \leftarrow E(\text{visible tokens})\) // no mask tokens in the encoder
  5. insert a shared mask token in every removed place; undo the shuffle; add position embeddings
  6. \(\hat{\mathbf{x}} \leftarrow D(\text{all } N \text{ tokens})\), then a linear layer to \(p^2 \cdot 3\) pixels per patch
  7. target \(\leftarrow\) per-patch normalized pixels // formula (165.2)
  8. loss \(\leftarrow\) MSE on masked patches only // formula (165.1); backward; AdamW step
  9. after pretraining: discard \(D\); fine-tune or probe \(E\)
Algorithm 2: one DINO step (DINOv2 adds lines 6 and 7)
  1. input: student \(\theta_s\), teacher \(\theta_t\), center \(\mathbf{c}\), temperatures \(\tau_s, \tau_t\), EMA rates \(\lambda, m\)
  2. for each image: 2 global crops and several local crops, with color jitter, blur, flips
  3. teacher on global crops: \(P_t = \softmax((g_{\theta_t} - \mathbf{c})/\tau_t)\), no gradient // (165.5)
  4. student on all crops: \(P_s = \softmax(g_{\theta_s}/\tau_s)\) // (165.4)
  5. \(\mathcal{L} \leftarrow\) cross-entropy over all pairs of different crops // (165.6)
  6. DINOv2: mask some student patches of the global crops; add the iBOT loss on them // (165.10)
  7. DINOv2: add \(0.1 \times\) KoLeo on the student CLS features // (165.11)
  8. backward; optimizer step on \(\theta_s\) only
  9. \(\theta_t \leftarrow \lambda\theta_t + (1-\lambda)\theta_s\) // (165.7); \(\mathbf{c} \leftarrow m\mathbf{c} + (1-m)\,\text{mean}(g_{\theta_t})\) // (165.8)
  10. DINOv3 refinement: also add \(w_{\text{Gram}}\mathcal{L}_{\text{Gram}}\) against an early teacher // (165.13)

The core pieces in PyTorch. It runs on a CPU in a few seconds, and prints the same numbers every time:

import copy, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# ---- 1) MAE: patchify, random masking, loss on masked patches only ----
def patchify(img, p=16):                     # (B,3,H,W) -> (B,N,p*p*3)
    B, C, H, W = img.shape
    x = img.reshape(B, C, H // p, p, W // p, p)
    return x.permute(0, 2, 4, 3, 5, 1).reshape(B, (H // p) * (W // p), p * p * C)

def random_masking(tokens, ratio=0.75):      # the shuffle trick from the MAE paper
    B, N, D = tokens.shape
    keep = int(N * (1 - ratio))
    noise = torch.rand(B, N)                 # one random number per patch
    ids_shuffle = noise.argsort(dim=1)       # random order
    ids_restore = ids_shuffle.argsort(dim=1) # to undo the shuffle later
    ids_keep = ids_shuffle[:, :keep]
    visible = torch.gather(tokens, 1, ids_keep[..., None].expand(-1, -1, D))
    mask = torch.ones(B, N); mask[:, :keep] = 0
    mask = torch.gather(mask, 1, ids_restore)  # 1 = masked, in the original order
    return visible, mask, ids_restore

img = torch.randn(2, 3, 224, 224)
target = patchify(img)                       # (2, 196, 768)
mu, var = target.mean(-1, keepdim=True), target.var(-1, keepdim=True)
target = (target - mu) / (var + 1e-6).sqrt() # normalized pixels, formula (165.2)
emb = nn.Linear(768, 64)
visible, mask, ids_restore = random_masking(emb(target))
print("visible tokens:", visible.shape[1], "masked:", int(mask[0].sum()))   # 49, 147
pred = torch.randn_like(target)              # stand-in for the decoder output
loss = (((pred - target) ** 2).mean(-1) * mask).sum() / mask.sum()   # (165.1)
print("MAE loss on masked patches:", round(loss.item(), 3))           # 2.009

# ---- 2) DINO: centering + sharpening, cross-entropy, EMA teacher ----
K = 256
student = nn.Sequential(nn.Linear(32, 64), nn.GELU(), nn.Linear(64, K))
teacher = copy.deepcopy(student)
for p in teacher.parameters(): p.requires_grad = False
center = torch.zeros(K)

def dino_loss(s_out, t_out, center, tau_s=0.1, tau_t=0.04):
    t = F.softmax((t_out - center) / tau_t, dim=-1).detach()   # (165.5): center, sharpen, stop-grad
    log_s = F.log_softmax(s_out / tau_s, dim=-1)                # (165.4)
    return -(t * log_s).sum(-1).mean()

opt = torch.optim.AdamW(student.parameters(), lr=1e-3)
x = torch.randn(16, 32)
for step in range(3):
    v1 = x + 0.1 * torch.randn_like(x)        # two "views" of the same batch
    v2 = x + 0.1 * torch.randn_like(x)
    s1, s2 = student(v1), student(v2)
    with torch.no_grad():
        t1, t2 = teacher(v1), teacher(v2)
    loss = 0.5 * (dino_loss(s2, t1, center) + dino_loss(s1, t2, center))  # (165.6) with 2 views
    opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        lam = 0.996
        for pt, ps in zip(teacher.parameters(), student.parameters()):
            pt.mul_(lam).add_((1 - lam) * ps)                  # (165.7) EMA teacher
        center = 0.9 * center + 0.1 * torch.cat([t1, t2]).mean(0)   # (165.8)
    print(f"step {step}: DINO loss {loss.item():.3f}")       # 2.230, 2.129, 2.010

# ---- 3) KoLeo (165.11) and Gram anchoring (165.13) ----
def koleo(z):
    z = F.normalize(z, dim=-1)
    d = torch.cdist(z, z); d.fill_diagonal_(float("inf"))
    return -torch.log(d.min(dim=1).values + 1e-8).mean()

def gram_loss(xs, xg):                       # xs, xg: (P, d) patch features of one image
    xs, xg = F.normalize(xs, dim=-1), F.normalize(xg, dim=-1)
    return ((xs @ xs.T - xg @ xg.T) ** 2).sum()

z = torch.tensor([[1., 0.], [0.8, 0.6], [0., 1.], [-1., 0.]])
print("KoLeo:", round(koleo(z).item(), 4))   # 0.1703, as in the worked example
xg = torch.randn(196, 64)
Q, _ = torch.linalg.qr(torch.randn(64, 64))  # a random rotation
print("Gram loss after rotation:", round(gram_loss(xg @ Q, xg).item(), 6))  # 0.0
print("Gram loss vs noisy copy:", round(gram_loss(xg + torch.randn_like(xg), xg).item(), 1))  # 597.7

The printed values come from PyTorch 2.8 on a CPU; other versions may differ in the last digits. For real work, load pretrained weights instead of training: for example the DINOv2 models through torch.hub or Hugging Face, and take last_hidden_state[:, 0] as the global feature and the remaining tokens (after the registers, if any) as the dense map. [Likely]

Where this is used

ApplicationWhich model and which featuresNote
Backbone pretraining for a fine-tuned detector or classifierMAE encoder, fully fine-tunedSAM’s image encoder is an MAE-pretrained ViT-H [Likely]
Vision encoder inside a VLMDINOv2 patch tokens, often next to SigLIP tokensBetter spatial grounding than a CLIP-only encoder [Likely]; see session 178
Monocular depth estimationDINOv2 encoder with a DPT-style headDepth Anything builds on a DINOv2 encoder [Likely]
Video object segmentation and trackingDINO, DINOv2, DINOv3 dense tokens; label propagation across framesDINOv3 7B: 83.3 J&F on DAVIS at high resolution, with no video training [Certain]; see session 204
Video retrieval, near-duplicate detection, fingerprintingDINO global features per frame, L2-normalized, pooled over timeStrong copy-detection features [Certain]; see session 209
Action recognition and video classificationfrozen frame features plus a small attentive probeDINOv3 is evaluated this way on UCF101, Something-Something V2 and Kinetics-400 [Certain]
Key-moment and shot-change detectioncosine distance between global features of consecutive framesCheap and label-free; a sudden drop in similarity marks a cut
Domains without labels: satellite, medical, biologyDINO-style SSL trained on the domain itselfDINOv3 trains a 7B model on 493M satellite images [Certain]

Common mistakes

  • Using MAE features frozen and expecting CLIP-level linear-probe results. MAE is designed to be fine-tuned. Probe it with a few trainable blocks, or choose DINOv2 or DINOv3 for frozen use.
  • Feeding mask tokens into the MAE encoder. It wastes about 3× compute and creates a train–test mismatch. Drop the masked patches; add mask tokens only in the decoder.
  • Computing the MAE loss on all patches. The visible patches are trivial to copy and dilute the signal; the paper computes the loss on masked patches only.
  • Letting gradients flow into the DINO teacher, or copying the student into the teacher every step. The teacher must be an EMA with no gradient; the paper reports that copying the student fails to converge. [Certain]
  • Removing centering or using a high teacher temperature. You get one-dimension or uniform collapse (Figure 4). Monitor the teacher entropy and the KL of formula (165.9) during training.
  • Using the CLS token for a dense task. Segmentation, depth and tracking need the patch tokens. Also skip the register tokens when you reshape patch tokens into a 2-D map.
  • Wrong patch size or resolution when reshaping. DINOv2 uses 14-pixel patches, DINOv3 uses 16. A 224×224 input gives a 16×16 map in DINOv2 and a 14×14 map in DINOv3.
  • Forgetting normalization in retrieval. L2-normalize global features before cosine search and k-NN, as in the DINO evaluation.

Interview questions

Why does MAE mask 75% of patches while BERT masks 15% of tokens?basic
Language is dense in information: each word carries meaning, so 15% missing already makes a hard task. Images are highly redundant: a missing patch can often be filled by interpolating its neighbours, which teaches nothing. A high random masking ratio removes this shortcut and forces the model to use knowledge of objects and scenes. It also makes training cheap, because the encoder only processes the 25% visible patches.
Why does the MAE encoder not use mask tokens, and what does that buy?core
Two things. (1) No train–test mismatch: at test time the encoder never sees mask tokens, so it should not learn on inputs that are 75% mask tokens. (2) Compute and memory: the encoder runs on 49 of 196 tokens, so its linear layers do 1/4 and its attention 1/16 of the work. With ViT-L and an 8-block, 512-wide decoder, total FLOPs drop about 3.3× (the paper reports 2.8× wall-clock speedup). The mask tokens enter only the small decoder, which is thrown away after pretraining.
MAE fine-tunes very well but has weak linear-probe accuracy. Why?core
Pixel reconstruction rewards keeping low-level detail (texture, color, exact position) in the features, so the last-layer features are not organized so that classes are linearly separable. Fine-tuning can reshape them quickly: in the MAE paper, fine-tuning only the last block lifts accuracy from 73.5% (linear probe, ViT-L) to 81.0%. Joint-embedding methods like DINO directly train the output to be invariant to crops and colors, which gives more semantic, linearly separable features, so their frozen features are stronger (in the DINOv2 paper, k-NN 49.4% for MAE ViT-H vs 83.5% for DINOv2 ViT-g).
How does DINO avoid collapse without negative pairs?core
With a momentum (EMA) teacher plus centering and sharpening of the teacher output. Centering subtracts a running mean of the teacher logits, so no single dimension can dominate for all images; alone it pushes toward a uniform output. Sharpening uses a low teacher temperature (0.04 to 0.07), which makes outputs peaked; alone it pushes toward one dimension. Balanced, they keep the teacher output both confident per image and diverse across images. The cross-entropy splits into teacher entropy plus KL; collapse shows up as KL going to zero.
Why is the teacher an EMA of the student? Why not just use the student?advanced
Using the current student as its own target is unstable: targets move as fast as the student and the paper reports that copying the student weights fails to converge. The EMA teacher (\(\lambda\) from 0.996 to 1) changes slowly, so targets are stable, and it behaves like an ensemble of recent students (Polyak-style averaging), so it is actually more accurate than the student during training. The student learns from a slightly better, slowly moving version of itself, and this is what drives improvement without labels.
What is multi-crop and why does it help DINO?core
Multi-crop makes 2 large global crops (\(224^2\), a large part of the image) and several small local crops (\(96^2\)). The teacher sees only global crops; the student sees all. The loss asks a small local crop to predict the teacher’s output for the whole scene, which forces “local-to-global” understanding: a part must be recognized as belonging to the object. Small crops are cheap (\(96^2\) gives 36 patches with 16-pixel patches), so more views cost little. In the DINO paper (ViT-S/16, 100 epochs), adding 10 local crops to the 2 global ones raised linear-probe accuracy from 67.8% to 74.6%, for about 1.6× the training time (24.2 h vs 15.3 h); training the 2-crop model 3× longer reached only 72.5%.
What does DINOv2 add to DINO, and which part matters for segmentation?core
An iBOT patch-level masked-modeling loss (student predicts the teacher’s prototype distribution for masked patches), untied DINO and iBOT heads, Sinkhorn-Knopp centering, a KoLeo regularizer, a short \(518^2\) high-resolution phase, efficient training (FlashAttention-style kernels, sequence packing), the curated LVD-142M dataset, and a 1.1B ViT-g distilled into smaller models. For segmentation the iBOT term is critical: removing it costs about 3 mIoU on ADE20k in the paper’s ablation.
What is the KoLeo regularizer?advanced
It comes from the Kozachenko-Leonenko entropy estimator. For a batch of L2-normalized features it minimizes \(-\frac{1}{n}\sum_i \log d_{n,i}\), where \(d_{n,i}\) is the distance from point \(i\) to its nearest neighbour. It pushes features apart so they spread uniformly over the sphere. That helps nearest-neighbour search: in DINOv2 it raised Oxford-M retrieval from 55.6 to 63.9 mAP with little effect on classification. Weight 0.1, applied to CLS features within each GPU.
What is Gram anchoring and why does it act on the Gram matrix, not on the features?advanced
In long DINOv2-style training of large models, global accuracy keeps rising but patch features lose locality: similarity maps become noisy and dense-task scores fall after about 200k iterations. Gram anchoring adds \(\lVert X_S X_S^\top - X_G X_G^\top\rVert_F^2\) on L2-normalized patch features, pulling the student’s patch-to-patch similarity structure toward an early checkpoint with clean maps. Acting on the Gram matrix leaves the features free to change (any rotation keeps the Gram matrix), so the global objective can keep improving while the local structure is preserved. DINOv3 applies it as a refinement after 1M iterations, with a high-resolution Gram teacher for smoother targets.
You build a VLM. Do you use CLIP/SigLIP or DINOv2 as the vision encoder, and which tokens?core
Use the patch tokens, not one global vector, because the LLM needs local details and positions. For the encoder, a text-aligned model (SigLIP or CLIP) is the usual default, since its features already match language concepts and make the projector easy to train. DINOv2 or DINOv3 features add spatial detail, counting and fine visual differences that captions rarely describe; studies such as “Eyes Wide Shut” (CLIP-blind pairs), Prismatic and Cambrian-1 report gains from combining both, at the cost of a second encoder and more compute. The final choice should be made with ablations on your target tasks.
How would you track an object through a video using only a frozen DINO model?core
Label propagation. Extract the dense patch features of every frame (L2-normalized). Given the object mask in frame 1, for each patch of frame \(t+1\) compute cosine similarities to the patches of the previous frame(s), keep the top-k, and take a softmax-weighted vote of their labels, usually restricted to a spatial neighbourhood. Repeat frame by frame, and upsample the patch-level mask to pixels. No training is needed; DINOv3 evaluates exactly this way on DAVIS, YouTube-VOS and MOSE, and higher input resolution gives finer masks.
What are registers in a ViT, and why do DINOv2 and DINOv3 need them?advanced
Large ViTs, supervised or self-supervised, learn to use a few low-information background patches as storage for global information. Those tokens get very high norms (about 10× normal) and corrupt attention and feature maps. Registers are extra learnable tokens appended to the input that are not tied to any patch; the model uses them as scratch space instead, so the patch maps become clean. With 4 registers the FLOP increase is below 2%. They are dropped at the output for dense tasks.

Exercises

  1. An MAE uses 448×448 images and 16×16 patches, mask ratio 75%. How many patches are there, and how many does the encoder see? By what factor does the attention part of one encoder block shrink?
  2. A DINO teacher has \(\lambda = 0.999\). About how many past student versions does it average? Compare with \(\lambda = 0.996\).
  3. In Figure 4, set \(\tau_t = 0.07\) with centering on. Is the output healthy? What is the mean per-image entropy?
  4. For the worked DINO example, suppose the student instead outputs \(g_s = (0.6, 0.2, 0.0)\). Without computing exactly, will the cross-entropy go up or down, and why?
  5. Two student patch features are \(\mathbf{x}_1 = (1, 0)\) and \(\mathbf{x}_2 = (0.6, 0.8)\); the Gram teacher has \(\mathbf{x}_1 = (1, 0)\), \(\mathbf{x}_2 = (1, 0)\), and a third patch \((0, 1)\) in both, while the student’s third patch is \((0, 1)\). Compute the Gram loss (165.13).
  6. DINOv3 uses 16-pixel patches. How many patch tokens does a 1024×1024 image produce, and how many entries does its Gram matrix have?
Answerscheck
1. \(28 \times 28 = 784\) patches; the encoder sees 196. Attention cost scales with \(n^2\), so it shrinks by \(4^2 = 16\). 2. \(1/(1 - 0.999) = 1000\) versions, against 250 for 0.996: the teacher changes 4 times more slowly. DINOv3 uses a constant 0.999. 3. Yes, healthy: six distinct winners; mean entropy 0.629 nats (batch-average entropy 1.954). 4. Up: the teacher target puts 0.9933 on prototype 2, but now the student puts most mass on prototype 1, so \(-\log P_s^{(2)}\) becomes large. 5. Student Gram: rows \((1, 0.6, 0), (0.6, 1, 0.8), (0, 0.8, 1)\). Teacher Gram: \((1, 1, 0), (1, 1, 0), (0, 0, 1)\). Differences: \(-0.4\) twice and \(0.8\) twice; loss \(= 2(0.16) + 2(0.64) = 1.6\). 6. \(64 \times 64 = 4096\) tokens; the Gram matrix is \(4096 \times 4096 \approx 16.8\) million entries, which is why the loss is computed per crop and only on global crops.

Key takeaways

  • MAE removes 75% of patches, runs a large encoder on the rest, and a small decoder rebuilds the pixels; the loss is on masked patches only (165.1). It is about 3× cheaper to train and excellent after fine-tuning, but weaker as frozen features.
  • DINO trains a student to match an EMA teacher (165.7) across global and local crops (165.6). Centering plus sharpening (165.5, 165.8) avoid collapse without negatives.
  • DINOv2 adds a patch-level iBOT loss (165.10), KoLeo (165.11), Sinkhorn-Knopp centering, curated LVD-142M data and a 1.1B model distilled into smaller ones; its frozen features are strong for both global and dense tasks.
  • DINOv3 scales to 6.7B parameters and 1.7B images and adds Gram anchoring (165.13), which keeps patch similarity structure clean during long training.
  • Use global features for retrieval and near-duplicates; use dense patch tokens for segmentation, depth, tracking and as VLM input. DINO features complement CLIP-style features in VLMs.

Key formulas

\[ \mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \tfrac{1}{D_p}\lVert \hat{\mathbf{x}}_i - \tilde{\mathbf{x}}_i \rVert^2, \qquad P_t = \softmax\!\Big(\frac{g_{\theta_t} - \mathbf{c}}{\tau_t}\Big) \]
\[ \theta_t \leftarrow \lambda\theta_t + (1-\lambda)\theta_s, \qquad \mathcal{L}_{\text{Gram}} = \lVert \mathbf{X}_S\mathbf{X}_S^\top - \mathbf{X}_G\mathbf{X}_G^\top \rVert_F^2 \]

References for this session

  1. He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
  2. Caron et al. (2021). Emerging Properties in Self-Supervised Vision Transformers
  3. Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision
  4. Siméoni et al. (2025). DINOv3
  5. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  6. Zhou et al. (2021). iBOT: Image BERT Pre-Training with Online Tokenizer
  7. Caron et al. (2020). Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
  8. Sablayrolles et al. (2018). Spreading vectors for similarity search
  9. Darcet, Oquab, Mairal, Bojanowski (2023). Vision Transformers Need Registers
  10. Tong et al. (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  11. Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  12. Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
  13. Kirillov et al. (2023). Segment Anything
  14. Yang et al. (2024). Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
  15. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language