The Vision Transformer (ViT)
The Vision Transformer cuts an image into small square patches, turns each patch into one token with a linear layer, and then runs the same transformer encoder that reads text. This session builds it piece by piece: patchify, patch embedding (and why it is a convolution with stride P), the CLS token and mean pooling, 2D position embeddings and how to stretch them to a new resolution, and the most important number for any VLM: how many tokens one image costs.
What you will learn
- How patchify turns an H × W × C image into N = HW/P² tokens, and what one token holds
- Why the patch embedding is both a linear layer on flattened patches and a Conv2d with kernel = stride = P, and how many weights it has
- How the CLS token, mean pooling and attention pooling turn many tokens into one image vector
- Why ViT needs position embeddings, why 1D and 2D versions work about equally, and how to interpolate them for a new resolution (bilinear vs bicubic)
- Counting a ViT by hand: 86,567,656 parameters for ViT-B/16 and 17.6 G multiply-adds at 224 × 224
- How resolution and patch size decide the token count and cost, and why this number drives VLM design
- Why the first ViT needed 300 million images, and how DeiT trained it on ImageNet alone
Why this matters
Almost every vision-language model you will meet (LLaVA, Qwen-VL, InternVL, PaliGemma, SmolVLM) reads images with a Vision Transformer. The idea is short: cut the image into small squares, call each square a token, and give the tokens to a normal transformer. The title of the paper says it: “An Image is Worth 16x16 Words”.
The previous session (session 163) ended with what CNNs miss: long-range relations, and a token interface. The Vision Transformer (ViT) answers both. Self-attention connects any two patches in one layer, and the output is a sequence of vectors with positions, which is exactly the format a language model already reads. This is why the ViT became the image encoder of CLIP, SigLIP and nearly all VLMs (session 166, session 167, session 177).
The ViT also brings the single most important number for VLM cost: tokens per image. One 336 × 336 image through CLIP ViT-L/14 becomes 576 tokens. One 384 × 384 image through SigLIP So400m/14 becomes 729 tokens. [Certain] An LLM must read every one of them, so resolution and patch size decide latency, memory and price. Later sessions spend a lot of effort on reducing this number (session 183, session 180).
The idea in one picture
A ViT has four parts. First, patchify: cut the image into a grid of \(P \times P\) squares. Second, patch embedding: flatten each square into a vector and map it with one linear layer to the model width \(D\). Third, add a position embedding to each token and put an extra learned CLS token in front. Fourth, run a standard transformer encoder (bidirectional attention, no causal mask) and read the image vector from the CLS token or from the mean of all tokens.
Everything after the first step is the transformer of Track 2. That is the point of the design: the authors kept the text transformer “as closely as possible”, so its code, its scaling behaviour and its fast kernels could be reused almost unchanged. [Certain]
Patchify: an image becomes a sequence
Take an image \(\mathbf{x} \in \R^{H \times W \times C}\) (height, width, channels; \(C = 3\) for RGB). Choose a patch size \(P\). Cut the image into non-overlapping \(P \times P\) squares, read them in raster order (left to right, then top to bottom, like reading a page), and flatten each square into one vector of \(P^2 C\) numbers. The number of patches is
- Read the symbols
- \(H, W, C\)
- image height, width (in pixels) and number of channels
- \(P\)
- patch side in pixels (16 in “ViT-B/16”, 14 in “ViT-L/14”)
- \(N\)
- number of patches, which is the sequence length of the transformer
- \(\mathbf{x}_p\)
- the matrix of flattened patches: one row per patch, \(P^2 C\) numbers per row
For a 224 × 224 RGB image and \(P = 16\): \(224/16 = 14\), so \(N = 14 \times 14 = 196\) patches, each with \(16 \cdot 16 \cdot 3 = 768\) numbers. If the side is not a multiple of \(P\), the usual implementation (a strided convolution, below) silently drops the last pixels: SigLIP at 384 with \(P = 14\) uses \(\lfloor 384/14 \rfloor = 27\) patches per side and ignores the last 6 pixel rows and columns. [Certain]
Figure 2 runs this cut as a loop, one patch per step. The pixel values are real: the small picture is sampled at the centre of every pixel, so each patch gives its true \(P^2 C\) numbers. Change the height, the width or the patch size and watch formula (164.1) recompute.
- input: image \(\mathbf{x}\) of size \(H \times W \times C\), patch size \(P\)
- \(g_h \leftarrow \lfloor H/P \rfloor,\ g_w \leftarrow \lfloor W/P \rfloor,\ N \leftarrow g_h g_w\) // (164.1); the rest is dropped
- for \(r = 0\) to \(g_h - 1\):
- for \(c = 0\) to \(g_w - 1\):
- \(i \leftarrow 1 + r\,g_w + c\) // index 0 is kept for CLS
- \(\mathbf{x}_p^{\,i} \leftarrow \operatorname{flatten}(\mathbf{x}[rP{:}rP{+}P,\ cP{:}cP{+}P,\ :])\) // \(P^2 C\) numbers
- return \(\mathbf{x}_p \in \R^{N \times P^2 C}\)
unfold and by the conv weight), and the token sequence, where slot 0 is the CLS token. Right: the loop, with the current line highlighted and its formula filled in with the live numbers.How to read Figure 2
Press Play. Each step reads one patch in raster order and writes it into slot \(i = 1 + r g_w + c\). When \(N\) is larger than 256, one step reads a whole row of patches. Click any patch to jump to it. Halve the patch size and \(N\) grows by 4 (formula 164.1); double both \(H\) and \(W\) and it also grows by 4. Set \(H = W = 384\) with \(P = 14\): a red band appears at the right and bottom edges. Those 6 pixel rows and columns do not fill a whole patch and the model never sees them. The image does not have to be square: \(H = 224\), \(W = 448\), \(P = 16\) gives \(14 \times 28 = 392\) patches.
Patch embedding: a linear layer, or a convolution with stride P
A raw patch has \(P^2 C\) numbers, but the transformer works with width \(D\). So each flattened patch is multiplied by one shared, trainable matrix \(\mathbf{E}\):
The same \(\mathbf{E}\) is used for every patch. Each column of \(\mathbf{E}\) is a \(P \times P \times C\) filter: it gives one number per patch, the dot product of the filter with the patch. That is exactly what a convolution with kernel size \(P\) and stride \(P\) computes. Write the conv weight as \(\mathbf{W} \in \R^{D \times C \times P \times P}\). For the patch in grid row \(r\) and column \(c\):
- Read the symbols
- \(d\)
- output channel, which becomes one dimension of the token (1 to \(D\))
- \(r, c\)
- row and column of the patch in the grid (from 0); \(g = W/P\) patches per row
- \(u, v\)
- pixel offset inside the patch
- \(\mathbf{E}\)
- the conv weight reshaped to \((C P^2) \times D\), in the same order used to flatten the patch
Because the stride equals the kernel size, the windows never overlap and never skip a pixel: every pixel belongs to exactly one patch. So “patchify + linear” and “Conv2d(C, D, kernel_size=P, stride=P)” are the same function. Implementations use the conv because it does the cutting, flattening and multiplying in one fast call. [Certain] Calling this a “CNN” is misleading, though: it is one layer with no overlap and no nonlinearity. All later mixing between patches is done by attention.
How to read Figure 3
Look at the last two patches. Both have mean 4.667, so a model that only looked at brightness could not tell them apart. The second filter fires on the bottom-right patch (a change from left to right) and the third filter fires on the bottom-left patch (a change from top to bottom). A real ViT-B/16 learns 768 such filters of size 16 × 16 × 3. The ViT paper shows that the main components of the learned filters look like smooth basis patterns, similar to the first layer of a CNN. [Likely]
Each patch has \(16 \cdot 16 \cdot 3 = 768\) numbers and \(D = 768\). The matrix \(\mathbf{E}\) has \(768 \times 768 = 589{,}824\) weights, plus 768 biases: 590,592 parameters. That is 0.7% of the 86.6M parameters of the model. For ViT-L/14 (\(D = 1024\)): \(14 \cdot 14 \cdot 3 = 588\) numbers per patch and \(588 \cdot 1024 + 1024 = 603{,}136\) parameters. The embedding is cheap; the cost of a ViT is in the encoder.
The input sequence: CLS token and position embeddings
Two more pieces finish the input. A learned CLS token \(\mathbf{x}_{\text{cls}} \in \R^D\) is put in front of the patch tokens (the idea comes from BERT, session 33). It is the same vector for every image; it carries no pixels. Through attention it can collect information from all patches, and its final state is used as the image representation. And a learned position embedding is added to every token, because attention by itself does not know where a token came from (session 13):
- Read the symbols
- \(\mathbf{z}_0\)
- the input to the first encoder layer: \(N + 1\) rows of width \(D\)
- \([\,\cdot\,;\,\cdot\,]\)
- stacking the vectors as rows of one matrix
- \(\mathbf{E}_{\text{pos}}\)
- one trainable vector per position (CLS slot plus each grid cell); for ViT-B/16 at 224 that is \(197 \times 768 = 151{,}296\) numbers
In the original ViT the position embedding is a plain learned table indexed by the raster index (“1D”). The model has to learn from data that index 15 is just below index 1. It does learn it: the paper shows that, after training, each position embedding is most similar to its neighbours in the same row and column. [Certain] We return to positions, and to what happens when the grid changes size, two sections below.
The encoder: the same transformer that reads text
The encoder is \(L\) identical pre-LN blocks, as in session 22: LayerNorm before each sub-layer, a residual connection around it, multi-head self-attention (MSA) and a two-layer MLP with GELU:
Two differences from a decoder LLM. There is no causal mask: every patch attends to every other patch, in both directions, because the whole image is known at once. And there is no next-token loss: the output is used for classification, for contrastive training (CLIP), or as input to an LLM. The model sizes were taken directly from BERT for Base and Large: [Certain]
| Model | Layers \(L\) | Width \(D\) | MLP size | Heads | Parameters |
|---|---|---|---|---|---|
| ViT-Base | 12 | 768 | 3072 | 12 | 86M |
| ViT-Large | 24 | 1024 | 4096 | 16 | 307M |
| ViT-Huge | 32 | 1280 | 5120 | 16 | 632M |
| SigLIP So400m (vision tower) | 27 | 1152 | 4304 | 16 | about 400M |
The name “ViT-L/14” means the Large model with \(P = 14\). The patch size does not change the parameter count of the encoder (only the small embedding), but it changes the number of tokens, and so the compute. Each block has about \(4D^2\) weights in attention (\(W_Q, W_K, W_V, W_O\)) and \(2 D F\) in the MLP; with \(F = 4D\):
One block with \(D = 768\), \(F = 3072\): \(4 \cdot 768^2 + 4 \cdot 768 = 2{,}362{,}368\) (attention) \(+\ 2 \cdot 768 \cdot 3072 + 3072 + 768 = 4{,}722{,}432\) (MLP) \(+\ 3{,}072\) (LayerNorms) \(= 7{,}087{,}872\). Twelve blocks: 85,054,464.
Add the patch embedding 590,592, the CLS token 768, the position table 151,296, the final LayerNorm 1,536 and a 1000-class head \(768 \cdot 1000 + 1000 = 769{,}000\). Total: 86,567,656, the “86M” of the table. The PyTorch code below builds this model and prints this number.
Reading out: CLS token, mean pooling, attention pooling
After \(L\) blocks we have \(N + 1\) output vectors. For a single image label, or for a CLIP embedding, we need one vector. There are three common choices:
- CLS token (ViT, DeiT, CLIP): take the final state of token 0. It has learned, through attention, to gather what the task needs.
- Mean pooling (also called global average pooling, GAP; used for example by MAE for fine-tuning [Likely]): average the patch outputs, just like a ResNet averages its last feature map. No extra token needed.
- Attention pooling (the “MAP head” of SigLIP): a small learned query attends once over all patch outputs. It is a CLS token that only acts at the end. [Likely]
Which is better? The ViT authors first found GAP “performed very poorly”, then showed the gap was fully explained by the learning rate: with its own learning rate, GAP works as well as the CLS token. [Certain] So the choice is mostly about convenience. For VLMs the question often disappears: LLaVA-style models feed all patch tokens to the LLM, not one pooled vector (LLaVA-1.5 takes the 576 patch outputs of the second-to-last layer of CLIP ViT-L/14-336). [Likely]
Now put all the pieces together and run them. Figure 4 is a complete ViT, small enough to read every number: an 8 × 8 grey image, \(D = 4\), one encoder block with one attention head and an MLP of size 8. The weights are random but fixed (seeded), so the output is not a useful prediction; what matters is the path each number takes. Every value is computed in your browser by the same formulas as above.
- input: image \(\mathbf{x}\) (8 × 8 × 1, pixel / 9), patch size \(P\)
- for each patch \(i\): \(\mathbf{e}_i \leftarrow \mathbf{x}_p^{\,i} \mathbf{E} + \mathbf{b}\) // (164.2)
- \(\mathbf{z}_0 \leftarrow [\mathbf{x}_{\text{cls}}; \mathbf{e}_1; \dots; \mathbf{e}_N] + \mathbf{E}_{\text{pos}}\) // (164.4)
- \(\mathbf{h} \leftarrow \operatorname{LN}(\mathbf{z}_0)\)
- \(Q, K, V \leftarrow \mathbf{h} W_Q,\ \mathbf{h} W_K,\ \mathbf{h} W_V\)
- \(A \leftarrow \softmax(Q K^\top / \sqrt{D})\) // every row, no causal mask
- \(\mathbf{z}' \leftarrow \mathbf{z}_0 + (A V) W_O\) // (164.5), first line
- \(\mathbf{z}_1 \leftarrow \mathbf{z}' + \operatorname{MLP}(\operatorname{LN}(\mathbf{z}'))\) // (164.5), second line
- \(\mathbf{y} \leftarrow \operatorname{LN}(\mathbf{z}_1[0])\) or \(\tfrac1N \sum_i \operatorname{LN}(\mathbf{z}_1[i])\) // (164.7)
How to read Figure 4
Press Play and follow the CLS row (the top row of the middle panel). It starts as a learned vector with no pixels in it. After line 6 its attention row (outlined) says how much it reads from each patch: with \(P = 4\) the weights are 0.144 for itself and 0.072, 0.232, 0.069 and 0.483 for patches 1 to 4. The read-out is \(\mathbf{y} = (0.330, 1.329, -0.217, -1.442)\). Now set the patch order to “shuffled”: the same four patches enter the sequence in another order, the positions no longer match the patches, and \(\mathbf{y}\) changes to \((0.946, 0.966, -0.542, -1.370)\). Turn positions off and compare the two orders again: \(\mathbf{y} = (0.040, 1.348, 0.086, -1.474)\) for both, the difference is exactly 0. Without positions the model cannot tell where a patch was: it sees a bag of patches. The mean read-out behaves the same way.
Position embeddings in 2D, and changing the resolution
Without positions, the encoder sees a bag of patches: shuffle them and the CLS output does not change (attention is permutation equivariant, session 16). Figure 4 shows this with real numbers: with positions off, the shuffled image gives exactly the same output. The ViT paper measured this on ImageNet 5-shot linear accuracy with ViT-B/16: no position embedding 0.614, learned 1D 0.642, learned 2D (separate row and column tables, each \(D/2\), concatenated) 0.640, relative positions 0.640. [Certain] So positions matter, but the exact kind matters little at a 14 × 14 grid. The authors’ explanation: the grid is small, so all methods learn it equally easily. [Speculation, by the authors]
The real problem appears when the resolution changes. A model trained at 224 with \(P = 16\) has a 14 × 14 table. Fine-tuning at 384 gives a 24 × 24 grid, so 576 positions that do not exist in the table. Fine-tuning at a higher resolution than pretraining is a standard trick (it is cheap and raises accuracy), so this happens all the time. [Certain] The fix from the ViT paper: treat the \(14 \times 14 \times D\) table as a small image with \(D\) channels, and resize it with 2D interpolation to \(24 \times 24 \times D\). The CLS position is kept as it is. With the usual “align corners = False” convention, target cell \(j\) of a new grid of size \(g'\) reads from the old grid at
and blends the old vectors around \(u_j\) (2 neighbours per axis for bilinear, 4 for bicubic). Example: \(g = 14\), \(g' = 24\): cell 0 reads at \(-0.208\) (clamped to the border), cell 23 at \(13.208\). The method matters. A blend of two different unit vectors is shorter than either (picture the midpoint of a chord in a circle). DeiT reports that bilinear interpolation shrinks the norm of the embeddings, that these low-norm vectors cause “a significant drop in accuracy” before fine-tuning, and so uses bicubic, which roughly keeps the norm. [Certain] Figure 5 shows the effect.
How to read Figure 5
“Roughness” scales the frequencies: low roughness means neighbours are almost equal, high roughness means the table changes fast from cell to cell, as learned tables often do. At roughness 1.5 and \(g' = 24\), bilinear vectors have average length 0.920 (8% too short), while bicubic gives 1.012. Raise the roughness and the bilinear loss grows. In both cases the similarity pattern keeps its shape: the resized table still says “this cell is near those cells”, which is why a short fine-tune at the new resolution recovers the accuracy. This is a stand-in, not real ViT weights; the numbers show the mechanism, not a measured accuracy drop.
Interpolation is a patch, not a cure: the model saw one grid during training, and very different grids still hurt until it is fine-tuned. Newer encoders change the position scheme instead. NaViT trains on many resolutions and aspect ratios at once (“Patch n’ Pack”), and Qwen2-VL replaces the learned table with 2D rotary embeddings (RoPE on the row and the column), so any grid size works without interpolation (session 14, session 180). [Certain]
- input: trained position table \(\mathbf{E}_{\text{pos}} \in \R^{(1 + g^2) \times D}\), new image side \(R'\), patch size \(P\)
- \(g' \leftarrow \lfloor R' / P \rfloor\) // the patch size stays the same; the grid grows
- split \(\mathbf{E}_{\text{pos}}\) into the CLS row \(\mathbf{p}_{\text{cls}}\) and the grid rows \(\mathbf{G} \in \R^{g^2 \times D}\)
- reshape \(\mathbf{G}\) to an image of shape \(D \times g \times g\) (row-major, the same raster order as the patches)
- \(\mathbf{G}' \leftarrow\) bicubic resize to \(D \times g' \times g'\) // formula (164.8) for each target cell
- reshape back to \(g'^2 \times D\); \(\mathbf{E}'_{\text{pos}} \leftarrow [\mathbf{p}_{\text{cls}}; \mathbf{G}']\)
- replace the table in the model; patch embedding and all blocks are unchanged
- fine-tune briefly at the new resolution if accuracy matters
Tokens per image: resolution decides the cost
The parameters of a ViT do not depend on the image size. The compute does, and strongly. Formula (164.1) says the token count grows with the square of the image side: \(N = \lfloor R/P \rfloor^2\) for an \(R \times R\) image. Per layer, with sequence length \(N\) (including CLS), width \(D\) and MLP size \(F\), the forward pass costs about
(A MAC is one multiply-add, which is 2 FLOPs; see session 25.) The first two terms grow like \(N\), so like \(R^2\). The attention term grows like \(N^2\), so like \(R^4\). With \(F = 4D\) the fraction of compute spent in attention scores has a very simple form:
For ViT-B/16 at 224 (\(N = 197\), \(6D = 4608\)) attention is only 4.1% of the compute; the MLP and projections dominate. At 1024 (\(N = 4097\)) it is 47.1%. So at normal resolutions a ViT is a “linear” model in practice, and the quadratic term only takes over for large images or many video frames.
How to read Figure 6
Start with ViT-B/16 at 224: 197 tokens and 17.56 G multiply-adds, which is the “17.6 GFLOPs” printed in many tables (those tables count multiply-adds) [Likely]. Move to 384: the tokens grow 2.93 times (197 to 577) but the compute grows 3.16 times, because the attention share rises from 4.1% to 11.1%. Switch to SigLIP So400m/14 at 384: 729 tokens. Set 4 images per prompt: the LLM now reads 2,916 visual tokens before the first word of text.
| Encoder, input | Grid | Patch tokens | Encoder G MACs | Attention share |
|---|---|---|---|---|
| ViT-B/16, 224 | 14 × 14 | 196 (+ CLS) | 17.56 | 4.1% |
| ViT-B/16, 384 | 24 × 24 | 576 (+ CLS) | 55.48 | 11.1% |
| CLIP ViT-L/14, 336 | 24 × 24 | 576 (+ CLS) | 190.96 | 8.6% |
| SigLIP So400m/14, 384 | 27 × 27 | 729 (no CLS) | 333.22 | 9.9% |
The shapes are from the Hugging Face configs (CLIP: 24 layers, width 1024, MLP 4096; SigLIP: 27 layers, width 1152, MLP 4304). [Certain] The compute numbers follow from formula (164.9); they are not measured speeds. Note that the encoder is not the expensive part for long answers: a 7B LLM reading 576 tokens costs about \(7 \times 10^9 \times 576 \approx 4\) T multiply-adds for the prefill alone, roughly 20 times the CLIP encoder. This is why VLMs try so hard to send fewer visual tokens to the LLM (pixel shuffle, pooling, Q-Former, session 178).
The patch size is the main knob. ViT-B/32 at 224 gives only 49 tokens: fast, but each token covers 32 × 32 pixels and small text disappears. ViT-B/8 gives 784 tokens. The ViT paper found that smaller patches give “surprisingly robust improvements without introducing parameters”: you pay in compute, not in model size. [Certain] For video the numbers multiply: 16 frames of ViT-B/16 at 224 are 3,136 tokens. Encoding the frames separately costs about 278 G MACs in the layers; one joint attention over all 3,136 tokens costs about 448 G, because attention then takes 40% of the compute (formula 164.10). That is the reason video encoders factor attention into space and time (session 202).
- input: image tensor \(\mathbf{x}\) of shape \(C \times H \times W\), already normalized with the encoder’s own mean and standard deviation
- \(\mathbf{T} \leftarrow \operatorname{Conv2d}(\mathbf{x};\ \text{kernel} = P, \text{stride} = P)\) // shape \(D \times H/P \times W/P\), formula (164.3)
- flatten the grid in raster order to \(N \times D\) // \(N = HW/P^2\), formula (164.1)
- prepend \(\mathbf{x}_{\text{cls}}\); add \(\mathbf{E}_{\text{pos}}\) // formula (164.4); resize \(\mathbf{E}_{\text{pos}}\) first if the grid changed (Algorithm 1)
- for \(\ell = 1\) to \(L\): \(\mathbf{z} \leftarrow \mathbf{z} + \operatorname{MSA}(\operatorname{LN}(\mathbf{z}))\); \(\;\mathbf{z} \leftarrow \mathbf{z} + \operatorname{MLP}(\operatorname{LN}(\mathbf{z}))\) // no causal mask
- \(\mathbf{z} \leftarrow \operatorname{LN}(\mathbf{z})\)
- return \(\mathbf{z}[0]\) (CLS), or the mean of \(\mathbf{z}[1{:}]\), or all of \(\mathbf{z}[1{:}]\) for a VLM // formula (164.7)
Training a ViT: data hunger, and how DeiT fixed it
A CNN builds in locality and translation equivariance in every layer. A ViT builds in almost nothing: only the patch cut and, at fine-tuning, the 2D interpolation of positions. The paper says: “all spatial relations between the patches have to be learned from scratch.” [Certain] With little data this hurts. Trained on ImageNet alone (1.3M images), ViTs were “a few percentage points below ResNets of comparable size”. Pretrained on JFT-300M (a private set of 300M images), they won: ViT-H/14 reached 88.55% ImageNet top-1 and ViT-L/16 87.76%, with less pretraining compute than the best CNN of the time (0.68k TPUv3-core-days for ViT-L/16 against 9.9k for BiT-L). [Certain]
DeiT (Data-efficient image Transformers) showed that the data hunger was mostly a training-recipe problem. It trained the same ViT-B architecture on ImageNet-1k only, on a single 8-GPU node in about three days (53 hours of pretraining). [Certain] The recipe: AdamW, 300 epochs, cosine schedule with 5 warmup epochs, weight decay 0.05, strong augmentation (RandAugment, Mixup, CutMix, random erasing, repeated augmentation), stochastic depth 0.1 and label smoothing 0.1. DeiT also added a distillation token: a second extra token, like CLS, whose output is trained to match the predictions of a CNN teacher (a RegNetY-16GF). [Certain]
| Model (ImageNet-1k only, as reported by DeiT) | Params | Resolution | Top-1 |
|---|---|---|---|
| ViT-B/16 (original recipe) | 86M | 384 | 77.9% |
| DeiT-Ti / DeiT-S / DeiT-B | 5M / 22M / 86M | 224 | 72.2% / 79.8% / 81.8% |
| DeiT-B, fine-tuned at 384 | 86M | 384 | 83.1% |
| DeiT-B with distillation token | 87M | 224 | 83.4% |
| DeiT-B distilled, 1000 epochs, fine-tuned at 384 | 87M | 384 | 85.2% |
[Certain, all numbers from the DeiT paper.] Today most ViTs used in VLMs are not trained with labels at all: they come from image–text contrastive training (CLIP, SigLIP) or self-supervised training (MAE, DINOv2), covered in the next sessions (session 165, session 166).
The PyTorch version
The code checks that the conv patch embedding equals “unfold + matrix multiply”, builds ViT-B/16 and counts its parameters, resizes the position table to run at 384, and prints the token counts from the worked example.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) patch embedding: Conv2d with kernel = stride = P is a linear layer on flattened patches (164.3) P, C, D = 16, 3, 768 conv = nn.Conv2d(C, D, kernel_size=P, stride=P) x = torch.randn(2, C, 224, 224) a = conv(x).flatten(2).transpose(1, 2) # (2, 196, 768) patches = F.unfold(x, kernel_size=P, stride=P).transpose(1, 2) # (2, 196, 768): P*P*C numbers each b = patches @ conv.weight.reshape(D, -1).T + conv.bias print(a.shape, torch.allclose(a, b, atol=1e-4)) # 2) a minimal ViT: patchify, CLS token, learned positions, pre-LN blocks (164.4, 164.5) class ViT(nn.Module): def __init__(self, img=224, P=16, D=768, depth=12, heads=12, mlp=3072, classes=1000, pool="cls"): super().__init__() self.P, self.pool, g = P, pool, img // P self.patch = nn.Conv2d(3, D, kernel_size=P, stride=P) self.cls = nn.Parameter(torch.zeros(1, 1, D)) self.pos = nn.Parameter(torch.randn(1, 1 + g * g, D) * 0.02) layer = nn.TransformerEncoderLayer(D, heads, mlp, dropout=0.0, activation="gelu", batch_first=True, norm_first=True) self.blocks = nn.TransformerEncoder(layer, depth, enable_nested_tensor=False) self.norm = nn.LayerNorm(D) self.head = nn.Linear(D, classes) def forward(self, x): z = self.patch(x).flatten(2).transpose(1, 2) # (B, N, D) cls = self.cls.expand(z.shape[0], -1, -1) z = torch.cat([cls, z], dim=1) + self.pos # formula (164.4) z = self.norm(self.blocks(z)) # no mask: bidirectional y = z[:, 0] if self.pool == "cls" else z[:, 1:].mean(dim=1) # formula (164.7) return self.head(y) model = ViT().eval() print(sum(p.numel() for p in model.parameters())) # 86,567,656 = ViT-B/16 with torch.no_grad(): print(model(torch.randn(1, 3, 224, 224)).shape) # 3) run at 384: resize the 14 x 14 position grid to 24 x 24 (Algorithm 1, bicubic as in DeiT) def resize_pos(pos, new_grid): cls_pos, grid_pos = pos[:, :1], pos[:, 1:] g = math.isqrt(grid_pos.shape[1]) grid_pos = grid_pos.reshape(1, g, g, -1).permute(0, 3, 1, 2) # (1, D, g, g) grid_pos = F.interpolate(grid_pos, size=(new_grid, new_grid), mode="bicubic", align_corners=False) grid_pos = grid_pos.permute(0, 2, 3, 1).reshape(1, new_grid * new_grid, -1) return torch.cat([cls_pos, grid_pos], dim=1) with torch.no_grad(): model.pos = nn.Parameter(resize_pos(model.pos, 384 // 16)) print(model.pos.shape, model(torch.randn(1, 3, 384, 384)).shape) # 4) tokens per image: resolution and patch size decide the cost (164.1) for side, p in [(224, 16), (384, 16), (336, 14), (384, 14)]: print(side, p, (side // p) ** 2)
Output with PyTorch 2.8 on CPU: torch.Size([2, 196, 768]) True; 86567656; torch.Size([1, 1000]); torch.Size([1, 577, 768]) torch.Size([1, 1000]); then 196, 576, 576 and 729 tokens. The same model with pool="mean" uses mean pooling and has the same parameters (the CLS token is still created but not read).
Where this is used
| Application | How the ViT is used |
|---|---|
| Image encoder of VLMs (LLaVA, Qwen-VL, InternVL, PaliGemma, SmolVLM) | A CLIP or SigLIP ViT; all patch outputs go through a connector into the LLM, so tokens per image decide the prompt length (session 177, session 178) |
| Image–text retrieval and zero-shot classification | The pooled ViT output (CLS or attention pooling) is the image embedding compared with text embeddings by cosine similarity (session 166, session 169) |
| Video retrieval, near-duplicate detection and fingerprinting | One pooled ViT embedding per sampled frame, averaged or matched over time; cheap because each frame is encoded once and cached |
| Video classification and action recognition | Frames are patchified into space–time “tubelets” (ViViT, VideoMAE) and attention is factored into space and time to keep the token count manageable (session 202) |
| Key-moment detection in long video | Per-frame ViT features scored over time; resolution and frame rate set the token budget |
| Detection and segmentation | Patch outputs reshaped back to a \(g \times g\) feature map; DINOv2 patch features are strong for dense tasks (session 168) |
| Self-supervised pretraining | MAE drops 75% of the patch tokens before the encoder, which is cheap precisely because the ViT input is a set of tokens (session 165) |
| Image generation | Diffusion transformers (DiT) patchify the latent image the same way (session 228) |
Common mistakes
- Feeding a resolution the model was not built for, without resizing the positions. A 384 image into a 224 ViT gives 577 tokens and only 197 position vectors: a shape error, or, in code that crops the table, silently wrong positions. Interpolate (Algorithm 1) and fine-tune.
- Bilinear interpolation of the position table. It shrinks the vectors (Figure 5). Use bicubic, as DeiT and timm do. [Likely for timm’s default]
- Forgetting the CLS slot when reshaping. The table has \(1 + g^2\) rows. Reshaping all of them to a \(g \times g\) grid fails, or worse, shifts every position by one if you drop the wrong row.
- Image side not a multiple of \(P\). The strided conv silently drops the remainder (6 pixels at 384 with \(P = 14\)). If edges matter (text near borders), resize to a multiple of \(P\) or pad.
- Reshape instead of permute when making patches by hand.
x.reshape(N, P*P*C)on a \(C \times H \times W\) tensor mixes pixels from different patches. Useunfoldor the conv, or permute to \((H/P, W/P, P, P, C)\) first. - Using a causal mask in the encoder. Copying LLM code gives patch 1 no view of patch 2. Vision encoders are bidirectional.
- Treating token count as linear in resolution. Doubling the side gives 4 times the tokens and, through the LLM prefill and the KV cache, roughly 4 times the cost of the visual part of the prompt, more where attention dominates.
- Training a ViT from scratch on a small dataset with a CNN recipe. Without strong augmentation, regularization and many epochs (DeiT) or a pretrained checkpoint, a ViT does worse than a ResNet.
Interview questions
How does a Vision Transformer turn an image into tokens?basic
Why is the patch embedding implemented as a convolution? Does that make ViT a CNN?core
How many tokens does a 336 × 336 image give with patch 14? What happens to the cost if you double the side?core
CLS token or mean pooling?core
How do you run a ViT trained at 224 on 384 × 384 images?core
Why did the first ViT need 300 million images, and how did DeiT train it on ImageNet alone?core
What happens without position embeddings? Why do 1D and 2D work about equally?basic
What is the trade-off between patch sizes 32, 16, 14 and 8?core
Count the parameters of ViT-B/16.core
At what point does attention dominate the compute of a ViT?advanced
What are the high-norm “artifact” tokens in large ViTs, and what are registers?advanced
In a VLM, why is the ViT token count a problem, and what do real systems do about it?advanced
Exercises
- How many tokens does a 448 × 448 image give with \(P = 14\), with and without a CLS token?
- Compute the number of parameters of the patch embedding of ViT-L/14.
- A 1280 × 720 video frame is fed to a ViT with \(P = 16\) at its native size. How many patch tokens? Compare with 224 × 224.
- With formula (164.8), where does a 24-cell target grid read from a 14-cell source grid for target cells 0 and 23?
- Use formula (164.10) to find the attention share of ViT-B/16 at 1024 × 1024 (with CLS).
- In Figure 3, change the bottom-right patch to all 5s. What are its three token values, and which other patch’s token is it now closest to?
Answerscheck
Key takeaways
- A ViT cuts the image into \(N = HW/P^2\) patches (164.1); each becomes one token through a shared linear map (164.2), which is the same thing as a Conv2d with kernel = stride = \(P\) (164.3).
- A CLS token and a learned position embedding complete the input (164.4); the encoder is a standard bidirectional pre-LN transformer (164.5) with about \(12D^2\) parameters per block (164.6): 86,567,656 for ViT-B/16.
- CLS, mean pooling and attention pooling work about equally (164.7); VLMs usually take all patch tokens instead.
- Positions are needed (0.614 without vs 0.642 with), 1D and 2D work alike, and a new resolution needs interpolation of the table (164.8), preferably bicubic; 2D RoPE and NaViT avoid it.
- Tokens grow with the square of the image side; compute per layer is \(4ND^2 + 2NDF + 2N^2D\) (164.9) and attention’s share is \(N/(N + 6D)\) (164.10): 4.1% at 224 for ViT-B/16. Tokens per image is the number that drives VLM cost.
- ViT has weak inductive bias: it needs large pretraining or a strong recipe (DeiT: 81.8% on ImageNet alone, 83.4% with distillation).
Key formulas
References for this session
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Touvron et al. (2020). Training data-efficient image transformers & distillation through attention
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
- Darcet, Oquab, Mairal, Bojanowski (2023). Vision Transformers Need Registers
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
- Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision