LLM & VLM courseBlog
164Track 7: Vision and VLMs · session 2 of 38

The Vision Transformer (ViT)

The Vision Transformer cuts an image into small square patches, turns each patch into one token with a linear layer, and then runs the same transformer encoder that reads text. This session builds it piece by piece: patchify, patch embedding (and why it is a convolution with stride P), the CLS token and mean pooling, 2D position embeddings and how to stretch them to a new resolution, and the most important number for any VLM: how many tokens one image costs.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 13, 19, 22 and 163 (position, multi-head attention, residual stream, images and CNNs)

What you will learn

  • How patchify turns an H × W × C image into N = HW/P² tokens, and what one token holds
  • Why the patch embedding is both a linear layer on flattened patches and a Conv2d with kernel = stride = P, and how many weights it has
  • How the CLS token, mean pooling and attention pooling turn many tokens into one image vector
  • Why ViT needs position embeddings, why 1D and 2D versions work about equally, and how to interpolate them for a new resolution (bilinear vs bicubic)
  • Counting a ViT by hand: 86,567,656 parameters for ViT-B/16 and 17.6 G multiply-adds at 224 × 224
  • How resolution and patch size decide the token count and cost, and why this number drives VLM design
  • Why the first ViT needed 300 million images, and how DeiT trained it on ImageNet alone

Why this matters

Almost every vision-language model you will meet (LLaVA, Qwen-VL, InternVL, PaliGemma, SmolVLM) reads images with a Vision Transformer. The idea is short: cut the image into small squares, call each square a token, and give the tokens to a normal transformer. The title of the paper says it: “An Image is Worth 16x16 Words”.

The previous session (session 163) ended with what CNNs miss: long-range relations, and a token interface. The Vision Transformer (ViT) answers both. Self-attention connects any two patches in one layer, and the output is a sequence of vectors with positions, which is exactly the format a language model already reads. This is why the ViT became the image encoder of CLIP, SigLIP and nearly all VLMs (session 166, session 167, session 177).

The ViT also brings the single most important number for VLM cost: tokens per image. One 336 × 336 image through CLIP ViT-L/14 becomes 576 tokens. One 384 × 384 image through SigLIP So400m/14 becomes 729 tokens. [Certain] An LLM must read every one of them, so resolution and patch size decide latency, memory and price. Later sessions spend a lot of effort on reducing this number (session 183, session 180).

The idea in one picture

A ViT has four parts. First, patchify: cut the image into a grid of \(P \times P\) squares. Second, patch embedding: flatten each square into a vector and map it with one linear layer to the model width \(D\). Third, add a position embedding to each token and put an extra learned CLS token in front. Fourth, run a standard transformer encoder (bidirectional attention, no causal mask) and read the image vector from the CLS token or from the mean of all tokens.

patch tokensCLS tokenposition embedding (added)
Figure 1. The ViT pipeline for ViT-B/16 at 224 × 224. The image becomes \(14 \times 14 = 196\) patches; each patch (768 numbers) is mapped to a 768-dimensional token; a CLS token is added in front, so the encoder reads 197 tokens. Nothing in the encoder is specific to images.

Everything after the first step is the transformer of Track 2. That is the point of the design: the authors kept the text transformer “as closely as possible”, so its code, its scaling behaviour and its fast kernels could be reused almost unchanged. [Certain]

Patchify: an image becomes a sequence

Take an image \(\mathbf{x} \in \R^{H \times W \times C}\) (height, width, channels; \(C = 3\) for RGB). Choose a patch size \(P\). Cut the image into non-overlapping \(P \times P\) squares, read them in raster order (left to right, then top to bottom, like reading a page), and flatten each square into one vector of \(P^2 C\) numbers. The number of patches is

\[ N = \frac{H}{P} \cdot \frac{W}{P} = \frac{HW}{P^2}, \qquad \mathbf{x}_p \in \R^{N \times (P^2 C)} \tag{164.1} \]
Read the symbols
\(H, W, C\)
image height, width (in pixels) and number of channels
\(P\)
patch side in pixels (16 in “ViT-B/16”, 14 in “ViT-L/14”)
\(N\)
number of patches, which is the sequence length of the transformer
\(\mathbf{x}_p\)
the matrix of flattened patches: one row per patch, \(P^2 C\) numbers per row

For a 224 × 224 RGB image and \(P = 16\): \(224/16 = 14\), so \(N = 14 \times 14 = 196\) patches, each with \(16 \cdot 16 \cdot 3 = 768\) numbers. If the side is not a multiple of \(P\), the usual implementation (a strided convolution, below) silently drops the last pixels: SigLIP at 384 with \(P = 14\) uses \(\lfloor 384/14 \rfloor = 27\) patches per side and ignores the last 6 pixel rows and columns. [Certain]

Figure 2 runs this cut as a loop, one patch per step. The pixel values are real: the small picture is sampled at the centre of every pixel, so each patch gives its true \(P^2 C\) numbers. Change the height, the width or the patch size and watch formula (164.1) recompute.

Patch size \(P\)
  1. input: image \(\mathbf{x}\) of size \(H \times W \times C\), patch size \(P\)
  2. \(g_h \leftarrow \lfloor H/P \rfloor,\ g_w \leftarrow \lfloor W/P \rfloor,\ N \leftarrow g_h g_w\) // (164.1); the rest is dropped
  3. for \(r = 0\) to \(g_h - 1\):
  4. for \(c = 0\) to \(g_w - 1\):
  5. \(i \leftarrow 1 + r\,g_w + c\) // index 0 is kept for CLS
  6. \(\mathbf{x}_p^{\,i} \leftarrow \operatorname{flatten}(\mathbf{x}[rP{:}rP{+}P,\ cP{:}cP{+}P,\ :])\) // \(P^2 C\) numbers
  7. return \(\mathbf{x}_p \in \R^{N \times P^2 C}\)
Figure 2. Patchify as a loop. Left: the picture at \(H \times W\) pixels, cut into \(P \times P\) patches (blue = already read). Below it: the current patch zoomed to its \(P \times P\) pixels, the same patch flattened into its red, green and blue planes (\(P^2\) numbers each, the order used by unfold and by the conv weight), and the token sequence, where slot 0 is the CLS token. Right: the loop, with the current line highlighted and its formula filled in with the live numbers.
How to read Figure 2

Press Play. Each step reads one patch in raster order and writes it into slot \(i = 1 + r g_w + c\). When \(N\) is larger than 256, one step reads a whole row of patches. Click any patch to jump to it. Halve the patch size and \(N\) grows by 4 (formula 164.1); double both \(H\) and \(W\) and it also grows by 4. Set \(H = W = 384\) with \(P = 14\): a red band appears at the right and bottom edges. Those 6 pixel rows and columns do not fill a whole patch and the model never sees them. The image does not have to be square: \(H = 224\), \(W = 448\), \(P = 16\) gives \(14 \times 28 = 392\) patches.

Patch embedding: a linear layer, or a convolution with stride P

A raw patch has \(P^2 C\) numbers, but the transformer works with width \(D\). So each flattened patch is multiplied by one shared, trainable matrix \(\mathbf{E}\):

\[ \mathbf{e}_i = \mathbf{x}_p^{\,i}\, \mathbf{E} + \mathbf{b}, \qquad \mathbf{E} \in \R^{(P^2 C) \times D},\quad \mathbf{b} \in \R^{D}, \quad i = 1, \dots, N \tag{164.2} \]

The same \(\mathbf{E}\) is used for every patch. Each column of \(\mathbf{E}\) is a \(P \times P \times C\) filter: it gives one number per patch, the dot product of the filter with the patch. That is exactly what a convolution with kernel size \(P\) and stride \(P\) computes. Write the conv weight as \(\mathbf{W} \in \R^{D \times C \times P \times P}\). For the patch in grid row \(r\) and column \(c\):

\[ \operatorname{Conv}(\mathbf{x})[d, r, c] = b_d + \sum_{k=1}^{C} \sum_{u=0}^{P-1} \sum_{v=0}^{P-1} W[d, k, u, v]\; x[k,\, rP + u,\, cP + v] = \big(\mathbf{x}_p^{\,rg + c + 1}\, \mathbf{E} + \mathbf{b}\big)_d \tag{164.3} \]
Read the symbols
\(d\)
output channel, which becomes one dimension of the token (1 to \(D\))
\(r, c\)
row and column of the patch in the grid (from 0); \(g = W/P\) patches per row
\(u, v\)
pixel offset inside the patch
\(\mathbf{E}\)
the conv weight reshaped to \((C P^2) \times D\), in the same order used to flatten the patch

Because the stride equals the kernel size, the windows never overlap and never skip a pixel: every pixel belongs to exactly one patch. So “patchify + linear” and “Conv2d(C, D, kernel_size=P, stride=P)” are the same function. Implementations use the conv because it does the cutting, flattening and multiplying in one fast call. [Certain] Calling this a “CNN” is misleading, though: it is one layer with no overlap and no nonlinearity. All later mixing between patches is done by attention.

Figure 3. Patch embedding by hand. A 6 × 6 one-channel image, \(P = 3\), so \(N = 4\) patches of 9 numbers. The matrix \(\mathbf{E}\) has \(D = 3\) columns (three filters): the mean brightness, “left column minus right column”, and “top row minus bottom row”. Each step slides the window by 3 pixels (stride = kernel) and produces one token.
How to read Figure 3

Look at the last two patches. Both have mean 4.667, so a model that only looked at brightness could not tell them apart. The second filter fires on the bottom-right patch (a change from left to right) and the third filter fires on the bottom-left patch (a change from top to bottom). A real ViT-B/16 learns 768 such filters of size 16 × 16 × 3. The ViT paper shows that the main components of the learned filters look like smooth basis patterns, similar to the first layer of a CNN. [Likely]

Worked example: the patch embedding of ViT-B/16

Each patch has \(16 \cdot 16 \cdot 3 = 768\) numbers and \(D = 768\). The matrix \(\mathbf{E}\) has \(768 \times 768 = 589{,}824\) weights, plus 768 biases: 590,592 parameters. That is 0.7% of the 86.6M parameters of the model. For ViT-L/14 (\(D = 1024\)): \(14 \cdot 14 \cdot 3 = 588\) numbers per patch and \(588 \cdot 1024 + 1024 = 603{,}136\) parameters. The embedding is cheap; the cost of a ViT is in the encoder.

The input sequence: CLS token and position embeddings

Two more pieces finish the input. A learned CLS token \(\mathbf{x}_{\text{cls}} \in \R^D\) is put in front of the patch tokens (the idea comes from BERT, session 33). It is the same vector for every image; it carries no pixels. Through attention it can collect information from all patches, and its final state is used as the image representation. And a learned position embedding is added to every token, because attention by itself does not know where a token came from (session 13):

\[ \mathbf{z}_0 = \big[\, \mathbf{x}_{\text{cls}};\ \mathbf{e}_1;\ \mathbf{e}_2;\ \dots;\ \mathbf{e}_N \,\big] + \mathbf{E}_{\text{pos}}, \qquad \mathbf{E}_{\text{pos}} \in \R^{(N+1) \times D} \tag{164.4} \]
Read the symbols
\(\mathbf{z}_0\)
the input to the first encoder layer: \(N + 1\) rows of width \(D\)
\([\,\cdot\,;\,\cdot\,]\)
stacking the vectors as rows of one matrix
\(\mathbf{E}_{\text{pos}}\)
one trainable vector per position (CLS slot plus each grid cell); for ViT-B/16 at 224 that is \(197 \times 768 = 151{,}296\) numbers

In the original ViT the position embedding is a plain learned table indexed by the raster index (“1D”). The model has to learn from data that index 15 is just below index 1. It does learn it: the paper shows that, after training, each position embedding is most similar to its neighbours in the same row and column. [Certain] We return to positions, and to what happens when the grid changes size, two sections below.

The encoder: the same transformer that reads text

The encoder is \(L\) identical pre-LN blocks, as in session 22: LayerNorm before each sub-layer, a residual connection around it, multi-head self-attention (MSA) and a two-layer MLP with GELU:

\[ \begin{aligned} \mathbf{z}'_\ell &= \mathbf{z}_{\ell-1} + \operatorname{MSA}\big(\operatorname{LN}(\mathbf{z}_{\ell-1})\big) \\ \mathbf{z}_\ell &= \mathbf{z}'_\ell + \operatorname{MLP}\big(\operatorname{LN}(\mathbf{z}'_\ell)\big), \qquad \ell = 1, \dots, L \end{aligned} \tag{164.5} \]

Two differences from a decoder LLM. There is no causal mask: every patch attends to every other patch, in both directions, because the whole image is known at once. And there is no next-token loss: the output is used for classification, for contrastive training (CLIP), or as input to an LLM. The model sizes were taken directly from BERT for Base and Large: [Certain]

ModelLayers \(L\)Width \(D\)MLP sizeHeadsParameters
ViT-Base1276830721286M
ViT-Large241024409616307M
ViT-Huge321280512016632M
SigLIP So400m (vision tower)271152430416about 400M

The name “ViT-L/14” means the Large model with \(P = 14\). The patch size does not change the parameter count of the encoder (only the small embedding), but it changes the number of tokens, and so the compute. Each block has about \(4D^2\) weights in attention (\(W_Q, W_K, W_V, W_O\)) and \(2 D F\) in the MLP; with \(F = 4D\):

\[ \text{params per block} = \underbrace{4D^2 + 4D}_{\text{attention}} + \underbrace{2DF + F + D}_{\text{MLP}} + \underbrace{4D}_{\text{two LayerNorms}} \;\approx\; 12 D^2 \tag{164.6} \]
Worked example: count ViT-B/16 exactly

One block with \(D = 768\), \(F = 3072\): \(4 \cdot 768^2 + 4 \cdot 768 = 2{,}362{,}368\) (attention) \(+\ 2 \cdot 768 \cdot 3072 + 3072 + 768 = 4{,}722{,}432\) (MLP) \(+\ 3{,}072\) (LayerNorms) \(= 7{,}087{,}872\). Twelve blocks: 85,054,464.

Add the patch embedding 590,592, the CLS token 768, the position table 151,296, the final LayerNorm 1,536 and a 1000-class head \(768 \cdot 1000 + 1000 = 769{,}000\). Total: 86,567,656, the “86M” of the table. The PyTorch code below builds this model and prints this number.

Reading out: CLS token, mean pooling, attention pooling

After \(L\) blocks we have \(N + 1\) output vectors. For a single image label, or for a CLIP embedding, we need one vector. There are three common choices:

\[ \mathbf{y}_{\text{cls}} = \operatorname{LN}\big(\mathbf{z}_L^{0}\big) \qquad \text{or} \qquad \mathbf{y}_{\text{mean}} = \frac{1}{N} \sum_{i=1}^{N} \operatorname{LN}\big(\mathbf{z}_L^{i}\big) \tag{164.7} \]
  • CLS token (ViT, DeiT, CLIP): take the final state of token 0. It has learned, through attention, to gather what the task needs.
  • Mean pooling (also called global average pooling, GAP; used for example by MAE for fine-tuning [Likely]): average the patch outputs, just like a ResNet averages its last feature map. No extra token needed.
  • Attention pooling (the “MAP head” of SigLIP): a small learned query attends once over all patch outputs. It is a CLS token that only acts at the end. [Likely]

Which is better? The ViT authors first found GAP “performed very poorly”, then showed the gap was fully explained by the learning rate: with its own learning rate, GAP works as well as the CLS token. [Certain] So the choice is mostly about convenience. For VLMs the question often disappears: LLaVA-style models feed all patch tokens to the LLM, not one pooled vector (LLaVA-1.5 takes the 576 patch outputs of the second-to-last layer of CLIP ViT-L/14-336). [Likely]

Now put all the pieces together and run them. Figure 4 is a complete ViT, small enough to read every number: an 8 × 8 grey image, \(D = 4\), one encoder block with one attention head and an MLP of size 8. The weights are random but fixed (seeded), so the output is not a useful prediction; what matters is the path each number takes. Every value is computed in your browser by the same formulas as above.

Patch size \(P\)PositionsPatch orderRead out
Image and patches
Token matrix
Attention weights \(A\)
positive valuenegative value / CLSposition embedding added
  1. input: image \(\mathbf{x}\) (8 × 8 × 1, pixel / 9), patch size \(P\)
  2. for each patch \(i\): \(\mathbf{e}_i \leftarrow \mathbf{x}_p^{\,i} \mathbf{E} + \mathbf{b}\) // (164.2)
  3. \(\mathbf{z}_0 \leftarrow [\mathbf{x}_{\text{cls}}; \mathbf{e}_1; \dots; \mathbf{e}_N] + \mathbf{E}_{\text{pos}}\) // (164.4)
  4. \(\mathbf{h} \leftarrow \operatorname{LN}(\mathbf{z}_0)\)
  5. \(Q, K, V \leftarrow \mathbf{h} W_Q,\ \mathbf{h} W_K,\ \mathbf{h} W_V\)
  6. \(A \leftarrow \softmax(Q K^\top / \sqrt{D})\) // every row, no causal mask
  7. \(\mathbf{z}' \leftarrow \mathbf{z}_0 + (A V) W_O\) // (164.5), first line
  8. \(\mathbf{z}_1 \leftarrow \mathbf{z}' + \operatorname{MLP}(\operatorname{LN}(\mathbf{z}'))\) // (164.5), second line
  9. \(\mathbf{y} \leftarrow \operatorname{LN}(\mathbf{z}_1[0])\) or \(\tfrac1N \sum_i \operatorname{LN}(\mathbf{z}_1[i])\) // (164.7)
Figure 4. A tiny ViT, run step by step: patch embedding (164.2), CLS token and positions (164.4), one pre-LN encoder block (164.5) and the read-out (164.7). Sizes: \(D = 4\), one head, MLP size 8 with the tanh form of GELU, LayerNorm without its learned scale and shift, no biases in \(W_Q, W_K, W_V, W_O\). Weights are drawn from a seeded generator, so the numbers are always the same. The middle panel shows the token matrix after the highlighted line; the right panel shows the attention weights once line 6 has run.
How to read Figure 4

Press Play and follow the CLS row (the top row of the middle panel). It starts as a learned vector with no pixels in it. After line 6 its attention row (outlined) says how much it reads from each patch: with \(P = 4\) the weights are 0.144 for itself and 0.072, 0.232, 0.069 and 0.483 for patches 1 to 4. The read-out is \(\mathbf{y} = (0.330, 1.329, -0.217, -1.442)\). Now set the patch order to “shuffled”: the same four patches enter the sequence in another order, the positions no longer match the patches, and \(\mathbf{y}\) changes to \((0.946, 0.966, -0.542, -1.370)\). Turn positions off and compare the two orders again: \(\mathbf{y} = (0.040, 1.348, 0.086, -1.474)\) for both, the difference is exactly 0. Without positions the model cannot tell where a patch was: it sees a bag of patches. The mean read-out behaves the same way.

Position embeddings in 2D, and changing the resolution

Without positions, the encoder sees a bag of patches: shuffle them and the CLS output does not change (attention is permutation equivariant, session 16). Figure 4 shows this with real numbers: with positions off, the shuffled image gives exactly the same output. The ViT paper measured this on ImageNet 5-shot linear accuracy with ViT-B/16: no position embedding 0.614, learned 1D 0.642, learned 2D (separate row and column tables, each \(D/2\), concatenated) 0.640, relative positions 0.640. [Certain] So positions matter, but the exact kind matters little at a 14 × 14 grid. The authors’ explanation: the grid is small, so all methods learn it equally easily. [Speculation, by the authors]

The real problem appears when the resolution changes. A model trained at 224 with \(P = 16\) has a 14 × 14 table. Fine-tuning at 384 gives a 24 × 24 grid, so 576 positions that do not exist in the table. Fine-tuning at a higher resolution than pretraining is a standard trick (it is cheap and raises accuracy), so this happens all the time. [Certain] The fix from the ViT paper: treat the \(14 \times 14 \times D\) table as a small image with \(D\) channels, and resize it with 2D interpolation to \(24 \times 24 \times D\). The CLS position is kept as it is. With the usual “align corners = False” convention, target cell \(j\) of a new grid of size \(g'\) reads from the old grid at

\[ u_j = \Big(j + \tfrac12\Big) \frac{g}{g'} - \tfrac12, \qquad j = 0, \dots, g' - 1 \tag{164.8} \]

and blends the old vectors around \(u_j\) (2 neighbours per axis for bilinear, 4 for bicubic). Example: \(g = 14\), \(g' = 24\): cell 0 reads at \(-0.208\) (clamped to the border), cell 23 at \(13.208\). The method matters. A blend of two different unit vectors is shorter than either (picture the midpoint of a chord in a circle). DeiT reports that bilinear interpolation shrinks the norm of the embeddings, that these low-norm vectors cause “a significant drop in accuracy” before fine-tuning, and so uses bicubic, which roughly keeps the norm. [Certain] Figure 5 shows the effect.

trained grid 14 × 14 (click a cell)
after interpolation
Figure 5. A stand-in position table (64 dimensions, built from 32 sine–cosine pairs of fixed 2D frequencies, all vectors of length 1) on a 14 × 14 grid, resized to a new grid. The colour is the cosine similarity of the selected cell to every other cell. The readout gives the average length of the resized vectors.
How to read Figure 5

“Roughness” scales the frequencies: low roughness means neighbours are almost equal, high roughness means the table changes fast from cell to cell, as learned tables often do. At roughness 1.5 and \(g' = 24\), bilinear vectors have average length 0.920 (8% too short), while bicubic gives 1.012. Raise the roughness and the bilinear loss grows. In both cases the similarity pattern keeps its shape: the resized table still says “this cell is near those cells”, which is why a short fine-tune at the new resolution recovers the accuracy. This is a stand-in, not real ViT weights; the numbers show the mechanism, not a measured accuracy drop.

Interpolation is a patch, not a cure: the model saw one grid during training, and very different grids still hurt until it is fine-tuned. Newer encoders change the position scheme instead. NaViT trains on many resolutions and aspect ratios at once (“Patch n’ Pack”), and Qwen2-VL replaces the learned table with 2D rotary embeddings (RoPE on the row and the column), so any grid size works without interpolation (session 14, session 180). [Certain]

Algorithm 1: run a trained ViT at a new resolution
  1. input: trained position table \(\mathbf{E}_{\text{pos}} \in \R^{(1 + g^2) \times D}\), new image side \(R'\), patch size \(P\)
  2. \(g' \leftarrow \lfloor R' / P \rfloor\) // the patch size stays the same; the grid grows
  3. split \(\mathbf{E}_{\text{pos}}\) into the CLS row \(\mathbf{p}_{\text{cls}}\) and the grid rows \(\mathbf{G} \in \R^{g^2 \times D}\)
  4. reshape \(\mathbf{G}\) to an image of shape \(D \times g \times g\) (row-major, the same raster order as the patches)
  5. \(\mathbf{G}' \leftarrow\) bicubic resize to \(D \times g' \times g'\) // formula (164.8) for each target cell
  6. reshape back to \(g'^2 \times D\); \(\mathbf{E}'_{\text{pos}} \leftarrow [\mathbf{p}_{\text{cls}}; \mathbf{G}']\)
  7. replace the table in the model; patch embedding and all blocks are unchanged
  8. fine-tune briefly at the new resolution if accuracy matters

Tokens per image: resolution decides the cost

The parameters of a ViT do not depend on the image size. The compute does, and strongly. Formula (164.1) says the token count grows with the square of the image side: \(N = \lfloor R/P \rfloor^2\) for an \(R \times R\) image. Per layer, with sequence length \(N\) (including CLS), width \(D\) and MLP size \(F\), the forward pass costs about

\[ \text{MACs per layer} \approx \underbrace{4 N D^2}_{Q, K, V, O} + \underbrace{2 N D F}_{\text{MLP}} + \underbrace{2 N^2 D}_{QK^\top \text{ and } AV} \tag{164.9} \]

(A MAC is one multiply-add, which is 2 FLOPs; see session 25.) The first two terms grow like \(N\), so like \(R^2\). The attention term grows like \(N^2\), so like \(R^4\). With \(F = 4D\) the fraction of compute spent in attention scores has a very simple form:

\[ \text{attention share} = \frac{2N^2 D}{12 N D^2 + 2 N^2 D} = \frac{N}{N + 6D} \tag{164.10} \]

For ViT-B/16 at 224 (\(N = 197\), \(6D = 4608\)) attention is only 4.1% of the compute; the MLP and projections dominate. At 1024 (\(N = 4097\)) it is 47.1%. So at normal resolutions a ViT is a “linear” model in practice, and the quadratic term only takes over for large images or many video frames.

projections + MLP (grows like \(R^2\))attention scores (grows like \(R^4\))
Figure 6. Tokens and encoder compute (forward pass, multiply-adds, formula 164.9 summed over all layers plus the patch embedding; no pooling head) for three real encoder shapes. The dot is the chosen image side. The attention share in the readout counts the patch embedding in the total, so at large sizes it is a little below formula (164.10). The readout also gives the number of visual tokens an LLM would read if it takes every patch output.
How to read Figure 6

Start with ViT-B/16 at 224: 197 tokens and 17.56 G multiply-adds, which is the “17.6 GFLOPs” printed in many tables (those tables count multiply-adds) [Likely]. Move to 384: the tokens grow 2.93 times (197 to 577) but the compute grows 3.16 times, because the attention share rises from 4.1% to 11.1%. Switch to SigLIP So400m/14 at 384: 729 tokens. Set 4 images per prompt: the LLM now reads 2,916 visual tokens before the first word of text.

Worked example: common VLM encoders
Encoder, inputGridPatch tokensEncoder G MACsAttention share
ViT-B/16, 22414 × 14196 (+ CLS)17.564.1%
ViT-B/16, 38424 × 24576 (+ CLS)55.4811.1%
CLIP ViT-L/14, 33624 × 24576 (+ CLS)190.968.6%
SigLIP So400m/14, 38427 × 27729 (no CLS)333.229.9%

The shapes are from the Hugging Face configs (CLIP: 24 layers, width 1024, MLP 4096; SigLIP: 27 layers, width 1152, MLP 4304). [Certain] The compute numbers follow from formula (164.9); they are not measured speeds. Note that the encoder is not the expensive part for long answers: a 7B LLM reading 576 tokens costs about \(7 \times 10^9 \times 576 \approx 4\) T multiply-adds for the prefill alone, roughly 20 times the CLIP encoder. This is why VLMs try so hard to send fewer visual tokens to the LLM (pixel shuffle, pooling, Q-Former, session 178).

The patch size is the main knob. ViT-B/32 at 224 gives only 49 tokens: fast, but each token covers 32 × 32 pixels and small text disappears. ViT-B/8 gives 784 tokens. The ViT paper found that smaller patches give “surprisingly robust improvements without introducing parameters”: you pay in compute, not in model size. [Certain] For video the numbers multiply: 16 frames of ViT-B/16 at 224 are 3,136 tokens. Encoding the frames separately costs about 278 G MACs in the layers; one joint attention over all 3,136 tokens costs about 448 G, because attention then takes 40% of the compute (formula 164.10). That is the reason video encoders factor attention into space and time (session 202).

Algorithm 2: ViT forward pass
  1. input: image tensor \(\mathbf{x}\) of shape \(C \times H \times W\), already normalized with the encoder’s own mean and standard deviation
  2. \(\mathbf{T} \leftarrow \operatorname{Conv2d}(\mathbf{x};\ \text{kernel} = P, \text{stride} = P)\) // shape \(D \times H/P \times W/P\), formula (164.3)
  3. flatten the grid in raster order to \(N \times D\) // \(N = HW/P^2\), formula (164.1)
  4. prepend \(\mathbf{x}_{\text{cls}}\); add \(\mathbf{E}_{\text{pos}}\) // formula (164.4); resize \(\mathbf{E}_{\text{pos}}\) first if the grid changed (Algorithm 1)
  5. for \(\ell = 1\) to \(L\): \(\mathbf{z} \leftarrow \mathbf{z} + \operatorname{MSA}(\operatorname{LN}(\mathbf{z}))\); \(\;\mathbf{z} \leftarrow \mathbf{z} + \operatorname{MLP}(\operatorname{LN}(\mathbf{z}))\) // no causal mask
  6. \(\mathbf{z} \leftarrow \operatorname{LN}(\mathbf{z})\)
  7. return \(\mathbf{z}[0]\) (CLS), or the mean of \(\mathbf{z}[1{:}]\), or all of \(\mathbf{z}[1{:}]\) for a VLM // formula (164.7)

Training a ViT: data hunger, and how DeiT fixed it

A CNN builds in locality and translation equivariance in every layer. A ViT builds in almost nothing: only the patch cut and, at fine-tuning, the 2D interpolation of positions. The paper says: “all spatial relations between the patches have to be learned from scratch.” [Certain] With little data this hurts. Trained on ImageNet alone (1.3M images), ViTs were “a few percentage points below ResNets of comparable size”. Pretrained on JFT-300M (a private set of 300M images), they won: ViT-H/14 reached 88.55% ImageNet top-1 and ViT-L/16 87.76%, with less pretraining compute than the best CNN of the time (0.68k TPUv3-core-days for ViT-L/16 against 9.9k for BiT-L). [Certain]

DeiT (Data-efficient image Transformers) showed that the data hunger was mostly a training-recipe problem. It trained the same ViT-B architecture on ImageNet-1k only, on a single 8-GPU node in about three days (53 hours of pretraining). [Certain] The recipe: AdamW, 300 epochs, cosine schedule with 5 warmup epochs, weight decay 0.05, strong augmentation (RandAugment, Mixup, CutMix, random erasing, repeated augmentation), stochastic depth 0.1 and label smoothing 0.1. DeiT also added a distillation token: a second extra token, like CLS, whose output is trained to match the predictions of a CNN teacher (a RegNetY-16GF). [Certain]

Model (ImageNet-1k only, as reported by DeiT)ParamsResolutionTop-1
ViT-B/16 (original recipe)86M38477.9%
DeiT-Ti / DeiT-S / DeiT-B5M / 22M / 86M22472.2% / 79.8% / 81.8%
DeiT-B, fine-tuned at 38486M38483.1%
DeiT-B with distillation token87M22483.4%
DeiT-B distilled, 1000 epochs, fine-tuned at 38487M38485.2%

[Certain, all numbers from the DeiT paper.] Today most ViTs used in VLMs are not trained with labels at all: they come from image–text contrastive training (CLIP, SigLIP) or self-supervised training (MAE, DINOv2), covered in the next sessions (session 165, session 166).

The PyTorch version

The code checks that the conv patch embedding equals “unfold + matrix multiply”, builds ViT-B/16 and counts its parameters, resizes the position table to run at 384, and prints the token counts from the worked example.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) patch embedding: Conv2d with kernel = stride = P is a linear layer on flattened patches (164.3)
P, C, D = 16, 3, 768
conv = nn.Conv2d(C, D, kernel_size=P, stride=P)
x = torch.randn(2, C, 224, 224)
a = conv(x).flatten(2).transpose(1, 2)                            # (2, 196, 768)
patches = F.unfold(x, kernel_size=P, stride=P).transpose(1, 2)    # (2, 196, 768): P*P*C numbers each
b = patches @ conv.weight.reshape(D, -1).T + conv.bias
print(a.shape, torch.allclose(a, b, atol=1e-4))

# 2) a minimal ViT: patchify, CLS token, learned positions, pre-LN blocks (164.4, 164.5)
class ViT(nn.Module):
    def __init__(self, img=224, P=16, D=768, depth=12, heads=12, mlp=3072, classes=1000, pool="cls"):
        super().__init__()
        self.P, self.pool, g = P, pool, img // P
        self.patch = nn.Conv2d(3, D, kernel_size=P, stride=P)
        self.cls = nn.Parameter(torch.zeros(1, 1, D))
        self.pos = nn.Parameter(torch.randn(1, 1 + g * g, D) * 0.02)
        layer = nn.TransformerEncoderLayer(D, heads, mlp, dropout=0.0, activation="gelu",
                                           batch_first=True, norm_first=True)
        self.blocks = nn.TransformerEncoder(layer, depth, enable_nested_tensor=False)
        self.norm = nn.LayerNorm(D)
        self.head = nn.Linear(D, classes)
    def forward(self, x):
        z = self.patch(x).flatten(2).transpose(1, 2)               # (B, N, D)
        cls = self.cls.expand(z.shape[0], -1, -1)
        z = torch.cat([cls, z], dim=1) + self.pos                  # formula (164.4)
        z = self.norm(self.blocks(z))                              # no mask: bidirectional
        y = z[:, 0] if self.pool == "cls" else z[:, 1:].mean(dim=1)   # formula (164.7)
        return self.head(y)

model = ViT().eval()
print(sum(p.numel() for p in model.parameters()))          # 86,567,656 = ViT-B/16
with torch.no_grad():
    print(model(torch.randn(1, 3, 224, 224)).shape)

# 3) run at 384: resize the 14 x 14 position grid to 24 x 24 (Algorithm 1, bicubic as in DeiT)
def resize_pos(pos, new_grid):
    cls_pos, grid_pos = pos[:, :1], pos[:, 1:]
    g = math.isqrt(grid_pos.shape[1])
    grid_pos = grid_pos.reshape(1, g, g, -1).permute(0, 3, 1, 2)          # (1, D, g, g)
    grid_pos = F.interpolate(grid_pos, size=(new_grid, new_grid), mode="bicubic", align_corners=False)
    grid_pos = grid_pos.permute(0, 2, 3, 1).reshape(1, new_grid * new_grid, -1)
    return torch.cat([cls_pos, grid_pos], dim=1)

with torch.no_grad():
    model.pos = nn.Parameter(resize_pos(model.pos, 384 // 16))
    print(model.pos.shape, model(torch.randn(1, 3, 384, 384)).shape)

# 4) tokens per image: resolution and patch size decide the cost (164.1)
for side, p in [(224, 16), (384, 16), (336, 14), (384, 14)]:
    print(side, p, (side // p) ** 2)

Output with PyTorch 2.8 on CPU: torch.Size([2, 196, 768]) True; 86567656; torch.Size([1, 1000]); torch.Size([1, 577, 768]) torch.Size([1, 1000]); then 196, 576, 576 and 729 tokens. The same model with pool="mean" uses mean pooling and has the same parameters (the CLS token is still created but not read).

Where this is used

ApplicationHow the ViT is used
Image encoder of VLMs (LLaVA, Qwen-VL, InternVL, PaliGemma, SmolVLM)A CLIP or SigLIP ViT; all patch outputs go through a connector into the LLM, so tokens per image decide the prompt length (session 177, session 178)
Image–text retrieval and zero-shot classificationThe pooled ViT output (CLS or attention pooling) is the image embedding compared with text embeddings by cosine similarity (session 166, session 169)
Video retrieval, near-duplicate detection and fingerprintingOne pooled ViT embedding per sampled frame, averaged or matched over time; cheap because each frame is encoded once and cached
Video classification and action recognitionFrames are patchified into space–time “tubelets” (ViViT, VideoMAE) and attention is factored into space and time to keep the token count manageable (session 202)
Key-moment detection in long videoPer-frame ViT features scored over time; resolution and frame rate set the token budget
Detection and segmentationPatch outputs reshaped back to a \(g \times g\) feature map; DINOv2 patch features are strong for dense tasks (session 168)
Self-supervised pretrainingMAE drops 75% of the patch tokens before the encoder, which is cheap precisely because the ViT input is a set of tokens (session 165)
Image generationDiffusion transformers (DiT) patchify the latent image the same way (session 228)

Common mistakes

  • Feeding a resolution the model was not built for, without resizing the positions. A 384 image into a 224 ViT gives 577 tokens and only 197 position vectors: a shape error, or, in code that crops the table, silently wrong positions. Interpolate (Algorithm 1) and fine-tune.
  • Bilinear interpolation of the position table. It shrinks the vectors (Figure 5). Use bicubic, as DeiT and timm do. [Likely for timm’s default]
  • Forgetting the CLS slot when reshaping. The table has \(1 + g^2\) rows. Reshaping all of them to a \(g \times g\) grid fails, or worse, shifts every position by one if you drop the wrong row.
  • Image side not a multiple of \(P\). The strided conv silently drops the remainder (6 pixels at 384 with \(P = 14\)). If edges matter (text near borders), resize to a multiple of \(P\) or pad.
  • Reshape instead of permute when making patches by hand. x.reshape(N, P*P*C) on a \(C \times H \times W\) tensor mixes pixels from different patches. Use unfold or the conv, or permute to \((H/P, W/P, P, P, C)\) first.
  • Using a causal mask in the encoder. Copying LLM code gives patch 1 no view of patch 2. Vision encoders are bidirectional.
  • Treating token count as linear in resolution. Doubling the side gives 4 times the tokens and, through the LLM prefill and the KV cache, roughly 4 times the cost of the visual part of the prompt, more where attention dominates.
  • Training a ViT from scratch on a small dataset with a CNN recipe. Without strong augmentation, regularization and many epochs (DeiT) or a pretrained checkpoint, a ViT does worse than a ResNet.

Interview questions

How does a Vision Transformer turn an image into tokens?basic
It cuts the \(H \times W\) image into non-overlapping \(P \times P\) patches, \(N = HW/P^2\) of them, flattens each into \(P^2 C\) numbers and multiplies by one shared matrix \(\mathbf{E}\) to get a \(D\)-dimensional token. It prepends a learned CLS token and adds a learned position embedding to each token. For 224 × 224 and \(P = 16\): 196 patch tokens of 768 numbers each, 197 tokens with CLS.
Why is the patch embedding implemented as a convolution? Does that make ViT a CNN?core
A Conv2d with kernel size \(P\) and stride \(P\) applies \(D\) filters of size \(P \times P \times C\) to non-overlapping windows. That is exactly a dot product of each flattened patch with each column of \(\mathbf{E}\), so the two are identical (formula 164.3); the conv is just the fastest way to cut, flatten and project in one call. It is not a CNN in the usual sense: one layer, no overlap, no nonlinearity, no pooling. All spatial mixing afterwards comes from global self-attention.
How many tokens does a 336 × 336 image give with patch 14? What happens to the cost if you double the side?core
\(336/14 = 24\), so \(24^2 = 576\) patch tokens (577 with CLS). Doubling to 672 gives \(48^2 = 2304\) tokens, 4 times more. Projections and MLP grow 4 times; attention scores grow 16 times. For CLIP ViT-L/14, attention is 8.6% of the compute at 336, so the encoder grows a bit more than 4 times. In a VLM the LLM must also read 4 times as many visual tokens, which usually costs more than the encoder.
CLS token or mean pooling?core
The CLS token is a learned extra input whose final state is the image vector; mean pooling averages the patch outputs. The ViT paper found they work equally well once the learning rate is tuned for each (GAP looked “very poor” at first only because it needed a different learning rate). CLS costs one token and lets the model learn what to collect; mean pooling needs no extra token and treats patches evenly. SigLIP uses attention pooling (a learned query over all patches). Many VLMs use neither: they send all patch tokens to the LLM.
How do you run a ViT trained at 224 on 384 × 384 images?core
Keep the patch size, so the grid grows from 14 × 14 to 24 × 24 and the sequence from 197 to 577 tokens. Patch embedding and blocks work unchanged, but the position table has only 196 grid vectors. Reshape them to a \(14 \times 14 \times D\) image, resize with bicubic interpolation to \(24 \times 24\), keep the CLS position, and fine-tune briefly. Bicubic is preferred because bilinear blends shrink the vector norms, which DeiT found causes a large accuracy drop before fine-tuning. Alternatives that avoid the problem: 2D RoPE (Qwen2-VL) or training on mixed resolutions (NaViT).
Why did the first ViT need 300 million images, and how did DeiT train it on ImageNet alone?core
ViT has very weak inductive bias: no built-in locality or translation equivariance, so it must learn spatial structure from data. On ImageNet alone it was a few points below comparable ResNets (ViT-B/16 at 77.9% as reported by DeiT); with JFT-300M it surpassed them (ViT-H/14 88.55%). DeiT showed most of the gap was the recipe: AdamW for 300 epochs with heavy augmentation (RandAugment, Mixup, CutMix, random erasing, repeated augmentation), stochastic depth and label smoothing gave DeiT-B 81.8%, and a distillation token trained against a RegNetY CNN teacher gave 83.4% at 224, all on one 8-GPU node in about three days.
What happens without position embeddings? Why do 1D and 2D work about equally?basic
Self-attention is permutation equivariant, so without positions the model sees a bag of patches; with a CLS readout, shuffling the patches does not change the output. In the ViT ablation (ViT-B/16, ImageNet 5-shot linear) no positions gave 0.614, 1D learned 0.642, 2D learned 0.640, relative 0.640. The 14 × 14 grid is small, and the learned 1D table discovers the 2D structure by itself: trained embeddings are most similar to neighbours in the same row and column.
What is the trade-off between patch sizes 32, 16, 14 and 8?core
At 224: \(P = 32\) gives 49 tokens, 16 gives 196, 14 gives 256, 8 gives 784. Smaller patches mean finer detail (small text, thin objects) and better accuracy at the same parameter count, but compute grows like \(1/P^2\) in the linear parts and \(1/P^4\) in attention, and a VLM must read more tokens. The patch embedding changes size slightly; the encoder does not. Many VLM encoders use 14 (CLIP-L/14, SigLIP So400m/14) and then merge neighbouring tokens (for example 2 × 2) before the LLM.
Count the parameters of ViT-B/16.core
Per block: attention \(4D^2 + 4D = 2{,}362{,}368\), MLP \(2DF + F + D = 4{,}722{,}432\), two LayerNorms 3,072, total 7,087,872; times 12 = 85,054,464. Plus patch embedding \(768 \cdot 768 + 768 = 590{,}592\), CLS 768, positions \(197 \cdot 768 = 151{,}296\), final LayerNorm 1,536, head 769,000: 86,567,656. The rule of thumb \(12 D^2 L = 84.9\)M gets close.
At what point does attention dominate the compute of a ViT?advanced
With MLP size \(4D\), per layer the linear parts cost \(12ND^2\) multiply-adds and attention scores \(2N^2D\), so attention’s share is \(N/(N + 6D)\). It reaches one half at \(N = 6D\): 4,608 tokens for ViT-B (about a 1088 × 1088 image at \(P = 16\)), 6,144 for ViT-L. At 224 it is only about 4%. So for single images the ViT behaves like a linear-cost model, but for high-resolution documents or joint space–time attention over many video frames the quadratic term matters, which motivates windowed attention, token merging and factorized video attention.
What are the high-norm “artifact” tokens in large ViTs, and what are registers?advanced
In large, long-trained ViTs (for example DINOv2, OpenCLIP, DeiT-III) a few patch tokens, usually on low-information background, get very large norms and carry global information instead of local content; they show up as spikes in attention maps and hurt dense tasks. Darcet et al. add a few extra learned tokens (registers) to the input, like CLS but discarded at the output; the model uses them as scratch space and the artifacts disappear from the patch tokens.
In a VLM, why is the ViT token count a problem, and what do real systems do about it?advanced
Every patch output becomes an LLM input token: 576 for CLIP ViT-L/14-336, 729 for SigLIP So400m/14-384, multiplied by images, tiles and video frames. The LLM prefill and KV cache grow with this count, and for a 7B LLM the prefill of 576 tokens costs about 20 times the CLIP encoder. Systems reduce it with token merging before the LLM (2 × 2 pixel shuffle or merging in Qwen2-VL and SmolVLM), resamplers with a fixed number of queries (Q-Former, Perceiver), pooling for video frames, and dynamic resolution that spends tokens only where the image has detail.

Exercises

  1. How many tokens does a 448 × 448 image give with \(P = 14\), with and without a CLS token?
  2. Compute the number of parameters of the patch embedding of ViT-L/14.
  3. A 1280 × 720 video frame is fed to a ViT with \(P = 16\) at its native size. How many patch tokens? Compare with 224 × 224.
  4. With formula (164.8), where does a 24-cell target grid read from a 14-cell source grid for target cells 0 and 23?
  5. Use formula (164.10) to find the attention share of ViT-B/16 at 1024 × 1024 (with CLS).
  6. In Figure 3, change the bottom-right patch to all 5s. What are its three token values, and which other patch’s token is it now closest to?
Answerscheck
1. \(448/14 = 32\), so \(32^2 = 1024\) tokens, 1025 with CLS. 2. \(14 \cdot 14 \cdot 3 = 588\) inputs, \(588 \cdot 1024 + 1024 = 603{,}136\). 3. \(1280/16 = 80\), \(720/16 = 45\), so \(80 \times 45 = 3600\) tokens, about 18 times the 196 tokens of a 224 image (and attention grows about 340 times). 4. \(u_0 = 0.5 \cdot 14/24 - 0.5 = -0.208\) (clamped to cell 0) and \(u_{23} = 23.5 \cdot 14/24 - 0.5 = 13.208\) (beyond the last cell 13, clamped). 5. \(N = 64^2 + 1 = 4097\); share \(= 4097/(4097 + 4608) = 47.1\%\). 6. Mean 5, left minus right 0, top minus bottom 0: token (5, 0, 0). It is closest to the top-right patch (2, 0, 0); distance 3, against \(\sqrt{0.333^2 + 27^2} \approx 27.0\) to the bottom-left patch and about 21.1 to the top-left patch.

Key takeaways

  • A ViT cuts the image into \(N = HW/P^2\) patches (164.1); each becomes one token through a shared linear map (164.2), which is the same thing as a Conv2d with kernel = stride = \(P\) (164.3).
  • A CLS token and a learned position embedding complete the input (164.4); the encoder is a standard bidirectional pre-LN transformer (164.5) with about \(12D^2\) parameters per block (164.6): 86,567,656 for ViT-B/16.
  • CLS, mean pooling and attention pooling work about equally (164.7); VLMs usually take all patch tokens instead.
  • Positions are needed (0.614 without vs 0.642 with), 1D and 2D work alike, and a new resolution needs interpolation of the table (164.8), preferably bicubic; 2D RoPE and NaViT avoid it.
  • Tokens grow with the square of the image side; compute per layer is \(4ND^2 + 2NDF + 2N^2D\) (164.9) and attention’s share is \(N/(N + 6D)\) (164.10): 4.1% at 224 for ViT-B/16. Tokens per image is the number that drives VLM cost.
  • ViT has weak inductive bias: it needs large pretraining or a strong recipe (DeiT: 81.8% on ImageNet alone, 83.4% with distillation).

Key formulas

\[ N = \frac{HW}{P^2}, \quad \mathbf{z}_0 = [\mathbf{x}_{\text{cls}}; \mathbf{x}_p^{1}\mathbf{E}; \dots; \mathbf{x}_p^{N}\mathbf{E}] + \mathbf{E}_{\text{pos}}, \quad u_j = \Big(j + \tfrac12\Big)\frac{g}{g'} - \tfrac12 \]
\[ \text{MACs/layer} \approx 4ND^2 + 2NDF + 2N^2D, \qquad \text{attention share} = \frac{N}{N + 6D} \ \ (F = 4D) \]

References for this session

  1. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  2. Touvron et al. (2020). Training data-efficient image transformers & distillation through attention
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  4. Darcet, Oquab, Mairal, Bojanowski (2023). Vision Transformers Need Registers
  5. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  6. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  7. Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
  8. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  9. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  10. He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
  11. Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision