LLM & VLM courseBlog
163Track 7: Vision and VLMs · session 1 of 38

Images as data and a CNN refresher

Before a vision-language model can talk about a picture, the picture must become a tensor of numbers with the exact shape, scale and colour order that the vision encoder was trained on. This session shows how that happens (pixels, channels, normalization, resizing and aspect ratio), then refreshes convolutional networks: the sliding filter, the receptive field, pooling, ResNet and transfer learning. It ends with what CNNs do well and what they miss, which is the reason the next session moves to the Vision Transformer.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 3 to 5 (math, optimization, PyTorch)

What you will learn

  • How an image is stored (uint8, HWC vs CHW, RGB vs BGR) and how much memory images and video frames take
  • Why every encoder needs its own normalization (ImageNet, CLIP, SigLIP statistics) and what goes wrong when it does not match
  • Three ways to fit any aspect ratio into a square input (squash, crop, pad) and what each one loses
  • Convolution as a sliding dot product: output size, parameter count and multiply-adds of a layer, computed by hand
  • Receptive field growth through depth, stride and pooling, and why the effective receptive field is smaller
  • From LeNet to ResNet: residual blocks, the ResNet-50 stage layout, and transfer learning with frozen or fine-tuned backbones
  • What the CNN assumptions (locality, translation equivariance) give you, and where CNNs fail: aliasing, texture bias, limited global context

Why this matters

A vision-language model (VLM) never sees a picture. It sees a tensor: a block of numbers with a fixed shape, a fixed value range and a fixed colour order. A vision encoder turns that tensor into vectors, and only then can the language model read them. If the numbers are prepared in the wrong way, the model still answers, fluently and wrongly.

This session starts the vision track (Track 7). It has two parts. The first part is images as data: how a photo or a video frame becomes a tensor, and the small preprocessing choices (normalization, resizing, cropping, padding) that silently decide what the model can see. These choices come back in every later session: CLIP and SigLIP have their own normalization, and high-resolution VLMs are mostly about handling aspect ratio well (session 180).

The second part is a refresher on convolutional neural networks (CNNs). From 2012 to about 2020, CNNs were the standard way to turn images into features, and they are still everywhere: in detection and segmentation backbones, on phones, in video pipelines, and as the image encoder of some VLMs. The Vision Transformer of the next session (session 164) is easiest to understand as an answer to the limits of CNNs. So we build the CNN up from one filter, measure it with numbers, and end with what it does well and what it misses.

An image is a grid of numbers

A digital colour image is a grid of pixels (picture elements). Each pixel stores three numbers, one per channel: red, green and blue (RGB). In a normal photo file (JPEG, PNG) each number is an 8-bit unsigned integer (uint8) from 0 (no light) to 255 (full light). So an image of height \(H\) and width \(W\) is an array of \(H \times W \times 3\) integers.

  • Layout. Image libraries (PIL, OpenCV, NumPy) store the array as \((H, W, C)\): channels last. PyTorch models expect \((B, C, H, W)\): a batch dimension first and channels before height and width. The conversion is one permute(2, 0, 1).
  • Colour order. PIL and torchvision give RGB. OpenCV’s cv2.imread gives BGR (blue first). [Certain] A model trained on RGB that receives BGR sees red and blue swapped.
  • Size. A 224 × 224 RGB image is \(224 \cdot 224 \cdot 3 = 150{,}528\) numbers: 150,528 bytes as uint8, four times that (602,112 bytes) as float32. One 1920 × 1080 video frame is 6,220,800 bytes, and one second of 30 fps video is about 187 MB before compression. This is why video systems sample frames and shrink them first (session 201).
  • Grayscale and others. A grayscale image has one channel. Some images have a fourth alpha channel (transparency) that must be dropped or blended before an RGB model sees it.

Normalization

Neural networks train best when their inputs are centred near 0 with a spread near 1 (the same reason we normalize hidden states, see session 22). So each channel \(c\) is first scaled to \([0, 1]\), then shifted by a mean \(\mu_c\) and divided by a standard deviation \(\sigma_c\):

\[ \tilde{x}_{c,i,j} = \frac{x_{c,i,j} / 255 - \mu_c}{\sigma_c} \tag{163.1} \]
Read the symbols
\(x_{c,i,j}\)
the raw uint8 value of channel \(c\) at row \(i\), column \(j\)
\(\mu_c, \sigma_c\)
a mean and a standard deviation per channel, fixed by whoever trained the encoder
\(\tilde{x}_{c,i,j}\)
the number the network actually receives

The important point: \(\mu\) and \(\sigma\) are part of the model. They are not computed on your data. Three common choices [Certain]:

Encoder family\(\mu\) (R, G, B)\(\sigma\) (R, G, B)Range of \(\tilde{x}\)
ImageNet CNNs (torchvision ResNet, many others)0.485, 0.456, 0.4060.229, 0.224, 0.225about −2.1 to 2.6
OpenAI CLIP0.4815, 0.4578, 0.40820.2686, 0.2613, 0.2758about −1.8 to 2.1
SigLIP (Hugging Face config)0.5, 0.5, 0.50.5, 0.5, 0.5exactly −1 to 1

The ImageNet numbers are the average colour of the ImageNet training photos. CLIP measured its own statistics on its own data. SigLIP simply maps \([0, 1]\) to \([-1, 1]\). Figure 1 lets you push a few pixel colours through each recipe.

Figure 1. One pixel through formula (163.1). Pick a colour and a normalization recipe. The bars show the three numbers the network receives, one per input channel. The dashed bars show the correct values when the BGR mistake is switched on.
How to read Figure 1

Mid-gray (128, 128, 128) lands near 0 for every recipe: that is the point of subtracting the mean. Pure white lands near +2.2 to +2.6 for ImageNet but at exactly +1 for SigLIP, so feeding ImageNet-normalized images into SigLIP gives inputs more than twice too large. Turn on the BGR switch with the red pixel: the network now sees a strongly blue pixel. Nothing crashes, the shapes are correct, and the model quietly sees the wrong colours.

Worked example: pure white and pure black, ImageNet recipe

White is (255, 255, 255), so \(x/255 = 1\) in every channel. Red channel: \((1 - 0.485)/0.229 = 2.249\). Green: \((1 - 0.456)/0.224 = 2.429\). Blue: \((1 - 0.406)/0.225 = 2.640\).

Black is (0, 0, 0): red \(-0.485/0.229 = -2.118\), green \(-2.036\), blue \(-1.804\). So the ImageNet-normalized range is about \([-2.1, 2.6]\), not \([-1, 1]\). With SigLIP’s recipe the same two pixels become exactly \(+1\) and \(-1\).

Resizing and aspect ratio

Most vision encoders take a fixed square input: 224 × 224 for classic CNNs and many ViTs, 336 × 336 for CLIP ViT-L/14-336, 384 or 448 for others. Real images are not square: a phone photo is 4:3, a video frame is 16:9, a receipt can be 1:3. The aspect ratio is width divided by height. There are three standard ways to fit a \(W \times H\) image into an \(S \times S\) square:

  1. Squash: resize width and height separately to \(S\). Nothing is lost, but shapes are stretched: a circle becomes an ellipse.
  2. Resize and center crop: scale the shorter side to \(S\), then cut the central \(S \times S\) square. Shapes are correct, but the edges of the long side are thrown away. This is the default for ImageNet CNNs and for CLIP. [Certain]
  3. Pad (letterbox): scale the longer side to \(S\), then fill the rest with a constant colour. Nothing is lost and nothing is stretched, but part of the input is empty, and the content gets fewer pixels.
\[ s_{\text{crop}} = \frac{S}{\min(W, H)}, \qquad s_{\text{pad}} = \frac{S}{\max(W, H)}, \qquad \text{kept (crop)} = \text{content (pad)} = \frac{\min(W, H)}{\max(W, H)} \tag{163.2} \]
Read the symbols
\(W, H\)
width and height of the original image in pixels
\(S\)
side of the square input of the encoder
\(s\)
the scale factor applied to both sides (so shapes are not distorted)
kept, content
the fraction of the original image area that survives the crop; the same number is the fraction of the padded square that holds real content

For a 1920 × 1080 frame and \(S = 224\): the crop keeps \(1080/1920 = 56.25\%\) of the frame, and the pad leaves \(43.75\%\) of the square empty. Either way the encoder sees only \(224^2 / (1920 \cdot 1080) = 2.4\%\) as many pixels as the original. Small text, a distant face or a road sign may simply vanish. That single fact drives a large part of modern VLM design: tiling, dynamic resolution and native aspect ratio (session 180).

Figure 2. Fitting a \(W \times H\) image into an \(S \times S\) encoder input. Left: the original image; the highlighted region is what reaches the encoder. Right: the encoder input with its \(P \times P\) patch grid (patches become tokens in a ViT). The circle shows whether shapes are kept.
How to read Figure 2

Start with a 16:9 frame in crop mode: the left and right 22% of the frame are gone. Switch to pad: everything is kept, but almost half of the patch grid holds padding, so many tokens carry no information. Switch to squash: the circle becomes an ellipse with the aspect-ratio factor printed below. Now set \(W = 1000\), \(H = 3000\) (a tall receipt) and see how much each method loses. No single choice is right for every image, which is why newer VLMs avoid the fixed square.

Convolution: a sliding filter

Now the network. The simplest idea would be to flatten the 150,528 input numbers and use a fully connected layer. With only 1,000 output units, that one layer would already need 150 million weights, and it would treat a cat in the top-left corner as a completely different thing from the same cat in the bottom-right. Images have two properties we can use instead:

  • Locality. Nearby pixels belong together. An edge, a corner, a patch of fur: each is visible inside a small window.
  • Same pattern, any place. An edge detector that works in one corner should work everywhere.

A convolution uses both. It takes a small grid of weights, the filter or kernel (for example 3 × 3), slides it over the image, and at every position computes a dot product between the filter and the pixels under it. The same filter weights are used at every position: this is called weight sharing. The result is a new grid, the feature map, that is large where the image looks like the filter. With several input channels the filter has a depth too, and with several output channels we use several filters:

\[ Y_{o,i,j} = b_o + \sum_{c=1}^{C_{\text{in}}} \sum_{u=0}^{k-1} \sum_{v=0}^{k-1} W_{o,c,u,v} \; X_{c,\; s i + u - p,\; s j + v - p} \tag{163.3} \]
Read the symbols
\(X\)
input with \(C_{\text{in}}\) channels; positions outside the image count as 0 (zero padding)
\(W_{o,c,u,v}\)
weight of filter \(o\), input channel \(c\), offset \((u, v)\) inside the \(k \times k\) window
\(b_o\)
one bias per output channel
\(s\)
stride: how many pixels the window jumps between positions
\(p\)
padding: how many rows and columns of zeros are added around the input
\(Y_{o,i,j}\)
output channel \(o\) at output position \((i, j)\)

Strictly, (163.3) is a cross-correlation: a mathematical convolution flips the filter first. Deep learning libraries do not flip, and since the filter is learned, the difference does not matter. [Certain] Each output is just a dot product, so a whole conv layer can be written as one big matrix multiply (the PyTorch code below checks this with unfold).

Output size, parameters and compute

Along each side, the number of window positions is

\[ H_{\text{out}} = \left\lfloor \frac{H_{\text{in}} + 2p - k}{s} \right\rfloor + 1 \tag{163.4} \]

The ResNet first layer has \(k = 7\), \(s = 2\), \(p = 3\) on a 224 input: \(\lfloor (224 + 6 - 7)/2 \rfloor + 1 = 112\). A 3 × 3 conv with \(p = 1\), \(s = 1\) keeps the size (“same” padding). The cost of a layer is

\[ \text{params} = k^2 C_{\text{in}} C_{\text{out}} + C_{\text{out}}, \qquad \text{MACs} = k^2 C_{\text{in}} C_{\text{out}} \, H_{\text{out}} W_{\text{out}} \tag{163.5} \]

A MAC is one multiply-accumulate (about 2 FLOPs). Notice the difference: parameters do not depend on image size (weight sharing), but compute grows with the number of output positions.

Worked example: one ResNet-style layer

A 3 × 3 conv from 64 to 128 channels with an output of 56 × 56. Parameters: \(9 \cdot 64 \cdot 128 + 128 = 73{,}856\). Multiply-adds: \(9 \cdot 64 \cdot 128 \cdot 56 \cdot 56 = 231{,}211{,}008\), about 0.23 billion.

Now a 3 × 3 conv from 128 to 256 channels at 28 × 28 (one stage deeper: half the size, twice the channels). Parameters: 295,168, four times more. Multiply-adds: \(9 \cdot 128 \cdot 256 \cdot 784 = 231{,}211{,}008\), exactly the same. Halving the resolution while doubling the channels keeps the compute per layer constant. This is the design rule of VGG and ResNet. [Certain]

Figure 3 runs formula (163.3) on a tiny one-channel 8 × 8 image: dark on the left (value 1), bright on the right (value 7), with a small bright block in the dark half.

positive outputnegative outputcurrent window
Figure 3. A 3 × 3 filter slides over an 8 × 8 image with stride 1. Step through the positions or click any output cell. The readout writes out the nine products of formula (163.3).
How to read Figure 3

The vertical-edge filter (a Sobel filter) gives 0 on flat regions and 24 on the big dark-to-bright edge, because there the right column of the window is 7 and the left column is 1: \((7 - 1) \cdot (1 + 2 + 1) = 24\). The small block gives a +18 and a −18: a bright-to-dark edge has the opposite sign. The horizontal filter ignores the big vertical edge completely and only fires on the top and bottom of the block. Turn on padding: the output becomes 8 × 8 instead of 6 × 6, as formula (163.4) says, and the border cells show artifacts from the zeros. Learned CNN filters in the first layer look a lot like these: edges at many angles, blobs of colour. [Certain]

Algorithm 1: forward pass of one conv layer (formula 163.3)
  1. input: \(X\) of shape \((C_{\text{in}}, H, W)\), weights \(W\) of shape \((C_{\text{out}}, C_{\text{in}}, k, k)\), bias \(b\), stride \(s\), padding \(p\)
  2. pad \(X\) with \(p\) zeros on every side
  3. \(H_{\text{out}}, W_{\text{out}} \leftarrow\) formula (163.4)
  4. for \(o = 1 \dots C_{\text{out}}\), \(i = 0 \dots H_{\text{out}}-1\), \(j = 0 \dots W_{\text{out}}-1\):
  5.   \(\text{win} \leftarrow X[:, si : si + k, sj : sj + k]\) // a \(C_{\text{in}} \times k \times k\) block
  6.   \(Y[o, i, j] \leftarrow b_o + \sum \text{win} \odot W[o]\) // one dot product of length \(k^2 C_{\text{in}}\)
  7. apply a nonlinearity (ReLU) to \(Y\), usually after a normalization layer
  8. return \(Y\) of shape \((C_{\text{out}}, H_{\text{out}}, W_{\text{out}})\) // real kernels batch all windows into one GEMM

Two special kernel sizes are worth knowing. A 1 × 1 convolution looks at one position only: it is a linear layer applied to the channel vector of each pixel, used to change the number of channels cheaply. A depthwise convolution uses one filter per channel (no mixing between channels) and is the cheap building block of MobileNet and ConvNeXt. [Certain]

Receptive field and pooling

One output of a 3 × 3 conv depends on a 3 × 3 patch of its input. Its receptive field is the region of the original image that can affect it. Stack a second 3 × 3 conv on top: each of its outputs sees a 3 × 3 patch of the first map, which covers a 5 × 5 patch of the image. A stride or a pooling layer makes every later step count double. The rule, from the input upwards, is

\[ r_\ell = r_{\ell-1} + (k_\ell - 1)\, j_{\ell-1}, \qquad j_\ell = j_{\ell-1}\, s_\ell, \qquad r_0 = j_0 = 1 \tag{163.6} \]
Read the symbols
\(r_\ell\)
receptive field (in input pixels, along one side) of a unit after layer \(\ell\)
\(j_\ell\)
the jump: how many input pixels apart two neighbouring units of layer \(\ell\) are (the product of all strides so far)
\(k_\ell, s_\ell\)
kernel size and stride of layer \(\ell\) (pooling layers count as layers too)

So \(L\) stacked 3 × 3 convs with stride 1 reach \(1 + 2L\) pixels: 3, 5, 7, 9, … Growth is only linear, which is why every CNN also downsamples. With strides the jump doubles each time, and the receptive field grows fast. Applying (163.6) to all 13 conv and 5 pooling layers of VGG-16 gives 212 pixels; applying it to the torchvision ResNet-50 gives 427 pixels, larger than the 224 input. [Certain, computed]

stride-1 layerstride-2 layerthe 224-pixel image side
Figure 4. Receptive field of one unit in the centre, after each layer, from formula (163.6). Each bar is drawn to scale against a 224-pixel image side. The feature map size assumes “same” padding.
How to read Figure 4

With \(m = 0\) (no stride) and \(k = 3\), the bars grow by 2 pixels per layer: after 14 layers a unit still sees only 29 of 224 pixels. Set \(m = 1\) (stride 2 in every layer): the receptive field passes the full image after 7 layers, but the feature map shrinks to 2 × 2. Real CNNs sit in between, typically halving the map five times (\(224 \to 7\)). Watch the feature map size in the readout: more downsampling means a larger view but a coarser grid, which hurts tasks that need exact positions, like segmentation.

A large theoretical receptive field does not mean the unit uses all of it. Luo et al. showed that the effective receptive field, the part that really changes the output, has a roughly Gaussian shape and covers only a fraction of the theoretical one. [Certain] The centre pixels matter much more than the border ones. This is one reason CNNs struggle with relations between far-apart parts of an image.

Pooling

Pooling summarizes each small window with one number and has no weights. Max pooling keeps the largest value (the strongest detection in the window); average pooling keeps the mean. Global average pooling (GAP) averages over the whole feature map and turns a \(C \times H \times W\) tensor into a vector of \(C\) numbers:

\[ \text{maxpool}(X)_{c,i,j} = \max_{0 \le u, v \lt k} X_{c,\, si+u,\, sj+v}, \qquad \text{GAP}(X)_c = \frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} X_{c,i,j} \tag{163.7} \]

GAP is how a ResNet turns its final 7 × 7 × 2048 map into one 2048-number image vector. Because GAP works for any \(H\) and \(W\), a CNN can process images of any size, unlike a fully connected layer.

What convolution assumes: locality and equivariance

The design of (163.3) builds two assumptions, called inductive biases, into the network: locality (a unit sees only a small window) and translation equivariance. Equivariance means: if you shift the input, the output shifts by the same amount and otherwise stays the same. With \(T_\delta\) meaning “shift by \(\delta\) pixels”,

\[ \operatorname{conv}(T_\delta\, x) = T_\delta \operatorname{conv}(x) \qquad \text{(stride 1)}, \qquad \text{but in general} \quad \operatorname{pool}_{s=2}(T_1\, x) \ne T_{1/2} \operatorname{pool}_{s=2}(x) \tag{163.8} \]

The first part is exact for stride-1 convolution (up to the borders). Invariance (output does not change at all) would be a stronger property, and CNNs only get it approximately, from pooling and GAP at the end. The second part is the catch: any layer with stride 2 keeps only every second position. A shift by an odd number of pixels cannot be expressed on the coarser grid, so the output changes in a way that is not a clean shift. This is aliasing. Zhang (2019) showed that common CNNs are surprisingly sensitive to one-pixel shifts and that a small blur before each downsampling (“BlurPool”) helps both stability and accuracy. [Certain] Figure 5 shows it on a 1-D signal.

shifted input and its outputsunshifted output (reference)
Figure 5. A 32-sample signal (wrapping around at the ends) goes through a 3-tap “peak” filter \([-1, 2, -1]\) with ReLU, then max pooling with window 2 and stride 2. Gray outlines show the outputs for the unshifted input, moved by the same shift. On the pooled grid a shift of \(\delta\) is \(\delta/2\) positions, so for odd \(\delta\) the outline is the average of the two nearest positions.
How to read Figure 5

Row 2 always matches its gray outline exactly: stride-1 convolution is equivariant. Row 3 matches only for even shifts. For odd shifts, peaks merge or split on the coarse grid, and even the average of the pooled map changes (2.250 versus 2.000). Turn on the blur: the pooled map is smoother, and its average stays at 2.125 for every shift. That is the anti-aliasing idea in one picture. A real CNN has five such stride-2 steps.

From filters to backbones: LeNet to ResNet

A CNN backbone is a deep stack of conv layers that turns an image into feature maps; a small head on top turns the features into the answer (class scores, boxes, masks). The history is short and worth knowing for interviews:

ModelYearWhat it added
LeNet-51998Conv, subsampling, conv, then fully connected layers, trained with backprop to read handwritten digits. [Certain]
AlexNet2012A much bigger CNN trained on GPUs on ImageNet (about 1.28 million images, 1,000 classes), with ReLU and dropout. It won ILSVRC 2012 by a large margin and started the deep learning era. [Certain]
VGG2014Only 3 × 3 convs, stacked 16 to 19 layers deep. Two 3 × 3 layers see 5 × 5 and three see 7 × 7 (formula 163.6), with fewer weights and more nonlinearities than one big filter. [Certain]
ResNet2015Residual connections, which made 50, 101 and 152 layers trainable. An ensemble reached 3.57% top-5 error on the ImageNet test set and won ILSVRC 2015. [Certain]
ConvNeXt2022A ResNet modernized with ViT-era choices (large depthwise kernels, LayerNorm, fewer activations); its paper reports 87.8% ImageNet top-1, competitive with transformers. [Certain]

Why three 3 × 3 layers instead of one 7 × 7? Both have a 7 × 7 receptive field. With \(C\) channels in and out, the stack has \(3 \cdot 9 C^2 = 27C^2\) weights and the single layer \(49C^2\): the stack uses 55% of the weights and adds two more ReLUs.

Residual connections

Before ResNet, adding layers to a deep CNN eventually made training error go up, not only test error. That is an optimization problem, not overfitting: a deeper network could in principle copy the shallower one and set the extra layers to the identity, but gradient descent did not find that solution. He et al. made the identity the default. Each block learns only a residual, a correction added to its input:

\[ \mathbf{y} = \operatorname{ReLU}\big(\mathbf{x} + \mathcal{F}(\mathbf{x})\big), \qquad \frac{\partial \mathbf{y}}{\partial \mathbf{x}} \approx I + \frac{\partial \mathcal{F}}{\partial \mathbf{x}} \tag{163.9} \]
Read the symbols
\(\mathbf{x}\)
input feature map of the block
\(\mathcal{F}\)
two or three conv layers with BatchNorm and ReLU; if the shape changes, the skip path uses a 1 × 1 conv with stride 2
\(I\)
the identity: the gradient always has a direct path back through the skip connection (ignoring the final ReLU)

If \(\mathcal{F}\) learns nothing useful, it can output near 0 and the block passes \(\mathbf{x}\) through. The gradient flows back through the “+” without being multiplied by many weight matrices. This is exactly the residual stream of every transformer (session 22). ResNets also use BatchNorm: each channel is normalized with the mean and variance of the current batch during training, and with stored running averages at test time. [Certain]

Inside ResNet-50

ResNet-50 uses bottleneck blocks: a 1 × 1 conv reduces the channels by 4×, a 3 × 3 conv works on the small tensor, and a 1 × 1 conv expands again. Here is the shape of the tensor for a 224 × 224 input, measured on the torchvision model:

StageWhatOutput shape \((C, H, W)\)Parameters
stem7 × 7 conv, stride 2; 3 × 3 max pool, stride 264 × 56 × 569,536
conv2_x3 bottleneck blocks256 × 56 × 56215,808
conv3_x4 blocks, first with stride 2512 × 28 × 281,219,584
conv4_x6 blocks, first with stride 21024 × 14 × 147,098,368
conv5_x3 blocks, first with stride 22048 × 7 × 714,964,736
headglobal average pool, linear 2048 → 100010002,049,000
total50 weight layers25,557,032

The total downsampling is 32×: a 224 input gives a 7 × 7 grid, so each final position summarizes a 32 × 32 pixel cell. Keep this 7 × 7 grid in mind: it is the CNN version of the 14 × 14 grid of patch tokens in a ViT-B/16 (next session), and early VLMs fed such grids to the language model. The paper reports 3.8 billion FLOPs (multiply-adds) for ResNet-50. [Certain] Counting every conv and the final linear layer of the torchvision version gives 4.09 billion multiply-adds. [Certain, computed]

Transfer learning

Training ResNet-50 from scratch needs a large labeled dataset and days of GPU time. Most real projects instead reuse a backbone pretrained on a large dataset and adapt it. This works because features are layered: early layers learn edges, colours and textures that are useful for almost any image, and only the last layers become specific to the training classes. Yosinski et al. measured this: features move from general to specific with depth. [Certain]

The simplest form is the linear probe: keep the backbone \(h\) frozen and train only a linear layer on its output vector:

\[ \hat{\mathbf{y}} = \softmax\big( A\, h_{\theta}(\mathbf{x}) + \mathbf{c} \big), \qquad \theta \text{ frozen}, \quad A \in \R^{K \times d} \text{ trained} \tag{163.10} \]

For ResNet-50, \(d = 2048\). A probe for 10 classes has \(2048 \cdot 10 + 10 = 20{,}490\) trainable numbers, less than 0.1% of the backbone. Linear probes are also the standard way to compare vision encoders, because they measure how much useful information the frozen features already hold (session 171).

Algorithm 2: choosing and running a transfer-learning recipe
  1. input: pretrained backbone \(h_\theta\), its preprocessing (resize, crop, \(\mu\), \(\sigma\)), your labeled data with \(K\) classes
  2. preprocess every image exactly as in pretraining // formulas (163.1), (163.2)
  3. replace the old head with a new linear layer \(A \in \R^{K \times d}\) // formula (163.10)
  4. if little data or very similar domain: freeze \(\theta\), train only \(A\) // linear probe; keep BatchNorm in eval mode
  5. else if medium data: first train \(A\) with \(\theta\) frozen, then unfreeze the last stage(s)
  6. else (much data or a very different domain, e.g. X-rays): fine-tune everything
  7. use a small learning rate for the backbone (often 10× smaller than for the head) // protects pretrained features
  8. evaluate on a held-out set with the same preprocessing

What CNNs are good at, and what they miss

The inductive biases are a trade. They help a lot when data is limited, and they limit the model when data is huge.

Good atWhy
Learning from modest dataLocality and weight sharing are built in, so the network does not need to discover them from examples
Any input sizeConvolution and global pooling work on any \(H \times W\); compute grows linearly with the number of pixels
Dense, local tasksFeature maps keep a spatial grid, which suits detection, segmentation and edge-level work
Speed on small devicesConvolutions are very well optimized on phones and embedded chips; MobileNet-style models run in real time [Certain]
MissesWhy
Long-range relations (“is the cup to the left of the laptop?”)Information travels only through local windows; the effective receptive field is small and Gaussian [Certain]
Shape over textureImageNet-trained CNNs rely strongly on texture: a cat shape with elephant skin texture is usually called an elephant (Geirhos et al.) [Certain]
Exact shift invarianceStride and pooling cause aliasing (Figure 5)
Scaling with dataWith very large pretraining sets, Vision Transformers match or beat CNNs of similar cost, because they assume less and learn more from data [Certain for the ViT paper’s results; the general trend is Likely]
A token interfaceA CNN outputs a grid of feature vectors, not a sequence of tokens with positions; VLMs prefer the ViT format, which plugs straight into a transformer

These gaps are the motivation for the next session (session 164): cut the image into patches, treat each patch as a token, and let self-attention connect any two patches in one layer. CNNs did not disappear: CLIP was released with ResNet image encoders next to its ViTs [Certain], detection and segmentation still often use CNN backbones, and some VLMs use ConvNeXt encoders for high-resolution input. [Likely]

The PyTorch version

All five ideas in one deterministic script: preprocessing, a hand-written convolution checked against F.conv2d, the output-size and receptive-field formulas, a residual block, and a frozen-backbone transfer-learning step. It runs on CPU in a few seconds and needs no download.

import torch, torch.nn as nn, torch.nn.functional as F, torchvision
torch.manual_seed(0)

# 1) an image is a uint8 array of shape (H, W, 3); models want float (B, 3, H, W)
img = torch.randint(0, 256, (480, 640, 3), dtype=torch.uint8)      # stand-in for a decoded photo
mean = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)            # ImageNet statistics
std  = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)
x = img.permute(2, 0, 1).float() / 255.0                             # HWC -> CHW, [0, 255] -> [0, 1]
x = F.interpolate(x[None], size=(224, 298), mode="bilinear", antialias=True, align_corners=False)  # shorter side -> 224
left = (298 - 224) // 2
x = x[..., :, left:left + 224]                                       # center crop -> (1, 3, 224, 224)
x = (x - mean) / std                                                 # formula (163.1)
print(x.shape, round(x.mean().item(), 3))

# 2) convolution = sliding dot product; check a hand version against F.conv2d
w = torch.randn(8, 3, 3, 3)                                          # C_out=8, C_in=3, k=3
b = torch.randn(8)
y_ref = F.conv2d(x, w, b, stride=2, padding=1)
cols = F.unfold(x, kernel_size=3, padding=1, stride=2)               # (1, 3*3*3, L): every 3x3x3 window as a column
y_hand = (w.view(8, -1) @ cols + b[:, None]).view(1, 8, 112, 112)    # formula (163.3) for every position
print(y_ref.shape, (y_ref - y_hand).abs().max().item() < 1e-4)

def out_size(n, k, s, p):                                            # formula (163.4)
    return (n + 2 * p - k) // s + 1
print(out_size(224, 7, 2, 3), out_size(112, 3, 2, 1))                # ResNet stem: 112, then 56

# 3) receptive field of a stack, formula (163.6)
def receptive_field(layers):                                         # layers: list of (kernel, stride)
    r, j = 1, 1
    for k, s in layers:
        r, j = r + (k - 1) * j, j * s
    return r, j
print(receptive_field([(3, 1)] * 3))                                 # (7, 1): three 3x3 convs see 7x7

# 4) a residual block, formula (163.9)
class BasicBlock(nn.Module):
    def __init__(self, c):
        super().__init__()
        self.c1, self.b1 = nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c)
        self.c2, self.b2 = nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c)
    def forward(self, x):
        h = F.relu(self.b1(self.c1(x)))
        return F.relu(x + self.b2(self.c2(h)))                       # y = x + F(x)
blk = BasicBlock(64)
print(blk(torch.randn(2, 64, 56, 56)).shape, sum(p.numel() for p in blk.parameters()))

# 5) transfer learning: freeze the backbone, train a new head
net = torchvision.models.resnet18(weights=None)   # in practice: weights=torchvision.models.ResNet18_Weights.DEFAULT
for p in net.parameters():
    p.requires_grad = False
net.fc = nn.Linear(net.fc.in_features, 5)                            # new head for 5 classes, trainable
print(sum(p.numel() for p in net.parameters() if p.requires_grad))   # 512*5 + 5 = 2565
opt = torch.optim.AdamW(net.fc.parameters(), lr=1e-3)
net.eval()                                                           # frozen BatchNorm: use its stored statistics
xb, yb = torch.randn(16, 3, 224, 224), torch.randint(0, 5, (16,))
for step in range(3):
    loss = F.cross_entropy(net(xb), yb)
    opt.zero_grad(); loss.backward(); opt.step()
    print(step, round(loss.item(), 4))

Output with PyTorch 2.8 on CPU: shape (1, 3, 224, 224); the hand convolution matches (True); sizes 112 and 56; receptive field (7, 1); the block keeps the shape and has 73,984 parameters (two 3 × 3 convs of \(9 \cdot 64 \cdot 64\) plus four BatchNorm vectors); 2,565 trainable parameters; and the loss on the fixed batch falls from 2.78 to 1.96 to 1.65. Exact loss values can differ slightly across PyTorch versions. [Likely]

Where this is used

ApplicationWhat the CNN ideas do there
VLM preprocessing (LLaVA, Qwen-VL, InternVL)Formulas (163.1) and (163.2) decide what the encoder sees; a wrong mean, BGR order or crop silently hurts answers (session 177)
Object detection and segmentationResNet or ConvNeXt backbones produce multi-scale feature maps (strides 8, 16, 32) for box and mask heads [Certain]
Video classification and action recognitionI3D “inflates” 2D ImageNet filters into 3D (time × height × width) filters and reuses their weights [Certain] (session 202)
Video retrieval and near-duplicate detection, fingerprintingPer-frame CNN or ViT embeddings after GAP, pooled over time, then compared by cosine similarity; robustness to crops and small shifts matters here (session 169)
Key-moment and shot-boundary detectionCheap per-frame features from a small CNN; a large change between neighbouring frames marks a cut
On-device vision (phones, cameras)Depthwise and 1 × 1 convolutions keep compute low; CNNs remain common at the edge
Medical and industrial inspection with little dataTransfer learning from a pretrained backbone, often with full fine-tuning because the domain is far from ImageNet

Common mistakes

  • Wrong normalization. Using ImageNet \(\mu, \sigma\) for a CLIP or SigLIP encoder, or forgetting the division by 255. Always load the encoder’s own preprocessor config.
  • BGR instead of RGB. cv2.imread returns BGR. Convert with cv2.cvtColor(img, cv2.COLOR_BGR2RGB).
  • Wrong layout. Passing \((H, W, C)\) where \((C, H, W)\) is expected. With a 3 × 224 × 224 conv input, a 224 × 224 × 3 tensor fails loudly, but a reshape instead of a permute fails silently and scrambles the pixels.
  • Ignoring EXIF orientation. Phone photos store a rotation flag; if you do not apply it (for example with PIL.ImageOps.exif_transpose), the model sees the image sideways.
  • Center-cropping away the answer. For documents, charts and wide video frames, the important text is often at the edges. Pad or tile instead.
  • Different resize at training and serving. Bilinear without anti-aliasing in one place and PIL bicubic in another gives different inputs.
  • Frozen backbone left in train mode. BatchNorm running statistics drift; call eval() on frozen parts.
  • Confusing receptive field with understanding. A 427-pixel theoretical receptive field does not mean ResNet-50 relates opposite corners of the image; the effective field is much smaller.

Interview questions

Why do we normalize images, and why must we use the encoder’s own mean and standard deviation?basic
Centred inputs with unit spread make optimization better conditioned, like any feature scaling. But once a model is trained, its first-layer weights expect inputs with exactly the statistics used in training. Using other \(\mu, \sigma\) shifts and rescales every input: for example white is +2.25 to +2.64 with ImageNet statistics but +1 with SigLIP’s. The model still runs, so the bug is silent. Use the preprocessor config shipped with the model.
A conv layer has \(k = 3\), stride 2, padding 1, 64 input and 128 output channels, on a 56 × 56 input. Give the output size, parameters and multiply-adds.basic
Output side \(\lfloor (56 + 2 - 3)/2 \rfloor + 1 = 28\). Parameters \(9 \cdot 64 \cdot 128 + 128 = 73{,}856\), independent of image size. Multiply-adds \(9 \cdot 64 \cdot 128 \cdot 28 \cdot 28 = 57{,}802{,}752\), a quarter of the stride-1 version because there are four times fewer output positions.
Why stack 3 × 3 convolutions instead of using one 7 × 7?core
Three 3 × 3 layers have the same 7 × 7 receptive field (formula 163.6: 3, 5, 7). With \(C\) channels they cost \(27C^2\) weights versus \(49C^2\), about 55%, and they add two extra nonlinearities, so the function is more expressive. This was the main design idea of VGG and is used in ResNet.
What is the receptive field, and how does it differ from the effective receptive field?core
The (theoretical) receptive field is the input region that can influence a unit; it follows \(r_\ell = r_{\ell-1} + (k_\ell - 1) j_{\ell-1}\), with the jump \(j\) the product of strides. For ResNet-50 it is 427 pixels. The effective receptive field measures how much each input pixel actually changes the output (via the gradient); Luo et al. found it is roughly Gaussian and covers only a fraction of the theoretical one, dominated by the centre. So a big theoretical field does not guarantee global reasoning.
Why do residual connections make very deep networks trainable?core
Plain deep networks showed higher training error as depth grew, an optimization problem. With \(y = x + \mathcal{F}(x)\), the identity is the default and each block only learns a correction; the Jacobian is \(I + \partial\mathcal{F}/\partial x\), so gradients have a direct path back to early layers and do not shrink through many weight multiplications. The same idea is the residual stream of transformers.
Are CNNs translation invariant?advanced
Stride-1 convolution is translation equivariant (shift in, same shift out), up to border effects. Invariance comes only approximately, from pooling and global average pooling at the end. Strided convs and pooling subsample, which breaks equivariance for shifts that are not multiples of the stride (aliasing); Zhang (2019) showed one-pixel shifts can change predictions and proposed blurring before subsampling. Padding also leaks absolute position, so CNNs can partly know where things are.
Freeze the backbone or fine-tune it? What about BatchNorm?core
With little data or a domain close to pretraining, freeze and train a linear head (cheap, no overfitting of the backbone). With more data or a distant domain, unfreeze the last stages or everything, with a smaller learning rate for the backbone than for the head, often after first training the head alone so random head gradients do not damage the features. For frozen parts, keep BatchNorm in eval mode so its running statistics do not drift; with small batches in fine-tuning, consider freezing BN statistics too.
You must feed 16:9 video frames to a square 224 encoder for a VLM. Crop, pad or squash?core
Crop keeps correct shapes and full resolution of the centre but drops 43.75% of the frame (edges, captions, scoreboards). Pad keeps everything undistorted but wastes 43.75% of the input on padding and gives content fewer pixels. Squash keeps everything at full width but stretches shapes by 1.78×, which the encoder never saw in training. Choose by task: pad (or tile) when edges matter, crop when the subject is central; avoid squash for models trained with crops. Better: use an encoder that supports native aspect ratio or tiling.
What does a 1 × 1 convolution do?basic
It applies the same linear map to the channel vector at every pixel, with no spatial mixing: \(C_{\text{in}} C_{\text{out}}\) weights. It changes the number of channels cheaply, as in ResNet bottlenecks (1 × 1 down, 3 × 3, 1 × 1 up) and in the shortcut when shapes change. It is the convolutional version of a token-wise linear layer in a transformer.
Why did Vision Transformers overtake CNNs, and when would you still choose a CNN?advanced
CNNs hard-code locality and equivariance. That helps with small data but limits long-range interaction (small effective receptive field) and adds texture bias. ViTs assume less, connect any two patches in one attention layer, scale well with huge pretraining sets, and produce tokens that plug directly into LLMs, which made them the default VLM encoder. Choose a CNN for small datasets without a good pretrained ViT, for dense high-resolution tasks where linear cost in pixels matters, or for edge devices with strong conv support; modern CNNs like ConvNeXt are competitive at similar compute.
Global average pooling vs flattening into a fully connected layer?core
Flattening a 7 × 7 × 2048 map into an FC layer needs \(100{,}352 \times K\) weights and fixes the input size. GAP reduces it to a 2048-vector first, so the head has \(2048K\) weights, works for any input resolution and is roughly invariant to where the object is. The cost is losing spatial layout, which is why detection and VLM connectors use the grid instead.

Exercises

  1. Normalize the pure red pixel (255, 0, 0) with the ImageNet recipe.
  2. Give the output side of a conv with \(k = 5\), \(s = 2\), \(p = 2\) on a 224 input.
  3. Parameters and multiply-adds of a 3 × 3 conv from 128 to 256 channels with a 28 × 28 output. Compare with the 64 → 128 layer at 56 × 56.
  4. Receptive field and jump of: conv 3 × 3, conv 3 × 3, max pool 2 × 2 stride 2, conv 3 × 3, conv 3 × 3 (all convs stride 1).
  5. A 1920 × 1080 frame goes to a 336 encoder. Give the resized size for crop and for pad, and the fraction lost or padded.
  6. How many trainable parameters does a linear probe on ResNet-50 features need for 10 classes? For the original 1,000-class head?
Answerscheck
1. \(((1 - 0.485)/0.229,\ (0 - 0.456)/0.224,\ (0 - 0.406)/0.225) = (2.249, -2.036, -1.804)\). 2. \(\lfloor (224 + 4 - 5)/2 \rfloor + 1 = 111 + 1 = 112\). 3. \(9 \cdot 128 \cdot 256 + 256 = 295{,}168\) parameters and \(9 \cdot 128 \cdot 256 \cdot 784 = 231{,}211{,}008\) multiply-adds: four times the parameters of the 64 → 128 layer but exactly the same compute. 4. \(r\): \(1 \to 3 \to 5 \to 6\) (pool: \(+1 \cdot 1\), jump becomes 2) \(\to 10 \to 14\). Receptive field 14, jump 2. 5. Crop: scale \(336/1080 = 0.311\), resized to about 597 × 336, then cut to 336 × 336, losing 43.75% of the frame. Pad: scale \(336/1920 = 0.175\), resized to 336 × 189, so 43.75% of the square is padding. 6. \(2048 \cdot 10 + 10 = 20{,}490\); the 1,000-class head has \(2048 \cdot 1000 + 1000 = 2{,}049{,}000\).

Key takeaways

  • An image is a \(H \times W \times 3\) uint8 array; models want float \((B, 3, H, W)\) in RGB order, normalized with the encoder’s own \(\mu, \sigma\) (163.1). ImageNet, CLIP and SigLIP use different numbers.
  • Square encoders force a choice between cropping, padding and squashing (163.2); each loses something, and small details vanish at 224 pixels.
  • A convolution is a shared sliding dot product (163.3); its output size (163.4), parameters and compute (163.5) can be computed by hand. Halving the map and doubling channels keeps compute per layer constant.
  • Receptive fields grow linearly with depth and multiply with stride (163.6); the effective field is smaller. Pooling and GAP (163.7) summarize and give approximate invariance; stride causes aliasing (163.8).
  • Residual blocks (163.9) make deep CNNs trainable; ResNet-50 has 25.6 million parameters and turns 224 × 224 into a 7 × 7 × 2048 grid.
  • Transfer learning reuses a pretrained backbone with a linear probe (163.10) or fine-tuning. The CNN’s locality bias helps with small data but limits global reasoning, which motivates the ViT.

Key formulas

\[ \tilde{x} = \frac{x/255 - \mu_c}{\sigma_c}, \quad H_{\text{out}} = \left\lfloor \frac{H_{\text{in}} + 2p - k}{s} \right\rfloor + 1, \quad \text{params} = k^2 C_{\text{in}} C_{\text{out}} + C_{\text{out}} \]
\[ r_\ell = r_{\ell-1} + (k_\ell - 1)\, j_{\ell-1}, \quad j_\ell = j_{\ell-1} s_\ell, \quad \mathbf{y} = \operatorname{ReLU}\big(\mathbf{x} + \mathcal{F}(\mathbf{x})\big) \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  2. He et al. (2015). Deep Residual Learning for Image Recognition
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
  4. He et al. (2015). Deep Residual Learning for Image Recognition
  5. LeCun, Bottou, Bengio & Haffner (1998). Gradient-Based Learning Applied to Document Recognition
  6. Krizhevsky, Sutskever & Hinton (2012). ImageNet Classification with Deep Convolutional Neural Networks
  7. Simonyan & Zisserman (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition
  8. Yosinski, Clune, Bengio & Lipson (2014). How transferable are features in deep neural networks?
  9. Luo, Li, Urtasun & Zemel (2016). Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
  10. Geirhos et al. (2019). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
  11. Zhang (2019). Making Convolutional Networks Shift-Invariant Again
  12. Liu et al. (2022). A ConvNet for the 2020s
  13. Carreira & Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
  14. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  15. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale