Images as data and a CNN refresher
Before a vision-language model can talk about a picture, the picture must become a tensor of numbers with the exact shape, scale and colour order that the vision encoder was trained on. This session shows how that happens (pixels, channels, normalization, resizing and aspect ratio), then refreshes convolutional networks: the sliding filter, the receptive field, pooling, ResNet and transfer learning. It ends with what CNNs do well and what they miss, which is the reason the next session moves to the Vision Transformer.
What you will learn
- How an image is stored (uint8, HWC vs CHW, RGB vs BGR) and how much memory images and video frames take
- Why every encoder needs its own normalization (ImageNet, CLIP, SigLIP statistics) and what goes wrong when it does not match
- Three ways to fit any aspect ratio into a square input (squash, crop, pad) and what each one loses
- Convolution as a sliding dot product: output size, parameter count and multiply-adds of a layer, computed by hand
- Receptive field growth through depth, stride and pooling, and why the effective receptive field is smaller
- From LeNet to ResNet: residual blocks, the ResNet-50 stage layout, and transfer learning with frozen or fine-tuned backbones
- What the CNN assumptions (locality, translation equivariance) give you, and where CNNs fail: aliasing, texture bias, limited global context
Why this matters
A vision-language model (VLM) never sees a picture. It sees a tensor: a block of numbers with a fixed shape, a fixed value range and a fixed colour order. A vision encoder turns that tensor into vectors, and only then can the language model read them. If the numbers are prepared in the wrong way, the model still answers, fluently and wrongly.
This session starts the vision track (Track 7). It has two parts. The first part is images as data: how a photo or a video frame becomes a tensor, and the small preprocessing choices (normalization, resizing, cropping, padding) that silently decide what the model can see. These choices come back in every later session: CLIP and SigLIP have their own normalization, and high-resolution VLMs are mostly about handling aspect ratio well (session 180).
The second part is a refresher on convolutional neural networks (CNNs). From 2012 to about 2020, CNNs were the standard way to turn images into features, and they are still everywhere: in detection and segmentation backbones, on phones, in video pipelines, and as the image encoder of some VLMs. The Vision Transformer of the next session (session 164) is easiest to understand as an answer to the limits of CNNs. So we build the CNN up from one filter, measure it with numbers, and end with what it does well and what it misses.
An image is a grid of numbers
A digital colour image is a grid of pixels (picture elements). Each pixel stores three numbers, one per channel: red, green and blue (RGB). In a normal photo file (JPEG, PNG) each number is an 8-bit unsigned integer (uint8) from 0 (no light) to 255 (full light). So an image of height \(H\) and width \(W\) is an array of \(H \times W \times 3\) integers.
- Layout. Image libraries (PIL, OpenCV, NumPy) store the array as \((H, W, C)\): channels last. PyTorch models expect \((B, C, H, W)\): a batch dimension first and channels before height and width. The conversion is one
permute(2, 0, 1). - Colour order. PIL and torchvision give RGB. OpenCV’s
cv2.imreadgives BGR (blue first). [Certain] A model trained on RGB that receives BGR sees red and blue swapped. - Size. A 224 × 224 RGB image is \(224 \cdot 224 \cdot 3 = 150{,}528\) numbers: 150,528 bytes as
uint8, four times that (602,112 bytes) asfloat32. One 1920 × 1080 video frame is 6,220,800 bytes, and one second of 30 fps video is about 187 MB before compression. This is why video systems sample frames and shrink them first (session 201). - Grayscale and others. A grayscale image has one channel. Some images have a fourth alpha channel (transparency) that must be dropped or blended before an RGB model sees it.
Normalization
Neural networks train best when their inputs are centred near 0 with a spread near 1 (the same reason we normalize hidden states, see session 22). So each channel \(c\) is first scaled to \([0, 1]\), then shifted by a mean \(\mu_c\) and divided by a standard deviation \(\sigma_c\):
- Read the symbols
- \(x_{c,i,j}\)
- the raw
uint8value of channel \(c\) at row \(i\), column \(j\) - \(\mu_c, \sigma_c\)
- a mean and a standard deviation per channel, fixed by whoever trained the encoder
- \(\tilde{x}_{c,i,j}\)
- the number the network actually receives
The important point: \(\mu\) and \(\sigma\) are part of the model. They are not computed on your data. Three common choices [Certain]:
| Encoder family | \(\mu\) (R, G, B) | \(\sigma\) (R, G, B) | Range of \(\tilde{x}\) |
|---|---|---|---|
| ImageNet CNNs (torchvision ResNet, many others) | 0.485, 0.456, 0.406 | 0.229, 0.224, 0.225 | about −2.1 to 2.6 |
| OpenAI CLIP | 0.4815, 0.4578, 0.4082 | 0.2686, 0.2613, 0.2758 | about −1.8 to 2.1 |
| SigLIP (Hugging Face config) | 0.5, 0.5, 0.5 | 0.5, 0.5, 0.5 | exactly −1 to 1 |
The ImageNet numbers are the average colour of the ImageNet training photos. CLIP measured its own statistics on its own data. SigLIP simply maps \([0, 1]\) to \([-1, 1]\). Figure 1 lets you push a few pixel colours through each recipe.
How to read Figure 1
Mid-gray (128, 128, 128) lands near 0 for every recipe: that is the point of subtracting the mean. Pure white lands near +2.2 to +2.6 for ImageNet but at exactly +1 for SigLIP, so feeding ImageNet-normalized images into SigLIP gives inputs more than twice too large. Turn on the BGR switch with the red pixel: the network now sees a strongly blue pixel. Nothing crashes, the shapes are correct, and the model quietly sees the wrong colours.
White is (255, 255, 255), so \(x/255 = 1\) in every channel. Red channel: \((1 - 0.485)/0.229 = 2.249\). Green: \((1 - 0.456)/0.224 = 2.429\). Blue: \((1 - 0.406)/0.225 = 2.640\).
Black is (0, 0, 0): red \(-0.485/0.229 = -2.118\), green \(-2.036\), blue \(-1.804\). So the ImageNet-normalized range is about \([-2.1, 2.6]\), not \([-1, 1]\). With SigLIP’s recipe the same two pixels become exactly \(+1\) and \(-1\).
Resizing and aspect ratio
Most vision encoders take a fixed square input: 224 × 224 for classic CNNs and many ViTs, 336 × 336 for CLIP ViT-L/14-336, 384 or 448 for others. Real images are not square: a phone photo is 4:3, a video frame is 16:9, a receipt can be 1:3. The aspect ratio is width divided by height. There are three standard ways to fit a \(W \times H\) image into an \(S \times S\) square:
- Squash: resize width and height separately to \(S\). Nothing is lost, but shapes are stretched: a circle becomes an ellipse.
- Resize and center crop: scale the shorter side to \(S\), then cut the central \(S \times S\) square. Shapes are correct, but the edges of the long side are thrown away. This is the default for ImageNet CNNs and for CLIP. [Certain]
- Pad (letterbox): scale the longer side to \(S\), then fill the rest with a constant colour. Nothing is lost and nothing is stretched, but part of the input is empty, and the content gets fewer pixels.
- Read the symbols
- \(W, H\)
- width and height of the original image in pixels
- \(S\)
- side of the square input of the encoder
- \(s\)
- the scale factor applied to both sides (so shapes are not distorted)
- kept, content
- the fraction of the original image area that survives the crop; the same number is the fraction of the padded square that holds real content
For a 1920 × 1080 frame and \(S = 224\): the crop keeps \(1080/1920 = 56.25\%\) of the frame, and the pad leaves \(43.75\%\) of the square empty. Either way the encoder sees only \(224^2 / (1920 \cdot 1080) = 2.4\%\) as many pixels as the original. Small text, a distant face or a road sign may simply vanish. That single fact drives a large part of modern VLM design: tiling, dynamic resolution and native aspect ratio (session 180).
How to read Figure 2
Start with a 16:9 frame in crop mode: the left and right 22% of the frame are gone. Switch to pad: everything is kept, but almost half of the patch grid holds padding, so many tokens carry no information. Switch to squash: the circle becomes an ellipse with the aspect-ratio factor printed below. Now set \(W = 1000\), \(H = 3000\) (a tall receipt) and see how much each method loses. No single choice is right for every image, which is why newer VLMs avoid the fixed square.
Convolution: a sliding filter
Now the network. The simplest idea would be to flatten the 150,528 input numbers and use a fully connected layer. With only 1,000 output units, that one layer would already need 150 million weights, and it would treat a cat in the top-left corner as a completely different thing from the same cat in the bottom-right. Images have two properties we can use instead:
- Locality. Nearby pixels belong together. An edge, a corner, a patch of fur: each is visible inside a small window.
- Same pattern, any place. An edge detector that works in one corner should work everywhere.
A convolution uses both. It takes a small grid of weights, the filter or kernel (for example 3 × 3), slides it over the image, and at every position computes a dot product between the filter and the pixels under it. The same filter weights are used at every position: this is called weight sharing. The result is a new grid, the feature map, that is large where the image looks like the filter. With several input channels the filter has a depth too, and with several output channels we use several filters:
- Read the symbols
- \(X\)
- input with \(C_{\text{in}}\) channels; positions outside the image count as 0 (zero padding)
- \(W_{o,c,u,v}\)
- weight of filter \(o\), input channel \(c\), offset \((u, v)\) inside the \(k \times k\) window
- \(b_o\)
- one bias per output channel
- \(s\)
- stride: how many pixels the window jumps between positions
- \(p\)
- padding: how many rows and columns of zeros are added around the input
- \(Y_{o,i,j}\)
- output channel \(o\) at output position \((i, j)\)
Strictly, (163.3) is a cross-correlation: a mathematical convolution flips the filter first. Deep learning libraries do not flip, and since the filter is learned, the difference does not matter. [Certain] Each output is just a dot product, so a whole conv layer can be written as one big matrix multiply (the PyTorch code below checks this with unfold).
Output size, parameters and compute
Along each side, the number of window positions is
The ResNet first layer has \(k = 7\), \(s = 2\), \(p = 3\) on a 224 input: \(\lfloor (224 + 6 - 7)/2 \rfloor + 1 = 112\). A 3 × 3 conv with \(p = 1\), \(s = 1\) keeps the size (“same” padding). The cost of a layer is
A MAC is one multiply-accumulate (about 2 FLOPs). Notice the difference: parameters do not depend on image size (weight sharing), but compute grows with the number of output positions.
A 3 × 3 conv from 64 to 128 channels with an output of 56 × 56. Parameters: \(9 \cdot 64 \cdot 128 + 128 = 73{,}856\). Multiply-adds: \(9 \cdot 64 \cdot 128 \cdot 56 \cdot 56 = 231{,}211{,}008\), about 0.23 billion.
Now a 3 × 3 conv from 128 to 256 channels at 28 × 28 (one stage deeper: half the size, twice the channels). Parameters: 295,168, four times more. Multiply-adds: \(9 \cdot 128 \cdot 256 \cdot 784 = 231{,}211{,}008\), exactly the same. Halving the resolution while doubling the channels keeps the compute per layer constant. This is the design rule of VGG and ResNet. [Certain]
Figure 3 runs formula (163.3) on a tiny one-channel 8 × 8 image: dark on the left (value 1), bright on the right (value 7), with a small bright block in the dark half.
How to read Figure 3
The vertical-edge filter (a Sobel filter) gives 0 on flat regions and 24 on the big dark-to-bright edge, because there the right column of the window is 7 and the left column is 1: \((7 - 1) \cdot (1 + 2 + 1) = 24\). The small block gives a +18 and a −18: a bright-to-dark edge has the opposite sign. The horizontal filter ignores the big vertical edge completely and only fires on the top and bottom of the block. Turn on padding: the output becomes 8 × 8 instead of 6 × 6, as formula (163.4) says, and the border cells show artifacts from the zeros. Learned CNN filters in the first layer look a lot like these: edges at many angles, blobs of colour. [Certain]
- input: \(X\) of shape \((C_{\text{in}}, H, W)\), weights \(W\) of shape \((C_{\text{out}}, C_{\text{in}}, k, k)\), bias \(b\), stride \(s\), padding \(p\)
- pad \(X\) with \(p\) zeros on every side
- \(H_{\text{out}}, W_{\text{out}} \leftarrow\) formula (163.4)
- for \(o = 1 \dots C_{\text{out}}\), \(i = 0 \dots H_{\text{out}}-1\), \(j = 0 \dots W_{\text{out}}-1\):
- \(\text{win} \leftarrow X[:, si : si + k, sj : sj + k]\) // a \(C_{\text{in}} \times k \times k\) block
- \(Y[o, i, j] \leftarrow b_o + \sum \text{win} \odot W[o]\) // one dot product of length \(k^2 C_{\text{in}}\)
- apply a nonlinearity (ReLU) to \(Y\), usually after a normalization layer
- return \(Y\) of shape \((C_{\text{out}}, H_{\text{out}}, W_{\text{out}})\) // real kernels batch all windows into one GEMM
Two special kernel sizes are worth knowing. A 1 × 1 convolution looks at one position only: it is a linear layer applied to the channel vector of each pixel, used to change the number of channels cheaply. A depthwise convolution uses one filter per channel (no mixing between channels) and is the cheap building block of MobileNet and ConvNeXt. [Certain]
Receptive field and pooling
One output of a 3 × 3 conv depends on a 3 × 3 patch of its input. Its receptive field is the region of the original image that can affect it. Stack a second 3 × 3 conv on top: each of its outputs sees a 3 × 3 patch of the first map, which covers a 5 × 5 patch of the image. A stride or a pooling layer makes every later step count double. The rule, from the input upwards, is
- Read the symbols
- \(r_\ell\)
- receptive field (in input pixels, along one side) of a unit after layer \(\ell\)
- \(j_\ell\)
- the jump: how many input pixels apart two neighbouring units of layer \(\ell\) are (the product of all strides so far)
- \(k_\ell, s_\ell\)
- kernel size and stride of layer \(\ell\) (pooling layers count as layers too)
So \(L\) stacked 3 × 3 convs with stride 1 reach \(1 + 2L\) pixels: 3, 5, 7, 9, … Growth is only linear, which is why every CNN also downsamples. With strides the jump doubles each time, and the receptive field grows fast. Applying (163.6) to all 13 conv and 5 pooling layers of VGG-16 gives 212 pixels; applying it to the torchvision ResNet-50 gives 427 pixels, larger than the 224 input. [Certain, computed]
How to read Figure 4
With \(m = 0\) (no stride) and \(k = 3\), the bars grow by 2 pixels per layer: after 14 layers a unit still sees only 29 of 224 pixels. Set \(m = 1\) (stride 2 in every layer): the receptive field passes the full image after 7 layers, but the feature map shrinks to 2 × 2. Real CNNs sit in between, typically halving the map five times (\(224 \to 7\)). Watch the feature map size in the readout: more downsampling means a larger view but a coarser grid, which hurts tasks that need exact positions, like segmentation.
A large theoretical receptive field does not mean the unit uses all of it. Luo et al. showed that the effective receptive field, the part that really changes the output, has a roughly Gaussian shape and covers only a fraction of the theoretical one. [Certain] The centre pixels matter much more than the border ones. This is one reason CNNs struggle with relations between far-apart parts of an image.
Pooling
Pooling summarizes each small window with one number and has no weights. Max pooling keeps the largest value (the strongest detection in the window); average pooling keeps the mean. Global average pooling (GAP) averages over the whole feature map and turns a \(C \times H \times W\) tensor into a vector of \(C\) numbers:
GAP is how a ResNet turns its final 7 × 7 × 2048 map into one 2048-number image vector. Because GAP works for any \(H\) and \(W\), a CNN can process images of any size, unlike a fully connected layer.
What convolution assumes: locality and equivariance
The design of (163.3) builds two assumptions, called inductive biases, into the network: locality (a unit sees only a small window) and translation equivariance. Equivariance means: if you shift the input, the output shifts by the same amount and otherwise stays the same. With \(T_\delta\) meaning “shift by \(\delta\) pixels”,
The first part is exact for stride-1 convolution (up to the borders). Invariance (output does not change at all) would be a stronger property, and CNNs only get it approximately, from pooling and GAP at the end. The second part is the catch: any layer with stride 2 keeps only every second position. A shift by an odd number of pixels cannot be expressed on the coarser grid, so the output changes in a way that is not a clean shift. This is aliasing. Zhang (2019) showed that common CNNs are surprisingly sensitive to one-pixel shifts and that a small blur before each downsampling (“BlurPool”) helps both stability and accuracy. [Certain] Figure 5 shows it on a 1-D signal.
How to read Figure 5
Row 2 always matches its gray outline exactly: stride-1 convolution is equivariant. Row 3 matches only for even shifts. For odd shifts, peaks merge or split on the coarse grid, and even the average of the pooled map changes (2.250 versus 2.000). Turn on the blur: the pooled map is smoother, and its average stays at 2.125 for every shift. That is the anti-aliasing idea in one picture. A real CNN has five such stride-2 steps.
From filters to backbones: LeNet to ResNet
A CNN backbone is a deep stack of conv layers that turns an image into feature maps; a small head on top turns the features into the answer (class scores, boxes, masks). The history is short and worth knowing for interviews:
| Model | Year | What it added |
|---|---|---|
| LeNet-5 | 1998 | Conv, subsampling, conv, then fully connected layers, trained with backprop to read handwritten digits. [Certain] |
| AlexNet | 2012 | A much bigger CNN trained on GPUs on ImageNet (about 1.28 million images, 1,000 classes), with ReLU and dropout. It won ILSVRC 2012 by a large margin and started the deep learning era. [Certain] |
| VGG | 2014 | Only 3 × 3 convs, stacked 16 to 19 layers deep. Two 3 × 3 layers see 5 × 5 and three see 7 × 7 (formula 163.6), with fewer weights and more nonlinearities than one big filter. [Certain] |
| ResNet | 2015 | Residual connections, which made 50, 101 and 152 layers trainable. An ensemble reached 3.57% top-5 error on the ImageNet test set and won ILSVRC 2015. [Certain] |
| ConvNeXt | 2022 | A ResNet modernized with ViT-era choices (large depthwise kernels, LayerNorm, fewer activations); its paper reports 87.8% ImageNet top-1, competitive with transformers. [Certain] |
Why three 3 × 3 layers instead of one 7 × 7? Both have a 7 × 7 receptive field. With \(C\) channels in and out, the stack has \(3 \cdot 9 C^2 = 27C^2\) weights and the single layer \(49C^2\): the stack uses 55% of the weights and adds two more ReLUs.
Residual connections
Before ResNet, adding layers to a deep CNN eventually made training error go up, not only test error. That is an optimization problem, not overfitting: a deeper network could in principle copy the shallower one and set the extra layers to the identity, but gradient descent did not find that solution. He et al. made the identity the default. Each block learns only a residual, a correction added to its input:
- Read the symbols
- \(\mathbf{x}\)
- input feature map of the block
- \(\mathcal{F}\)
- two or three conv layers with BatchNorm and ReLU; if the shape changes, the skip path uses a 1 × 1 conv with stride 2
- \(I\)
- the identity: the gradient always has a direct path back through the skip connection (ignoring the final ReLU)
If \(\mathcal{F}\) learns nothing useful, it can output near 0 and the block passes \(\mathbf{x}\) through. The gradient flows back through the “+” without being multiplied by many weight matrices. This is exactly the residual stream of every transformer (session 22). ResNets also use BatchNorm: each channel is normalized with the mean and variance of the current batch during training, and with stored running averages at test time. [Certain]
Inside ResNet-50
ResNet-50 uses bottleneck blocks: a 1 × 1 conv reduces the channels by 4×, a 3 × 3 conv works on the small tensor, and a 1 × 1 conv expands again. Here is the shape of the tensor for a 224 × 224 input, measured on the torchvision model:
| Stage | What | Output shape \((C, H, W)\) | Parameters |
|---|---|---|---|
| stem | 7 × 7 conv, stride 2; 3 × 3 max pool, stride 2 | 64 × 56 × 56 | 9,536 |
| conv2_x | 3 bottleneck blocks | 256 × 56 × 56 | 215,808 |
| conv3_x | 4 blocks, first with stride 2 | 512 × 28 × 28 | 1,219,584 |
| conv4_x | 6 blocks, first with stride 2 | 1024 × 14 × 14 | 7,098,368 |
| conv5_x | 3 blocks, first with stride 2 | 2048 × 7 × 7 | 14,964,736 |
| head | global average pool, linear 2048 → 1000 | 1000 | 2,049,000 |
| total | 50 weight layers | 25,557,032 |
The total downsampling is 32×: a 224 input gives a 7 × 7 grid, so each final position summarizes a 32 × 32 pixel cell. Keep this 7 × 7 grid in mind: it is the CNN version of the 14 × 14 grid of patch tokens in a ViT-B/16 (next session), and early VLMs fed such grids to the language model. The paper reports 3.8 billion FLOPs (multiply-adds) for ResNet-50. [Certain] Counting every conv and the final linear layer of the torchvision version gives 4.09 billion multiply-adds. [Certain, computed]
Transfer learning
Training ResNet-50 from scratch needs a large labeled dataset and days of GPU time. Most real projects instead reuse a backbone pretrained on a large dataset and adapt it. This works because features are layered: early layers learn edges, colours and textures that are useful for almost any image, and only the last layers become specific to the training classes. Yosinski et al. measured this: features move from general to specific with depth. [Certain]
The simplest form is the linear probe: keep the backbone \(h\) frozen and train only a linear layer on its output vector:
For ResNet-50, \(d = 2048\). A probe for 10 classes has \(2048 \cdot 10 + 10 = 20{,}490\) trainable numbers, less than 0.1% of the backbone. Linear probes are also the standard way to compare vision encoders, because they measure how much useful information the frozen features already hold (session 171).
- input: pretrained backbone \(h_\theta\), its preprocessing (resize, crop, \(\mu\), \(\sigma\)), your labeled data with \(K\) classes
- preprocess every image exactly as in pretraining // formulas (163.1), (163.2)
- replace the old head with a new linear layer \(A \in \R^{K \times d}\) // formula (163.10)
- if little data or very similar domain: freeze \(\theta\), train only \(A\) // linear probe; keep BatchNorm in eval mode
- else if medium data: first train \(A\) with \(\theta\) frozen, then unfreeze the last stage(s)
- else (much data or a very different domain, e.g. X-rays): fine-tune everything
- use a small learning rate for the backbone (often 10× smaller than for the head) // protects pretrained features
- evaluate on a held-out set with the same preprocessing
What CNNs are good at, and what they miss
The inductive biases are a trade. They help a lot when data is limited, and they limit the model when data is huge.
| Good at | Why |
|---|---|
| Learning from modest data | Locality and weight sharing are built in, so the network does not need to discover them from examples |
| Any input size | Convolution and global pooling work on any \(H \times W\); compute grows linearly with the number of pixels |
| Dense, local tasks | Feature maps keep a spatial grid, which suits detection, segmentation and edge-level work |
| Speed on small devices | Convolutions are very well optimized on phones and embedded chips; MobileNet-style models run in real time [Certain] |
| Misses | Why |
|---|---|
| Long-range relations (“is the cup to the left of the laptop?”) | Information travels only through local windows; the effective receptive field is small and Gaussian [Certain] |
| Shape over texture | ImageNet-trained CNNs rely strongly on texture: a cat shape with elephant skin texture is usually called an elephant (Geirhos et al.) [Certain] |
| Exact shift invariance | Stride and pooling cause aliasing (Figure 5) |
| Scaling with data | With very large pretraining sets, Vision Transformers match or beat CNNs of similar cost, because they assume less and learn more from data [Certain for the ViT paper’s results; the general trend is Likely] |
| A token interface | A CNN outputs a grid of feature vectors, not a sequence of tokens with positions; VLMs prefer the ViT format, which plugs straight into a transformer |
These gaps are the motivation for the next session (session 164): cut the image into patches, treat each patch as a token, and let self-attention connect any two patches in one layer. CNNs did not disappear: CLIP was released with ResNet image encoders next to its ViTs [Certain], detection and segmentation still often use CNN backbones, and some VLMs use ConvNeXt encoders for high-resolution input. [Likely]
The PyTorch version
All five ideas in one deterministic script: preprocessing, a hand-written convolution checked against F.conv2d, the output-size and receptive-field formulas, a residual block, and a frozen-backbone transfer-learning step. It runs on CPU in a few seconds and needs no download.
import torch, torch.nn as nn, torch.nn.functional as F, torchvision torch.manual_seed(0) # 1) an image is a uint8 array of shape (H, W, 3); models want float (B, 3, H, W) img = torch.randint(0, 256, (480, 640, 3), dtype=torch.uint8) # stand-in for a decoded photo mean = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1) # ImageNet statistics std = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1) x = img.permute(2, 0, 1).float() / 255.0 # HWC -> CHW, [0, 255] -> [0, 1] x = F.interpolate(x[None], size=(224, 298), mode="bilinear", antialias=True, align_corners=False) # shorter side -> 224 left = (298 - 224) // 2 x = x[..., :, left:left + 224] # center crop -> (1, 3, 224, 224) x = (x - mean) / std # formula (163.1) print(x.shape, round(x.mean().item(), 3)) # 2) convolution = sliding dot product; check a hand version against F.conv2d w = torch.randn(8, 3, 3, 3) # C_out=8, C_in=3, k=3 b = torch.randn(8) y_ref = F.conv2d(x, w, b, stride=2, padding=1) cols = F.unfold(x, kernel_size=3, padding=1, stride=2) # (1, 3*3*3, L): every 3x3x3 window as a column y_hand = (w.view(8, -1) @ cols + b[:, None]).view(1, 8, 112, 112) # formula (163.3) for every position print(y_ref.shape, (y_ref - y_hand).abs().max().item() < 1e-4) def out_size(n, k, s, p): # formula (163.4) return (n + 2 * p - k) // s + 1 print(out_size(224, 7, 2, 3), out_size(112, 3, 2, 1)) # ResNet stem: 112, then 56 # 3) receptive field of a stack, formula (163.6) def receptive_field(layers): # layers: list of (kernel, stride) r, j = 1, 1 for k, s in layers: r, j = r + (k - 1) * j, j * s return r, j print(receptive_field([(3, 1)] * 3)) # (7, 1): three 3x3 convs see 7x7 # 4) a residual block, formula (163.9) class BasicBlock(nn.Module): def __init__(self, c): super().__init__() self.c1, self.b1 = nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c) self.c2, self.b2 = nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c) def forward(self, x): h = F.relu(self.b1(self.c1(x))) return F.relu(x + self.b2(self.c2(h))) # y = x + F(x) blk = BasicBlock(64) print(blk(torch.randn(2, 64, 56, 56)).shape, sum(p.numel() for p in blk.parameters())) # 5) transfer learning: freeze the backbone, train a new head net = torchvision.models.resnet18(weights=None) # in practice: weights=torchvision.models.ResNet18_Weights.DEFAULT for p in net.parameters(): p.requires_grad = False net.fc = nn.Linear(net.fc.in_features, 5) # new head for 5 classes, trainable print(sum(p.numel() for p in net.parameters() if p.requires_grad)) # 512*5 + 5 = 2565 opt = torch.optim.AdamW(net.fc.parameters(), lr=1e-3) net.eval() # frozen BatchNorm: use its stored statistics xb, yb = torch.randn(16, 3, 224, 224), torch.randint(0, 5, (16,)) for step in range(3): loss = F.cross_entropy(net(xb), yb) opt.zero_grad(); loss.backward(); opt.step() print(step, round(loss.item(), 4))
Output with PyTorch 2.8 on CPU: shape (1, 3, 224, 224); the hand convolution matches (True); sizes 112 and 56; receptive field (7, 1); the block keeps the shape and has 73,984 parameters (two 3 × 3 convs of \(9 \cdot 64 \cdot 64\) plus four BatchNorm vectors); 2,565 trainable parameters; and the loss on the fixed batch falls from 2.78 to 1.96 to 1.65. Exact loss values can differ slightly across PyTorch versions. [Likely]
Where this is used
| Application | What the CNN ideas do there |
|---|---|
| VLM preprocessing (LLaVA, Qwen-VL, InternVL) | Formulas (163.1) and (163.2) decide what the encoder sees; a wrong mean, BGR order or crop silently hurts answers (session 177) |
| Object detection and segmentation | ResNet or ConvNeXt backbones produce multi-scale feature maps (strides 8, 16, 32) for box and mask heads [Certain] |
| Video classification and action recognition | I3D “inflates” 2D ImageNet filters into 3D (time × height × width) filters and reuses their weights [Certain] (session 202) |
| Video retrieval and near-duplicate detection, fingerprinting | Per-frame CNN or ViT embeddings after GAP, pooled over time, then compared by cosine similarity; robustness to crops and small shifts matters here (session 169) |
| Key-moment and shot-boundary detection | Cheap per-frame features from a small CNN; a large change between neighbouring frames marks a cut |
| On-device vision (phones, cameras) | Depthwise and 1 × 1 convolutions keep compute low; CNNs remain common at the edge |
| Medical and industrial inspection with little data | Transfer learning from a pretrained backbone, often with full fine-tuning because the domain is far from ImageNet |
Common mistakes
- Wrong normalization. Using ImageNet \(\mu, \sigma\) for a CLIP or SigLIP encoder, or forgetting the division by 255. Always load the encoder’s own preprocessor config.
- BGR instead of RGB.
cv2.imreadreturns BGR. Convert withcv2.cvtColor(img, cv2.COLOR_BGR2RGB). - Wrong layout. Passing \((H, W, C)\) where \((C, H, W)\) is expected. With a 3 × 224 × 224 conv input, a 224 × 224 × 3 tensor fails loudly, but a reshape instead of a permute fails silently and scrambles the pixels.
- Ignoring EXIF orientation. Phone photos store a rotation flag; if you do not apply it (for example with
PIL.ImageOps.exif_transpose), the model sees the image sideways. - Center-cropping away the answer. For documents, charts and wide video frames, the important text is often at the edges. Pad or tile instead.
- Different resize at training and serving. Bilinear without anti-aliasing in one place and PIL bicubic in another gives different inputs.
- Frozen backbone left in train mode. BatchNorm running statistics drift; call
eval()on frozen parts. - Confusing receptive field with understanding. A 427-pixel theoretical receptive field does not mean ResNet-50 relates opposite corners of the image; the effective field is much smaller.
Interview questions
Why do we normalize images, and why must we use the encoder’s own mean and standard deviation?basic
A conv layer has \(k = 3\), stride 2, padding 1, 64 input and 128 output channels, on a 56 × 56 input. Give the output size, parameters and multiply-adds.basic
Why stack 3 × 3 convolutions instead of using one 7 × 7?core
What is the receptive field, and how does it differ from the effective receptive field?core
Why do residual connections make very deep networks trainable?core
Are CNNs translation invariant?advanced
Freeze the backbone or fine-tune it? What about BatchNorm?core
You must feed 16:9 video frames to a square 224 encoder for a VLM. Crop, pad or squash?core
What does a 1 × 1 convolution do?basic
Why did Vision Transformers overtake CNNs, and when would you still choose a CNN?advanced
Global average pooling vs flattening into a fully connected layer?core
Exercises
- Normalize the pure red pixel (255, 0, 0) with the ImageNet recipe.
- Give the output side of a conv with \(k = 5\), \(s = 2\), \(p = 2\) on a 224 input.
- Parameters and multiply-adds of a 3 × 3 conv from 128 to 256 channels with a 28 × 28 output. Compare with the 64 → 128 layer at 56 × 56.
- Receptive field and jump of: conv 3 × 3, conv 3 × 3, max pool 2 × 2 stride 2, conv 3 × 3, conv 3 × 3 (all convs stride 1).
- A 1920 × 1080 frame goes to a 336 encoder. Give the resized size for crop and for pad, and the fraction lost or padded.
- How many trainable parameters does a linear probe on ResNet-50 features need for 10 classes? For the original 1,000-class head?
Answerscheck
Key takeaways
- An image is a \(H \times W \times 3\)
uint8array; models want float \((B, 3, H, W)\) in RGB order, normalized with the encoder’s own \(\mu, \sigma\) (163.1). ImageNet, CLIP and SigLIP use different numbers. - Square encoders force a choice between cropping, padding and squashing (163.2); each loses something, and small details vanish at 224 pixels.
- A convolution is a shared sliding dot product (163.3); its output size (163.4), parameters and compute (163.5) can be computed by hand. Halving the map and doubling channels keeps compute per layer constant.
- Receptive fields grow linearly with depth and multiply with stride (163.6); the effective field is smaller. Pooling and GAP (163.7) summarize and give approximate invariance; stride causes aliasing (163.8).
- Residual blocks (163.9) make deep CNNs trainable; ResNet-50 has 25.6 million parameters and turns 224 × 224 into a 7 × 7 × 2048 grid.
- Transfer learning reuses a pretrained backbone with a linear probe (163.10) or fine-tuning. The CNN’s locality bias helps with small data but limits global reasoning, which motivates the ViT.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
- He et al. (2015). Deep Residual Learning for Image Recognition
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 1: Introduction to Vision and Language
- He et al. (2015). Deep Residual Learning for Image Recognition
- LeCun, Bottou, Bengio & Haffner (1998). Gradient-Based Learning Applied to Document Recognition
- Krizhevsky, Sutskever & Hinton (2012). ImageNet Classification with Deep Convolutional Neural Networks
- Simonyan & Zisserman (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition
- Yosinski, Clune, Bengio & Lipson (2014). How transferable are features in deep neural networks?
- Luo, Li, Urtasun & Zemel (2016). Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
- Geirhos et al. (2019). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- Zhang (2019). Making Convolutional Networks Shift-Invariant Again
- Liu et al. (2022). A ConvNet for the 2020s
- Carreira & Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale