LLM & VLM courseBlog
Track 7 · sessions 163–200

Vision and VLMs

ViT, DINO, CLIP and SigLIP, early VLMs, connectors and fusion, VLM training and deployment.

Vision foundations for VLMs

ViT, self-supervised vision, CLIP and SigLIP, open-vocabulary detection, visual tokens.

163

Images as data and a CNN refresher

  • Pixels, channels, normalization, resizing and aspect ratio
  • Convolution as a sliding filter; receptive field; pooling
  • From filters to CNN backbones (ResNet); transfer learning
  • What CNNs are good at, and what they miss
Ready
164

The Vision Transformer (ViT)

  • Patchify: cut the image into P×P patches; each patch becomes a token
  • Patch embedding: a linear layer, or a convolution with stride P
  • CLS token vs mean pooling; 2D position embeddings and interpolation
  • Tokens per image: resolution decides the cost
Ready
165

Self-supervised vision: MAE, DINO, DINOv2, DINOv3

  • MAE: hide most patches, rebuild the pixels
  • DINO: a student matches a teacher on different views of the same image
  • DINOv2 and DINOv3: strong dense features for detection, segmentation, tracking
  • Global vs dense features: which to use for VLMs and for video
Ready
166

CLIP: learning images and text together

  • Two encoders, one shared space
  • Symmetric InfoNCE over a batch of image–text pairs; learned temperature
  • Zero-shot classification with prompts (“a photo of a {label}”)
  • Limits: counting, spatial relations, small text, web-data bias
Ready
167

SigLIP and SigLIP 2

  • Sigmoid loss: each image–text pair is its own yes/no question
  • No softmax over the whole batch: easier scaling and smaller batches
  • SigLIP 2: better localization and dense features, flexible-resolution variants, many languages
  • Why many open VLMs use a SigLIP-family encoder
Ready
168

Open-vocabulary detection and segmentation

  • Grounding: connect words to boxes and masks
  • OWL-ViT and Grounding DINO: detect anything you can name
  • SAM and SAM 2: promptable segmentation for images and video
  • How VLMs output boxes and points as text
Ready
169

Vision applications: retrieval, zero-shot classification, near-duplicates

  • Image retrieval with embeddings; indexing; re-ranking
  • Zero-shot vs few-shot vs linear probe for classification
  • Copy detection: global descriptors (SSCD) and robustness to edits
  • Metrics: recall@k, mAP, micro-AP
Ready
170

zero-shot classification and text-to-image search

  • Run CLIP and SigLIP zero-shot on a labeled image set; test prompt templates
  • Build text-to-image search with a FAISS index
  • Measure recall@k; study the failures (counting, text, small objects)
Ready
171

compare vision features with linear probes

  • Extract DINOv2, CLIP and SigLIP features on the same dataset
  • Train linear probes; compare accuracy and speed
  • Test dense patch features on a simple localization task
  • Decide which encoder you would put in a VLM, and why
Ready

First-generation vision-language models

The ideas before LLM-based VLMs: two-stream and one-stream models, ALBEF and BLIP, grounded detectors, early video–language models.

172

Two-stream and single-stream VLMs: ViLBERT, LXMERT, UNITER

  • Image regions from an object detector become visual tokens
  • Two streams with co-attention (ViLBERT, LXMERT) vs one joint stream (UNITER)
  • Pretraining tasks: masked words with the image, masked regions, image–text matching
  • Why they faded: slow detectors, small paired data, no generation
Ready
173

ALBEF and BLIP: contrastive, matching and captioning losses together

  • Image–text contrastive loss (ITC) aligns the two encoders
  • Image–text matching (ITM) with hard negatives checks fine details
  • Captioning loss (LM) lets the model write text
  • BLIP’s CapFilt: generate new captions and filter noisy web captions
Ready
174

Grounded pretraining: GLIP and YOLO-World

  • Detection rewritten as phrase grounding: match regions to words
  • Region–word alignment scores replace a fixed class head
  • YOLO-World: real-time open-vocabulary detection; encode the vocabulary once, then detect
  • Where these beat VLMs: speed, boxes, small objects
Ready
175

Early video–language pretraining: VideoBERT, VideoCLIP, MERLOT

  • VideoBERT: turn video features into visual words, then train BERT on video words + speech text
  • VideoCLIP: contrastive learning with overlapping clips and harder negatives
  • MERLOT: frames + transcripts; learn what happens and in what order
  • Lessons kept by modern Video-LLMs: speech as free labels, time alignment
Ready
176

compare CLIP, BLIP and BLIP-2 on retrieval, captioning and VQA

  • Run all three on the same small image set
  • Retrieval recall@k, caption quality (CIDEr or a judge), VQA accuracy
  • Look at where each one fails and why (architecture, losses, data)
Ready

VLM architectures

Encoders, connectors and fusion; high resolution; case studies; any-to-any and omni models.

177

Anatomy of a vision-language model

  • Three parts: vision encoder → connector → language model
  • Two main families: one unified sequence, or cross-attention adapters
  • What is pretrained, what is new, and what is trained when
  • A tour of the design space
Ready
178

Connectors: projector, Q-Former, Perceiver resampler, pixel shuffle

  • Linear or MLP projector (LLaVA): simple and strong
  • Q-Former (BLIP-2): learned queries pull a fixed number of tokens from the image
  • Perceiver resampler (Flamingo): a fixed set of latents via cross-attention
  • Pixel shuffle and patch merging: fewer tokens by folding neighbors into channels
Ready
179

Fusion: early, late, unified sequence and cross-attention

  • Early vs late fusion, with examples
  • Unified sequence: image and text tokens in one self-attention stream
  • Cross-attention adapters with tanh gates: the LLM can stay frozen and safe
  • Trade-offs: tokens in the context vs extra layers
Ready
180

High resolution and native aspect ratio

  • Why 224×224 is not enough for documents, charts and small objects
  • Tiling (AnyRes): a thumbnail plus high-resolution crops
  • Dynamic resolution (Qwen2-VL): any size, 2×2 patches merged; NaViT-style packing
  • M-RoPE: separate position parts for time, height and width
Ready
181

Case studies 1: LLaVA, BLIP-2, Flamingo, IDEFICS

  • LLaVA: CLIP + MLP + LLM, trained on GPT-generated instruction data
  • BLIP-2: frozen encoder and frozen LLM; train only the Q-Former
  • Flamingo: interleaved images and text; few-shot multimodal learning
  • IDEFICS 2: what matters when building a VLM (ablations)
Ready
182

Case studies 2: Qwen2.5-VL, InternVL, PaliGemma and Gemma, SmolVLM

  • Qwen2.5-VL: native-resolution ViT, window attention, absolute time for video
  • InternVL: a large vision encoder and dynamic tiling
  • PaliGemma and Gemma vision: SigLIP + Gemma, prefix-LM attention
  • SmolVLM: design choices for small and edge models
Ready
183

Visual token efficiency

  • Visual tokens often far outnumber text tokens, so they dominate cost
  • Token merging (ToMe) and pruning by attention (FastV)
  • Fixed-budget resamplers vs dynamic token counts
  • Measuring the accuracy vs cost trade-off honestly
Ready
184

Audio and omni models

  • Speech encoders (Whisper-style) and audio tokens
  • Aligning audio and video in time
  • Omni models: text, image, audio and video in; text and speech out
  • Applications: speaker detection, audio events, commentary understanding
Ready
185

build a mini-VLM

  • Frozen SigLIP + MLP projector + small LLM
  • Stage 1: train only the projector on image–caption pairs
  • Stage 2: instruction-tune with LoRA on a small VQA set
  • Look at attention from text tokens to image tokens
Ready

VLM training, data and post-training

Training stages, multimodal data, batching, SFT, preference tuning, tasks, evaluation and failure modes.

186

VLM training stages

  • Stage 1: alignment (projector only)
  • Stage 2: multimodal pretraining (unfreeze more)
  • Stage 3: instruction tuning; optional stage 4: preference tuning or RL
  • What to freeze, when, and why
Ready
187

Multimodal data at scale

  • Image–text pairs (LAION, DataComp), interleaved web documents (OBELICS), video–text
  • Filtering with CLIP score; deduplication; safety filtering
  • Synthetic and re-captioned data (dense captions)
  • The data mixture as a hidden hyperparameter; designing ablations
Ready
188

Batching and masking for multimodal sequences

  • A different number of image tokens per sample
  • Packing samples, with attention masks so samples do not see each other
  • Loss only on the text answer, not on image tokens
  • Throughput: sort by length, balance tokens per GPU
Ready
189

VLM SFT and PEFT

  • Freeze or unfreeze the vision encoder; different learning rates per part
  • LoRA on the LLM, the connector, or the vision tower
  • QLoRA for VLMs
  • Keeping text-only skills while learning vision tasks
Ready
190

VLM preference optimization and RL

  • DPO and MPO for VLMs; preference data from humans or AI
  • Fine-grained corrections against hallucination (RLHF-V style)
  • GRPO for visual reasoning with checkable answers (counting, charts, geometry)
  • Reward design for vision tasks: IoU, exact match, format
Ready
191

VLM tasks and how models answer them

  • Captioning, VQA, visual reasoning
  • Grounding and localization: boxes and points as text
  • Counting, OCR, charts, documents
  • Prompt formats for each task
Ready
192

Evaluating VLMs

  • VQA accuracy, CIDEr for captions, ANLS for documents
  • MMMU, DocVQA, POPE (hallucination), MVBench and Video-MME
  • A domain test set sampled from your own data, with clear labels
  • Judge-based scoring for open answers, and its limits
Ready
193

VLM failure modes

  • Object hallucination: the language prior beats the image
  • Counting and spatial relations
  • Small text and fine details lost by resizing
  • Temporal confusion in video; causes and fixes for each
Ready
194

LoRA fine-tune a small VLM

  • Pick a small open VLM and a domain VQA dataset
  • Fine-tune with LoRA; compare frozen vs unfrozen vision tower
  • Evaluate before and after on held-out data; check text-only regressions
Ready
195

preference tuning a VLM against hallucination

  • Create chosen and rejected answers (correct vs hallucinated objects)
  • Train with DPO; tune β
  • Measure with a POPE-style yes/no test
Ready

VLM inference and deployment

Make VLMs fast: the inference path, quantization, serving, and a latency budget on an L4.

196

The VLM inference path

  • Preprocessing on CPU or GPU: image decode and resize
  • Vision encoder cost and the long prefill of image tokens
  • Image tokens in the KV cache; encoder caching and prefix caching for repeated images
  • Where the time goes: a breakdown
Ready
197

Quantizing VLMs

  • Why decode is bandwidth-bound and weight-only quantization helps
  • The VLM asymmetry: the vision encoder and connector are often more sensitive than the LLM
  • Outliers; calibration data must include images
  • Test quality after quantization on vision tasks, not only text
Ready
198

Serving VLMs: vLLM, TensorRT-LLM and the edge

  • Multimodal serving in vLLM and TensorRT-LLM; limits per request (images, frames)
  • Separating the encoder from prefill and decode
  • Edge: ONNX, transformers.js, llama.cpp with a vision projector, MLX
  • PEFT adapters on edge devices
Ready
199

A latency budget on an NVIDIA L4: worked example

  • Inputs: image size, frames, visual tokens, prompt and output length
  • Estimate encoder time, prefill time (compute-bound) and decode time (bandwidth-bound)
  • Choose resolution, frames, quantization and batch size to meet the budget
  • Check the estimate with a real measurement
Ready
200

serve a VLM and measure

  • Serve a small VLM with vLLM
  • Measure TTFT vs image resolution and number of video frames
  • Try FP8 or INT4 and prefix caching; record quality and speed
Ready