Track 7 · sessions 163–200
Vision and VLMs
ViT, DINO, CLIP and SigLIP, early VLMs, connectors and fusion, VLM training and deployment.
Vision foundations for VLMs
ViT, self-supervised vision, CLIP and SigLIP, open-vocabulary detection, visual tokens.
163Ready
Images as data and a CNN refresher
- Pixels, channels, normalization, resizing and aspect ratio
- Convolution as a sliding filter; receptive field; pooling
- From filters to CNN backbones (ResNet); transfer learning
- What CNNs are good at, and what they miss
164Ready
The Vision Transformer (ViT)
- Patchify: cut the image into P×P patches; each patch becomes a token
- Patch embedding: a linear layer, or a convolution with stride P
- CLS token vs mean pooling; 2D position embeddings and interpolation
- Tokens per image: resolution decides the cost
165Ready
Self-supervised vision: MAE, DINO, DINOv2, DINOv3
- MAE: hide most patches, rebuild the pixels
- DINO: a student matches a teacher on different views of the same image
- DINOv2 and DINOv3: strong dense features for detection, segmentation, tracking
- Global vs dense features: which to use for VLMs and for video
166Ready
CLIP: learning images and text together
- Two encoders, one shared space
- Symmetric InfoNCE over a batch of image–text pairs; learned temperature
- Zero-shot classification with prompts (“a photo of a {label}”)
- Limits: counting, spatial relations, small text, web-data bias
167Ready
SigLIP and SigLIP 2
- Sigmoid loss: each image–text pair is its own yes/no question
- No softmax over the whole batch: easier scaling and smaller batches
- SigLIP 2: better localization and dense features, flexible-resolution variants, many languages
- Why many open VLMs use a SigLIP-family encoder
168Ready
Open-vocabulary detection and segmentation
- Grounding: connect words to boxes and masks
- OWL-ViT and Grounding DINO: detect anything you can name
- SAM and SAM 2: promptable segmentation for images and video
- How VLMs output boxes and points as text
169Ready
Vision applications: retrieval, zero-shot classification, near-duplicates
- Image retrieval with embeddings; indexing; re-ranking
- Zero-shot vs few-shot vs linear probe for classification
- Copy detection: global descriptors (SSCD) and robustness to edits
- Metrics: recall@k, mAP, micro-AP
170Ready
zero-shot classification and text-to-image search
- Run CLIP and SigLIP zero-shot on a labeled image set; test prompt templates
- Build text-to-image search with a FAISS index
- Measure recall@k; study the failures (counting, text, small objects)
171Ready
compare vision features with linear probes
- Extract DINOv2, CLIP and SigLIP features on the same dataset
- Train linear probes; compare accuracy and speed
- Test dense patch features on a simple localization task
- Decide which encoder you would put in a VLM, and why
First-generation vision-language models
The ideas before LLM-based VLMs: two-stream and one-stream models, ALBEF and BLIP, grounded detectors, early video–language models.
172Ready
Two-stream and single-stream VLMs: ViLBERT, LXMERT, UNITER
- Image regions from an object detector become visual tokens
- Two streams with co-attention (ViLBERT, LXMERT) vs one joint stream (UNITER)
- Pretraining tasks: masked words with the image, masked regions, image–text matching
- Why they faded: slow detectors, small paired data, no generation
173Ready
ALBEF and BLIP: contrastive, matching and captioning losses together
- Image–text contrastive loss (ITC) aligns the two encoders
- Image–text matching (ITM) with hard negatives checks fine details
- Captioning loss (LM) lets the model write text
- BLIP’s CapFilt: generate new captions and filter noisy web captions
174Ready
Grounded pretraining: GLIP and YOLO-World
- Detection rewritten as phrase grounding: match regions to words
- Region–word alignment scores replace a fixed class head
- YOLO-World: real-time open-vocabulary detection; encode the vocabulary once, then detect
- Where these beat VLMs: speed, boxes, small objects
175Ready
Early video–language pretraining: VideoBERT, VideoCLIP, MERLOT
- VideoBERT: turn video features into visual words, then train BERT on video words + speech text
- VideoCLIP: contrastive learning with overlapping clips and harder negatives
- MERLOT: frames + transcripts; learn what happens and in what order
- Lessons kept by modern Video-LLMs: speech as free labels, time alignment
176Ready
compare CLIP, BLIP and BLIP-2 on retrieval, captioning and VQA
- Run all three on the same small image set
- Retrieval recall@k, caption quality (CIDEr or a judge), VQA accuracy
- Look at where each one fails and why (architecture, losses, data)
VLM architectures
Encoders, connectors and fusion; high resolution; case studies; any-to-any and omni models.
177Ready
Anatomy of a vision-language model
- Three parts: vision encoder → connector → language model
- Two main families: one unified sequence, or cross-attention adapters
- What is pretrained, what is new, and what is trained when
- A tour of the design space
178Ready
Connectors: projector, Q-Former, Perceiver resampler, pixel shuffle
- Linear or MLP projector (LLaVA): simple and strong
- Q-Former (BLIP-2): learned queries pull a fixed number of tokens from the image
- Perceiver resampler (Flamingo): a fixed set of latents via cross-attention
- Pixel shuffle and patch merging: fewer tokens by folding neighbors into channels
179Ready
Fusion: early, late, unified sequence and cross-attention
- Early vs late fusion, with examples
- Unified sequence: image and text tokens in one self-attention stream
- Cross-attention adapters with tanh gates: the LLM can stay frozen and safe
- Trade-offs: tokens in the context vs extra layers
180Ready
High resolution and native aspect ratio
- Why 224×224 is not enough for documents, charts and small objects
- Tiling (AnyRes): a thumbnail plus high-resolution crops
- Dynamic resolution (Qwen2-VL): any size, 2×2 patches merged; NaViT-style packing
- M-RoPE: separate position parts for time, height and width
181Ready
Case studies 1: LLaVA, BLIP-2, Flamingo, IDEFICS
- LLaVA: CLIP + MLP + LLM, trained on GPT-generated instruction data
- BLIP-2: frozen encoder and frozen LLM; train only the Q-Former
- Flamingo: interleaved images and text; few-shot multimodal learning
- IDEFICS 2: what matters when building a VLM (ablations)
182Ready
Case studies 2: Qwen2.5-VL, InternVL, PaliGemma and Gemma, SmolVLM
- Qwen2.5-VL: native-resolution ViT, window attention, absolute time for video
- InternVL: a large vision encoder and dynamic tiling
- PaliGemma and Gemma vision: SigLIP + Gemma, prefix-LM attention
- SmolVLM: design choices for small and edge models
183Ready
Visual token efficiency
- Visual tokens often far outnumber text tokens, so they dominate cost
- Token merging (ToMe) and pruning by attention (FastV)
- Fixed-budget resamplers vs dynamic token counts
- Measuring the accuracy vs cost trade-off honestly
184Ready
Audio and omni models
- Speech encoders (Whisper-style) and audio tokens
- Aligning audio and video in time
- Omni models: text, image, audio and video in; text and speech out
- Applications: speaker detection, audio events, commentary understanding
185Ready
build a mini-VLM
- Frozen SigLIP + MLP projector + small LLM
- Stage 1: train only the projector on image–caption pairs
- Stage 2: instruction-tune with LoRA on a small VQA set
- Look at attention from text tokens to image tokens
VLM training, data and post-training
Training stages, multimodal data, batching, SFT, preference tuning, tasks, evaluation and failure modes.
186Ready
VLM training stages
- Stage 1: alignment (projector only)
- Stage 2: multimodal pretraining (unfreeze more)
- Stage 3: instruction tuning; optional stage 4: preference tuning or RL
- What to freeze, when, and why
187Ready
Multimodal data at scale
- Image–text pairs (LAION, DataComp), interleaved web documents (OBELICS), video–text
- Filtering with CLIP score; deduplication; safety filtering
- Synthetic and re-captioned data (dense captions)
- The data mixture as a hidden hyperparameter; designing ablations
188Ready
Batching and masking for multimodal sequences
- A different number of image tokens per sample
- Packing samples, with attention masks so samples do not see each other
- Loss only on the text answer, not on image tokens
- Throughput: sort by length, balance tokens per GPU
189Ready
VLM SFT and PEFT
- Freeze or unfreeze the vision encoder; different learning rates per part
- LoRA on the LLM, the connector, or the vision tower
- QLoRA for VLMs
- Keeping text-only skills while learning vision tasks
190Ready
VLM preference optimization and RL
- DPO and MPO for VLMs; preference data from humans or AI
- Fine-grained corrections against hallucination (RLHF-V style)
- GRPO for visual reasoning with checkable answers (counting, charts, geometry)
- Reward design for vision tasks: IoU, exact match, format
191Ready
VLM tasks and how models answer them
- Captioning, VQA, visual reasoning
- Grounding and localization: boxes and points as text
- Counting, OCR, charts, documents
- Prompt formats for each task
192Ready
Evaluating VLMs
- VQA accuracy, CIDEr for captions, ANLS for documents
- MMMU, DocVQA, POPE (hallucination), MVBench and Video-MME
- A domain test set sampled from your own data, with clear labels
- Judge-based scoring for open answers, and its limits
193Ready
VLM failure modes
- Object hallucination: the language prior beats the image
- Counting and spatial relations
- Small text and fine details lost by resizing
- Temporal confusion in video; causes and fixes for each
194Ready
LoRA fine-tune a small VLM
- Pick a small open VLM and a domain VQA dataset
- Fine-tune with LoRA; compare frozen vs unfrozen vision tower
- Evaluate before and after on held-out data; check text-only regressions
195Ready
preference tuning a VLM against hallucination
- Create chosen and rejected answers (correct vs hallucinated objects)
- Train with DPO; tune β
- Measure with a POPE-style yes/no test
VLM inference and deployment
Make VLMs fast: the inference path, quantization, serving, and a latency budget on an L4.
196Ready
The VLM inference path
- Preprocessing on CPU or GPU: image decode and resize
- Vision encoder cost and the long prefill of image tokens
- Image tokens in the KV cache; encoder caching and prefix caching for repeated images
- Where the time goes: a breakdown
197Ready
Quantizing VLMs
- Why decode is bandwidth-bound and weight-only quantization helps
- The VLM asymmetry: the vision encoder and connector are often more sensitive than the LLM
- Outliers; calibration data must include images
- Test quality after quantization on vision tasks, not only text
198Ready
Serving VLMs: vLLM, TensorRT-LLM and the edge
- Multimodal serving in vLLM and TensorRT-LLM; limits per request (images, frames)
- Separating the encoder from prefill and decode
- Edge: ONNX, transformers.js, llama.cpp with a vision projector, MLX
- PEFT adapters on edge devices
199Ready
A latency budget on an NVIDIA L4: worked example
- Inputs: image size, frames, visual tokens, prompt and output length
- Estimate encoder time, prefill time (compute-bound) and decode time (bandwidth-bound)
- Choose resolution, frames, quantization and batch size to meet the budget
- Check the estimate with a real measurement
200Ready
serve a VLM and measure
- Serve a small VLM with vLLM
- Measure TTFT vs image resolution and number of video frames
- Try FP8 or INT4 and prefix caching; record quality and speed