LLM & VLM courseBlog
Track 8 · sessions 201–221

Video and multimodal systems

Video encoders, action recognition, segmentation, key moments, retrieval, fingerprinting, Video-RAG, documents.

Video understanding

Video encoders, Video-LLMs, action recognition, key moments, retrieval, fingerprinting and Video-RAG.

201

Video as data

  • Frames, fps, resolution; codecs and GOPs (I, P, B frames)
  • Decoding cost is often the hidden bottleneck; GPU decode (NVDEC)
  • Sampling: uniform, keyframes, scene-based, adaptive
  • Tokens per second of video: the budget problem
Ready
202

Video encoders: from 3D CNNs to video transformers

  • I3D (inflated 2D filters) and SlowFast (two speeds); axial and factorized attention
  • ViViT and TimeSformer: divided space–time attention
  • VideoMAE: masked video pretraining with very high masking ratios
  • Video–text encoders such as InternVideo2
Ready
203

Action recognition

  • Clip-level vs video-level classification; datasets such as Kinetics
  • Temporal action localization: start, end and class
  • Zero-shot action recognition with video–text models
  • Sports actions are short, fast and fine-grained: what makes them hard
Ready
204

Video object segmentation and tracking: from VOS to SAM 2

  • Semi-supervised video object segmentation: mask in frame 1, follow the object
  • Memory-based matching across frames (XMem)
  • SAM 2: promptable segmentation with a streaming memory; points, boxes and masks as prompts
  • Referring segmentation from a sentence; metrics J and F
Ready
205

Video-LLMs

  • Frame sampling + per-frame encoder + temporal merging + LLM
  • Time encoding: frame index, timestamps as text, absolute time in M-RoPE
  • Video instruction data; joint image and video training
  • Case studies: Video-LLaVA, LLaVA-Video, Qwen2.5-VL video
Ready
206

Long video

  • Token compression across frames: merge similar tokens over time
  • Memory banks and streaming video LLMs
  • Agentic search: the model chooses which segments to look at
  • Captions as a compressed memory of the video
Ready
207

Temporal grounding and key-moment detection

  • Moment retrieval: given text, return the start and end time
  • Highlight detection: score each clip by importance
  • DETR-style detectors (Moment-DETR) vs LLMs that write timestamps (TimeChat)
  • Metrics: R1@IoU and mAP; label noise in “key moments”
Ready
208

Text-to-video retrieval

  • Frame embeddings + pooling (CLIP4Clip): mean, sequential, attention
  • Clip-level vs segment-level indexes; choosing the segment length
  • Video–text encoders (ViCLIP, InternVideo2)
  • Scaling to millions of segments; re-ranking with a VLM
Ready
209

Video fingerprinting and near-duplicate detection

  • Goal: the same content after crop, resize, re-encode, logo or speed change; not the same meaning
  • Perceptual hashes vs learned per-frame descriptors (SSCD-style)
  • Temporal alignment of matches (similarity matrices, ViSiL-style)
  • Evaluation: precision–recall, micro-AP, robustness tests
Ready
210

Video question answering and Video-RAG

  • Retrieve then read: find the right segments, then let a VLM answer
  • Captions, ASR transcripts and OCR as searchable memory
  • Indexes at several levels: shot, scene, whole game
  • When to use Video-RAG vs a long-context video model
Ready
211

Production video understanding pipelines

  • Shot and scene boundary detection
  • Captioning and classification per shot or scene, for example IAB and brand-safety labels
  • Active speaker detection with audio–visual fusion
  • Streaming vs batch pipelines; latency, cost and monitoring
Ready
212

Training video-language models

  • Video–text data: ASR-aligned, captioned, synthetic
  • Temporal annotations and dense captions
  • Domain fine-tuning (for example sports) with LoRA
  • Efficiency: fewer frames, lower resolution, token merging, packing
Ready
213

Evaluating video understanding

  • Multiple-choice video QA (Video-MME, MVBench)
  • Grounding (R1@IoU) and retrieval (R@k) metrics
  • Long-video pitfalls: answers guessed from text alone
  • Building a domain video test set
Ready
214

text-to-video retrieval

  • Cut videos into segments; embed frames with SigLIP; pool per segment
  • Index with FAISS and search with text queries
  • Measure recall@k; compare mean pooling with a small temporal model
Ready
215

key-moment detection

  • Score each segment with (a) similarity to event prompts and (b) a VLM yes/no question
  • Combine the scores; pick top moments with non-maximum suppression in time
  • Evaluate against labeled moments with tIoU-based metrics
Ready
216

video fingerprinting

  • Build per-frame descriptors and a temporal matcher
  • Attack your own system: crop, resize, re-encode, speed up, add a logo
  • Plot precision–recall and find the weakest transform
Ready
217

segment and track players with SAM 2

  • Prompt SAM 2 with boxes from a detector on a sports clip
  • Track through occlusions and camera cuts; find where it fails
  • Measure J&F on a few hand-labeled frames
Ready

Documents, multimodal RAG and agents

Document AI, multimodal RAG, agents that see and act, and the generation side.

218

Document AI

  • OCR pipelines vs OCR-free models (Donut, VLMs)
  • Layout, tables, charts; parsing documents to markdown
  • DocVQA and information extraction
  • Page-image retrieval with multi-vector embeddings (ColPali)
Ready
219

Multimodal RAG

  • Indexes for text, images, video segments and document pages
  • Cross-modal retrieval; late interaction vs single vectors
  • Feeding retrieved images and frames to a VLM; token budgets
  • Evaluation across modalities
Ready
220

Agentic VLMs and vision-language-action models

  • Computer-use agents: read screenshots, click, type
  • Tool-using VLMs: crop, zoom, detect, search
  • VLA models (RT-2, OpenVLA, π0): actions as tokens or continuous outputs
  • Evaluation and safety for agents that act
Ready
221

multimodal RAG over PDFs

  • Index PDF pages as images (ColPali-style) and as text
  • Answer questions with a VLM using the retrieved pages
  • Compare text-only RAG and page-image RAG on tables and charts
Ready