Track 8 · sessions 201–221
Video and multimodal systems
Video encoders, action recognition, segmentation, key moments, retrieval, fingerprinting, Video-RAG, documents.
Video understanding
Video encoders, Video-LLMs, action recognition, key moments, retrieval, fingerprinting and Video-RAG.
201Ready
Video as data
- Frames, fps, resolution; codecs and GOPs (I, P, B frames)
- Decoding cost is often the hidden bottleneck; GPU decode (NVDEC)
- Sampling: uniform, keyframes, scene-based, adaptive
- Tokens per second of video: the budget problem
202Ready
Video encoders: from 3D CNNs to video transformers
- I3D (inflated 2D filters) and SlowFast (two speeds); axial and factorized attention
- ViViT and TimeSformer: divided space–time attention
- VideoMAE: masked video pretraining with very high masking ratios
- Video–text encoders such as InternVideo2
203Ready
Action recognition
- Clip-level vs video-level classification; datasets such as Kinetics
- Temporal action localization: start, end and class
- Zero-shot action recognition with video–text models
- Sports actions are short, fast and fine-grained: what makes them hard
204Ready
Video object segmentation and tracking: from VOS to SAM 2
- Semi-supervised video object segmentation: mask in frame 1, follow the object
- Memory-based matching across frames (XMem)
- SAM 2: promptable segmentation with a streaming memory; points, boxes and masks as prompts
- Referring segmentation from a sentence; metrics J and F
205Ready
Video-LLMs
- Frame sampling + per-frame encoder + temporal merging + LLM
- Time encoding: frame index, timestamps as text, absolute time in M-RoPE
- Video instruction data; joint image and video training
- Case studies: Video-LLaVA, LLaVA-Video, Qwen2.5-VL video
206Ready
Long video
- Token compression across frames: merge similar tokens over time
- Memory banks and streaming video LLMs
- Agentic search: the model chooses which segments to look at
- Captions as a compressed memory of the video
207Ready
Temporal grounding and key-moment detection
- Moment retrieval: given text, return the start and end time
- Highlight detection: score each clip by importance
- DETR-style detectors (Moment-DETR) vs LLMs that write timestamps (TimeChat)
- Metrics: R1@IoU and mAP; label noise in “key moments”
208Ready
Text-to-video retrieval
- Frame embeddings + pooling (CLIP4Clip): mean, sequential, attention
- Clip-level vs segment-level indexes; choosing the segment length
- Video–text encoders (ViCLIP, InternVideo2)
- Scaling to millions of segments; re-ranking with a VLM
209Ready
Video fingerprinting and near-duplicate detection
- Goal: the same content after crop, resize, re-encode, logo or speed change; not the same meaning
- Perceptual hashes vs learned per-frame descriptors (SSCD-style)
- Temporal alignment of matches (similarity matrices, ViSiL-style)
- Evaluation: precision–recall, micro-AP, robustness tests
210Ready
Video question answering and Video-RAG
- Retrieve then read: find the right segments, then let a VLM answer
- Captions, ASR transcripts and OCR as searchable memory
- Indexes at several levels: shot, scene, whole game
- When to use Video-RAG vs a long-context video model
211Ready
Production video understanding pipelines
- Shot and scene boundary detection
- Captioning and classification per shot or scene, for example IAB and brand-safety labels
- Active speaker detection with audio–visual fusion
- Streaming vs batch pipelines; latency, cost and monitoring
212Ready
Training video-language models
- Video–text data: ASR-aligned, captioned, synthetic
- Temporal annotations and dense captions
- Domain fine-tuning (for example sports) with LoRA
- Efficiency: fewer frames, lower resolution, token merging, packing
213Ready
Evaluating video understanding
- Multiple-choice video QA (Video-MME, MVBench)
- Grounding (R1@IoU) and retrieval (R@k) metrics
- Long-video pitfalls: answers guessed from text alone
- Building a domain video test set
214Ready
text-to-video retrieval
- Cut videos into segments; embed frames with SigLIP; pool per segment
- Index with FAISS and search with text queries
- Measure recall@k; compare mean pooling with a small temporal model
215Ready
key-moment detection
- Score each segment with (a) similarity to event prompts and (b) a VLM yes/no question
- Combine the scores; pick top moments with non-maximum suppression in time
- Evaluate against labeled moments with tIoU-based metrics
216Ready
video fingerprinting
- Build per-frame descriptors and a temporal matcher
- Attack your own system: crop, resize, re-encode, speed up, add a logo
- Plot precision–recall and find the weakest transform
217Ready
segment and track players with SAM 2
- Prompt SAM 2 with boxes from a detector on a sports clip
- Track through occlusions and camera cuts; find where it fails
- Measure J&F on a few hand-labeled frames
Documents, multimodal RAG and agents
Document AI, multimodal RAG, agents that see and act, and the generation side.
218Ready
Document AI
- OCR pipelines vs OCR-free models (Donut, VLMs)
- Layout, tables, charts; parsing documents to markdown
- DocVQA and information extraction
- Page-image retrieval with multi-vector embeddings (ColPali)
219Ready
Multimodal RAG
- Indexes for text, images, video segments and document pages
- Cross-modal retrieval; late interaction vs single vectors
- Feeding retrieved images and frames to a VLM; token budgets
- Evaluation across modalities
220Ready
Agentic VLMs and vision-language-action models
- Computer-use agents: read screenshots, click, type
- Tool-using VLMs: crop, zoom, detect, search
- VLA models (RT-2, OpenVLA, π0): actions as tokens or continuous outputs
- Evaluation and safety for agents that act
221Ready
multimodal RAG over PDFs
- Index PDF pages as images (ColPali-style) and as text
- Answer questions with a VLM using the retrieved pages
- Compare text-only RAG and page-image RAG on tables and charts