Production video understanding pipelines
Real video systems are pipelines: decode once, cut the video into shots, describe and label each shot, find who is speaking, and sum the results up for scenes and whole videos. This session builds such a pipeline stage by stage, with TransNet V2 for shot boundaries and TalkNet for active speakers, and then compares streaming and batch designs by latency, cost and monitoring.
What you will learn
- The stages of a production video understanding pipeline and why the shot is the unit of work
- How shot boundaries are found with histogram differences, why dissolves, flashes and fast motion break simple rules, and how TransNet V2 learns them
- How to group shots into scenes, and how to caption and classify shots with IAB topic and brand-safety labels
- Why topics should be aggregated by time and safety labels by max, with the numbers worked out
- How active speaker detection fuses audio and video through sync, and what TalkNet adds with cross-attention and long context
- How to size a streaming or batch pipeline: GPU count, latency parts, queueing at high utilization
- What to monitor: lag, per-stage latency, label drift with PSI, and human-reviewed samples
Why this matters
A video platform, a broadcaster or an ad company does not run one model on one clip. It runs a pipeline: many models, one after another, on thousands of hours of video every day. The pipeline cuts each video into pieces, describes each piece, labels it, finds who is talking, and stores the results so that search, ads and safety teams can use them. This session is about how such a pipeline is built, what each stage does, and how to keep it fast, cheap and correct.
In the earlier sessions of this track you met the models one by one: video encoders, Video-LLMs, temporal grounding, retrieval, fingerprinting, and in session 210 a system that answers questions over a video archive. Video-RAG needs an index of the archive. Production pipelines build that index. They also feed many other products:
- Contextual advertising. An ad server wants to know “this part of the video is about cars” so it can show a car ad, and “this part shows an accident” so it does not show a car ad there. Labels from a shared list, the IAB Content Taxonomy, are the common language between publishers and advertisers.
- Brand safety and moderation. Find violence, adult content or hate speech, so that ads are not placed next to it and so that human reviewers see the risky parts first.
- Search and recommendation. Captions, transcripts and embeddings per shot make an archive searchable by text (session 208).
- Editing tools. Shot lists, chapter marks, speaker labels, and automatic reframing of a wide video into a vertical one that follows the person who speaks.
The key ideas are in four groups, and the page follows them. (1) Shot and scene boundary detection: where does the camera cut, and which shots belong to one scene? TransNet V2 (Souček and Lokoč, 2020) is the standard open model. (2) Captioning and classification per shot or scene, and how to turn many shot labels into one label for a video. (3) Active speaker detection: which visible face is speaking right now? TalkNet (Tao et al., 2021) fuses audio and video for this. (4) Streaming vs batch: the same models, run as a live system or as an offline job, with very different latency, cost and monitoring. The book’s chapter on video (Noyan et al., chapter 9) gives the model side; this page is about the system around the models.
The pipeline, end to end
Figure 1 shows a typical design. Read it from left to right. A video enters, is decoded once, and is cut into shots. Most of the expensive work is done once per shot, not once per frame, because frames inside one shot look almost the same. Results are joined per shot, grouped into scenes, and summed up for the whole video.
Three design rules come from this picture, and we will meet them again:
- Decode once. Decoding compressed video is often a big part of the cost (session 201). Decode each video once, at the lowest resolution and frame rate that each stage needs, and share the frames.
- The shot is the unit of work. A caption per shot, not per frame, cuts the number of expensive VLM calls by a large factor. If shots last a few seconds on average, as in much edited TV and film [Likely], an hour of video has on the order of \(3600/4 = 900\) shots.
- Store every intermediate result. Shot lists, captions, transcripts and embeddings are stored with the model version that made them. Then a new label set needs only a cheap re-run of the last stage, not of the whole pipeline.
Shot boundary detection
A shot is a continuous piece of video from one camera take. A shot transition (or shot boundary) is where one shot ends and the next begins. There are two families of transitions:
- Hard cuts: the last frame of shot A is followed directly by the first frame of shot B. Most transitions in edited video are cuts.
- Gradual transitions: a dissolve (A fades into B over many frames), a fade to or from black, a wipe (B slides over A). They can last from a few frames to a second or more.
A scene is a larger unit: a group of shots that share a place, a time and a story, for example a dialogue filmed as A, B, A, B close-ups. Scenes are harder to define and to detect; we come back to them below.
The classic method: compare color histograms
The simplest detector compares each frame with an earlier frame. A good comparison is a color histogram: count how many pixels fall into each color bin, then divide by the number of pixels, so the bins sum to 1. A histogram ignores where colors are, so small camera or object motion changes it only a little, while a cut to a new place changes it a lot. The difference between frame \(t\) and frame \(t-k\) is
- Read the symbols
- \(h_t(b)\)
- the fraction of pixels of frame \(t\) that fall into color bin \(b\); the bins sum to 1
- \(B\)
- number of bins, for example 8 per color channel
- \(k\)
- the frame gap: \(k = 1\) compares neighbors, a larger \(k\) compares frames further apart
- \(d_t^{(k)}\)
- a number between 0 (same histogram) and 1 (no color in common); the factor \(\tfrac12\) makes the maximum 1
The detector reports a boundary where the difference is above a threshold \(\theta\). Neighboring frames above the threshold form one run, and each run is one detection, placed at the middle of the run:
The shift \(-(k-1)/2\) is needed because, with a gap \(k\), one cut makes a run of \(k\) frames: every frame in the first \(k\) frames of shot B is compared with a frame of shot A. Open-source tools work this way. For example, PySceneDetect’s ContentDetector compares neighboring frames in HSV color space with a default threshold of 27 (on its own scale), and its AdaptiveDetector compares each score with a rolling average of nearby scores, to reduce false cuts in fast motion. [Certain]
Why simple rules break
Each kind of hard video breaks a different rule:
- Dissolves spread the change over many frames. With \(k = 1\), each step is small, so a dissolve hides below any threshold that ignores noise.
- Flashes (a camera flash, an explosion, lightning) change one or two frames a lot and then go back. They look like two cuts.
- Fast motion (a camera pan, sport) changes the histogram steadily. With a large \(k\), the change over \(k\) frames looks like a cut.
Figure 2 lets you try three hand-made detectors on a synthetic 300-frame video with five true transitions (four cuts and one 20-frame dissolve), one flash and one fast camera pan.
How to read Figure 2
Start with “Neighbors, \(k = 1\)” and move \(\theta\). Below about 0.12 the camera pan makes many false cuts. From 0.14 to 0.48 you get the four cuts plus two false detections from the flash, and the dissolve is always missed: \(F_1 = 0.727\). Switch to “Gap \(k = 8\)”: at \(\theta = 0.30\) the dissolve is found, but it breaks into three runs, the flash still gives two false detections, and lower thresholds let the pan create many more (\(F_1 = 0.714\) at 0.30). The third detector subtracts the local median score (adaptive) and drops a detection if the frames 3 before and 3 after it look the same (flash check): it is clean, \(F_1 = 0.889\) for \(\theta\) from 0.06 to 0.42, but it still misses the dissolve. No single rule handles every trap.
This is the honest summary of decades of hand-made detectors: each fix helps one case and hurts another, and thresholds must be tuned per type of content. This is why learned detectors replaced them.
TransNet V2: a learned shot detector
TransNet V2 is a small 3D convolutional network that looks at many frames at once and outputs, for every frame, the probability that it is the middle of a transition. Its main design choices are [Certain]:
- Tiny frames. Frames are resized to \(48 \times 27\) pixels. Shot changes are visible at this size, and it makes the network very cheap.
- A window of 100 frames. The network sees 100 frames and predicts all 100, but at test time only the middle 50 predictions are used, so each prediction has at least 25 frames of context on each side.
- Dilated, factorized 3D convolutions (DDCNN cells). Each cell runs convolutions with temporal dilation 1, 2, 4 and 8 in parallel, so it sees short and long changes. V2 splits each 3D convolution into a 2D spatial one and a 1D temporal one, and adds batch normalization.
- Hand-made features inside the network. RGB color histograms (\(8^3 = 512\) bins) and learned features of each frame are compared by cosine similarity with the 50 frames before and the 50 frames after it. So the old idea of Figure 2 is kept, but as an input to a learned model.
- Two heads. A “single-frame” head predicts the middle frame of each transition (used at test time). An “all-frame” head predicts every frame that belongs to a transition (used only in training, as extra signal).
- Synthetic training data. Transitions are made by joining real shots from TRECVID IACC.3 and ClipShots: the training mix is 35% hard cuts, 50% dissolves (2 to 30 frames long), and 15% real transitions from ClipShots.
The training loss is a weighted cross-entropy on both heads plus weight decay:
- Read the symbols
- \(y^{\text{mid}}, \hat{y}^{\text{mid}}\)
- true and predicted labels of the single-frame head: 1 at the middle frame of a transition, 0 elsewhere
- \(\mathrm{CE}_{w=5}\)
- binary cross-entropy where positive frames count 5 times, because transitions are rare
- \(y^{\text{all}}, \hat{y}^{\text{all}}\)
- labels of the all-frame head: 1 for every frame inside a transition
- \(\mathbf{w}\)
- all network weights; the last term is L2 regularization
At test time a frame is a boundary when its probability is above 0.5. The paper trains with SGD (momentum 0.9, learning rate 0.01) for 50 epochs of 750 batches of 16, about 17 hours on one V100 GPU. [Certain]
Shot detection is scored with F1, with a small tolerance: a detection counts as correct if it is within 2 frames of a true transition. With \(\mathrm{TP}\) correct detections, \(\mathrm{FP}\) false ones and \(\mathrm{FN}\) missed transitions,
The paper reports F1 of 77.9 on ClipShots, 96.2 on BBC Planet Earth and 93.9 on RAI. The first TransNet had 73.5, 92.9 and 94.3, so V2 is better on two of the three sets. ClipShots is the hardest: it is made of short user videos with many gradual transitions. [Certain]
\(k = 1\), \(\theta = 0.20\). Six detections at frames 50, 80, 82, 110, 220, 280. The flash makes two false ones (80 and 82), and the dissolve is missed. So \(\mathrm{TP} = 4\), \(\mathrm{FP} = 2\), \(\mathrm{FN} = 1\). \(P = 4/6 = 0.667\), \(R = 4/5 = 0.8\), \(F_1 = 2 \cdot 0.667 \cdot 0.8 / 1.467 = 0.727\).
Adaptive + flash check, \(\theta = 0.20\). Four detections (50, 110, 220, 280), all correct, dissolve missed: \(P = 1\), \(R = 0.8\), \(F_1 = 1.6/1.8 = 0.889\).
Gap \(k = 8\), \(\theta = 0.30\). Nine detections: 50, 77, 85, 110, three inside the dissolve (164.5, 168.5, 176.5), 220 and 280. Only one detection can match the dissolve (the closest to its middle, 169.5), so \(\mathrm{TP} = 5\), \(\mathrm{FP} = 4\), \(\mathrm{FN} = 0\). \(P = 5/9 = 0.556\), \(R = 1\), \(F_1 = 0.714\). Finding the dissolve did not raise F1, because the same gap also creates false detections.
Streaming shot detection and the window
The 100-frame window matters for live video. The reference inference code pads the start of the video with 25 copies of the first frame, slides the window by 50 frames, and keeps predictions 25 to 74 of each window. [Certain] So the prediction for a frame needs between 25 and 74 frames of the future. At 25 frames per second that is a lookahead of 1 to 3 seconds before a shot boundary is final.
- input: a live stream of frames; window \(L = 100\), step \(L/2 = 50\), threshold 0.5
- resize each new frame to \(48 \times 27\) and push it into a buffer; at the start, push 25 copies of the first frame
- while the buffer holds 100 frames do
- run the network on the 100 frames; keep probabilities of positions 25 to 74 // each kept frame has 25+ frames of context on both sides
- mark frames with probability \(\gt 0.5\); a run of marked frames is one transition
- if a transition is found, close the current shot and emit it downstream with its start and end time
- drop the oldest 50 frames from the buffer
- at the end of the stream: pad with copies of the last frame and flush; close the last shot
From shots to scenes
Shots are a technical unit; scenes are a story unit. A dialogue alternates two close-ups (A, B, A, B): four shots, one scene. A simple and robust rule links a new shot to the current scene if it looks like any of the last few shots, not only the previous one. Represent each shot \(i\) by an embedding \(\mathbf{e}_i\), for example the mean CLIP embedding of its keyframes. Then
- Read the symbols
- \(\mathbf{e}_i\)
- embedding of shot \(i\) (visual, and often also audio or transcript features)
- \(M\)
- how many earlier shots to look back, for example 3 to 5, so that A, B, A patterns stay together
- \(\tau\)
- similarity threshold, tuned on labeled scenes
Learned scene segmentation works on the same idea with more context. LGSS (Rao et al., 2020) combines place, cast, action and audio features at the level of shots, segments and the whole movie, and introduced the MovieScenes dataset with 21K annotated scenes from 150 movies. [Certain] In practice, many products use shots for precise work (ads, cut points) and scenes or fixed-length chapters for summaries.
Captions and labels per shot or scene
Once the video is cut into shots, each shot gets a description. A common recipe has three steps:
- Pick keyframes. One to three frames per shot (for example the middle frame, or frames spread over a long shot). A single frame is enough for “what place, what objects”; motion and actions need several frames or a short clip (session 205).
- Describe. A VLM writes a short caption; CLIP or SigLIP gives an embedding for search; OCR reads on-screen text; ASR (for example Whisper) gives the speech of the shot’s time range.
- Classify. Map the shot to labels. There are three common ways, from cheap to expensive: a zero-shot CLIP classifier with one text prompt per label; a small classifier head trained on the embeddings; or an LLM that reads the caption, transcript and OCR text and returns labels in a fixed JSON format (session 146).
Which labels?
Two families of labels matter in advertising:
- Topic labels. The IAB Tech Lab Content Taxonomy is a shared, tree-shaped list of content categories (for example “Automotive”, “Sports”, “Food & Drink” with finer children). Version 3.1 is the latest listed by IAB Tech Lab, and its stated uses are contextual targeting and brand safety. [Certain] Its trees have up to four tiers. [Likely]
- Brand safety and suitability labels. The GARM framework from the World Federation of Advertisers defined sensitive categories (for example adult content, arms, crime, death and injury, hate speech, terrorism) with a “floor” that no ad should run next to and lower risk levels that each brand can choose to accept. GARM itself was discontinued on 8 August 2024 [Certain], but many platforms and vendors still use its categories and risk tiers. [Likely]
From shot labels to a video label
Each shot \(i\) has a duration \(\delta_i\) and a probability \(p_{i,c}\) for each label \(c\). An advertiser often needs one decision per video or per ad break. There are several ways to aggregate:
- Read the symbols
- \(n\)
- number of shots (or scenes) in the video or ad break
- \(\delta_i, w_i\)
- duration of shot \(i\) in seconds, and its share of the total time
- \(p_{i,c}\)
- the classifier’s probability that shot \(i\) has label \(c\)
- \(\bar{p}_c\)
- duration-weighted mean: “how much of the video, on average, is about \(c\)”
- \(\mathrm{share}_c\)
- time share: the fraction of time in shots that are confidently \(c\)
- \(p^{\max}_c\)
- the strongest single shot: “is \(c\) anywhere in the video?”
The right rule depends on the question. A topic answers “what is this video mostly about?”, so a time-weighted rule is right: one 3-second shot of food does not make a football match a food video. A safety label answers “is there any harmful moment?”, so the max rule is right: one 3-second crash shot is enough to keep a car ad away. Figure 3 shows this on a 100-second video with 8 shots.
How to read Figure 3
Darker cells mean higher shot probability. With “Max”, every label passes 0.5, so the video is called Sports, Automotive and Food: too many topics. With “Duration-weighted” or “Time share”, only Sports passes, which is right for topics, but the injury score falls to 0.060 (weighted) or 0.03 (time share): the crash is missed. The policy rule gives the right answer at 0.5: topic Sports, and the safety flag raised by the 3-second crash shot (max 0.92). Lower the threshold to 0.10 to see secondary topics (Automotive 0.14 and Food 0.15 by time share).
Durations 12, 8, 20, 3, 15, 6, 25, 11 s sum to 100 s, so \(w_i = \delta_i / 100\). Sports probabilities are 0.90, 0.80, 0.85, 0.20, 0.90, 0.30, 0.88, 0.70. The plain mean is \(5.53/8 = 0.691\). The weighted mean is \((10.8 + 6.4 + 17 + 0.6 + 13.5 + 1.8 + 22 + 7.7)/100 = 0.798\). The time share counts shots with \(p \gt 0.5\): \(12 + 8 + 20 + 15 + 25 + 11 = 91\) s, so 0.91.
The injury probabilities are small except shot 4 (0.92). The weighted mean is only 0.060, because shot 4 is 3% of the time; the max is 0.92. A rule that averages would never flag this video.
- input: a video file; label set with a rule per label (topic: time share; safety: max)
- decode once; run the shot detector; split long shots into sub-shots of at most 20 s
- for each shot \(i\) (in parallel, in batches) do
- pick keyframes; run the VLM caption, CLIP embedding and OCR on them // one GPU batch for many shots
- cut the ASR transcript to the time range of the shot
- \(p_{i,c} \leftarrow\) classifier(caption, transcript, OCR, embedding) for every label \(c\)
- group shots into scenes with rule (211.5)
- aggregate with (211.6): \(\mathrm{share}_c\) for topics, \(p^{\max}_c\) for safety; apply per-label thresholds
- store shots, captions, embeddings, \(p_{i,c}\) and video labels with model versions; send uncertain or flagged items to human review
Active speaker detection
Active speaker detection (ASD) answers, for every visible face and every frame: is this person speaking right now? It is needed for speaker-attributed captions, for joining a transcript to the right person, for automatic reframing (follow the speaker in a vertical crop), and for diarization (“who spoke when”) when faces are visible.
Neither signal alone is enough. Audio alone says that someone is speaking, but not which face. Video alone sees lips move, but people also move their lips when they chew, laugh, smile or sing along, and a speaker’s face may be partly hidden. The answer is to check whether the lip motion and the sound are in sync. An old and simple measure of sync is the correlation between the loudness of the audio and the opening of the mouth over a short time window:
- Read the symbols
- \(a_\tau\)
- audio loudness (energy envelope) at video frame \(\tau\)
- \(m_{i,\tau}\)
- mouth opening of face \(i\) at frame \(\tau\), from a face crop
- \(\mathcal{W}_t\)
- a window of \(W\) frames centered on frame \(t\); \(\bar{a}, \bar{m}_i\) are the means over the window
- \(r_{i,t}\)
- Pearson correlation between \(-1\) and 1; high when the mouth opens exactly when the sound is loud
Face \(i\) is called “speaking” at frame \(t\) if \(r_{i,t}\) is above a threshold. The window length is the key trade-off. A short window has too few syllables to measure sync reliably. A long window is reliable inside a speech turn but blurs the boundaries where one person stops and another starts. Figure 4 shows both effects.
How to read Figure 4
“Audio only” gives the same answer for both faces, so it is wrong for one of them whenever someone talks, and the music also fools it: about 0.53 to 0.56 frame accuracy. “Video only” is fooled by chewing and laughing: best about 0.87 at \(W = 5\). The sync score ignores both traps, because chewing is not in time with the sound. With \(W = 1\) it is useless (a correlation needs several frames); accuracy rises to 0.948 at \(W = 15\) (0.6 s), then falls again for long windows (0.823 at \(W = 101\)), because the window crosses the speaker changes.
TalkNet: learned audio-visual fusion
Real systems learn the sync from data. TalkNet (Tao et al., 2021) has the following parts [Certain]:
- Visual encoder. Face crops of \(112 \times 112\) pixels at 25 fps go through a 3D convolution and a ResNet-18, then a temporal network (V-TCN: five residual blocks of depthwise-separable 1D convolutions) with a receptive field of 21 frames (840 ms).
- Audio encoder. 13-dimensional MFCC features at 100 frames per second go through a ResNet-34 with squeeze-and-excitation blocks; dilated convolutions bring the audio down to the video’s time resolution. Its receptive field is 189 audio frames (1,890 ms). Both encoders output 128-dimensional features per video frame.
- Cross-attention. Two attention layers: audio features attend to visual features, and visual features attend to audio features (8 heads each). This is where the model checks whether the sound and the lips agree.
- Self-attention backend. One transformer layer over the joined audio-visual sequence models long-term evidence (“this person has been talking for 3 seconds”), then a linear layer gives a speaking probability for every frame.
In the audio-to-visual cross-attention, the audio features ask the questions and the visual features answer:
- Read the symbols
- \(\mathbf{F}_a, \mathbf{F}_v\)
- audio and visual features of one face track, shape \(T \times 128\) (one row per video frame)
- \(\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V\)
- learned projections for queries (from audio), keys and values (from video)
- \(d\)
- dimension of one attention head
The paper reports 92.3 mAP on the AVA-ActiveSpeaker validation set (3.5 points above the previous best, MAAS-TAN at 88.8) and 90.8 on the test set, and an average F1 of 96.2 on Columbia ASD. AVA-ActiveSpeaker has about 3.65 million labeled frames and about 38.5 hours of face tracks. [Certain] Two ablations of TalkNet match Figure 4:
| TalkNet input length (validation mAP) | 5 frames | 10 | 25 | 50 | 100 | variable, 1 to 10 s |
|---|---|---|---|---|---|---|
| mAP | 75.2 | 82.8 | 87.9 | 89.5 | 89.4 | 92.3 |
Short inputs are much worse: 0.2 s of video gives 75.2 mAP, 1 s gives 87.9. And removing both attention modules drops the result from 92.3 to 90.0. [Certain] Long-term context and audio-visual interaction both matter.
- input: decoded frames (25 fps) and audio (16 kHz) of one shot
- detect faces in every frame (or every few frames); link boxes over time into face tracks with a tracker; never link across a shot boundary
- for each track: crop and resize faces to \(112 \times 112\); cut the audio of the same time range; compute MFCCs
- run the ASD model on (face crops, audio) // one score per face per frame
- smooth scores over time (for example a median over 5 to 10 frames); threshold
- join with ASR word times: each word goes to the face that is speaking at that time, or to “off-screen” if none
Streaming vs batch
The same models can run in two very different ways.
- Batch (offline): process whole files, for example every upload of the day. Latency is minutes to hours, which is fine for archives, search indexes and ad targeting of video-on-demand. The system can use big batches, cheap interruptible machines and full-video context (for example, scenes and video-level labels).
- Streaming (online): process live video, for example sports, news or live streams, where labels are needed within seconds (moderation, live ad breaks). The video is cut into chunks; each chunk must be processed before the next arrives, on average, or the system falls behind.
How many GPUs?
Let \(\lambda\) be how many seconds of video arrive per second of wall-clock time (100 live channels give \(\lambda = 100\)), and \(c\) the GPU-seconds of work per second of video for the whole pipeline (you measure it per stage on your own hardware). The GPU count is
- Read the symbols
- \(\lambda\)
- arrival rate: seconds of video per second (equal to the number of live streams)
- \(c\)
- cost: GPU-seconds needed to process one second of video, summed over all stages
- \(\rho\)
- target utilization of each GPU, between 0 and 1
A batch system can run close to full utilization (\(\rho\) near 0.95), because nobody waits for a single result. A streaming system cannot, because of queueing.
Latency of a streaming pipeline
The time from “a frame was filmed” to “its label is ready” has four parts:
- Read the symbols
- \(T_{\text{chunk}}\)
- chunk length: the first frame of a chunk waits until the whole chunk has arrived
- \(T_{\text{look}}\)
- lookahead: extra future frames that a stage needs, for example 1 to 3 s for TransNet V2’s window
- \(S\)
- service time: GPU time to process one chunk
- \(W_q\)
- average waiting time in the queue; the formula is the classic single-server (M/M/1) result, a rough guide
The queue term grows without limit as \(\rho \to 1\): at \(\rho = 0.5\) the wait equals one service time, at 0.8 it is 4 service times, at 0.95 it is 19. Real systems with many GPUs sharing one queue wait less than this formula says, but the shape is the same: the last few percent of utilization cost a lot of latency. Figure 5 lets you explore it.
How to read Figure 5
With the defaults (100 streams, \(c = 0.1\), \(\rho = 0.8\), 10 s chunks) you need 13 GPUs and labels arrive after 18 s. Push \(\rho\) to 0.95: you save GPUs (11), but the queue wait jumps from 4 s to 19 s. Shorter chunks cut the chunk wait and the service time, but every chunk pays fixed costs in a real system (model calls, a keyframe at least, network), so very short chunks cost more GPU per second of video. The readout also shows the batch cost at \(\rho = 0.95\) for comparison.
Suppose you measured \(c = 0.1\) GPU-seconds per second of video for the whole pipeline. With \(\lambda = 100\) streams the offered load is \(100 \times 0.1 = 10\) GPUs of work. At \(\rho = 0.8\): \(N = \lceil 10 / 0.8 \rceil = \lceil 12.5 \rceil = 13\) GPUs.
With 10-second chunks, \(S = 0.1 \times 10 = 1\) s, and \(W_q \approx 0.8/0.2 \times 1 = 4\) s. With 3 s of lookahead, \(L = 10 + 3 + 1 + 4 = 18\) s.
The same work in batch at \(\rho = 0.95\) needs \(\lceil 10/0.95 \rceil = 11\) GPUs and \(0.1/0.95 = 0.105\) GPU-hours per video-hour, versus \(0.1/0.8 = 0.125\) for streaming: about 16% less. Over a day, 100 channels in streaming use \(100 \times 24 \times 0.125 = 300\) GPU-hours. Multiply by your own price per GPU-hour.
Where the cost goes
The cost \(c\) is a sum over stages, and the stages scale differently:
- Decode scales with frames and resolution. Use hardware decoders, and decode at low resolution for shot detection (\(48 \times 27\) is enough for TransNet V2).
- Shot detection is cheap per frame because the frames are tiny.
- VLM captions scale with the number of shots and keyframes, not with frames. This is usually the largest cost, so the keyframe policy (how many per shot, at what resolution) is the main knob (session 199 shows how to budget one VLM call).
- ASR scales with audio length; it can skip music and silence with a voice activity detector.
- ASD scales with the number of face tracks times their length: a crowd scene costs much more than a news anchor. Cap the number of tracks per shot (for example, the largest faces only).
Monitoring
A video pipeline can fail quietly: a decoder change drops frames, a new camera style breaks the shot detector, or a model update shifts label rates. Watch four groups of signals:
- System: throughput (video-hours processed per hour), streaming lag (wall-clock time minus media time of the last labeled frame; it must not grow), p50 and p95 latency per stage, queue lengths, GPU utilization, error and retry counts, decode failures.
- Data: input resolution, frame rate, audio presence, and duration distributions. A new source with 60 fps or no audio changes the cost and the models’ inputs.
- Model outputs: shots per minute (a sudden change often means a broken detector), caption length, the share of each label, the confidence histogram.
- Quality: a small random sample sent to human reviewers every day, plus all safety flags near the threshold. This is the only signal that measures accuracy directly.
A standard number for “has the label distribution changed?” is the population stability index (PSI) between a reference period \(q\) and today \(p\):
- Read the symbols
- \(q_c\)
- share of label (or bin) \(c\) in the reference period, for example last month
- \(p_c\)
- share of label \(c\) today; both must be above 0 (add a small constant to empty bins)
PSI is a symmetric form of the KL divergence: \(\mathrm{PSI} = \KL(p \,\|\, q) + \KL(q \,\|\, p)\). A common rule of thumb reads PSI below 0.1 as stable, 0.1 to 0.25 as a moderate change and above 0.25 as a large change. [Likely]
Last month the video labels were Sports 0.40, News 0.30, Entertainment 0.20, Other 0.10. Today they are 0.25, 0.30, 0.25, 0.20. The terms are \((0.25 - 0.40)\ln(0.25/0.40) = 0.0705\), \(0\) for News, \((0.25 - 0.20)\ln(1.25) = 0.0112\), and \((0.20 - 0.10)\ln 2 = 0.0693\). PSI = 0.151: a moderate change. It may be real (the football season ended) or a bug (a new source that the classifier calls “Other”). Look at samples before you retrain.
Code: a toy pipeline in PyTorch
The code below runs on a CPU in about a second and gives the same output on every run. It builds a synthetic video of four colored shots with a cut, a dissolve and another cut, and then (1) finds boundaries with histogram differences (211.1, 211.2) at gaps 1 and 8, (2) aggregates shot labels with the duration-weighted mean and the max (211.6), and (3) computes the audio-visual sync score (211.7) for a speaking face and a chewing face.
import torch import torch.nn.functional as F torch.manual_seed(0) # 1) A toy video: 4 shots of different colors, a hard cut at 40, a dissolve 80..95, a cut at 130 T, H, W = 160, 27, 48 # TransNet V2 also works on 48x27 frames colors = torch.tensor([[0.8, 0.2, 0.2], [0.2, 0.7, 0.3], [0.2, 0.3, 0.9], [0.9, 0.9, 0.3]]) def shot_of(t): return 0 if t < 40 else 1 if t < 80 else 2 if t < 130 else 3 frames = torch.empty(T, 3, H, W) for t in range(T): c = colors[shot_of(t)] if 80 <= t < 96: # dissolve from shot 1 to shot 2 a = (t - 79) / 17 c = (1 - a) * colors[1] + a * colors[2] frames[t] = c[:, None, None] + 0.05 * torch.randn(3, H, W) frames = frames.clamp(0, 1) # 2) Color histograms, 8 bins per channel, normalized to sum 1 def hist(f, bins=8): h = torch.stack([torch.histc(f[ch], bins=bins, min=0, max=1) for ch in range(3)]) return (h / h.sum()).flatten() # (24,) Hs = torch.stack([hist(f) for f in frames]) # (T, 24) # 3) Frame difference d_t^(k) = 1/2 * sum |h_t - h_{t-k}| (formula 211.1) def diff(Hs, k): d = torch.zeros(len(Hs)) d[k:] = 0.5 * (Hs[k:] - Hs[:-k]).abs().sum(1) return d def detect(d, th, k): above = (d > th).int().tolist() + [0] out, start = [], None for t, a in enumerate(above): if a and start is None: start = t if not a and start is not None: out.append((start + t - 1) / 2 - (k - 1) / 2); start = None return out d1, d8 = diff(Hs, 1), diff(Hs, 8) print("k=1:", detect(d1, 0.3, 1), " max inside dissolve:", round(d1[81:96].max().item(), 3)) print("k=8:", detect(d8, 0.3, 8)) # 4) Shot labels -> video labels (formula 211.6) dur = torch.tensor([12., 8, 20, 3, 15, 6, 25, 11]) p = torch.tensor([[0.90, 0.80, 0.85, 0.20, 0.90, 0.30, 0.88, 0.70], # Sports [0.02, 0.05, 0.03, 0.92, 0.04, 0.02, 0.03, 0.05]]) # injury risk w = dur / dur.sum() print("duration-weighted:", [round(x, 3) for x in (p @ w).tolist()], " max:", [round(x, 3) for x in p.max(1).values.tolist()]) # 5) Audio-visual sync: windowed Pearson correlation (formula 211.7) def win_corr(a, m, W=15): pad = W // 2 A = F.pad(a[None], (pad, pad), mode="replicate")[0].unfold(0, W, 1) M = F.pad(m[None], (pad, pad), mode="replicate")[0].unfold(0, W, 1) A, M = A - A.mean(1, keepdim=True), M - M.mean(1, keepdim=True) return (A * M).sum(1) / (A.norm(dim=1) * M.norm(dim=1) + 1e-8) t = torch.arange(250.) speech = ((t >= 50) & (t < 150)).float() env = speech * (0.5 - 0.5 * torch.cos(2 * torch.pi * t / 5)) * (0.3 + 0.7 * torch.rand(250)) audio = env + 0.08 * torch.rand(250) lips_speaker = 0.9 * env + 0.06 * torch.rand(250) lips_chewing = 0.6 * (0.5 - 0.5 * torch.cos(2 * torch.pi * t / 16)) + 0.06 * torch.rand(250) for name, m in [("speaker", lips_speaker), ("chewing", lips_chewing)]: r = win_corr(audio, m) print(name, "mean sync while speech:", round(r[60:140].mean().item(), 3), " lip motion:", round(m[60:140].mean().item(), 3))
Output:
k=1: [40.0, 130.0] max inside dissolve: 0.161 k=8: [40.0, 88.0, 130.0] duration-weighted: [0.798, 0.06] max: [0.9, 0.92] speaker mean sync while speech: 0.996 lip motion: 0.323 chewing mean sync while speech: 0.018 lip motion: 0.328
Read it: with \(k = 1\) the dissolve is missed (the largest step inside it is only 0.161, below the threshold 0.3), while \(k = 8\) finds it at frame 88 (the true middle is 87.5). The safety score is 0.06 by duration-weighted mean but 0.92 by max. Most important, the speaking face and the chewing face have almost the same lip motion (0.323 and 0.328), so a video-only rule cannot tell them apart; the sync score separates them completely (0.996 vs 0.018).
Where this is used
| Application | Pipeline stages that matter | Typical mode |
|---|---|---|
| Contextual ads on video-on-demand | Shots, keyframe captions, IAB topic labels by time share, safety by max; labels per ad break | Batch, at upload time |
| Live sports and news ad breaks | Streaming shot detection, fast small classifiers, safety labels of the last minute before the break | Streaming, seconds |
| Content moderation | Safety labels by max, human review queue ordered by risk, appeal logs | Both: fast pass, then a slower re-check |
| Archive search and Video-RAG | Per-shot captions, transcripts and embeddings in a vector index (session 210) | Batch |
| Key-moment detection and highlights | Shot boundaries as cut points, audio events (crowd noise), temporal grounding (session 207) | Streaming for live, batch for VOD |
| Speaker-attributed subtitles and reframing | Face tracks, active speaker detection, ASR word times | Batch, or streaming with a few seconds of delay |
| Duplicate and re-upload detection | Shot-level fingerprints and embeddings (session 209) | Batch at upload |
| Action recognition in video analytics | Clips cut at shot boundaries, so a clip never mixes two shots (session 203) | Both |
Common mistakes
- Running expensive models per frame. Captions and labels per frame are mostly repeats. Work per shot (or sub-shot), and only add frames where motion matters.
- Tuning the shot threshold on one kind of content. A threshold tuned on movies fails on sport, music videos or screen recordings. Use a learned detector and evaluate per content type.
- Averaging safety labels. One 3-second harmful shot disappears in a duration-weighted mean (Figure 3: 0.92 becomes 0.060). Use the max (or “any shot above threshold”) for safety, and averages only for topics.
- Linking face tracks across shot cuts. A tracker can jump from one person to another at a cut. Always reset tracks at shot boundaries.
- Trusting video-only speaker detection. Chewing, laughing and singing along move the lips. Use an audio-visual model, and check audio-video sync first.
- Running a streaming system at 95% utilization. The queue wait explodes (formula 211.10). Keep headroom, and autoscale on lag, not only on GPU utilization.
- Not storing intermediate results with model versions. Without them, every label change means re-decoding and re-running the whole archive.
- Monitoring only system metrics. A pipeline can be fast and wrong. Track label shares (PSI), shots per minute, and a daily human-reviewed sample.
Interview questions
What is the difference between a shot and a scene, and why does a pipeline need both?basic
Why does a frame-difference shot detector miss dissolves and fire on flashes? How do you fix it?core
Describe TransNet V2.core
You have per-shot probabilities for “Automotive” and for “violence”. How do you get video-level labels?core
How does active speaker detection work, and why do you need both audio and video?core
TalkNet gets 75.2 mAP with 5-frame inputs and 92.3 with variable 1 to 10 s inputs. Why, and what does it mean for a streaming system?advanced
Estimate the GPUs and latency for labeling 100 live channels.advanced
When would you choose streaming over batch, and what changes in the design?core
How do you monitor a video labeling pipeline?core
Why should shot detection run on tiny frames, and why decode only once?basic
Your label shares changed suddenly overnight. How do you debug it?advanced
Exercises
- Two 4-bin histograms are \(h_{t-1} = (0.4, 0.3, 0.2, 0.1)\) and \(h_t = (0.1, 0.2, 0.3, 0.4)\). Compute \(d_t^{(1)}\) with formula (211.1).
- A detector finds 9 boundaries in a clip with 10 true transitions; 7 detections are correct. Compute precision, recall and F1.
- With gap \(k = 8\), a cut at frame 100 makes a run of scores above the threshold over frames 100 to 107. Where does formula (211.2) place the detection?
- In Figure 3, compute the duration-weighted mean of “Food & Drink” (probabilities 0.05, 0.10, 0.10, 0.05, 0.70, 0.10, 0.05, 0.10) and its max. Which rule would you use for this label?
- With TransNet V2’s reference windowing at 30 fps, what is the largest lookahead in seconds before a boundary decision is final?
- A streaming pipeline has \(c = 0.2\), chunks of 5 s, \(\rho = 0.9\) and 3 s of lookahead. Compute \(S\), \(W_q\) and \(L\) with formula (211.10). How many GPUs for 50 streams?
Answerscheck
Key takeaways
- A production pipeline decodes once, cuts the video into shots, does the expensive work (captions, embeddings, OCR, ASR, ASD) once per shot, then aggregates to scenes and videos, and stores every intermediate result with its model version.
- Hand-made shot detectors (histogram differences, 211.1) handle cuts but fail on dissolves, flashes and fast motion; TransNet V2 learns them from 100-frame windows of \(48 \times 27\) frames and reaches F1 96.2 on BBC and 77.9 on ClipShots.
- Aggregate topics by time (weighted mean or time share) and safety by max (211.6); one short harmful shot must not be averaged away.
- Active speaker detection needs audio-visual sync, not lip motion alone; TalkNet fuses them with cross-attention and gains a lot from long context (75.2 mAP at 0.2 s vs 92.3 with 1 to 10 s inputs).
- Streaming trades cost for latency: queue wait grows as \(\rho/(1-\rho)\), so live systems keep headroom, while batch runs near full utilization; monitor lag, per-stage latency, label drift (PSI) and a human-reviewed sample.
Key formulas
References for this session
- Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
- Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
- Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Roth et al. (2019). AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection
- Rao, Xu, Xiong, Xu, Huang, Zhou & Lin (2020). A Local-to-Global Approach to Multi-modal Movie Scene Segmentation
- Chung & Zisserman (2016). Out of Time: Automated Lip Sync in the Wild
- PySceneDetect developers (2024). PySceneDetect documentation: scene detection algorithms (ContentDetector, AdaptiveDetector)
- IAB Tech Lab (2024). Content Taxonomy (versions up to 3.1)
- World Federation of Advertisers (GARM) (2022). GARM Brand Safety Floor + Suitability Framework
- Souček & Lokoč (2020). TransNet V2 reference code: inference/transnetv2.py (100-frame windows, step 50)
- Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision