LLM & VLM courseBlog
211Track 8: Video and multimodal systems · session 11 of 21

Production video understanding pipelines

Real video systems are pipelines: decode once, cut the video into shots, describe and label each shot, find who is speaking, and sum the results up for scenes and whole videos. This session builds such a pipeline stage by stage, with TransNet V2 for shot boundaries and TalkNet for active speakers, and then compares streaming and batch designs by latency, cost and monitoring.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on video as data, video encoders, Video-LLMs and serving; basic attention and cross-entropy

What you will learn

  • The stages of a production video understanding pipeline and why the shot is the unit of work
  • How shot boundaries are found with histogram differences, why dissolves, flashes and fast motion break simple rules, and how TransNet V2 learns them
  • How to group shots into scenes, and how to caption and classify shots with IAB topic and brand-safety labels
  • Why topics should be aggregated by time and safety labels by max, with the numbers worked out
  • How active speaker detection fuses audio and video through sync, and what TalkNet adds with cross-attention and long context
  • How to size a streaming or batch pipeline: GPU count, latency parts, queueing at high utilization
  • What to monitor: lag, per-stage latency, label drift with PSI, and human-reviewed samples

Why this matters

A video platform, a broadcaster or an ad company does not run one model on one clip. It runs a pipeline: many models, one after another, on thousands of hours of video every day. The pipeline cuts each video into pieces, describes each piece, labels it, finds who is talking, and stores the results so that search, ads and safety teams can use them. This session is about how such a pipeline is built, what each stage does, and how to keep it fast, cheap and correct.

In the earlier sessions of this track you met the models one by one: video encoders, Video-LLMs, temporal grounding, retrieval, fingerprinting, and in session 210 a system that answers questions over a video archive. Video-RAG needs an index of the archive. Production pipelines build that index. They also feed many other products:

  • Contextual advertising. An ad server wants to know “this part of the video is about cars” so it can show a car ad, and “this part shows an accident” so it does not show a car ad there. Labels from a shared list, the IAB Content Taxonomy, are the common language between publishers and advertisers.
  • Brand safety and moderation. Find violence, adult content or hate speech, so that ads are not placed next to it and so that human reviewers see the risky parts first.
  • Search and recommendation. Captions, transcripts and embeddings per shot make an archive searchable by text (session 208).
  • Editing tools. Shot lists, chapter marks, speaker labels, and automatic reframing of a wide video into a vertical one that follows the person who speaks.

The key ideas are in four groups, and the page follows them. (1) Shot and scene boundary detection: where does the camera cut, and which shots belong to one scene? TransNet V2 (Souček and Lokoč, 2020) is the standard open model. (2) Captioning and classification per shot or scene, and how to turn many shot labels into one label for a video. (3) Active speaker detection: which visible face is speaking right now? TalkNet (Tao et al., 2021) fuses audio and video for this. (4) Streaming vs batch: the same models, run as a live system or as an offline job, with very different latency, cost and monitoring. The book’s chapter on video (Noyan et al., chapter 9) gives the model side; this page is about the system around the models.

The pipeline, end to end

Figure 1 shows a typical design. Read it from left to right. A video enters, is decoded once, and is cut into shots. Most of the expensive work is done once per shot, not once per frame, because frames inside one shot look almost the same. Results are joined per shot, grouped into scenes, and summed up for the whole video.

ingest file or livestream decode frames + audioonce, on GPU shots TransNet V2 on48×27 frames keyframes per shot VLM caption, CLIPembedding, OCR audio speech to text (ASR),music, sound events faces detect, track, activespeaker (TalkNet) fuse + label per shot: IABtopics, safetyrisk, speakers aggregate shots → scenes→ video;index, store monitoring: throughput, lag, p95 latency per stage, errors, label distribution drift, human review samples every stage writes metrics; every stage can be re-run alone from stored outputs expensive work happensonce per shot, not per frame
Figure 1. A typical production video understanding pipeline. The three colored branches are visual work, audio work and face work. The shot detector decides the unit of work for everything after it. The monitoring layer covers every stage.

Three design rules come from this picture, and we will meet them again:

  1. Decode once. Decoding compressed video is often a big part of the cost (session 201). Decode each video once, at the lowest resolution and frame rate that each stage needs, and share the frames.
  2. The shot is the unit of work. A caption per shot, not per frame, cuts the number of expensive VLM calls by a large factor. If shots last a few seconds on average, as in much edited TV and film [Likely], an hour of video has on the order of \(3600/4 = 900\) shots.
  3. Store every intermediate result. Shot lists, captions, transcripts and embeddings are stored with the model version that made them. Then a new label set needs only a cheap re-run of the last stage, not of the whole pipeline.

Shot boundary detection

A shot is a continuous piece of video from one camera take. A shot transition (or shot boundary) is where one shot ends and the next begins. There are two families of transitions:

  • Hard cuts: the last frame of shot A is followed directly by the first frame of shot B. Most transitions in edited video are cuts.
  • Gradual transitions: a dissolve (A fades into B over many frames), a fade to or from black, a wipe (B slides over A). They can last from a few frames to a second or more.

A scene is a larger unit: a group of shots that share a place, a time and a story, for example a dialogue filmed as A, B, A, B close-ups. Scenes are harder to define and to detect; we come back to them below.

The classic method: compare color histograms

The simplest detector compares each frame with an earlier frame. A good comparison is a color histogram: count how many pixels fall into each color bin, then divide by the number of pixels, so the bins sum to 1. A histogram ignores where colors are, so small camera or object motion changes it only a little, while a cut to a new place changes it a lot. The difference between frame \(t\) and frame \(t-k\) is

\[ d_t^{(k)} = \frac{1}{2} \sum_{b=1}^{B} \left| h_t(b) - h_{t-k}(b) \right| \tag{211.1} \]
Read the symbols
\(h_t(b)\)
the fraction of pixels of frame \(t\) that fall into color bin \(b\); the bins sum to 1
\(B\)
number of bins, for example 8 per color channel
\(k\)
the frame gap: \(k = 1\) compares neighbors, a larger \(k\) compares frames further apart
\(d_t^{(k)}\)
a number between 0 (same histogram) and 1 (no color in common); the factor \(\tfrac12\) makes the maximum 1

The detector reports a boundary where the difference is above a threshold \(\theta\). Neighboring frames above the threshold form one run, and each run is one detection, placed at the middle of the run:

\[ \text{boundary in run } [s, e] \iff d_t^{(k)} \gt \theta \ \text{ for all } t \in [s, e], \qquad \hat{t} = \frac{s + e}{2} - \frac{k - 1}{2} \tag{211.2} \]

The shift \(-(k-1)/2\) is needed because, with a gap \(k\), one cut makes a run of \(k\) frames: every frame in the first \(k\) frames of shot B is compared with a frame of shot A. Open-source tools work this way. For example, PySceneDetect’s ContentDetector compares neighboring frames in HSV color space with a default threshold of 27 (on its own scale), and its AdaptiveDetector compares each score with a rolling average of nearby scores, to reduce false cuts in fast motion. [Certain]

Why simple rules break

Each kind of hard video breaks a different rule:

  • Dissolves spread the change over many frames. With \(k = 1\), each step is small, so a dissolve hides below any threshold that ignores noise.
  • Flashes (a camera flash, an explosion, lightning) change one or two frames a lot and then go back. They look like two cuts.
  • Fast motion (a camera pan, sport) changes the histogram steadily. With a large \(k\), the change over \(k\) frames looks like a cut.

Figure 2 lets you try three hand-made detectors on a synthetic 300-frame video with five true transitions (four cuts and one 20-frame dissolve), one flash and one fast camera pan.

scoretrue transitioncorrect detectionfalse or missed
Figure 2. Three hand-made shot detectors on one synthetic video (8-bin color histograms, 300 frames). True transitions: cuts at frames 50, 110, 220 and 280, and a dissolve over frames 160 to 179. Traps: a 2-frame flash at frame 80 and a fast camera pan over frames 220 to 279. A detection is correct if it is within 2 frames of a cut, or inside the dissolve (with 2 frames of slack).
How to read Figure 2

Start with “Neighbors, \(k = 1\)” and move \(\theta\). Below about 0.12 the camera pan makes many false cuts. From 0.14 to 0.48 you get the four cuts plus two false detections from the flash, and the dissolve is always missed: \(F_1 = 0.727\). Switch to “Gap \(k = 8\)”: at \(\theta = 0.30\) the dissolve is found, but it breaks into three runs, the flash still gives two false detections, and lower thresholds let the pan create many more (\(F_1 = 0.714\) at 0.30). The third detector subtracts the local median score (adaptive) and drops a detection if the frames 3 before and 3 after it look the same (flash check): it is clean, \(F_1 = 0.889\) for \(\theta\) from 0.06 to 0.42, but it still misses the dissolve. No single rule handles every trap.

This is the honest summary of decades of hand-made detectors: each fix helps one case and hurts another, and thresholds must be tuned per type of content. This is why learned detectors replaced them.

TransNet V2: a learned shot detector

TransNet V2 is a small 3D convolutional network that looks at many frames at once and outputs, for every frame, the probability that it is the middle of a transition. Its main design choices are [Certain]:

  • Tiny frames. Frames are resized to \(48 \times 27\) pixels. Shot changes are visible at this size, and it makes the network very cheap.
  • A window of 100 frames. The network sees 100 frames and predicts all 100, but at test time only the middle 50 predictions are used, so each prediction has at least 25 frames of context on each side.
  • Dilated, factorized 3D convolutions (DDCNN cells). Each cell runs convolutions with temporal dilation 1, 2, 4 and 8 in parallel, so it sees short and long changes. V2 splits each 3D convolution into a 2D spatial one and a 1D temporal one, and adds batch normalization.
  • Hand-made features inside the network. RGB color histograms (\(8^3 = 512\) bins) and learned features of each frame are compared by cosine similarity with the 50 frames before and the 50 frames after it. So the old idea of Figure 2 is kept, but as an input to a learned model.
  • Two heads. A “single-frame” head predicts the middle frame of each transition (used at test time). An “all-frame” head predicts every frame that belongs to a transition (used only in training, as extra signal).
  • Synthetic training data. Transitions are made by joining real shots from TRECVID IACC.3 and ClipShots: the training mix is 35% hard cuts, 50% dissolves (2 to 30 frames long), and 15% real transitions from ClipShots.

The training loss is a weighted cross-entropy on both heads plus weight decay:

\[ \mathcal{L} = \mathrm{CE}_{w=5}\!\left(y^{\text{mid}}, \hat{y}^{\text{mid}}\right) + 0.1\, \mathrm{CE}\!\left(y^{\text{all}}, \hat{y}^{\text{all}}\right) + 10^{-4} \lVert \mathbf{w} \rVert_2^2 \tag{211.3} \]
Read the symbols
\(y^{\text{mid}}, \hat{y}^{\text{mid}}\)
true and predicted labels of the single-frame head: 1 at the middle frame of a transition, 0 elsewhere
\(\mathrm{CE}_{w=5}\)
binary cross-entropy where positive frames count 5 times, because transitions are rare
\(y^{\text{all}}, \hat{y}^{\text{all}}\)
labels of the all-frame head: 1 for every frame inside a transition
\(\mathbf{w}\)
all network weights; the last term is L2 regularization

At test time a frame is a boundary when its probability is above 0.5. The paper trains with SGD (momentum 0.9, learning rate 0.01) for 50 epochs of 750 batches of 16, about 17 hours on one V100 GPU. [Certain]

Shot detection is scored with F1, with a small tolerance: a detection counts as correct if it is within 2 frames of a true transition. With \(\mathrm{TP}\) correct detections, \(\mathrm{FP}\) false ones and \(\mathrm{FN}\) missed transitions,

\[ P = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}, \qquad R = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}, \qquad F_1 = \frac{2 P R}{P + R} \tag{211.4} \]

The paper reports F1 of 77.9 on ClipShots, 96.2 on BBC Planet Earth and 93.9 on RAI. The first TransNet had 73.5, 92.9 and 94.3, so V2 is better on two of the three sets. ClipShots is the hardest: it is made of short user videos with many gradual transitions. [Certain]

Worked example: F1 for the detectors of Figure 2

\(k = 1\), \(\theta = 0.20\). Six detections at frames 50, 80, 82, 110, 220, 280. The flash makes two false ones (80 and 82), and the dissolve is missed. So \(\mathrm{TP} = 4\), \(\mathrm{FP} = 2\), \(\mathrm{FN} = 1\). \(P = 4/6 = 0.667\), \(R = 4/5 = 0.8\), \(F_1 = 2 \cdot 0.667 \cdot 0.8 / 1.467 = 0.727\).

Adaptive + flash check, \(\theta = 0.20\). Four detections (50, 110, 220, 280), all correct, dissolve missed: \(P = 1\), \(R = 0.8\), \(F_1 = 1.6/1.8 = 0.889\).

Gap \(k = 8\), \(\theta = 0.30\). Nine detections: 50, 77, 85, 110, three inside the dissolve (164.5, 168.5, 176.5), 220 and 280. Only one detection can match the dissolve (the closest to its middle, 169.5), so \(\mathrm{TP} = 5\), \(\mathrm{FP} = 4\), \(\mathrm{FN} = 0\). \(P = 5/9 = 0.556\), \(R = 1\), \(F_1 = 0.714\). Finding the dissolve did not raise F1, because the same gap also creates false detections.

Streaming shot detection and the window

The 100-frame window matters for live video. The reference inference code pads the start of the video with 25 copies of the first frame, slides the window by 50 frames, and keeps predictions 25 to 74 of each window. [Certain] So the prediction for a frame needs between 25 and 74 frames of the future. At 25 frames per second that is a lookahead of 1 to 3 seconds before a shot boundary is final.

Algorithm 1: streaming shot detection with a sliding window (TransNet V2 style)
  1. input: a live stream of frames; window \(L = 100\), step \(L/2 = 50\), threshold 0.5
  2. resize each new frame to \(48 \times 27\) and push it into a buffer; at the start, push 25 copies of the first frame
  3. while the buffer holds 100 frames do
  4.   run the network on the 100 frames; keep probabilities of positions 25 to 74 // each kept frame has 25+ frames of context on both sides
  5.   mark frames with probability \(\gt 0.5\); a run of marked frames is one transition
  6.   if a transition is found, close the current shot and emit it downstream with its start and end time
  7.   drop the oldest 50 frames from the buffer
  8. at the end of the stream: pad with copies of the last frame and flush; close the last shot

From shots to scenes

Shots are a technical unit; scenes are a story unit. A dialogue alternates two close-ups (A, B, A, B): four shots, one scene. A simple and robust rule links a new shot to the current scene if it looks like any of the last few shots, not only the previous one. Represent each shot \(i\) by an embedding \(\mathbf{e}_i\), for example the mean CLIP embedding of its keyframes. Then

\[ \text{same scene}(i) \iff \max_{j \in \{i-M, \dots, i-1\}} \cos(\mathbf{e}_i, \mathbf{e}_j) \gt \tau \tag{211.5} \]
Read the symbols
\(\mathbf{e}_i\)
embedding of shot \(i\) (visual, and often also audio or transcript features)
\(M\)
how many earlier shots to look back, for example 3 to 5, so that A, B, A patterns stay together
\(\tau\)
similarity threshold, tuned on labeled scenes

Learned scene segmentation works on the same idea with more context. LGSS (Rao et al., 2020) combines place, cast, action and audio features at the level of shots, segments and the whole movie, and introduced the MovieScenes dataset with 21K annotated scenes from 150 movies. [Certain] In practice, many products use shots for precise work (ads, cut points) and scenes or fixed-length chapters for summaries.

Captions and labels per shot or scene

Once the video is cut into shots, each shot gets a description. A common recipe has three steps:

  1. Pick keyframes. One to three frames per shot (for example the middle frame, or frames spread over a long shot). A single frame is enough for “what place, what objects”; motion and actions need several frames or a short clip (session 205).
  2. Describe. A VLM writes a short caption; CLIP or SigLIP gives an embedding for search; OCR reads on-screen text; ASR (for example Whisper) gives the speech of the shot’s time range.
  3. Classify. Map the shot to labels. There are three common ways, from cheap to expensive: a zero-shot CLIP classifier with one text prompt per label; a small classifier head trained on the embeddings; or an LLM that reads the caption, transcript and OCR text and returns labels in a fixed JSON format (session 146).

Which labels?

Two families of labels matter in advertising:

  • Topic labels. The IAB Tech Lab Content Taxonomy is a shared, tree-shaped list of content categories (for example “Automotive”, “Sports”, “Food & Drink” with finer children). Version 3.1 is the latest listed by IAB Tech Lab, and its stated uses are contextual targeting and brand safety. [Certain] Its trees have up to four tiers. [Likely]
  • Brand safety and suitability labels. The GARM framework from the World Federation of Advertisers defined sensitive categories (for example adult content, arms, crime, death and injury, hate speech, terrorism) with a “floor” that no ad should run next to and lower risk levels that each brand can choose to accept. GARM itself was discontinued on 8 August 2024 [Certain], but many platforms and vendors still use its categories and risk tiers. [Likely]

From shot labels to a video label

Each shot \(i\) has a duration \(\delta_i\) and a probability \(p_{i,c}\) for each label \(c\). An advertiser often needs one decision per video or per ad break. There are several ways to aggregate:

\[ \bar{p}_c = \sum_{i=1}^{n} w_i\, p_{i,c}, \quad w_i = \frac{\delta_i}{\sum_j \delta_j}; \qquad \mathrm{share}_c = \sum_{i:\, p_{i,c} \gt 0.5} w_i; \qquad p^{\max}_c = \max_i p_{i,c} \tag{211.6} \]
Read the symbols
\(n\)
number of shots (or scenes) in the video or ad break
\(\delta_i, w_i\)
duration of shot \(i\) in seconds, and its share of the total time
\(p_{i,c}\)
the classifier’s probability that shot \(i\) has label \(c\)
\(\bar{p}_c\)
duration-weighted mean: “how much of the video, on average, is about \(c\)”
\(\mathrm{share}_c\)
time share: the fraction of time in shots that are confidently \(c\)
\(p^{\max}_c\)
the strongest single shot: “is \(c\) anywhere in the video?”

The right rule depends on the question. A topic answers “what is this video mostly about?”, so a time-weighted rule is right: one 3-second shot of food does not make a football match a food video. A safety label answers “is there any harmful moment?”, so the max rule is right: one 3-second crash shot is enough to keep a car ad away. Figure 3 shows this on a 100-second video with 8 shots.

Figure 3. Shot-level probabilities for three IAB-style topics and one safety label (“death and injury”) on a 100-second sports video. Shot widths are proportional to their durations (12, 8, 20, 3, 15, 6, 25, 11 s). Shot 4 (3 s) shows a crash; shot 6 (6 s) is a car advert in the stream. The right column is the video-level score under the chosen rule; the dashed line is the decision threshold.
How to read Figure 3

Darker cells mean higher shot probability. With “Max”, every label passes 0.5, so the video is called Sports, Automotive and Food: too many topics. With “Duration-weighted” or “Time share”, only Sports passes, which is right for topics, but the injury score falls to 0.060 (weighted) or 0.03 (time share): the crash is missed. The policy rule gives the right answer at 0.5: topic Sports, and the safety flag raised by the 3-second crash shot (max 0.92). Lower the threshold to 0.10 to see secondary topics (Automotive 0.14 and Food 0.15 by time share).

Worked example: the numbers behind Figure 3

Durations 12, 8, 20, 3, 15, 6, 25, 11 s sum to 100 s, so \(w_i = \delta_i / 100\). Sports probabilities are 0.90, 0.80, 0.85, 0.20, 0.90, 0.30, 0.88, 0.70. The plain mean is \(5.53/8 = 0.691\). The weighted mean is \((10.8 + 6.4 + 17 + 0.6 + 13.5 + 1.8 + 22 + 7.7)/100 = 0.798\). The time share counts shots with \(p \gt 0.5\): \(12 + 8 + 20 + 15 + 25 + 11 = 91\) s, so 0.91.

The injury probabilities are small except shot 4 (0.92). The weighted mean is only 0.060, because shot 4 is 3% of the time; the max is 0.92. A rule that averages would never flag this video.

Algorithm 2: batch labeling of one video
  1. input: a video file; label set with a rule per label (topic: time share; safety: max)
  2. decode once; run the shot detector; split long shots into sub-shots of at most 20 s
  3. for each shot \(i\) (in parallel, in batches) do
  4.   pick keyframes; run the VLM caption, CLIP embedding and OCR on them // one GPU batch for many shots
  5.   cut the ASR transcript to the time range of the shot
  6.   \(p_{i,c} \leftarrow\) classifier(caption, transcript, OCR, embedding) for every label \(c\)
  7. group shots into scenes with rule (211.5)
  8. aggregate with (211.6): \(\mathrm{share}_c\) for topics, \(p^{\max}_c\) for safety; apply per-label thresholds
  9. store shots, captions, embeddings, \(p_{i,c}\) and video labels with model versions; send uncertain or flagged items to human review

Active speaker detection

Active speaker detection (ASD) answers, for every visible face and every frame: is this person speaking right now? It is needed for speaker-attributed captions, for joining a transcript to the right person, for automatic reframing (follow the speaker in a vertical crop), and for diarization (“who spoke when”) when faces are visible.

Neither signal alone is enough. Audio alone says that someone is speaking, but not which face. Video alone sees lips move, but people also move their lips when they chew, laugh, smile or sing along, and a speaker’s face may be partly hidden. The answer is to check whether the lip motion and the sound are in sync. An old and simple measure of sync is the correlation between the loudness of the audio and the opening of the mouth over a short time window:

\[ r_{i,t} = \frac{\sum_{\tau \in \mathcal{W}_t} (a_\tau - \bar{a})(m_{i,\tau} - \bar{m}_i)}{\sqrt{\sum_{\tau \in \mathcal{W}_t} (a_\tau - \bar{a})^2 \; \sum_{\tau \in \mathcal{W}_t} (m_{i,\tau} - \bar{m}_i)^2}} \tag{211.7} \]
Read the symbols
\(a_\tau\)
audio loudness (energy envelope) at video frame \(\tau\)
\(m_{i,\tau}\)
mouth opening of face \(i\) at frame \(\tau\), from a face crop
\(\mathcal{W}_t\)
a window of \(W\) frames centered on frame \(t\); \(\bar{a}, \bar{m}_i\) are the means over the window
\(r_{i,t}\)
Pearson correlation between \(-1\) and 1; high when the mouth opens exactly when the sound is loud

Face \(i\) is called “speaking” at frame \(t\) if \(r_{i,t}\) is above a threshold. The window length is the key trade-off. A short window has too few syllables to measure sync reliably. A long window is reliable inside a speech turn but blurs the boundaries where one person stops and another starts. Figure 4 shows both effects.

audiolip motiontruthpredicted speaking
Figure 4. A synthetic 20-second conversation at 25 fps between face A and face B. A speaks at 1 to 6 s and 13 to 16 s; B at 7 to 12 s and 17 to 19.5 s. Traps: A chews (lips move, no sound) from 8 to 11.5 s; B laughs silently from 13.5 to 15.5 s; there is music with nobody speaking from 12 to 13 s. Thresholds: 0.15 on mean lip motion or mean loudness, 0.5 on the sync correlation.
How to read Figure 4

“Audio only” gives the same answer for both faces, so it is wrong for one of them whenever someone talks, and the music also fools it: about 0.53 to 0.56 frame accuracy. “Video only” is fooled by chewing and laughing: best about 0.87 at \(W = 5\). The sync score ignores both traps, because chewing is not in time with the sound. With \(W = 1\) it is useless (a correlation needs several frames); accuracy rises to 0.948 at \(W = 15\) (0.6 s), then falls again for long windows (0.823 at \(W = 101\)), because the window crosses the speaker changes.

TalkNet: learned audio-visual fusion

Real systems learn the sync from data. TalkNet (Tao et al., 2021) has the following parts [Certain]:

  • Visual encoder. Face crops of \(112 \times 112\) pixels at 25 fps go through a 3D convolution and a ResNet-18, then a temporal network (V-TCN: five residual blocks of depthwise-separable 1D convolutions) with a receptive field of 21 frames (840 ms).
  • Audio encoder. 13-dimensional MFCC features at 100 frames per second go through a ResNet-34 with squeeze-and-excitation blocks; dilated convolutions bring the audio down to the video’s time resolution. Its receptive field is 189 audio frames (1,890 ms). Both encoders output 128-dimensional features per video frame.
  • Cross-attention. Two attention layers: audio features attend to visual features, and visual features attend to audio features (8 heads each). This is where the model checks whether the sound and the lips agree.
  • Self-attention backend. One transformer layer over the joined audio-visual sequence models long-term evidence (“this person has been talking for 3 seconds”), then a linear layer gives a speaking probability for every frame.

In the audio-to-visual cross-attention, the audio features ask the questions and the visual features answer:

\[ \mathbf{F}_{a \to v} = \softmax\!\left( \frac{(\mathbf{F}_a \mathbf{W}^Q)(\mathbf{F}_v \mathbf{W}^K)^\top}{\sqrt{d}} \right) \mathbf{F}_v \mathbf{W}^V \tag{211.8} \]
Read the symbols
\(\mathbf{F}_a, \mathbf{F}_v\)
audio and visual features of one face track, shape \(T \times 128\) (one row per video frame)
\(\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V\)
learned projections for queries (from audio), keys and values (from video)
\(d\)
dimension of one attention head

The paper reports 92.3 mAP on the AVA-ActiveSpeaker validation set (3.5 points above the previous best, MAAS-TAN at 88.8) and 90.8 on the test set, and an average F1 of 96.2 on Columbia ASD. AVA-ActiveSpeaker has about 3.65 million labeled frames and about 38.5 hours of face tracks. [Certain] Two ablations of TalkNet match Figure 4:

TalkNet input length (validation mAP)5 frames102550100variable, 1 to 10 s
mAP75.282.887.989.589.492.3

Short inputs are much worse: 0.2 s of video gives 75.2 mAP, 1 s gives 87.9. And removing both attention modules drops the result from 92.3 to 90.0. [Certain] Long-term context and audio-visual interaction both matter.

Algorithm 3: active speaker detection in a pipeline
  1. input: decoded frames (25 fps) and audio (16 kHz) of one shot
  2. detect faces in every frame (or every few frames); link boxes over time into face tracks with a tracker; never link across a shot boundary
  3. for each track: crop and resize faces to \(112 \times 112\); cut the audio of the same time range; compute MFCCs
  4. run the ASD model on (face crops, audio) // one score per face per frame
  5. smooth scores over time (for example a median over 5 to 10 frames); threshold
  6. join with ASR word times: each word goes to the face that is speaking at that time, or to “off-screen” if none

Streaming vs batch

The same models can run in two very different ways.

  • Batch (offline): process whole files, for example every upload of the day. Latency is minutes to hours, which is fine for archives, search indexes and ad targeting of video-on-demand. The system can use big batches, cheap interruptible machines and full-video context (for example, scenes and video-level labels).
  • Streaming (online): process live video, for example sports, news or live streams, where labels are needed within seconds (moderation, live ad breaks). The video is cut into chunks; each chunk must be processed before the next arrives, on average, or the system falls behind.

How many GPUs?

Let \(\lambda\) be how many seconds of video arrive per second of wall-clock time (100 live channels give \(\lambda = 100\)), and \(c\) the GPU-seconds of work per second of video for the whole pipeline (you measure it per stage on your own hardware). The GPU count is

\[ N_{\text{GPU}} = \left\lceil \frac{\lambda\, c}{\rho} \right\rceil, \qquad \text{GPU-hours per video-hour} = \frac{c}{\rho} \tag{211.9} \]
Read the symbols
\(\lambda\)
arrival rate: seconds of video per second (equal to the number of live streams)
\(c\)
cost: GPU-seconds needed to process one second of video, summed over all stages
\(\rho\)
target utilization of each GPU, between 0 and 1

A batch system can run close to full utilization (\(\rho\) near 0.95), because nobody waits for a single result. A streaming system cannot, because of queueing.

Latency of a streaming pipeline

The time from “a frame was filmed” to “its label is ready” has four parts:

\[ L = T_{\text{chunk}} + T_{\text{look}} + S + W_q, \qquad S = c\, T_{\text{chunk}}, \qquad W_q \approx \frac{\rho}{1 - \rho}\, S \tag{211.10} \]
Read the symbols
\(T_{\text{chunk}}\)
chunk length: the first frame of a chunk waits until the whole chunk has arrived
\(T_{\text{look}}\)
lookahead: extra future frames that a stage needs, for example 1 to 3 s for TransNet V2’s window
\(S\)
service time: GPU time to process one chunk
\(W_q\)
average waiting time in the queue; the formula is the classic single-server (M/M/1) result, a rough guide

The queue term grows without limit as \(\rho \to 1\): at \(\rho = 0.5\) the wait equals one service time, at 0.8 it is 4 service times, at 0.95 it is 19. Real systems with many GPUs sharing one queue wait less than this formula says, but the shape is the same: the last few percent of utilization cost a lot of latency. Figure 5 lets you explore it.

Figure 5. A toy model of a streaming pipeline (formulas 211.9 and 211.10), with a fixed lookahead of 3 s (the worst case of TransNet V2’s sliding window at 25 fps). Top: latency against utilization for the chosen chunk length and cost; the dot is the current setting. Bottom: the four parts of the latency. The cost \(c\) is an input you measure; the figure does not assume any GPU model.
How to read Figure 5

With the defaults (100 streams, \(c = 0.1\), \(\rho = 0.8\), 10 s chunks) you need 13 GPUs and labels arrive after 18 s. Push \(\rho\) to 0.95: you save GPUs (11), but the queue wait jumps from 4 s to 19 s. Shorter chunks cut the chunk wait and the service time, but every chunk pays fixed costs in a real system (model calls, a keyframe at least, network), so very short chunks cost more GPU per second of video. The readout also shows the batch cost at \(\rho = 0.95\) for comparison.

Worked example: 100 live channels

Suppose you measured \(c = 0.1\) GPU-seconds per second of video for the whole pipeline. With \(\lambda = 100\) streams the offered load is \(100 \times 0.1 = 10\) GPUs of work. At \(\rho = 0.8\): \(N = \lceil 10 / 0.8 \rceil = \lceil 12.5 \rceil = 13\) GPUs.

With 10-second chunks, \(S = 0.1 \times 10 = 1\) s, and \(W_q \approx 0.8/0.2 \times 1 = 4\) s. With 3 s of lookahead, \(L = 10 + 3 + 1 + 4 = 18\) s.

The same work in batch at \(\rho = 0.95\) needs \(\lceil 10/0.95 \rceil = 11\) GPUs and \(0.1/0.95 = 0.105\) GPU-hours per video-hour, versus \(0.1/0.8 = 0.125\) for streaming: about 16% less. Over a day, 100 channels in streaming use \(100 \times 24 \times 0.125 = 300\) GPU-hours. Multiply by your own price per GPU-hour.

Where the cost goes

The cost \(c\) is a sum over stages, and the stages scale differently:

  • Decode scales with frames and resolution. Use hardware decoders, and decode at low resolution for shot detection (\(48 \times 27\) is enough for TransNet V2).
  • Shot detection is cheap per frame because the frames are tiny.
  • VLM captions scale with the number of shots and keyframes, not with frames. This is usually the largest cost, so the keyframe policy (how many per shot, at what resolution) is the main knob (session 199 shows how to budget one VLM call).
  • ASR scales with audio length; it can skip music and silence with a voice activity detector.
  • ASD scales with the number of face tracks times their length: a crowd scene costs much more than a news anchor. Cap the number of tracks per shot (for example, the largest faces only).

Monitoring

A video pipeline can fail quietly: a decoder change drops frames, a new camera style breaks the shot detector, or a model update shifts label rates. Watch four groups of signals:

  • System: throughput (video-hours processed per hour), streaming lag (wall-clock time minus media time of the last labeled frame; it must not grow), p50 and p95 latency per stage, queue lengths, GPU utilization, error and retry counts, decode failures.
  • Data: input resolution, frame rate, audio presence, and duration distributions. A new source with 60 fps or no audio changes the cost and the models’ inputs.
  • Model outputs: shots per minute (a sudden change often means a broken detector), caption length, the share of each label, the confidence histogram.
  • Quality: a small random sample sent to human reviewers every day, plus all safety flags near the threshold. This is the only signal that measures accuracy directly.

A standard number for “has the label distribution changed?” is the population stability index (PSI) between a reference period \(q\) and today \(p\):

\[ \mathrm{PSI} = \sum_{c} (p_c - q_c) \ln \frac{p_c}{q_c} \tag{211.11} \]
Read the symbols
\(q_c\)
share of label (or bin) \(c\) in the reference period, for example last month
\(p_c\)
share of label \(c\) today; both must be above 0 (add a small constant to empty bins)

PSI is a symmetric form of the KL divergence: \(\mathrm{PSI} = \KL(p \,\|\, q) + \KL(q \,\|\, p)\). A common rule of thumb reads PSI below 0.1 as stable, 0.1 to 0.25 as a moderate change and above 0.25 as a large change. [Likely]

Worked example: a drift alert

Last month the video labels were Sports 0.40, News 0.30, Entertainment 0.20, Other 0.10. Today they are 0.25, 0.30, 0.25, 0.20. The terms are \((0.25 - 0.40)\ln(0.25/0.40) = 0.0705\), \(0\) for News, \((0.25 - 0.20)\ln(1.25) = 0.0112\), and \((0.20 - 0.10)\ln 2 = 0.0693\). PSI = 0.151: a moderate change. It may be real (the football season ended) or a bug (a new source that the classifier calls “Other”). Look at samples before you retrain.

Code: a toy pipeline in PyTorch

The code below runs on a CPU in about a second and gives the same output on every run. It builds a synthetic video of four colored shots with a cut, a dissolve and another cut, and then (1) finds boundaries with histogram differences (211.1, 211.2) at gaps 1 and 8, (2) aggregates shot labels with the duration-weighted mean and the max (211.6), and (3) computes the audio-visual sync score (211.7) for a speaking face and a chewing face.

import torch
import torch.nn.functional as F
torch.manual_seed(0)

# 1) A toy video: 4 shots of different colors, a hard cut at 40, a dissolve 80..95, a cut at 130
T, H, W = 160, 27, 48                      # TransNet V2 also works on 48x27 frames
colors = torch.tensor([[0.8, 0.2, 0.2], [0.2, 0.7, 0.3], [0.2, 0.3, 0.9], [0.9, 0.9, 0.3]])
def shot_of(t):
    return 0 if t < 40 else 1 if t < 80 else 2 if t < 130 else 3
frames = torch.empty(T, 3, H, W)
for t in range(T):
    c = colors[shot_of(t)]
    if 80 <= t < 96:                       # dissolve from shot 1 to shot 2
        a = (t - 79) / 17
        c = (1 - a) * colors[1] + a * colors[2]
    frames[t] = c[:, None, None] + 0.05 * torch.randn(3, H, W)
frames = frames.clamp(0, 1)

# 2) Color histograms, 8 bins per channel, normalized to sum 1
def hist(f, bins=8):
    h = torch.stack([torch.histc(f[ch], bins=bins, min=0, max=1) for ch in range(3)])
    return (h / h.sum()).flatten()          # (24,)
Hs = torch.stack([hist(f) for f in frames])  # (T, 24)

# 3) Frame difference d_t^(k) = 1/2 * sum |h_t - h_{t-k}|  (formula 211.1)
def diff(Hs, k):
    d = torch.zeros(len(Hs))
    d[k:] = 0.5 * (Hs[k:] - Hs[:-k]).abs().sum(1)
    return d
def detect(d, th, k):
    above = (d > th).int().tolist() + [0]
    out, start = [], None
    for t, a in enumerate(above):
        if a and start is None: start = t
        if not a and start is not None:
            out.append((start + t - 1) / 2 - (k - 1) / 2); start = None
    return out
d1, d8 = diff(Hs, 1), diff(Hs, 8)
print("k=1:", detect(d1, 0.3, 1), " max inside dissolve:", round(d1[81:96].max().item(), 3))
print("k=8:", detect(d8, 0.3, 8))

# 4) Shot labels -> video labels (formula 211.6)
dur = torch.tensor([12., 8, 20, 3, 15, 6, 25, 11])
p = torch.tensor([[0.90, 0.80, 0.85, 0.20, 0.90, 0.30, 0.88, 0.70],    # Sports
                  [0.02, 0.05, 0.03, 0.92, 0.04, 0.02, 0.03, 0.05]])   # injury risk
w = dur / dur.sum()
print("duration-weighted:", [round(x, 3) for x in (p @ w).tolist()],
      " max:", [round(x, 3) for x in p.max(1).values.tolist()])

# 5) Audio-visual sync: windowed Pearson correlation (formula 211.7)
def win_corr(a, m, W=15):
    pad = W // 2
    A = F.pad(a[None], (pad, pad), mode="replicate")[0].unfold(0, W, 1)
    M = F.pad(m[None], (pad, pad), mode="replicate")[0].unfold(0, W, 1)
    A, M = A - A.mean(1, keepdim=True), M - M.mean(1, keepdim=True)
    return (A * M).sum(1) / (A.norm(dim=1) * M.norm(dim=1) + 1e-8)
t = torch.arange(250.)
speech = ((t >= 50) & (t < 150)).float()
env = speech * (0.5 - 0.5 * torch.cos(2 * torch.pi * t / 5)) * (0.3 + 0.7 * torch.rand(250))
audio = env + 0.08 * torch.rand(250)
lips_speaker = 0.9 * env + 0.06 * torch.rand(250)
lips_chewing = 0.6 * (0.5 - 0.5 * torch.cos(2 * torch.pi * t / 16)) + 0.06 * torch.rand(250)
for name, m in [("speaker", lips_speaker), ("chewing", lips_chewing)]:
    r = win_corr(audio, m)
    print(name, "mean sync while speech:", round(r[60:140].mean().item(), 3),
          " lip motion:", round(m[60:140].mean().item(), 3))

Output:

k=1: [40.0, 130.0]  max inside dissolve: 0.161
k=8: [40.0, 88.0, 130.0]
duration-weighted: [0.798, 0.06]  max: [0.9, 0.92]
speaker mean sync while speech: 0.996  lip motion: 0.323
chewing mean sync while speech: 0.018  lip motion: 0.328

Read it: with \(k = 1\) the dissolve is missed (the largest step inside it is only 0.161, below the threshold 0.3), while \(k = 8\) finds it at frame 88 (the true middle is 87.5). The safety score is 0.06 by duration-weighted mean but 0.92 by max. Most important, the speaking face and the chewing face have almost the same lip motion (0.323 and 0.328), so a video-only rule cannot tell them apart; the sync score separates them completely (0.996 vs 0.018).

Where this is used

ApplicationPipeline stages that matterTypical mode
Contextual ads on video-on-demandShots, keyframe captions, IAB topic labels by time share, safety by max; labels per ad breakBatch, at upload time
Live sports and news ad breaksStreaming shot detection, fast small classifiers, safety labels of the last minute before the breakStreaming, seconds
Content moderationSafety labels by max, human review queue ordered by risk, appeal logsBoth: fast pass, then a slower re-check
Archive search and Video-RAGPer-shot captions, transcripts and embeddings in a vector index (session 210)Batch
Key-moment detection and highlightsShot boundaries as cut points, audio events (crowd noise), temporal grounding (session 207)Streaming for live, batch for VOD
Speaker-attributed subtitles and reframingFace tracks, active speaker detection, ASR word timesBatch, or streaming with a few seconds of delay
Duplicate and re-upload detectionShot-level fingerprints and embeddings (session 209)Batch at upload
Action recognition in video analyticsClips cut at shot boundaries, so a clip never mixes two shots (session 203)Both

Common mistakes

  • Running expensive models per frame. Captions and labels per frame are mostly repeats. Work per shot (or sub-shot), and only add frames where motion matters.
  • Tuning the shot threshold on one kind of content. A threshold tuned on movies fails on sport, music videos or screen recordings. Use a learned detector and evaluate per content type.
  • Averaging safety labels. One 3-second harmful shot disappears in a duration-weighted mean (Figure 3: 0.92 becomes 0.060). Use the max (or “any shot above threshold”) for safety, and averages only for topics.
  • Linking face tracks across shot cuts. A tracker can jump from one person to another at a cut. Always reset tracks at shot boundaries.
  • Trusting video-only speaker detection. Chewing, laughing and singing along move the lips. Use an audio-visual model, and check audio-video sync first.
  • Running a streaming system at 95% utilization. The queue wait explodes (formula 211.10). Keep headroom, and autoscale on lag, not only on GPU utilization.
  • Not storing intermediate results with model versions. Without them, every label change means re-decoding and re-running the whole archive.
  • Monitoring only system metrics. A pipeline can be fast and wrong. Track label shares (PSI), shots per minute, and a daily human-reviewed sample.

Interview questions

What is the difference between a shot and a scene, and why does a pipeline need both?basic
A shot is one continuous camera take, bounded by cuts or gradual transitions. A scene is a group of shots that share place, time and story, such as an A, B, A, B dialogue. Shots are precise and easy to detect, so they are the unit of work (one caption per shot) and the safe cut points for ads and clips. Scenes are better units for summaries, chapters and topic labels, because one shot is often too short to understand alone.
Why does a frame-difference shot detector miss dissolves and fire on flashes? How do you fix it?core
A dissolve spreads the change over many frames, so each neighbor difference is small and stays under a threshold that ignores noise. A flash changes one or two frames a lot and then goes back, so it looks like two cuts. Comparing frames further apart (gap \(k\)) catches dissolves but turns fast motion into false cuts; a before/after check removes flashes. Each hand rule fixes one case and breaks another, so production systems use a learned detector such as TransNet V2, which sees 100 frames and learns all these patterns, trained with many synthetic dissolves.
Describe TransNet V2.core
A small 3D CNN on \(48 \times 27\) frames. It takes 100-frame windows and uses the middle 50 predictions. Its DDCNN cells use temporal dilations 1, 2, 4, 8, with each 3D convolution split into 2D spatial and 1D temporal parts, plus batch norm. It also computes cosine similarities of RGB histograms and learned features between each frame and its 50 neighbors on each side. One head predicts the middle frame of each transition (used at test time with threshold 0.5), another predicts all transition frames (training only). It is trained on synthetic cuts and dissolves built from TRECVID IACC.3 and ClipShots shots plus real ClipShots transitions, and reaches F1 77.9 on ClipShots, 96.2 on BBC and 93.9 on RAI.
You have per-shot probabilities for “Automotive” and for “violence”. How do you get video-level labels?core
Use different rules. Topics such as Automotive answer “what is the video mostly about”, so use a duration-weighted mean or the time share of confident shots; a short car shot in a cooking show should not make it a car video. Safety labels answer “is there any harmful moment”, so use the max over shots (or any shot above a threshold); averaging would hide a short violent shot. Apply per-label thresholds tuned on labeled data, and let safety override topic targeting when they conflict.
How does active speaker detection work, and why do you need both audio and video?core
For each face track and each frame, the model predicts whether that face is speaking. Audio alone knows that someone speaks but not who. Video alone sees lip motion but cannot tell speaking from chewing, laughing or singing along. The key signal is synchrony: mouth opening and sound energy move together for the real speaker. TalkNet encodes face crops (ResNet-18 + temporal conv) and MFCCs (ResNet-34 with SE), lets audio attend to video and video attend to audio with cross-attention, and uses self-attention over time for long-term evidence. It reports 92.3 mAP on the AVA-ActiveSpeaker validation set.
TalkNet gets 75.2 mAP with 5-frame inputs and 92.3 with variable 1 to 10 s inputs. Why, and what does it mean for a streaming system?advanced
Speaking is a pattern over syllables, and a syllable is roughly a fifth of a second, so 5 frames (0.2 s) hold too little evidence to measure sync or to tell speech from other mouth motion. Long inputs let self-attention gather evidence over a whole turn. In streaming this means a trade: either wait for a few seconds of context (more latency, better labels) or decide on short windows and accept errors, maybe correcting them later when more context has arrived. Long windows also blur speaker changes, so boundaries need care (smoothing, or a model trained on variable lengths).
Estimate the GPUs and latency for labeling 100 live channels.advanced
First measure the pipeline cost \(c\) in GPU-seconds per second of video. Say \(c = 0.1\). The offered load is \(\lambda c = 10\) GPUs of work. At 80% utilization you need \(\lceil 12.5 \rceil = 13\) GPUs. With 10 s chunks, service time is 1 s and the queue wait is about \(\rho/(1-\rho) \cdot S = 4\) s, so with 3 s of shot-detector lookahead the latency is about 18 s. Running at 95% saves two GPUs but makes the queue wait about 19 s. Then add headroom for traffic spikes, and autoscale on lag.
When would you choose streaming over batch, and what changes in the design?core
Streaming when labels are needed within seconds: live moderation, live ad breaks, live highlights. Then you process chunks, keep utilization lower to bound queueing, use small models, cap shot length so shots close quickly, and accept that scene and video-level context is not available yet. Batch when minutes or hours are fine: archives, VOD ads, search indexes. Then you use whole-video context, large batches, higher utilization and cheaper interruptible machines. Many systems run both: a fast streaming pass and a later batch pass that refines the labels.
How do you monitor a video labeling pipeline?core
System metrics: throughput, streaming lag (it must not grow), per-stage p50 and p95 latency, queue length, errors and decode failures. Input metrics: resolution, fps, duration, audio presence per source. Output metrics: shots per minute, caption length, label shares, confidence histograms, with drift scores such as PSI against a reference period. Quality: a daily random sample reviewed by humans, plus all near-threshold safety cases. Alert on sudden changes, and keep model versions with every stored output so you can find which change caused a shift.
Why should shot detection run on tiny frames, and why decode only once?basic
A cut changes the whole frame, so it is visible at very low resolution; TransNet V2 uses \(48 \times 27\) frames, which makes it very cheap per frame. Decoding compressed video is expensive and often costs as much as the models, so decoding once and sharing frames (low resolution for shot detection, a few full-resolution keyframes for the VLM, the audio track for ASR and ASD) avoids paying it several times.
Your label shares changed suddenly overnight. How do you debug it?advanced
Compute PSI per label and per source to find where the change is. Check whether a model, a prompt, a decoder or a preprocessing step changed (stored model versions help). Check input metrics: a new source, new resolution or missing audio. Check shots per minute: a broken shot detector changes every downstream count. Then look at real samples of the shifted labels. Only if the content truly changed (a new season, a news event) do you accept it or retrain; if a component changed, roll it back and re-run only the affected stages from stored intermediate outputs.

Exercises

  1. Two 4-bin histograms are \(h_{t-1} = (0.4, 0.3, 0.2, 0.1)\) and \(h_t = (0.1, 0.2, 0.3, 0.4)\). Compute \(d_t^{(1)}\) with formula (211.1).
  2. A detector finds 9 boundaries in a clip with 10 true transitions; 7 detections are correct. Compute precision, recall and F1.
  3. With gap \(k = 8\), a cut at frame 100 makes a run of scores above the threshold over frames 100 to 107. Where does formula (211.2) place the detection?
  4. In Figure 3, compute the duration-weighted mean of “Food & Drink” (probabilities 0.05, 0.10, 0.10, 0.05, 0.70, 0.10, 0.05, 0.10) and its max. Which rule would you use for this label?
  5. With TransNet V2’s reference windowing at 30 fps, what is the largest lookahead in seconds before a boundary decision is final?
  6. A streaming pipeline has \(c = 0.2\), chunks of 5 s, \(\rho = 0.9\) and 3 s of lookahead. Compute \(S\), \(W_q\) and \(L\) with formula (211.10). How many GPUs for 50 streams?
Answerscheck
1. The absolute differences are 0.3, 0.1, 0.1, 0.3; their sum is 0.8; half is \(d = 0.4\). 2. \(P = 7/9 = 0.778\), \(R = 7/10 = 0.7\), \(F_1 = 2 \cdot 0.778 \cdot 0.7 / 1.478 = 0.737\). 3. The middle of the run is \((100 + 107)/2 = 103.5\); minus \((8-1)/2 = 3.5\) gives frame 100, the cut. 4. Weighted: \((0.6 + 0.8 + 2 + 0.15 + 10.5 + 0.6 + 1.25 + 1.1)/100 = 0.17\); max 0.70. Food is a topic, so use the weighted mean or the time share (0.15): it is a secondary topic, not the main one. 5. A kept prediction can need up to 74 future frames (it may be the first of the 50 kept frames, and the window extends 25 frames after the last kept one): \(74/30 = 2.47\) s. 6. \(S = 0.2 \times 5 = 1\) s; \(W_q = 0.9/0.1 \times 1 = 9\) s; \(L = 5 + 3 + 1 + 9 = 18\) s. GPUs: \(\lceil 50 \times 0.2 / 0.9 \rceil = \lceil 11.1 \rceil = 12\).

Key takeaways

  • A production pipeline decodes once, cuts the video into shots, does the expensive work (captions, embeddings, OCR, ASR, ASD) once per shot, then aggregates to scenes and videos, and stores every intermediate result with its model version.
  • Hand-made shot detectors (histogram differences, 211.1) handle cuts but fail on dissolves, flashes and fast motion; TransNet V2 learns them from 100-frame windows of \(48 \times 27\) frames and reaches F1 96.2 on BBC and 77.9 on ClipShots.
  • Aggregate topics by time (weighted mean or time share) and safety by max (211.6); one short harmful shot must not be averaged away.
  • Active speaker detection needs audio-visual sync, not lip motion alone; TalkNet fuses them with cross-attention and gains a lot from long context (75.2 mAP at 0.2 s vs 92.3 with 1 to 10 s inputs).
  • Streaming trades cost for latency: queue wait grows as \(\rho/(1-\rho)\), so live systems keep headroom, while batch runs near full utilization; monitor lag, per-stage latency, label drift (PSI) and a human-reviewed sample.

Key formulas

\[ d_t^{(k)} = \tfrac12 \textstyle\sum_b |h_t(b) - h_{t-k}(b)|, \qquad F_1 = \frac{2PR}{P+R}, \qquad \bar{p}_c = \textstyle\sum_i w_i p_{i,c}, \quad p^{\max}_c = \max_i p_{i,c} \]
\[ r_{i,t} = \mathrm{corr}_{\tau \in \mathcal{W}_t}(a_\tau, m_{i,\tau}), \qquad N_{\text{GPU}} = \left\lceil \frac{\lambda c}{\rho} \right\rceil, \qquad L = T_{\text{chunk}} + T_{\text{look}} + S + \frac{\rho}{1-\rho} S \]
\[ \mathrm{PSI} = \textstyle\sum_c (p_c - q_c) \ln \frac{p_c}{q_c} \]

References for this session

  1. Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
  2. Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  4. Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
  5. Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
  6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  7. Roth et al. (2019). AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection
  8. Rao, Xu, Xiong, Xu, Huang, Zhou & Lin (2020). A Local-to-Global Approach to Multi-modal Movie Scene Segmentation
  9. Chung & Zisserman (2016). Out of Time: Automated Lip Sync in the Wild
  10. PySceneDetect developers (2024). PySceneDetect documentation: scene detection algorithms (ContentDetector, AdaptiveDetector)
  11. IAB Tech Lab (2024). Content Taxonomy (versions up to 3.1)
  12. World Federation of Advertisers (GARM) (2022). GARM Brand Safety Floor + Suitability Framework
  13. Souček & Lokoč (2020). TransNet V2 reference code: inference/transnetv2.py (100-frame windows, step 50)
  14. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  15. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision