video fingerprinting
In this lab you build a video copy detector on a laptop and then attack it. You cut 45 reference clips and 630 attacked queries from three open movies, describe every frame with a perceptual hash, CLIP and the SSCD copy-detection model, match videos with a temporal line search that also finds speed and location, and measure everything with micro-AP. Then you find the weakest transform of each descriptor, fix it, and see where the weakness moves.
What you will learn
- How to turn a video into a sequence of frame descriptors: a 64-bit perceptual hash, a CLIP embedding and an SSCD copy-detection descriptor
- How to compare two videos with a frame similarity matrix, and how max, Chamfer and a temporal line search turn it into one score
- How a line search over speeds and offsets handles sped-up copies and reports where the copy sits in the reference
- How to measure copy detection honestly with micro-AP over all pairs, with distractor queries and one global threshold
- How to attack your own system (crop, resize, re-encode, speed, logo, flip, combinations) at several strengths and find the weakest transform
- Why score normalization and descriptor fusion help in some settings and hurt in others
- Why many false alarms are a label-policy question (the same shot a few seconds later), not a descriptor bug
Why this matters
In session 209 you saw the ideas: a fingerprint is a compact description of a video that stays the same when someone crops, shrinks, re-encodes, speeds up or brands the video, but changes when the content changes. In this lab you build one, end to end, and then you attack it. You will write per-frame descriptors, a temporal matcher that finds where a copy sits in time and at what speed, and an evaluation that gives one honest number, micro-AP. Then you apply seven attacks of your own, measure every one of them, find the weakest, fix it, and find the next weakest.
The task is called video copy detection: given a query video, find the reference videos that it was copied from. It is not the same as semantic search (session 208). Two different videos of a rabbit in a meadow are semantically similar but are not copies. A re-uploaded clip with a TV logo, a black-and-white filter and half the bitrate is a copy, even though it looks worse. Platforms run this at very large scale: to match uploads against a catalog of protected content, to find re-uploads of content that was already removed, and to remove near-duplicate clips from training data (session 187).
Everything here runs on a laptop CPU. The data is cut from three open movies of the Blender Foundation, the descriptors are a 64-bit perceptual hash, CLIP ViT-B/32 and SSCD (a ResNet-50 trained for image copy detection), and every number on this page comes from our own run of the code below, unless it is tagged as coming from a paper. In short, what you will find: the copy-detection descriptor (SSCD) beats the semantic one (CLIP) by a wide margin (micro-AP 0.931 against 0.478); the perceptual hash breaks completely under crop and mirror; SSCD’s weakest transform is heavy re-encoding; a simple fusion with the hash fixes that (0.969); and most of the remaining false alarms are not copies at all but the same scene a few seconds later.
Setup
You need Python 3.10 or newer, FFmpeg, and about 1 GB of disk. We used Python 3.12, torch 2.15, torchvision 0.29, transformers 5.23, SciPy 1.18 and FFmpeg 9.0 on an Apple-silicon laptop that was busy with other jobs; any recent versions should work.
python -m venv venv && source venv/bin/activate pip install torch torchvision transformers scipy pillow # three open movies (CC-BY 3.0, Blender Foundation) from the Internet Archive curl -L -o data/bbb.mp4 https://archive.org/download/BigBuckBunny_124/Content/big_buck_bunny_720p_surround.mp4 curl -L -o data/ed.mp4 https://archive.org/download/ElephantsDream/ed_1024_512kb.mp4 curl -L -o data/sintel.mp4 https://archive.org/download/Sintel/sintel-2048-stereo_512kb.mp4 # the SSCD descriptor (ResNet-50, 512-d), TorchScript file from the authors curl -L -o data/sscd.pt https://dl.fbaipublicfiles.com/sscd-copy-detection/sscd_disc_mixup.torchscript.pt
The three movies are Big Buck Bunny (2008), Elephants Dream (2006) and Sintel (2010), © Blender Foundation, released under the Creative Commons Attribution 3.0 license [Certain]. The files we used are 640 × 360, 426 × 240 and 564 × 240 pixels, all at 24 frames per second (read with ffprobe). Animated films are a good test because characters, colors and backgrounds repeat across a whole movie: many clips look alike without being copies.
The database and the queries. From each movie we cut 30 clips of 10 seconds, starting at second 15, 30, 45, …, 450 (so there is a 5-second gap between clips). The clips with even numbers (15 per movie, 45 in total) form the reference database. The clips with odd numbers are held out: they are never in the database, so queries made from them are distractors, queries with no true match. Every query is a 6-second excerpt that starts 2 seconds into its clip, with one attack applied. With 90 source clips and 7 attacks, there are 630 queries: 315 with exactly one true reference and 315 distractors.
| Attack | What it does (FFmpeg) | Why a pirate or a re-uploader does it |
|---|---|---|
| crop | keep the central 60% of the width and height (36% of the area) | remove borders and watermarks, zoom on the action |
| resize | a quarter of the width and height (for example 640 × 360 → 160 × 90) | small previews, thumbnails, low-quality re-uploads |
| reencode | H.264 at CRF 45 (about 32 kbit/s on average; the references are about 600 kbit/s) | repeated uploads and transcodes; strong block artifacts |
| speed | 1.5 times faster (setpts=PTS/1.5): 6 seconds become 4 | beat simple time-aligned matchers |
| logo | a red “CH 7” box at the top left and a semi-transparent news banner at the bottom (18% of the height) | re-broadcast, branding |
| flip | mirror left–right (hflip) | a classic trick against copyright filters |
| combo | crop 80%, logo and banner, 1.5 times faster, CRF 38 | real copies usually stack several edits |
The CRF (constant rate factor) is the quality knob of the x264 encoder: 18 is close to visually lossless, 51 is the worst. The reference clips are re-encoded at CRF 18 so that the database and the queries go through the same decoder.
The data script cuts the clips and applies the attacks. The logo is a PNG drawn with Pillow and laid over the video with FFmpeg’s overlay filter. Note the -t 6 before -i: it limits the input to 6 seconds, so the 1.5× query covers the same 6 seconds of story and lasts 4 seconds. (Placed after the input, it would limit the output and silently take 9 seconds of source. We made exactly this mistake in our first run and caught it with ffprobe.)
# make_data.py: reference clips and attacked queries (needs FFmpeg) import subprocess, json, os from concurrent.futures import ThreadPoolExecutor from PIL import Image, ImageDraw, ImageFont MOVIES = {"bbb": (640, 360), "ed": (426, 240), "sintel": (564, 240)} os.makedirs("work/ref", exist_ok=True); os.makedirs("work/qry", exist_ok=True) def logo_png(name, w, h): # a channel logo (top left) and a news banner (bottom) im = Image.new("RGBA", (w, h), (0, 0, 0, 0)); d = ImageDraw.Draw(im) d.rounded_rectangle([int(.03*w), int(.04*h), int(.30*w), int(.20*h)], radius=6, fill=(220, 30, 40, 255)) d.text((int(.05*w), int(.06*h)), "CH 7", font=ImageFont.load_default(size=int(.09*h)), fill="white") d.rectangle([0, int(.82*h), w, h], fill=(0, 0, 0, 170)) d.text((int(.03*w), int(.85*h)), "BREAKING: copied clip", font=ImageFont.load_default(size=int(.07*h)), fill="yellow") im.save(p := f"work/logo_{name}.png"); return p LOGO = {m: logo_png(m, *wh) for m, wh in MOVIES.items()} ENC = ["-c:v", "libx264", "-crf", "18", "-preset", "veryfast", "-an"] def crf(c): return ["-c:v", "libx264", "-crf", str(c), "-preset", "veryfast", "-an"] ATTACKS = {"crop": ("crop=iw*0.6:ih*0.6", ENC), "resize": ("scale=trunc(iw/8)*2:trunc(ih/8)*2", ENC), "reencode": (None, crf(45)), "speed": ("setpts=PTS/1.5", ENC), "logo": ("LOGO", ENC), "flip": ("hflip", ENC), "combo": ("COMBO", crf(38))} jobs, meta = [], {"ref": [], "qry": []} for m in MOVIES: for k in range(30): s, cid, in_db = 15 + 15 * k, f"{m}{k:02d}", k % 2 == 0 # even clips: database; odd clips: held out if in_db: meta["ref"].append({"id": cid}) jobs.append(["ffmpeg", "-y", "-ss", str(s), "-i", f"data/{m}.mp4", "-t", "10", *ENC, f"work/ref/{cid}.mp4"]) for a, (vf, enc) in ATTACKS.items(): out = f"work/qry/{cid}_{a}.mp4"; meta["qry"].append({"id": f"{cid}_{a}", "src": cid, "attack": a}) base = ["ffmpeg", "-y", "-ss", str(s + 2), "-t", "6", "-i", f"data/{m}.mp4"] # 6 s of source if vf == "LOGO": cmd = base + ["-i", LOGO[m], "-filter_complex", "[0:v][1:v]overlay=0:0", *enc, out] elif vf == "COMBO": # crop 80%, logo, 1.5x faster, CRF 38 cmd = base + ["-i", LOGO[m], "-filter_complex", "[0:v]crop=iw*0.8:ih*0.8[c];" "[1:v][c]scale2ref[l][c2];[c2][l]overlay=0:0,setpts=PTS/1.5", *enc, out] else: cmd = base + (["-vf", vf] if vf else []) + enc + [out] jobs.append(cmd) run = lambda c: subprocess.run(c, check=True, capture_output=True) with ThreadPoolExecutor(8) as ex: list(ex.map(run, jobs)) # sample every video at 2 frames per second into PNG files def frames(p): os.makedirs(p[:-4] + "_f", exist_ok=True); run(["ffmpeg", "-y", "-i", p, "-vf", "fps=2", p[:-4] + "_f/%03d.png"]) with ThreadPoolExecutor(8) as ex: list(ex.map(frames, [j[-1] for j in jobs])) json.dump(meta, open("work/meta.json", "w"))
At 2 frames per second a reference clip gives 20 frames and a normal query 12; the 1.5× queries (speed and combo) give 8. Because each query starts 2 seconds into its clip, query frame \(i\) shows the same moment as reference frame \(4 + \rho\, i\), where \(\rho\) is the speed factor (1 or 1.5). This is our ground truth for localization later.
Step 1: three per-frame descriptors
A descriptor is a fixed-size vector (or bit string) computed from one frame. We compare three kinds, because they fail in different ways.
Perceptual hash (pHash). A hand-made 64-bit code. Convert the frame to gray, shrink it to 32 × 32 pixels, take the two-dimensional discrete cosine transform (DCT, the transform JPEG uses), keep the 8 × 8 lowest frequencies, and write one bit per coefficient: 1 if it is above the median of the 64, else 0. Low frequencies describe the coarse layout of light and dark, which survives resizing and compression. We compare two hashes by their Hamming distance \(d_H\), the number of bits that differ, and turn it into a similarity between −1 and 1:
- Read the symbols
- \(\mathbf{h}, \mathbf{h}'\)
- two 64-bit hashes (bits 0 or 1)
- \(d_H\)
- Hamming distance: the number of positions where the bits differ (0 to 64)
- \(\tilde{\mathbf{h}}\)
- the same hash written with −1 and +1; then the similarity is a scaled dot product, so the same matcher code works for hashes and for float vectors
- \(s_{\text{pHash}}\)
- 1 for identical hashes, about 0 for unrelated frames (32 of 64 bits differ by chance), −1 for exactly opposite hashes
CLIP ViT-B/32 (session 166): a 512-dimensional image embedding trained to match images with captions. It knows what is in a frame (a rabbit, a meadow), not which exact frame it is. We include it as the semantic baseline, because many teams try it first.
SSCD (Pizzi et al., 2022): a ResNet-50 that outputs a 512-dimensional descriptor trained for image copy detection [Certain]. It is trained with a contrastive self-supervised loss: two strongly edited versions of the same image must be close, and all other images far apart. The paper changes the SimCLR recipe for this task and adds an entropy regularization term that spreads the descriptors out in the space, so that unrelated images are not accidentally close [Certain]. Its training edits are close to our attacks: crops, resizes, overlays, blur, JPEG compression and more [Likely]. We use the sscd_disc_mixup model, which was trained on the DISC21 image copy-detection dataset, with the preprocessing that the authors recommend: resize the short side to 288 pixels and normalize with the ImageNet mean and standard deviation [Certain].
- input: a video file, a frame rate \(f\) (2 per second here), a descriptor model
- decode the frames at times \(0, 1/f, 2/f, \dots\) // FFmpeg fps filter
- for each frame: preprocess exactly as the descriptor was trained // SSCD: short side 288, ImageNet normalization
- pHash: gray \(\to 32 \times 32 \to\) 2-D DCT \(\to 8 \times 8\) low frequencies \(\to\) 64 bits (above the median)
- CLIP or SSCD: one forward pass \(\to \mathbf{x} \in \R^{512} \to \mathbf{x} / \lVert \mathbf{x} \rVert\) (divide by the length: a unit vector)
- return the matrix of frame descriptors, one row per sampled frame, in time order // keep the order: the matcher needs it
# embed.py: per-frame descriptors for every reference clip and query (pHash, CLIP, SSCD) import glob, json, numpy as np, torch, torch.nn.functional as F from PIL import Image from scipy.fft import dctn from torchvision import transforms as T from transformers import CLIPModel, CLIPImageProcessor torch.manual_seed(0); torch.set_grad_enabled(False) def phash(im): # 64-bit perceptual hash, as in the imagehash library g = np.asarray(im.convert("L").resize((32, 32), Image.LANCZOS), dtype=np.float64) low = dctn(g, type=2, norm="ortho")[:8, :8] # the 8 x 8 lowest frequencies return (low > np.median(low)).flatten() # 64 bits sscd = torch.jit.load("data/sscd.pt").eval() # ResNet-50, 512-d sscd_tf = T.Compose([T.Resize(288), T.ToTensor(), T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]) clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").eval() clip_pp = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32") def describe(frame_dir): ims = [Image.open(p).convert("RGB") for p in sorted(glob.glob(frame_dir + "/*.png"))] h = np.stack([phash(im) for im in ims]) s = F.normalize(sscd(torch.stack([sscd_tf(im) for im in ims])), dim=-1) c = clip.get_image_features(pixel_values=clip_pp(images=ims, return_tensors="pt")["pixel_values"]) c = F.normalize(c if isinstance(c, torch.Tensor) else c.pooler_output, dim=-1) return h, s.numpy(), c.numpy() meta = json.load(open("work/meta.json")); out = {} for kind in ("ref", "qry"): for v in meta[kind]: out[v["id"]] = describe(f"work/{kind}/{v['id']}_f") np.savez("work/desc.npz", **{f"{k}|{n}": a for k, (h, s, c) in out.items() for n, a in (("phash", h), ("sscd", s), ("clip", c))})
The 675 videos (7,740 frames) took about 19 minutes on our busy laptop CPU, almost all of it in the two neural networks; pHash is nearly free. One frame from Sintel shows the three personalities at once. We compare query frame 0 of each attack with reference frame 4, which is the same moment:
| Attack (Sintel clip 24, same moment) | pHash: bits that differ (of 64) | pHash similarity (216.1) | CLIP cosine | SSCD cosine |
|---|---|---|---|---|
| resize to 1/4 | 2 | 0.94 | 0.909 | 0.741 |
| reencode, CRF 45 | 2 | 0.94 | 0.824 | 0.334 |
| logo and banner | 18 | 0.44 | 0.869 | 0.735 |
| crop 60% | 32 | 0.00 | 0.943 | 0.668 |
| flip | 32 | 0.00 | 0.973 | 0.901 |
The pHash survives resizing and heavy compression almost perfectly, but after a crop or a flip it is no closer to the original than a random frame. SSCD keeps all five above 0.3. CLIP gives high cosines to everything, and this is its problem, as Step 3 shows: CLIP also gives high cosines to other frames of the same movie.
Step 2: the similarity matrix and a temporal matcher
A video is a sequence of frame descriptors, so comparing two videos means comparing two sequences. The basic object is the frame similarity matrix: one row per query frame, one column per reference frame.
If the query is a copy of part of the reference, played at speed \(\rho\), its frame \(i\) matches reference frame \(o + \rho\, i\) for some start offset \(o\). So a copy shows up as a straight line of high values in \(S\), with slope \(\rho\). We compare four ways to turn \(S\) into one video score:
- Read the symbols
- \(\mathbf{q}_i, \mathbf{r}_j\)
- descriptor of query frame \(i\) and of reference frame \(j\) (unit vectors, or ±1 hash vectors scaled as in 216.1)
- \(n_q, n_r\)
- number of sampled frames in the query (12, or 8 at 1.5×) and in the reference (20)
- max
- the single best frame pair: the rule of the VSC 2022 descriptor track [Certain]; one lucky frame decides
- Chamfer
- every query frame finds its best reference frame anywhere, and we average; the order of frames is ignored. ViSiL uses Chamfer similarity to aggregate [Certain]
- \(\rho, o\)
- a speed (slope) and a start offset in reference frames; \(\lfloor x \rceil\) rounds to the nearest integer (halves go up)
- \(I_{\rho,o}\)
- the query frames whose matching reference frame falls inside the reference; we require at least half of \(\min(n_q, n_r/\rho)\) frames, so a line that just touches a corner cannot win
- \(\mathcal{P}\)
- the speeds we try: \(\{0.5, \tfrac{2}{3}, 1, 1.5, 2\}\); “temporal, \(\rho = 1\)” tries only normal speed
- pool (not in the formula)
- the simplest baseline: average all frame descriptors of a video into one vector and take a cosine; no matrix at all
The temporal matcher is a small version of temporal Hough voting, an idea from earlier video copy-detection systems (for example Douze, Jégou and Schmid, 2010 [Likely]): every (speed, offset) pair is a candidate line, and the line with the highest average similarity wins. It gives two things the other scores cannot give: the speed \(\rho\) and the location \(o\) of the copy inside the reference. Learned matchers such as ViSiL go further: they run a small CNN over \(S\) to learn which patterns (lines, broken lines, blocks) mean “copy” [Certain]; the VSC 2022 baseline localizes copies with a temporal network over \(S\) [Certain].
- input: similarity matrix \(S\) of shape \((n_q, n_r)\), speeds \(\mathcal{P}\), minimum coverage \(c = 0.5\)
- best \(\leftarrow (-\infty, \text{none}, \text{none})\)
- for each \(\rho \in \mathcal{P}\):
- for each integer offset \(o\) from \(-\lceil \rho (n_q - 1) \rceil\) to \(n_r - 1\): // vectorized in practice
- \(j_i \leftarrow \lfloor o + \rho i \rceil\) for every query frame \(i\); keep the \(i\) with \(0 \le j_i \lt n_r\)
- if fewer than \(c \cdot \min(n_q, n_r / \rho)\) frames are kept: skip
- \(L \leftarrow\) mean of \(S_{i, j_i}\) over the kept frames // formula (216.4)
- if \(L \gt\) best score: best \(\leftarrow (L, \rho, o)\)
- return best // score, speed, and where the copy starts in the reference
# fp.py: the matchers and micro-AP import numpy as np, torch RHOS = [0.5, 2 / 3, 1.0, 1.5, 2.0] def to_mat(x, d): # pHash bits -> +-1/8, so a dot product = 1 - 2 * Hamming / 64 x = torch.from_numpy(np.asarray(x, dtype=np.float32)) return (2 * x - 1) / 8.0 if d == "phash" else x def temporal(S, rhos=RHOS, min_cover=0.5): # best line j = o + rho * i through S, formula (216.4) nq, nr = S.shape; i = torch.arange(nq, dtype=torch.float32); best = (-9.0, None, None) for rho in rhos: o = torch.arange(-int(rho * (nq - 1)) - 1, nr) # every integer offset J = torch.floor(o[:, None] + rho * i[None, :] + 0.5).long() # round half up (torch.round rounds half to even) ok = (J >= 0) & (J < nr) vals = S[i.long()[None, :].expand_as(J), J.clamp(0, nr - 1)] * ok n = ok.sum(1); sc = vals.sum(1) / n.clamp(min=1) sc[n < min_cover * min(nq, nr / rho)] = -9.0 # the line must cover enough frames k = int(sc.argmax()) if sc[k] > best[0]: best = (float(sc[k]), rho, int(o[k])) return best def video_scores(S): t = temporal(S) return dict(maxmax=S.max().item(), chamfer=S.max(1).values.mean().item(), temporal1=temporal(S, rhos=[1.0])[0], temporal=t[0], rho=t[1], off=t[2]) def ap(y, s): # micro-AP of one ranked list of (query, reference) pairs, formula (216.5) o = np.argsort(-np.asarray(s), kind="stable"); y = np.asarray(y, float)[o] prec = np.cumsum(y) / np.arange(1, len(y) + 1) return float((prec * y).sum() / y.sum()) # quick self-test: a perfect line with slope 1.5 starting at reference frame 4 S = torch.zeros(8, 20) for i in range(8): S[i, int(np.floor(4 + 1.5 * i + 0.5))] = 1.0 print(temporal(S), temporal(S, rhos=[1.0])) # (1.0, 1.5, 4) (0.375, 1.0, 5)
A detail that cost us a bug: torch.round rounds halves to the nearest even number (2.5 becomes 2), but JavaScript’s Math.round and most people round halves up. With \(\rho = 1.5\) half values appear on every second frame, so the page’s figure and the Python code disagreed until both used “add 0.5, then floor”.
The speed query has 8 frames. With the true line \(\rho = 1.5\), \(o = 4\), the reference frames are \(\lfloor 4 + 1.5 i \rceil\) for \(i = 0..7\): 4, 6, 7, 9, 10, 12, 13, 15. The similarities on these cells (rounded to two decimals, as Figure 2 shows them) are 0.95, 0.71, 0.93, 0.90, 0.97, 0.94, 0.97 and 0.82. Their mean is \(7.19 / 8 = 0.899\): this is \(L(1.5, 4)\).
A matcher that only tries normal speed must use \(\rho = 1\). Its line \(o = 4\) visits frames 4 to 11; after the first few frames it falls behind the action, and the mean drops to 0.565. Its best offset is \(o = 5\) with 0.751. So without the speed search the score of a true copy falls from 0.899 to 0.751, and the estimated location is wrong. The Chamfer score of this pair is 0.905, a little higher than the line, because each query frame may pick its best reference frame freely, with no line constraint.
How to read Figure 2
It opens on the worked example: Sintel 24, speed attack, SSCD, with \(\rho = 1\) and \(o = 4\). The circles leave the dark cells after a few rows, and the line score is 0.565. Click \(\rho = 1.5\): the circles now sit on the dark diagonal and the score is 0.899. Click “Jump to the best line” for any setting to see what the matcher would choose. Then switch the descriptor to pHash and the attack to crop or flip: there is no line at all. Switch to CLIP: almost the whole matrix is dark, because CLIP finds every frame of the clip similar; the line is barely darker than the rest. Finally choose “Bunny 05 → Bunny 04”: these are two different clips, but the end of clip 04 and the start of clip 05 show the same meadow shot, and that shot is almost static, so SSCD finds a dark block in the right half of the matrix and a high line through it (best line score 0.559 for crop). That is the most common false alarm in this lab (Step 5).
Step 3: score every pair and measure micro-AP
A copy-detection system outputs a list of (query, reference, score) predictions, and the product sets one threshold: everything above it is reported as a copy. The right metric therefore ranks all predictions of all queries together, not each query separately. Sort the predictions by score, walk down the list, and at each position \(k\) compute the precision \(P(k)\) (the fraction of the first \(k\) predictions that are true copies) and the recall \(R(k)\) (the fraction of all true copies found so far). The area under this precision–recall curve is the micro average precision:
- Read the symbols
- \(N\)
- the number of predictions; here every (query, reference) pair: \(630 \times 45 = 28{,}350\)
- \(N_{+}\)
- the number of true copy pairs: 315 (every non-distractor query has exactly one source)
- \(P(k), R(k)\)
- precision and recall of the top \(k\) predictions; \(R\) only grows at a true copy, which gives the right-hand form
This is the metric of the DISC21 image challenge and of the VSC 2022 video challenge, where it is called micro-AP (µAP) [Certain]. It punishes two things that per-query metrics hide. First, calibration: a score of 0.5 must mean the same for every query, because one threshold serves all of them. Second, distractors: a distractor query has no true match, so every prediction it makes can only lower precision [Certain]. In VSC 2022 most queries are distractors, like in real upload traffic [Certain].
# analysis.py (core): score all pairs, then micro-AP per descriptor, matcher and attack import json, numpy as np from fp import to_mat, video_scores, ap meta = json.load(open("work/meta.json")); Z = np.load("work/desc.npz") REFS = [r["id"] for r in meta["ref"]] for d in ("phash", "clip", "sscd"): R = {r: to_mat(Z[f"{r}|{d}"], d) for r in REFS}; rows = [] for q in meta["qry"]: Q = to_mat(Z[f"{q['id']}|{d}"], d) for r in REFS: rows.append(dict(att=q["attack"], pos=(q["src"] == r), **video_scores(Q @ R[r].T))) for m in ("maxmax", "chamfer", "temporal1", "temporal"): per = {a: ap([x["pos"] for x in rows if x["att"] == a], [x[m] for x in rows if x["att"] == a]) for a in ("crop", "resize", "reencode", "speed", "logo", "flip", "combo")} print(d, m, round(ap([x["pos"] for x in rows], [x[m] for x in rows]), 3), per)
All 85,050 matrices (three descriptors) take about 40 seconds. Our output, micro-AP over all 28,350 pairs:
| Descriptor | pool (one vector per video) | max (216.3) | Chamfer (216.3) | temporal, \(\rho = 1\) only | temporal, 5 speeds (216.4) |
|---|---|---|---|---|---|
| pHash (64 bits) | 0.647 | 0.659 | 0.678 | 0.653 | 0.670 |
| CLIP B/32 | 0.447 | 0.455 | 0.499 | 0.491 | 0.478 |
| SSCD | 0.874 | 0.907 | 0.930 | 0.929 | 0.931 |
The descriptor matters much more than the matcher. SSCD is far ahead with every matcher; CLIP is the worst, below even the 64-bit hash. Among the matchers, pooling a whole video into one vector is clearly the worst, and the three matrix-based scores are close: Chamfer and the temporal line are within 0.002 for SSCD. The temporal matcher does not win on this metric here, because our queries are clean continuous excerpts, so Chamfer’s freedom costs little. Its value shows elsewhere: on the speed attack, searching five speeds raises SSCD from 0.979 to 0.995, pHash from 0.932 to 0.994 and CLIP from 0.837 to 0.969 (compared with \(\rho = 1\) only); and it is the only one of the four that says where the copy is. For the 311 copies that SSCD with the temporal matcher ranks first (98.7% of the 315), the found speed is right and the found offset is within one frame (0.5 seconds) for 94.5%.
How to read Figure 3
The default is SSCD with the temporal matcher on all attacks, µAP 0.931, and the dot sits at recall 0.90 where precision is 0.757: to catch 90% of copies at one global threshold (0.381) you accept 91 false alarms. Slide left to recall 0.80: precision rises to 0.944 with 15 false alarms. Compare the thin curves: CLIP (µAP 0.478) loses precision almost at once. Now pick single attacks. For pHash, “crop” and “flip” curves lie on the floor, while “reencode” is perfect. For SSCD, “reencode” is the lowest single-attack curve (0.867). Pick “SSCD + pHash”: the reencode curve becomes perfect, and “combo” becomes the weakest. That is the loop of this lab.
Step 4: attack your own system
An average hides the failures. The table splits the result by attack (temporal matcher, raw scores). Each cell is a µAP over the 45 true copies and the 4,005 other pairs of the same attack (90 queries × 45 references):
| Attack | pHash | CLIP | SSCD | SSCD + pHash (Step 6) |
|---|---|---|---|---|
| crop 60% | 0.066 | 0.545 | 1.000 | 1.000 |
| resize to 1/4 | 1.000 | 0.395 | 0.991 | 1.000 |
| reencode CRF 45 | 1.000 | 0.277 | 0.867 | 1.000 |
| speed 1.5× | 0.994 | 0.969 | 0.995 | 0.995 |
| logo and banner | 0.976 | 0.957 | 1.000 | 1.000 |
| flip | 0.012 | 0.973 | 1.000 | 1.000 |
| combo | 0.402 | 0.180 | 0.918 | 0.836 |
| all (µAP) | 0.670 | 0.478 | 0.931 | 0.969 |
Each descriptor has a different weakest transform, and each failure has a clear reason:
- pHash: crop and flip. The hash describes where light and dark areas sit in the full frame. A crop moves and scales everything; a mirror swaps left and right. Both change about half of the 64 bits, like an unrelated frame. But pHash is perfect on re-encoding: block artifacts are high-frequency noise, and the hash keeps only the 8 × 8 lowest frequencies.
- CLIP: re-encoding, resize and combo. CLIP’s problem is not that copies look different to it (in the Sintel table above, every copy cosine is above 0.8) but that non-copies look the same: the median temporal score of a pair that is not a copy is 0.721 for CLIP, against 0.100 for SSCD. A blurred, blocky frame loses the details that separate two frames of the same scene, so a copy falls into the crowd of similar clips from the same movie.
- SSCD: heavy re-encoding. At CRF 45 the queries average about 32 kbit/s. The weakest copies are from the two low-resolution movies (426 × 240 and 564 × 240), where the encoder leaves only blocks; the lowest copy score is 0.153, below many distractor pairs. SSCD learned compression from JPEG-style training edits [Likely], but not this level of video block artifacts.
One setting per attack is not enough: you need to know how fast each descriptor breaks. We generated four more crop levels and four more CRF levels for the same 90 source clips (720 more queries) and measured the µAP of each:
# sweep.py (core): the same 90 sources, stronger and weaker versions of two attacks SW = {f"crop{int(c*100)}": ["-vf", f"crop=iw*{c}:ih*{c}"] + ENC for c in (0.9, 0.8, 0.5, 0.4)} SW.update({f"crf{c}": crf(c) for c in (30, 35, 40, 51)}) # then: cut, sample at 2 fps, describe, and score against the same 45 references as before
How to read Figure 4
Left panel: pHash is fine at 90% (0.992), already hurt at 80% (0.834) and useless from 60% on. SSCD stays at 1.000 down to 50% and only drops to 0.987 at 40%. CLIP declines steadily. Right panel: the picture flips. pHash stays at 1.000 down to CRF 45 and 0.996 at CRF 51 (about 23 kbit/s), while SSCD falls from 0.968 at CRF 40 to 0.867 at CRF 45 and 0.671 at CRF 51, and CLIP collapses first. The violet line, the max of SSCD and pHash, follows the better of the two in both panels. Switch to Chamfer: the shapes stay the same.
Step 5: look at the false alarms
Before you fix anything, read the top of the ranked list. With SSCD and the temporal matcher, the highest-scoring wrong pair is the speed query of Bunny clip 05 against reference Bunny 04, at 0.847, higher than the median true copy (0.767). Of the 50 highest-scoring wrong pairs, 42 are a held-out clip against its neighbour reference: the clip that ends 5 seconds before it or starts 5 seconds after it. Figure 2 shows why: the second half of Bunny 04 and all of Bunny 05 belong to one long, almost static meadow shot, with the same background and the same rabbit. The detector is not wrong about the pixels; our labels say “not a copy” because the two clips do not share a single frame.
This is the real definition problem of video copy detection. Is “the same shot, 5 seconds later” a copy? For a copyright system it usually is (it is the same film); for training-data deduplication it may be; for a news-verification system it is not. The VSC 2022 dataset adds 500 hard negative pairs labeled by humans to each split, to measure exactly this kind of confusion [Certain]. Decide the policy first, write it into the labels, and only then tune thresholds. Two more lessons from this list:
- Mirror invariance cuts both ways. SSCD is almost blind to flips (flip µAP 1.000), so a flipped neighbour shot also matches: the second-highest wrong pair is the flipped Bunny 05 against Bunny 04 (0.832).
- Static shots make blocks, not lines. In an almost static shot every frame looks like every other, so the similarity matrix has a dark block instead of a thin line, and any line through the block scores high. In Figure 2 the best line for Bunny 05 against Bunny 04 starts at reference frame 11 (second 80.5 of the movie) and matches query frame 0 (second 92): every matched pair is 11.5 seconds apart. A line is real evidence only if it stands out from the cells next to it; a matcher can subtract the average of its neighbourhood, and a longer minimum coverage helps too. Both have a cost: a real copy of a static shot, or a short real copy inside a long query, then scores lower.
Step 6: fix the weakest transform, then find the next one
Try 1: score normalization. The VSC 2022 baseline subtracts from every frame similarity the similarity of that query frame to its nearest neighbour in a separate background set, scaled by \(\beta\) [Certain]. A query frame that looks like many things (a dark frame, a sky) is then pushed down:
- Read the symbols
- \(\mathcal{B}\)
- background descriptors of frames that are surely not in the database; we used 270 frames at 1 per second from seconds 470 to 560 of each movie, outside every clip
- \(\beta\)
- how much of the background similarity to remove; VSC used the nearest neighbour (\(k = 1\)) and \(\beta = 1.2\) [Certain]
In the VSC paper this step raised the SSCD baseline from µAP 0.010 to 0.605 on the challenge test set, because without it empty and solid-color frames matched everything [Certain]. In our lab the effect depends on the descriptor: CLIP rises from 0.478 to 0.653 (its scores were badly calibrated), pHash moves from 0.670 to 0.656, and SSCD drops from 0.931 to 0.913. Our clips have no empty frames, and our small background set contains scenes from the same movies, so for SSCD it mostly adds noise. A per-query alternative that uses the database itself, “score minus the best score of any other reference”, gave SSCD 0.956 and CLIP 0.788, but it cannot report two true sources for one query. Normalization is a calibration tool; measure it on your own distribution.
Try 2: combine two descriptors that fail differently. SSCD’s weakness (heavy compression) is pHash’s strength, and pHash’s weaknesses (crop, flip) are SSCD’s strengths. Both scores live on the same scale (about 0 for unrelated frames, near 1 for copies), so the simplest fusion is the maximum of the two video scores:
This raises the overall µAP from 0.931 to 0.969, and the re-encoding µAP from 0.867 to 1.000. Its cost is tiny: 8 more bytes per frame and no neural network. But the attack table shows the next weakest transform at once: combo drops from 0.918 to 0.836. The reason is the max: the fused score of a pair is high if either descriptor is fooled, so pHash adds its own false alarms (for example 0.719 for the Bunny 05 → Bunny 04 neighbour) while it cannot help on cropped combo queries. A max is a good fusion when the second descriptor is rarely wrong with high confidence; otherwise learn the fusion on held-out data, or use the second descriptor only to re-rank. And then attack again: this loop never ends, which is why public benchmarks add transforms to the validation and test sets that are not in the training set [Certain].
- input: a reference set, held-out distractor clips, a list of attacks with strength levels
- repeat:
- make queries: every source × every attack × every strength; keep distractors in the same proportion
- score all (query, reference) pairs; compute µAP overall, per attack and per strength // formula (216.5)
- read the top wrong pairs and the lowest true copies; label the cause (descriptor, matcher, label policy)
- fix the weakest attack (new descriptor, fusion, normalization, training edits) // one change at a time
- re-run everything; check that no other attack got worse // combo got worse in Step 6
- until every attack in your threat list meets its target, then add a new attack
Where this is used
| Application | What from this lab carries over |
|---|---|
| Copyright matching of uploads against a catalog | Frame descriptors at about 1 per second, a vector index, a temporal check that returns the matched segment; thresholds set on µAP with many distractors |
| Re-upload detection of removed content (trust and safety) | Robustness to crop, overlay, speed and re-encoding; hash-sharing between companies uses compact codes such as perceptual hashes for exactly this reason [Likely] |
| Training-data deduplication for video and image–text models | Near-duplicate removal before training (session 187); the “same shot, a few seconds later” policy decides how much is removed |
| News and fact checking: finding the original of a viral clip | Localization (offset and speed) tells you which part of which source was reused, and at what speed |
| Video-RAG and archive search | Fingerprints group duplicate segments before indexing, so retrieval does not return ten copies of one clip (session 210) |
| Broadcast monitoring (ads, sports highlights) | Logo and banner robustness, speed changes in replays, continuous streams instead of clips |
Common mistakes
- Using a semantic embedding as a fingerprint. CLIP placed last here (µAP 0.478), below a 64-bit hash, because it finds all frames of a scene similar. Use a descriptor trained for copy detection.
- Evaluating per query, without distractors. If every query has a true match and you report top-1 accuracy, SSCD looks almost perfect (98.7%). With half the queries as distractors and one global threshold, it needs 91 false alarms to reach 90% recall.
- Testing only the attacks you trained for, at one strength. The SSCD re-encoding curve is flat until CRF 40 and then falls fast; a single test at CRF 30 would never show it.
- Labeling neighbours of the same shot as negatives without deciding the policy. 42 of the 50 top false alarms here are “the same scene a few seconds later”. Fixing the descriptor will not help; fix the labels or the coverage rule.
- Mean-pooling a video into one vector. It is the worst matcher in every row of the table, and it cannot localize.
- Getting the clip timing wrong in FFmpeg.
-tafter the input limits the output, so a 1.5× query silently covers 9 seconds of source. Check every generated file withffprobe. - Mismatched rounding between components.
torch.roundrounds halves to even, JavaScript rounds them up; with \(\rho = 1.5\) the line visits different frames. Pin the rule in one place. - Copying a normalization trick without measuring it. Background score normalization was essential in VSC 2022 but lowered SSCD here (0.931 to 0.913).
Interview questions
What is video copy detection, and how is it different from text-to-video retrieval?basic
How does a perceptual hash work, and which attacks break it?basic
You have a frame similarity matrix between a query and a reference. How do you turn it into a video score?core
How do you handle a copy that was sped up or slowed down?core
What is micro-AP, and why is it used instead of per-query mAP or top-1 accuracy?core
Your SSCD-based system is weakest on heavily compressed uploads. What do you do?core
What is score normalization in copy detection, and when does it help?advanced
Most of your false alarms are “the same shot, 5 seconds later”. Is the detector wrong?core
Design a fingerprinting service for 100,000 hours of new video per day.advanced
How would you evaluate the robustness of a fingerprinting system honestly?advanced
Exercises
- Two pHashes differ in 18 bits. What is their similarity (216.1)? What similarity do you expect for two unrelated frames?
- A query has 12 frames and the matcher tries \(\rho = 2/3\), \(o = 4\). Which reference frames does the line visit (round halves up)? Which real edit does this slope describe?
- A ranked list of five predictions has labels copy, not, copy, copy, not, and there are 3 copies in total. Compute the micro-AP.
- Your service stores one 512-dimensional float32 descriptor per second for 100,000 hours of video. How many bytes is that, and how many with 64-byte product-quantization codes or with an 8-byte pHash?
- Add an eighth attack to
make_data.py: picture-in-picture (the source scaled to 40% and placed on a different clip). Predict, before you run it, which descriptor fails and why. Then run it. - Replace the max fusion (216.7) by the mean of the two scores and by “SSCD unless SSCD is below 0.3, then pHash”. Which attacks get better and which get worse?
Answerscheck
Key takeaways
- A fingerprint must be invariant to edits but specific to frames. A copy-detection descriptor (SSCD, µAP 0.931) beat a 64-bit pHash (0.670) and a semantic CLIP embedding (0.478) on the same 630 queries.
- Represent a video as a sequence of frame descriptors and compare two videos with a frame similarity matrix; a copy is a line, its slope is the speed and its start is the location.
- Chamfer and a temporal line search scored the same µAP here, but only the line search finds speed and offset; searching speeds fixed the 1.5× attack.
- Evaluate with micro-AP over all pairs, with distractors, at one global threshold. Top-1 accuracy of 98.7% hid a precision of 0.757 at 90% recall.
- Attack at several strengths. Each descriptor had its own weakest transform: crop and flip for pHash, re-encoding for SSCD and CLIP.
- Fusing descriptors that fail differently fixed SSCD’s weakest transform (re-encoding 0.867 → 1.000) but moved the weakness to the combined attack (0.918 → 0.836). Repeat the loop.
- Read the false alarms: most were the same shot a few seconds later, a label-policy question, not a descriptor bug.
Key formulas
References for this session
- Pizzi et al. (2022). A Self-Supervised Descriptor for Image Copy Detection
- Kordopatis-Zilos et al. (2019). ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning
- Pizzi et al. (2022). A Self-Supervised Descriptor for Image Copy Detection
- Kordopatis-Zilos et al. (2019). ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning
- Pizzi et al. (2023). The 2023 Video Similarity Dataset and Challenge
- Douze et al. (2021). The 2021 Image Similarity Dataset and Challenge
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Meta AI Research (code). SSCD: A Self-Supervised Descriptor for Image Copy Detection (code and pretrained TorchScript models)
- Blender Foundation (2006–2010). Elephants Dream, Big Buck Bunny and Sintel (open movies, CC-BY 3.0), Internet Archive copies
- J. Buchner (code). ImageHash: a Python perceptual image hashing module (pHash implementation)
- Douze, Jégou & Schmid (2010). An Image-Based Approach to Video Copy Detection With Spatio-Temporal Post-Filtering (IEEE Transactions on Multimedia)