segment and track players with SAM 2
In this lab a person detector finds the players in the first frame, its boxes become prompts for SAM 2, and SAM 2 follows every player through real sports clips from DAVIS 2017. You score the result with J&F, then break it on purpose with a camera cut and with bad boxes, fix the cut with a shot detector, and find out how many hand-labeled frames you need to trust the score.
What you will learn
- How to turn person detections into SAM 2 prompts, and why choosing players by detector score fails
- How to run SAM 2 on a clip with several objects through the Hugging Face video API, and read its masks and object scores
- How to compute DAVIS J, F and J&F per player and per clip, exactly as the official code does
- Where SAM 2 fails on sports video: occlusion by other players, motion blur, bad prompt boxes and camera cuts
- How a histogram cut detector and shot-by-shot tracking fix the camera-cut failure, and how absent frames inflate J&F
- How many labeled frames you need for a trustworthy J&F, using the standard error with the finite-population correction
- How to write a results sheet that someone else can reproduce
Why this matters
A sports analytics company wants the exact outline of every player in every frame: to measure speed and distance, to draw graphics on the broadcast, to cut highlight clips, or to build training data. Nobody will draw those masks by hand for 90 minutes of video. The modern recipe is short: a detector finds the players in one frame, its boxes become prompts for SAM 2, and SAM 2 follows each player through the clip. In this lab you build that pipeline, run it on real sports clips, break it on purpose, and measure it with \(\mathcal{J}\&\mathcal{F}\).
The theory is in Session 204: memory attention, the prompted and recent frames in SAM 2’s memory bank, the occlusion head, and the metrics \(\mathcal{J}\) and \(\mathcal{F}\). The detector side comes from session 168. The cut detector you write in Part 3 is the shot-boundary idea from session 201, and the habit of attacking your own system comes from the previous lab, session 216. The next session, session 218, leaves video for pages and scans.
The lab has four parts:
- Turn detector boxes into prompts, and choose which boxes are players.
- Prompt SAM 2 with those boxes and propagate through the clip; measure \(\mathcal{J}\), \(\mathcal{F}\) and \(\mathcal{J}\&\mathcal{F}\) per player.
- Find where it fails: occlusion, motion blur, bad prompts and a camera cut. Then fix the cut.
- Measure \(\mathcal{J}\&\mathcal{F}\) with only a few labeled frames, and learn how many you need.
transformers 5.23. Every number in the tables, figures and worked examples is from those runs and is tagged [Certain, our run]. The figures replay the saved results; they do not run SAM 2 in your browser. Speeds were measured on a laptop that was busy with other jobs, so treat them only as rough.The setup: two clips, one detector, one tracker
The clips. We need sports video with human-drawn masks on the frames, so that we can score our output. We use two sequences from DAVIS 2017, the standard benchmark for video object segmentation, at 854 × 480 pixels. [Certain] Every frame of every DAVIS sequence has a mask drawn by a person for each object, so the DAVIS masks play the role of your “hand-labeled frames”. In Part 4 we pretend we only have a few of them.
| Clip | Frames | Labeled objects | What makes it hard |
|---|---|---|---|
boxing-fisheye (DAVIS train split) | 87 | 3: blue-helmet boxer, red-helmet boxer, referee | fisheye lens, moving camera, the ring ropes and the other two people pass in front of each player |
judo (DAVIS val split) | 34 | 2: both players, both in white | players hold each other, one is thrown (fast motion, blur), and six bystanders in white judo suits stand around the mat |
The frame and object counts are from the files. [Certain, our run] The DAVIS readme says the sequences are under a Creative Commons Attribution 4.0 license, with the original video sources listed in the dataset. [Certain] We do not show the video frames on this page; the figures draw boxes and masks only.
The detector. RT-DETR with a ResNet-50 backbone (PekingU/rtdetr_r50vd, 42.9 M parameters by our count), trained on COCO, which has a “person” class. [Certain, our run] Any person detector works here; this one is a transformer detector with no extra dependencies in transformers.
The tracker. SAM 2.1 with the Hiera-Tiny image encoder (facebook/sam2.1-hiera-tiny, 38,962,498 parameters by our count), and for one comparison SAM 2.1 Hiera-B+ (80,850,178 parameters). [Certain, our run] The official repository lists 38.9 M and 80.8 M, and speeds of 91.2 and 64.1 frames per second on an A100. [Certain] The model resizes every frame to 1024 × 1024 and keeps a memory of 7 frames: the prompted frame plus the 6 most recent ones (num_maskmem = 7 in its config). [Certain, our run]
The metrics. We wrote \(\mathcal{J}\) and \(\mathcal{F}\) in NumPy, following the official DAVIS evaluation code: \(\mathcal{J}\) is mask IoU (1 when both masks are empty); \(\mathcal{F}\) matches boundaries within a tolerance of \(\lceil 0.008 \times \text{image diagonal} \rceil\), which is \(\lceil 0.008 \times 979.7 \rceil = 8\) pixels at 854 × 480. Like the official code, we score frames 1 to \(T-2\): the prompted first frame and the last frame are not scored. [Certain]
Part 1: from detector boxes to prompts
SAM 2 does not know what a “player” is. It segments whatever you point at. So the first job is to decide which detections are players. This sounds trivial, and it is where the first bug of this lab appears.
On frame 0 of the judo clip the detector returns 7 person boxes, all with scores of 0.726 or more. The two players get 0.925 and 0.904. Three bystanders get higher scores: 0.939, 0.933 and 0.932. [Certain, our run] A bystander standing still against a wall is an easier “person” than two athletes wrapped around each other. So the obvious rule, “take the top two by score”, tracks two spectators and no players.
In sports video we usually know more than the detector: the players are close to the camera and on the field. The simplest rule that uses this is to take the \(k\) largest boxes among those above a score threshold:
- Read the symbols
- \(\mathcal{D}\)
- all “person” detections on the prompt frame
- \(s_d, \mathbf{b}_d\)
- the score and the box \((x_0, y_0, x_1, y_1)\) of detection \(d\)
- \(s_{\min}\)
- the score threshold (0.5 in our runs)
- \(k\)
- the number of players you expect in the shot (3 for boxing, counting the referee; 2 for judo)
- \(\mathcal{P}\)
- the boxes that become SAM 2 prompts, one object id each
- \(\operatorname{IoU}\)
- box overlap; we use it only to check, after the fact, which human-labeled player each prompt belongs to
With this rule both clips are right: the boxing prompts overlap the true player boxes with IoU 0.939, 0.942 and 0.940, the judo prompts with 0.979 and 0.986. [Certain, our run] In a real system you would replace “largest” by something sturdier: a mask of the playing field, the team colors, or a jersey-number reader. Figure 1 lets you try the three rules.
How to read Figure 1
Start with Judo and “Top k by score”: both prompts land on bystanders at the edge of the mat. Switch to “Top k by area”: both players are found. Now choose Boxing and “Every box above threshold”: at 0.50 you get 8 prompts, 5 of them on spectators behind the ropes. Each one would become a SAM 2 track, cost compute, and produce masks you must later throw away. Move the threshold up: at 0.75 only the 3 people in the ring are left. A threshold alone works on this clip, but not on judo, where bystanders score higher than players.
One more detail matters on the judo clip. The two players’ true boxes overlap with IoU 0.40. [Certain, our run] A box is an ambiguous prompt when another person fills half of it, and SAM 2 must decide which of the two bodies you meant. Here it decided well (\(\mathcal{J} = 0.911\) and 0.917 on the prompt frame). [Certain, our run] When it does not, a mask prompt or an extra positive click on the right person removes the doubt.
Part 2: propagate and measure
With the prompts chosen, the tracking loop is short. All objects are added on frame 0 with their own ids, SAM 2 segments frame 0, then it propagates forward. For each frame and each object it returns mask logits and an object score logit \(\ell_t\), the output of the occlusion head: “is this object visible in this frame at all?”
- Read the symbols
- \(\ell_t\)
- the object score logit for this object in frame \(t\)
- \(z_t(x)\)
- the mask logit at pixel \(x\), resized to the original 854 × 480 frame
- \(\hat M_t\)
- the output mask: pixels with a positive logit, or nothing if the object is judged not visible
- input: frames \(I_0, \dots, I_{T-1}\); expected number of players \(k\); threshold \(s_{\min}\)
- \(\mathcal{D} \leftarrow\) detector\((I_0)\), keep class “person” // RT-DETR
- \(\mathcal{P} \leftarrow\) the \(k\) largest boxes with \(s_d \ge s_{\min}\) // formula (217.1)
- session \(\leftarrow\) SAM 2 video session on all frames; add box \(\mathbf{b}_i\) as object \(i\) on frame 0, for every \(\mathbf{b}_i \in \mathcal{P}\)
- segment frame 0; check each prompt-frame mask (predicted IoU, or \(\mathcal{J}\) if you have a label) before going on
- for \(t = 1, \dots, T-1\): for each object, \(\hat M_t, \ell_t \leftarrow\) SAM 2 step // memory attention over prompted + 6 recent frames
- apply formula (217.2); store \(\hat M_t\) and \(\ell_t\)
- score: for each object and each labeled frame, \(\mathcal{J}\) and \(\mathcal{F}\); average as in formula (217.3)
The clip score is the DAVIS one from session 204: average over frames for each object, then over objects, for \(\mathcal{J}\) and \(\mathcal{F}\) separately, then take the mean of the two:
- Read the symbols
- \(G_f\)
- the human mask of the object in frame \(f\)
- \(P_c, R_c\)
- contour precision and recall with the 8-pixel tolerance
- \(\mathcal{O}\)
- the tracked objects (players)
- \(S\)
- the scored frames: 1 to \(T-2\) here; in Part 4 only a few labeled frames
Results
| Clip, object | \(\mathcal{J}\) | \(\mathcal{F}\) | \(\mathcal{J}\&\mathcal{F}\) | Worst frame (\(\mathcal{J}\)) |
|---|---|---|---|---|
| Boxing, blue-helmet boxer | 0.913 | 0.969 | 0.941 | frame 12 (0.844) |
| Boxing, red-helmet boxer | 0.908 | 0.969 | 0.938 | frame 85 (0.678) |
| Boxing, referee | 0.884 | 0.969 | 0.926 | frame 26 (0.630) |
| Boxing, clip | 0.902 | 0.969 | 0.935 | |
| Judo, player 1 (thrower) | 0.875 | 0.907 | 0.891 | frame 16 (0.774) |
| Judo, player 2 (thrown) | 0.840 | 0.875 | 0.857 | frame 25 (0.680) |
| Judo, clip | 0.858 | 0.891 | 0.874 |
All numbers are SAM 2.1 Hiera-Tiny with the detector boxes as prompts. [Certain, our run] No player was ever lost: \(\mathcal{J}\) stayed above 0.5 on every scored frame (DAVIS “recall” = 1.00 for all five objects), and the object score logit never fell below 6.7 on these clips, so the occlusion head never switched a player off. [Certain, our run]
The bigger Hiera-B+ model gave \(\mathcal{J}\&\mathcal{F}\) 0.937 on boxing and 0.867 on judo: 0.002 better on one clip and 0.007 worse on the other. [Certain, our run] With 5 objects in 2 clips, a difference this small is noise, not a ranking. On these clips, the hard frames are hard for both models for the same reasons, which Part 3 looks at. Our laptop runs took 190 s for the 87 boxing frames with Tiny and 267 s with B+ (about 2.2 and 3.1 s per frame for 3 objects, including loading and resizing the frames). [Certain, our run] That is far from the 91 frames per second the authors measured on an A100 with the official code; a laptop GPU, float32 and a different implementation explain the gap, and you should not compare the two numbers.
How to read Figure 2
On boxing, step to frame 26: the referee (green) is partly behind the red-helmet boxer, and the missed pixels (dark) are the parts of him that show between the boxer’s arm and body. \(\mathcal{J}\) drops to 0.630, yet \(\mathcal{F}\) stays high, because most of his outline is still right. Step to frame 85: now the red-helmet boxer is behind the blue-helmet boxer. On judo, step to frame 25: player 2 is in the air, his legs are blurred, and SAM 2 misses most of them. In every case the track recovers a few frames later, because the prompted frame and the recent frames are still in the memory.
Part 3: where it fails
A high clip score hides the frames that matter. Read the per-frame curves, not just the average. In our two clips we saw three kinds of failure.
- Partial occlusion by another person. Boxing frame 26: the referee is behind the red-helmet boxer, \(\mathcal{J} = 0.630\). Frames 82 to 86: the red-helmet boxer is behind the blue-helmet boxer, \(\mathcal{J}\) between 0.68 and 0.82. [Certain, our run] The mask shrinks to the visible part, which is correct behavior, but SAM 2 also loses some pixels that are visible in thin gaps.
- Fast motion and blur. Judo frames 15 to 27, the throw: player 2’s \(\mathcal{J}\) stays between 0.68 and 0.83, lowest at frame 25, while his blurred legs swing through the air. [Certain, our run] Thin, blurred parts are where both \(\mathcal{J}\) and \(\mathcal{F}\) suffer.
- Prompt errors that never heal. If the box is bad, the first mask is bad, and every later frame copies it from memory. We show this with jittered boxes below.
The fourth failure is the one every sports broadcast has, and DAVIS clips do not: a camera cut.
A camera cut, made on purpose
We built a 121-frame clip: boxing frames 0 to 43, then all 34 judo frames, then boxing frames 44 to 86. This is a cut away to another shot and a cut back, as a broadcast does for a replay or a close-up. During the judo shot the three boxing people are not in the picture, so the correct output for all three tracks is an empty mask. We ran SAM 2 on the whole clip with the same three boxing prompts. [Certain, our run]
What happened:
- The blue-helmet boxer’s track jumped onto judo player 1 at the cut and stayed on him for all 34 judo frames. Its object score logit started at 3.7 and climbed to 8.1 by the end of the shot, as confident as it was on the real boxer (its lowest value on the 44 boxing frames before the cut was 7.9). [Certain, our run] The tracker had no idea anything was wrong.
- The red-helmet boxer’s track sat on judo player 2 for 3 frames, the referee’s track for 7 frames (as a small blob), and then their object scores went below 0 and their masks became empty, which is the right answer. [Certain, our run]
- At the cut back, all three tracks found their person again on the first frame: \(\mathcal{J} = 0.940\), 0.916 and 0.858. [Certain, our run] The prompted frame 0 is always in the memory, so SAM 2 could match the boxers again.
So the occlusion head sometimes catches a cut and sometimes does not. It was trained to say “this object is hidden”, not “this is a different camera”. A track that confidently follows the wrong person for a whole shot is the worst kind of error for analytics: the player’s distance and speed for those seconds are silently wrong.
The fix: shot-aware tracking
Cuts are easy to detect with a color histogram. Count the colors of each frame in \(8 \times 8 \times 8 = 512\) bins, normalize to sum 1, and compare neighboring frames:
- Read the symbols
- \(h_t(b)\)
- the fraction of pixels of frame \(t\) whose color falls in bin \(b\)
- \(\sum_b \min(\cdot,\cdot)\)
- histogram intersection: 1 for two frames with the same colors, 0 for two frames with no color in common
- \(\theta\)
- the cut threshold
On our spliced clip the two real cuts give \(d_t = 0.629\) and \(0.597\). Inside a shot, even with the moving fisheye camera, \(d_t\) is at most 0.035 (mean 0.016). [Certain, our run] Any threshold between 0.035 and 0.597 finds exactly the two cuts. The margin is wide here because the two sports have very different colors; two shots of the same field are much closer, and then a learned shot detector such as TransNet V2 is the safer choice. [Likely]
- input: frames, the prompted frame 0, cut threshold \(\theta\), same-scene threshold \(\delta\)
- compute \(d_t\) for every frame (formula 217.4); cuts \(\leftarrow \{t : d_t \gt \theta\}\); split the video into shots
- for each shot, compare the histogram of its first frame with frame 0: keep the shot if the distance is below \(\delta\) // same camera view
- run Algorithm 1 on the kept frames only, in order; give every other frame an empty mask
- at a kept shot that starts after a gap, run the detector again and check each track against a detection (box IoU); re-prompt when they disagree
With \(\theta = 0.3\) and \(\delta = 0.3\): the judo shot is 0.631 away from frame 0 and is dropped; the boxing shot after it is 0.149 away and is kept. [Certain, our run] SAM 2 then sees exactly the 87 boxing frames of Part 2, so its masks are the ones from Part 2, and every judo frame gets an empty mask. Figure 3 compares the two modes.
How to read Figure 3
In naive mode, follow the orange line through the yellow band: the blue-helmet boxer’s track has \(\mathcal{J} = 0\) for the whole judo shot, and its object score (bottom) stays well above 0. Its \(\mathcal{J}\&\mathcal{F}\) falls from 0.941 to 0.672. Switch to shot-aware: all three tracks are correct in the judo shot. Now move \(\theta\) above 0.597: only the first cut is found, the judo shot and the boxing shot after it look like one shot, the whole second half is dropped, and every player is missing for 43 frames (\(\mathcal{J}\&\mathcal{F}\) 0.625). Above 0.629 no cut is found, and you are back to the naive result. A threshold that is too high is as bad as no cut detection at all.
Naive run, \(\mathcal{J}\&\mathcal{F}\) over frames 1 to 119: red-helmet boxer 0.930, blue-helmet boxer 0.672, referee 0.887, mean 0.830. Shot-aware run: 0.956, 0.958, 0.947, mean 0.954. [Certain, our run]
Look closely at the shot-aware 0.954: it is higher than the 0.935 of the same masks on the boxing clip alone (Part 2). Nothing got better. The 34 judo frames are scored as perfect, because an empty prediction for an empty truth gives \(\mathcal{J} = 1\) and \(\mathcal{F} = 1\). Frames where the object is absent are easy points. When you compare two systems, compare them on the same frames, and also report the score on frames where the object is visible.
Prompt quality: jittered boxes and a single click
Detectors are not perfect, so we moved each corner of each prompt box by a random amount, up to a fraction \(a\) of the box width or height (two random seeds per level), and ran the whole clip again. We also replaced each box by one positive click at the box center. [Certain, our run]
| Prompt on frame 0 | Boxing \(\mathcal{J}\&\mathcal{F}\) (seed 0 / seed 1) | Judo \(\mathcal{J}\&\mathcal{F}\) (seed 0 / seed 1) |
|---|---|---|
| Detector box, as in Part 2 | 0.935 | 0.874 |
| Box corners moved up to \(a = 0.1\) | 0.477 / 0.897 | 0.825 / 0.847 |
| \(a = 0.2\) | 0.626 / 0.882 | 0.650 / 0.849 |
| \(a = 0.3\) | 0.437 / 0.774 | 0.384 / 0.806 |
| One positive click at the box center | 0.572 | 0.862 |
All values are clip \(\mathcal{J}\&\mathcal{F}\), averaged over the players. [Certain, our run] Three lessons come out of this table.
- Small box errors can be fatal, and it depends on luck. On boxing, a 10% jitter gave 0.897 with one seed and 0.477 with the other. In the bad case the red-helmet boxer’s box lost 24 pixels at the bottom and 13 at the right, and his first mask had \(\mathcal{J} = 0.203\); his track then scored 0.188 for the whole clip. [Certain, our run] Two seeds per level are too few to give a smooth curve; they are enough to show that the risk is real.
- The first mask decides the track. Over all 40 tracks in this section (5 players × 8 prompt settings), the \(\mathcal{J}\) of the first mask and the \(\mathcal{J}\&\mathcal{F}\) of the whole track have a correlation of 0.974. Every track whose first mask had \(\mathcal{J}\) below 0.7 ended at \(\mathcal{J}\&\mathcal{F}\) 0.809 or lower; every track whose first mask had \(\mathcal{J}\) of 0.85 or more ended at 0.813 or higher. [Certain, our run] SAM 2 does not repair a bad start by itself: the prompted frame stays in memory and keeps telling it what the object looks like.
- A click is ambiguous: part or whole? The center of a person’s box is often on the shirt. For the red-helmet boxer and the referee, SAM 2 segmented only the shirt (first-mask \(\mathcal{J}\) 0.232 and 0.416), and the tracks followed the shirts: \(\mathcal{J}\&\mathcal{F}\) 0.296 and 0.483. The blue-helmet boxer’s click gave the whole body (0.938), and on judo the clicks worked (0.862 for the clip). [Certain, our run] A box tells SAM 2 the extent of the object; a single click does not.
The practical rule: before you propagate, check every first mask. Without labels, use SAM 2’s predicted IoU for the prompt frame (the official image predictor returns it) and a simple sanity check, such as “a person mask should fill a good part of the person’s box”. If a mask fails, add a positive click on a missed body part or a negative click on the wrong person, and segment frame 0 again.
Part 4: J&F from a few labeled frames
DAVIS gives a human mask for every frame. Your own sports data will not: drawing a careful mask of a player takes minutes, so you will label a few frames per clip. How many do you need? Treat it as sampling. Let \(s_f\) be the per-frame score \(\tfrac12(\mathcal{J}_f + \mathcal{F}_f)\), averaged over players, for the \(n\) scorable frames. If you label a set \(S\) of \(k\) frames, your estimate and its standard error are:
- Read the symbols
- \(s_f\)
- the \(\mathcal{J}\&\mathcal{F}\) of frame \(f\), averaged over the players
- \(n\)
- the number of frames you could label (32 for judo, 85 for boxing)
- \(\sigma\)
- the standard deviation of \(s_f\) over all \(n\) frames: how much the quality changes from frame to frame
- \(\sqrt{(n-k)/(n-1)}\)
- the finite-population correction: labels are drawn without repetition, and when \(k = n\) the error is exactly 0
A range of about \(\pm 1.645\,\operatorname{SE}\) holds the true value 90% of the time when the frames are chosen at random. The formula says two things. The error shrinks only as \(1/\sqrt{k}\): to halve it you need 4 times more labels. And it scales with \(\sigma\): a clip whose quality changes a lot from frame to frame (a judo throw) needs more labels than a smooth one (boxing).
Judo, all 32 scorable frames: true \(\mathcal{J}\&\mathcal{F} = 0.874\), \(\sigma = 0.0558\). With \(k = 5\) random frames: \(\operatorname{SE} = 0.0558 / \sqrt{5} \times \sqrt{27/31} = 0.0233\), so the 90% range is about \(\pm 0.038\), a width of 0.077. In 400 random draws, 90% of the estimates fell in [0.838, 0.910], a width of 0.072: the formula is close. With 5 evenly spaced frames the estimate was 0.871, within 0.004 of the truth. [Certain, our run]
Boxing has a smaller \(\sigma = 0.0200\), so 5 random frames already give a 90% width of 0.027. [Certain, our run] In our 400 draws, the judo clip needed 17 random frames to get the 90% width below 0.03; boxing needed 5. [Certain, our run] Spreading the labels evenly over the clip helps, because it covers all phases of the action. But it can be unlucky too: with \(k = 1\) the single middle judo frame lands in the throw, and the estimate is 0.773, off by 0.101.
How to read Figure 4
The band is wide on the left and closes to zero at \(k = n\), as formula (217.5) says. Compare the two clips at \(k = 5\): judo’s band is almost three times wider than boxing’s, because judo’s per-frame quality swings during the throw. The green line is often closer to the truth than the band’s edges, but it jumps around: a single evenly spaced choice can land on a hard frame. In practice, label a few frames spread over the clip, plus the frames you expect to be hard (occlusions, fast motion, right after cuts), and report those separately.
- split the footage into shots (formula 217.4); sample shots, not only frames, so that one easy shot does not dominate
- in each sampled shot, label \(k\) frames spread over the shot (5 to 10), plus every frame within 5 frames after a cut back
- also label frames where the player is absent, but score them separately (they are “free” \(\mathcal{J} = 1\) points)
- compute \(\mathcal{J}\), \(\mathcal{F}\) and \(\mathcal{J}\&\mathcal{F}\) per player with the official tolerance; compute \(\sigma\) and the SE of formula (217.5)
- report: \(\mathcal{J}\&\mathcal{F}\) with a 90% range, \(\mathcal{J}\&\mathcal{F}\) on visible frames only, the number of frames with a mask on the wrong person, and the worst frames
The code
This is the code we ran, shortened only by removing logging. It needs torch, transformers 5.x, opencv-python, pillow and the DAVIS 2017 480p folder. Inference is deterministic for a fixed input (no sampling anywhere); the random box jitter uses a fixed NumPy seed.
import math, numpy as np, cv2, torch from PIL import Image from transformers import (AutoImageProcessor, AutoModelForObjectDetection, Sam2VideoModel, Sam2VideoProcessor) DEV = "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu") ROOT = "DAVIS/" def frame(seq, i): return Image.open(ROOT + f"JPEGImages/480p/{seq}/{i:05d}.jpg").convert("RGB") def label(seq, i): return np.array(Image.open(ROOT + f"Annotations/480p/{seq}/{i:05d}.png")) # 1) detector boxes -> prompts, formula (217.1): the k largest person boxes with score >= s_min det_proc = AutoImageProcessor.from_pretrained("PekingU/rtdetr_r50vd") det = AutoModelForObjectDetection.from_pretrained("PekingU/rtdetr_r50vd").eval() def player_boxes(img, k, s_min=0.5): with torch.no_grad(): out = det(**det_proc(images=img, return_tensors="pt")) r = det_proc.post_process_object_detection(out, target_sizes=[img.size[::-1]], threshold=s_min)[0] boxes = [b.tolist() for l, b in zip(r["labels"], r["boxes"]) if det.config.id2label[int(l)] == "person"] return sorted(boxes, key=lambda b: -(b[2] - b[0]) * (b[3] - b[1]))[:k] # 2) SAM 2: one object id per box on frame 0, then propagate (Algorithm 1) sam = Sam2VideoModel.from_pretrained("facebook/sam2.1-hiera-tiny").to(DEV).eval() proc = Sam2VideoProcessor.from_pretrained("facebook/sam2.1-hiera-tiny") def track(frames, boxes): sess = proc.init_video_session(video=[np.array(f) for f in frames], inference_device=DEV, dtype=torch.float32) ids = list(range(1, len(boxes) + 1)) proc.add_inputs_to_inference_session(sess, frame_idx=0, obj_ids=list(ids), input_boxes=[boxes]) # pass a copy: the session empties this list masks = {i: [None] * len(frames) for i in ids}; scores = {i: [None] * len(frames) for i in ids} with torch.no_grad(): sam(inference_session=sess, frame_idx=0) # segment the prompted frame first for out in sam.propagate_in_video_iterator(sess): m = proc.post_process_masks([out.pred_masks], original_sizes=[[sess.video_height, sess.video_width]], binarize=True)[0] for j, oid in enumerate(out.object_ids): masks[oid][out.frame_idx] = m[j, 0].cpu().numpy().astype(bool) scores[oid][out.frame_idx] = float(out.object_score_logits.reshape(-1)[j]) # l_t, formula (217.2) return masks, scores # 3) DAVIS metrics, formula (217.3): J = IoU, F = boundary F-measure with ceil(0.008 * diagonal) pixels def J(p, g): u = (p | g).sum(); return 1.0 if u == 0 else (p & g).sum() / u def bmap(m): # boundary pixels, as seg2bmap in the DAVIS code e = np.zeros_like(m); s = np.zeros_like(m); se = np.zeros_like(m) e[:, :-1] = m[:, 1:]; s[:-1, :] = m[1:, :]; se[:-1, :-1] = m[1:, 1:] b = (m ^ e) | (m ^ s) | (m ^ se); b[-1, :] = m[-1, :] ^ e[-1, :]; b[:, -1] = m[:, -1] ^ s[:, -1]; b[-1, -1] = 0 return b def F(p, g): r = math.ceil(0.008 * math.hypot(*p.shape)); yy, xx = np.mgrid[-r:r + 1, -r:r + 1] disk = (xx**2 + yy**2 <= r * r).astype(np.uint8) bp, bg = bmap(p), bmap(g); nP, nG = bp.sum(), bg.sum() if nP == 0 or nG == 0: return float(nP == nG) prec = (bp & cv2.dilate(bg.astype(np.uint8), disk).astype(bool)).sum() / nP rec = (bg & cv2.dilate(bp.astype(np.uint8), disk).astype(bool)).sum() / nG return 0.0 if prec + rec == 0 else 2 * prec * rec / (prec + rec) # 4) cut score, formula (217.4): 1 - histogram intersection of 8x8x8 color bins def hist(img): a = (np.asarray(img) // 32).reshape(-1, 3).astype(int) h = np.bincount(a[:, 0] * 64 + a[:, 1] * 8 + a[:, 2], minlength=512).astype(float) return h / h.sum() def cut_scores(frames): H = [hist(f) for f in frames] return [0.0] + [1 - np.minimum(H[t - 1], H[t]).sum() for t in range(1, len(H))] # run one clip seq, k = "judo", 2 frames = [frame(seq, i) for i in range(34)] boxes = player_boxes(frames[0], k) masks, scores = track(frames, boxes) g0 = label(seq, 0) # match each track to a labeled player by IoU on frame 0 match = {o: max(set(np.unique(g0)) - {0}, key=lambda L: J(masks[o][0], g0 == L)) for o in masks} Js = {o: [J(masks[o][f], label(seq, f) == match[o]) for f in range(1, 33)] for o in masks} Fs = {o: [F(masks[o][f], label(seq, f) == match[o]) for f in range(1, 33)] for o in masks} jf = (np.mean([np.mean(v) for v in Js.values()]) + np.mean([np.mean(v) for v in Fs.values()])) / 2 print(f"J&F = {jf:.3f}") # our run: 0.874
Three details in this code are easy to get wrong. The sam(inference_session=sess, frame_idx=0) call before propagation is needed; without it the Hugging Face iterator raises “Cannot determine the starting frame index”. [Certain, our run] The session keeps a reference to the obj_ids list you pass and removes ids from it once their prompt has been processed, so if you reuse that list later you get an empty list and no error; pass a copy. [Certain, our run, transformers 5.23] And the official sam2 package has the same loop under other names: init_state, add_new_points_or_box and propagate_in_video. [Certain]
The results sheet
| Setting | Value |
|---|---|
| Hardware, software | Apple M5 Pro, MPS, float32; torch 2.15.0, transformers 5.23.0 |
| Data | DAVIS 2017 480p: boxing-fisheye (87 frames, 3 objects), judo (34 frames, 2 objects) |
| Detector, prompt rule | RT-DETR R50 (COCO), class “person”, \(s_{\min} = 0.5\), the \(k\) largest boxes |
| Tracker | SAM 2.1 Hiera-Tiny, boxes on frame 0, forward propagation, default memory (prompted + 6 recent) |
| Metric | DAVIS \(\mathcal{J}\) and \(\mathcal{F}\), 8-pixel boundary tolerance, frames 1 to \(T-2\), mean over objects |
| \(\mathcal{J}\&\mathcal{F}\), boxing / judo | 0.935 / 0.874 (B+: 0.937 / 0.867) |
| Cut test, naive / shot-aware | 0.830 / 0.954; one track on the wrong person for 34 frames in the naive run |
| Cut detector margin | real cuts 0.597 and 0.629, largest within-shot score 0.035 |
| Prompt quality | box jitter of 10% gave boxing \(\mathcal{J}\&\mathcal{F}\) 0.477 (one seed of two); a center click gave 0.572 (boxing) and 0.862 (judo); first-mask \(\mathcal{J}\) vs track \(\mathcal{J}\&\mathcal{F}\) correlation 0.974 over 40 tracks |
| Labels needed (90% width under 0.03) | boxing: 5 random frames; judo: 17 |
Where this is used
| Application | Pipeline | What breaks first |
|---|---|---|
| Player tracking for sports analytics | detector → SAM 2 per player → field calibration → speed and distance | identity after cuts and in crowds (Part 3); a track on the wrong person gives silently wrong statistics |
| Broadcast graphics (highlighting a player, drawing behind players) | a click or box from an operator → SAM 2 masks → compositing | boundary quality (\(\mathcal{F}\)) on fast motion and blur |
| Highlight and key-moment clips | tracks + events from trajectories (session 207) | tracks that drift during the very action that matters (the judo throw) |
| Labeling data for training a faster model | detector + SAM 2 as a pseudo-labeler, humans fix a few frames, as in SAM 2’s own data engine | errors at the prompt frame copied into every frame |
| Object-level video search and Video-RAG | masklets stored per shot as index entries (session 210) | shot boundaries; the same person in two shots must be linked by re-identification, not by SAM 2’s memory |
| Privacy (blurring faces or bystanders) | face or person detector → SAM 2 → blur every frame | recall: one missed frame shows the face; objects coming into view after frame 0 need new prompts |
Common mistakes
- Choosing players by detector score. On our judo clip the three highest-scoring people are bystanders. Use what you know about the scene: size, position on the field, team colors.
- Not checking the prompt frame. If the first mask is wrong, every later frame copies it. Look at frame 0 (or at SAM 2’s predicted IoU) before propagating hundreds of frames.
- Running SAM 2 across camera cuts. The memory assumes one continuous shot. In our test one track followed a judo player for 34 frames with high confidence. Detect cuts first and track shot by shot.
- Trusting the object score as a cut detector. It turned off two of three tracks after the cut, and kept the third on the wrong person with a logit of 8. Motion cues help inside a shot (SAMURAI adds motion cues to SAM 2’s memory selection for this reason [Certain], with a Kalman filter that predicts the box [Likely]), but a cut breaks motion too.
- Letting absent frames inflate the score. Empty prediction on empty truth is \(\mathcal{J} = 1\) and \(\mathcal{F} = 1\). A clip with many absent frames looks better than it is. Report visible-frame scores too.
- Scoring the prompt frame. The DAVIS protocol skips the first frame (and the last). Including frame 0 rewards a good prompt, not good tracking.
- Calling a 0.005 difference a win. With 5 objects in 2 clips, Tiny and B+ were within 0.007 of each other in both directions. You need many more objects, and a confidence interval, to rank models.
- Labeling only easy frames. Few labeled frames give a noisy estimate (formula 217.5), and frames picked by convenience are biased. Spread them, add the hard frames on purpose, and report both.
Interview questions
Design a pipeline that gives you a mask for every player in a match video.core
Why can’t you take the top-k detections by score as the players?basic
What happens when SAM 2 tracks through a camera cut? Why?core
How did the tracks recover after the cut back, and when would they not?advanced
Write the DAVIS \(\mathcal{J}\&\mathcal{F}\) for a clip with several players. Which frames are scored?basic
You can afford to label only 5 frames per clip. How good is your \(\mathcal{J}\&\mathcal{F}\) estimate?core
Box prompt, click prompt or mask prompt: which would you use for players, and why?core
Hiera-B+ scored 0.937 and Tiny 0.935 on boxing, but B+ was worse on judo. What do you conclude?advanced
How would you detect camera cuts, and how do you pick the threshold?basic
Why did \(\mathcal{J}\) drop at boxing frame 26 while \(\mathcal{F}\) stayed high?advanced
Exercises
- In Figure 1, choose Boxing and “Every box above threshold”. What is the lowest threshold that gives exactly the 3 people in the ring? Why does no threshold work for judo?
- A track has object score logit \(\ell_t = 8.1\). What visibility probability does SAM 2 assign? What about \(\ell_t = -1.3\)?
- Using the results table, compute the boxing clip \(\mathcal{J}\&\mathcal{F}\) from the per-object \(\mathcal{J}\) and \(\mathcal{F}\) values.
- In the naive cut run, the blue-helmet boxer has \(\mathcal{J} = 0\) on 34 of the 119 scored frames. If its \(\mathcal{J}\) and \(\mathcal{F}\) on all other scored frames were 1, what would its \(\mathcal{J}\&\mathcal{F}\) be? Compare with the measured 0.672.
- For the boxing clip (\(\sigma = 0.0200\), \(n = 85\)), compute the SE of formula (217.5) for \(k = 10\) and the 90% range width.
- Run Algorithm 2 with a gap: boxing frames 0–43, then boxing frames 60–86 (16 frames skipped). Predict whether the tracks recover, then compare with our run below.
Answerscheck
Key takeaways
- The pipeline is detector → choose players → box prompts → SAM 2 propagation → \(\mathcal{J}\&\mathcal{F}\). Choosing players is a real step: by score alone, our judo prompts landed on bystanders (formula 217.1).
- On two DAVIS sports clips, SAM 2.1 Tiny with detector boxes reached \(\mathcal{J}\&\mathcal{F}\) 0.935 (boxing) and 0.874 (judo) and never lost a player; the hard frames were occlusions by other people and the blurred legs of a thrown player.
- Across a camera cut, one track followed the wrong person for the whole 34-frame shot with a high object score (formula 217.2 did not save it). A histogram cut detector (217.4) with shot-by-shot tracking fixed it.
- Absent frames are free points in \(\mathcal{J}\&\mathcal{F}\) (217.3); report visible-frame scores and wrong-person frames too.
- With \(k\) labeled frames, the error of your \(\mathcal{J}\&\mathcal{F}\) estimate is about \(\sigma/\sqrt{k}\) (217.5): 5 frames were enough for boxing, 17 for the judo throw.
Key formulas
References for this session
- Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
- Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
- Pont-Tuset et al. (2017). The 2017 DAVIS Challenge on Video Object Segmentation
- Cheng & Schwing (2022). XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
- Zhao, Lv, Xu, Wei, Wang, Dang, Liu & Chen (2023). DETRs Beat YOLOs on Real-time Object Detection (RT-DETR)
- Meta FAIR (2024). SAM 2 official repository: checkpoints, SAM 2.1 speed and accuracy table, video predictor
- Hugging Face (2025). transformers: SAM 2 Video (Sam2VideoModel, Sam2VideoProcessor) and the facebook/sam2.1-hiera-tiny checkpoint
- DAVIS challenge organizers (2017). DAVIS 2017 evaluation code (J and F measures)
- Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
- Zhang, Sun, Jiang, Yu, Weng, Yuan, Luo, Liu & Wang (2021). ByteTrack: Multi-Object Tracking by Associating Every Detection Box
- Yang, Huang, Chai, Jiang & Hwang (2024). SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory