LLM & VLM courseBlog
217Track 8: Video and multimodal systems · session 17 of 21

segment and track players with SAM 2

In this lab a person detector finds the players in the first frame, its boxes become prompts for SAM 2, and SAM 2 follows every player through real sports clips from DAVIS 2017. You score the result with J&F, then break it on purpose with a camera cut and with bad boxes, fix the cut with a shot detector, and find out how many hand-labeled frames you need to trust the score.

Time: 4 to 6 hours (mostly hands-on; a laptop GPU is enough)Level: Intermediate · hands-on labBefore this: Session 204 (video object segmentation and SAM 2), session 168 (open-vocabulary detection and segmentation), session 201 (video as data); Python

What you will learn

  • How to turn person detections into SAM 2 prompts, and why choosing players by detector score fails
  • How to run SAM 2 on a clip with several objects through the Hugging Face video API, and read its masks and object scores
  • How to compute DAVIS J, F and J&F per player and per clip, exactly as the official code does
  • Where SAM 2 fails on sports video: occlusion by other players, motion blur, bad prompt boxes and camera cuts
  • How a histogram cut detector and shot-by-shot tracking fix the camera-cut failure, and how absent frames inflate J&F
  • How many labeled frames you need for a trustworthy J&F, using the standard error with the finite-population correction
  • How to write a results sheet that someone else can reproduce

Why this matters

A sports analytics company wants the exact outline of every player in every frame: to measure speed and distance, to draw graphics on the broadcast, to cut highlight clips, or to build training data. Nobody will draw those masks by hand for 90 minutes of video. The modern recipe is short: a detector finds the players in one frame, its boxes become prompts for SAM 2, and SAM 2 follows each player through the clip. In this lab you build that pipeline, run it on real sports clips, break it on purpose, and measure it with \(\mathcal{J}\&\mathcal{F}\).

The theory is in Session 204: memory attention, the prompted and recent frames in SAM 2’s memory bank, the occlusion head, and the metrics \(\mathcal{J}\) and \(\mathcal{F}\). The detector side comes from session 168. The cut detector you write in Part 3 is the shot-boundary idea from session 201, and the habit of attacking your own system comes from the previous lab, session 216. The next session, session 218, leaves video for pages and scans.

The lab has four parts:

  1. Turn detector boxes into prompts, and choose which boxes are players.
  2. Prompt SAM 2 with those boxes and propagate through the clip; measure \(\mathcal{J}\), \(\mathcal{F}\) and \(\mathcal{J}\&\mathcal{F}\) per player.
  3. Find where it fails: occlusion, motion blur, bad prompts and a camera cut. Then fix the cut.
  4. Measure \(\mathcal{J}\&\mathcal{F}\) with only a few labeled frames, and learn how many you need.
What is measured on this pageWe ran every experiment ourselves, on an Apple M5 Pro laptop (PyTorch 2.15, MPS backend, float32) with Hugging Face transformers 5.23. Every number in the tables, figures and worked examples is from those runs and is tagged [Certain, our run]. The figures replay the saved results; they do not run SAM 2 in your browser. Speeds were measured on a laptop that was busy with other jobs, so treat them only as rough.

The setup: two clips, one detector, one tracker

The clips. We need sports video with human-drawn masks on the frames, so that we can score our output. We use two sequences from DAVIS 2017, the standard benchmark for video object segmentation, at 854 × 480 pixels. [Certain] Every frame of every DAVIS sequence has a mask drawn by a person for each object, so the DAVIS masks play the role of your “hand-labeled frames”. In Part 4 we pretend we only have a few of them.

ClipFramesLabeled objectsWhat makes it hard
boxing-fisheye (DAVIS train split)873: blue-helmet boxer, red-helmet boxer, refereefisheye lens, moving camera, the ring ropes and the other two people pass in front of each player
judo (DAVIS val split)342: both players, both in whiteplayers hold each other, one is thrown (fast motion, blur), and six bystanders in white judo suits stand around the mat

The frame and object counts are from the files. [Certain, our run] The DAVIS readme says the sequences are under a Creative Commons Attribution 4.0 license, with the original video sources listed in the dataset. [Certain] We do not show the video frames on this page; the figures draw boxes and masks only.

The detector. RT-DETR with a ResNet-50 backbone (PekingU/rtdetr_r50vd, 42.9 M parameters by our count), trained on COCO, which has a “person” class. [Certain, our run] Any person detector works here; this one is a transformer detector with no extra dependencies in transformers.

The tracker. SAM 2.1 with the Hiera-Tiny image encoder (facebook/sam2.1-hiera-tiny, 38,962,498 parameters by our count), and for one comparison SAM 2.1 Hiera-B+ (80,850,178 parameters). [Certain, our run] The official repository lists 38.9 M and 80.8 M, and speeds of 91.2 and 64.1 frames per second on an A100. [Certain] The model resizes every frame to 1024 × 1024 and keeps a memory of 7 frames: the prompted frame plus the 6 most recent ones (num_maskmem = 7 in its config). [Certain, our run]

The metrics. We wrote \(\mathcal{J}\) and \(\mathcal{F}\) in NumPy, following the official DAVIS evaluation code: \(\mathcal{J}\) is mask IoU (1 when both masks are empty); \(\mathcal{F}\) matches boundaries within a tolerance of \(\lceil 0.008 \times \text{image diagonal} \rceil\), which is \(\lceil 0.008 \times 979.7 \rceil = 8\) pixels at 854 × 480. Like the official code, we score frames 1 to \(T-2\): the prompted first frame and the last frame are not scored. [Certain]

Part 1: from detector boxes to prompts

SAM 2 does not know what a “player” is. It segments whatever you point at. So the first job is to decide which detections are players. This sounds trivial, and it is where the first bug of this lab appears.

On frame 0 of the judo clip the detector returns 7 person boxes, all with scores of 0.726 or more. The two players get 0.925 and 0.904. Three bystanders get higher scores: 0.939, 0.933 and 0.932. [Certain, our run] A bystander standing still against a wall is an easier “person” than two athletes wrapped around each other. So the obvious rule, “take the top two by score”, tracks two spectators and no players.

In sports video we usually know more than the detector: the players are close to the camera and on the field. The simplest rule that uses this is to take the \(k\) largest boxes among those above a score threshold:

\[ \mathcal{P} = \operatorname{top}_k \Big( \{\, d \in \mathcal{D} : s_d \ge s_{\min} \,\},\ \text{key} = \operatorname{area}(\mathbf{b}_d) \Big), \qquad \operatorname{IoU}(\mathbf{a}, \mathbf{b}) = \frac{\operatorname{area}(\mathbf{a} \cap \mathbf{b})}{\operatorname{area}(\mathbf{a} \cup \mathbf{b})} \tag{217.1} \]
Read the symbols
\(\mathcal{D}\)
all “person” detections on the prompt frame
\(s_d, \mathbf{b}_d\)
the score and the box \((x_0, y_0, x_1, y_1)\) of detection \(d\)
\(s_{\min}\)
the score threshold (0.5 in our runs)
\(k\)
the number of players you expect in the shot (3 for boxing, counting the referee; 2 for judo)
\(\mathcal{P}\)
the boxes that become SAM 2 prompts, one object id each
\(\operatorname{IoU}\)
box overlap; we use it only to check, after the fact, which human-labeled player each prompt belongs to

With this rule both clips are right: the boxing prompts overlap the true player boxes with IoU 0.939, 0.942 and 0.940, the judo prompts with 0.979 and 0.986. [Certain, our run] In a real system you would replace “largest” by something sturdier: a mask of the playing field, the team colors, or a jersey-number reader. Figure 1 lets you try the three rules.

true player box (from the human mask)prompt on a playerprompt on a non-playerother detections
Figure 1. Real RT-DETR detections on frame 0 of each clip (all “person” boxes with score ≥ 0.30: 22 for boxing, 7 for judo), and the prompts chosen by each rule of formula (217.1). A prompt counts as “on a player” when its box IoU with a true player box is at least 0.5.
How to read Figure 1

Start with Judo and “Top k by score”: both prompts land on bystanders at the edge of the mat. Switch to “Top k by area”: both players are found. Now choose Boxing and “Every box above threshold”: at 0.50 you get 8 prompts, 5 of them on spectators behind the ropes. Each one would become a SAM 2 track, cost compute, and produce masks you must later throw away. Move the threshold up: at 0.75 only the 3 people in the ring are left. A threshold alone works on this clip, but not on judo, where bystanders score higher than players.

One more detail matters on the judo clip. The two players’ true boxes overlap with IoU 0.40. [Certain, our run] A box is an ambiguous prompt when another person fills half of it, and SAM 2 must decide which of the two bodies you meant. Here it decided well (\(\mathcal{J} = 0.911\) and 0.917 on the prompt frame). [Certain, our run] When it does not, a mask prompt or an extra positive click on the right person removes the doubt.

Part 2: propagate and measure

With the prompts chosen, the tracking loop is short. All objects are added on frame 0 with their own ids, SAM 2 segments frame 0, then it propagates forward. For each frame and each object it returns mask logits and an object score logit \(\ell_t\), the output of the occlusion head: “is this object visible in this frame at all?”

\[ p_t^{\text{vis}} = \sigma(\ell_t) = \frac{1}{1 + e^{-\ell_t}}, \qquad \hat M_t = \begin{cases} \{\, x : z_t(x) \gt 0 \,\} & \text{if } \ell_t \gt 0 \\ \varnothing & \text{otherwise} \end{cases} \tag{217.2} \]
Read the symbols
\(\ell_t\)
the object score logit for this object in frame \(t\)
\(z_t(x)\)
the mask logit at pixel \(x\), resized to the original 854 × 480 frame
\(\hat M_t\)
the output mask: pixels with a positive logit, or nothing if the object is judged not visible
Algorithm 1: detect, prompt, propagate, score
  1. input: frames \(I_0, \dots, I_{T-1}\); expected number of players \(k\); threshold \(s_{\min}\)
  2. \(\mathcal{D} \leftarrow\) detector\((I_0)\), keep class “person” // RT-DETR
  3. \(\mathcal{P} \leftarrow\) the \(k\) largest boxes with \(s_d \ge s_{\min}\) // formula (217.1)
  4. session \(\leftarrow\) SAM 2 video session on all frames; add box \(\mathbf{b}_i\) as object \(i\) on frame 0, for every \(\mathbf{b}_i \in \mathcal{P}\)
  5. segment frame 0; check each prompt-frame mask (predicted IoU, or \(\mathcal{J}\) if you have a label) before going on
  6. for \(t = 1, \dots, T-1\): for each object, \(\hat M_t, \ell_t \leftarrow\) SAM 2 step // memory attention over prompted + 6 recent frames
  7.   apply formula (217.2); store \(\hat M_t\) and \(\ell_t\)
  8. score: for each object and each labeled frame, \(\mathcal{J}\) and \(\mathcal{F}\); average as in formula (217.3)

The clip score is the DAVIS one from session 204: average over frames for each object, then over objects, for \(\mathcal{J}\) and \(\mathcal{F}\) separately, then take the mean of the two:

\[ \begin{gathered} \mathcal{J}_f = \frac{\lvert \hat M_f \cap G_f \rvert}{\lvert \hat M_f \cup G_f \rvert}, \qquad \mathcal{F}_f = \frac{2 P_c R_c}{P_c + R_c} \\ \mathcal{J}\&\mathcal{F} = \frac12 \left( \frac{1}{\lvert \mathcal{O} \rvert} \sum_{o \in \mathcal{O}} \frac{1}{\lvert S \rvert} \sum_{f \in S} \mathcal{J}^{o}_f \;+\; \frac{1}{\lvert \mathcal{O} \rvert} \sum_{o \in \mathcal{O}} \frac{1}{\lvert S \rvert} \sum_{f \in S} \mathcal{F}^{o}_f \right) \end{gathered} \tag{217.3} \]
Read the symbols
\(G_f\)
the human mask of the object in frame \(f\)
\(P_c, R_c\)
contour precision and recall with the 8-pixel tolerance
\(\mathcal{O}\)
the tracked objects (players)
\(S\)
the scored frames: 1 to \(T-2\) here; in Part 4 only a few labeled frames

Results

Clip, object\(\mathcal{J}\)\(\mathcal{F}\)\(\mathcal{J}\&\mathcal{F}\)Worst frame (\(\mathcal{J}\))
Boxing, blue-helmet boxer0.9130.9690.941frame 12 (0.844)
Boxing, red-helmet boxer0.9080.9690.938frame 85 (0.678)
Boxing, referee0.8840.9690.926frame 26 (0.630)
Boxing, clip0.9020.9690.935
Judo, player 1 (thrower)0.8750.9070.891frame 16 (0.774)
Judo, player 2 (thrown)0.8400.8750.857frame 25 (0.680)
Judo, clip0.8580.8910.874

All numbers are SAM 2.1 Hiera-Tiny with the detector boxes as prompts. [Certain, our run] No player was ever lost: \(\mathcal{J}\) stayed above 0.5 on every scored frame (DAVIS “recall” = 1.00 for all five objects), and the object score logit never fell below 6.7 on these clips, so the occlusion head never switched a player off. [Certain, our run]

The bigger Hiera-B+ model gave \(\mathcal{J}\&\mathcal{F}\) 0.937 on boxing and 0.867 on judo: 0.002 better on one clip and 0.007 worse on the other. [Certain, our run] With 5 objects in 2 clips, a difference this small is noise, not a ranking. On these clips, the hard frames are hard for both models for the same reasons, which Part 3 looks at. Our laptop runs took 190 s for the 87 boxing frames with Tiny and 267 s with B+ (about 2.2 and 3.1 s per frame for 3 objects, including loading and resizing the frames). [Certain, our run] That is far from the 91 frames per second the authors measured on an A100 with the official code; a laptop GPU, float32 and a different implementation explain the gap, and you should not compare the two numbers.

track 1: correct pixelstrack 2: correct pixelstrack 3 (boxing): correct pixelspredicted, but not the playerplayer, but missed
Figure 2. Our SAM 2.1 Tiny run, replayed. Left: the predicted and human masks at 1/8 resolution for one frame (every 6th boxing frame and every 3rd judo frame are stored, plus the worst frames). Right: \(\mathcal{J}\) per frame for every player over the whole clip; the vertical line marks the chosen frame, and the ticks below show the stored frames.
How to read Figure 2

On boxing, step to frame 26: the referee (green) is partly behind the red-helmet boxer, and the missed pixels (dark) are the parts of him that show between the boxer’s arm and body. \(\mathcal{J}\) drops to 0.630, yet \(\mathcal{F}\) stays high, because most of his outline is still right. Step to frame 85: now the red-helmet boxer is behind the blue-helmet boxer. On judo, step to frame 25: player 2 is in the air, his legs are blurred, and SAM 2 misses most of them. In every case the track recovers a few frames later, because the prompted frame and the recent frames are still in the memory.

Part 3: where it fails

A high clip score hides the frames that matter. Read the per-frame curves, not just the average. In our two clips we saw three kinds of failure.

  • Partial occlusion by another person. Boxing frame 26: the referee is behind the red-helmet boxer, \(\mathcal{J} = 0.630\). Frames 82 to 86: the red-helmet boxer is behind the blue-helmet boxer, \(\mathcal{J}\) between 0.68 and 0.82. [Certain, our run] The mask shrinks to the visible part, which is correct behavior, but SAM 2 also loses some pixels that are visible in thin gaps.
  • Fast motion and blur. Judo frames 15 to 27, the throw: player 2’s \(\mathcal{J}\) stays between 0.68 and 0.83, lowest at frame 25, while his blurred legs swing through the air. [Certain, our run] Thin, blurred parts are where both \(\mathcal{J}\) and \(\mathcal{F}\) suffer.
  • Prompt errors that never heal. If the box is bad, the first mask is bad, and every later frame copies it from memory. We show this with jittered boxes below.

The fourth failure is the one every sports broadcast has, and DAVIS clips do not: a camera cut.

A camera cut, made on purpose

We built a 121-frame clip: boxing frames 0 to 43, then all 34 judo frames, then boxing frames 44 to 86. This is a cut away to another shot and a cut back, as a broadcast does for a replay or a close-up. During the judo shot the three boxing people are not in the picture, so the correct output for all three tracks is an empty mask. We ran SAM 2 on the whole clip with the same three boxing prompts. [Certain, our run]

What happened:

  • The blue-helmet boxer’s track jumped onto judo player 1 at the cut and stayed on him for all 34 judo frames. Its object score logit started at 3.7 and climbed to 8.1 by the end of the shot, as confident as it was on the real boxer (its lowest value on the 44 boxing frames before the cut was 7.9). [Certain, our run] The tracker had no idea anything was wrong.
  • The red-helmet boxer’s track sat on judo player 2 for 3 frames, the referee’s track for 7 frames (as a small blob), and then their object scores went below 0 and their masks became empty, which is the right answer. [Certain, our run]
  • At the cut back, all three tracks found their person again on the first frame: \(\mathcal{J} = 0.940\), 0.916 and 0.858. [Certain, our run] The prompted frame 0 is always in the memory, so SAM 2 could match the boxers again.

So the occlusion head sometimes catches a cut and sometimes does not. It was trained to say “this object is hidden”, not “this is a different camera”. A track that confidently follows the wrong person for a whole shot is the worst kind of error for analytics: the player’s distance and speed for those seconds are silently wrong.

The fix: shot-aware tracking

Cuts are easy to detect with a color histogram. Count the colors of each frame in \(8 \times 8 \times 8 = 512\) bins, normalize to sum 1, and compare neighboring frames:

\[ d_t = 1 - \sum_{b=1}^{512} \min\big( h_{t-1}(b),\ h_t(b) \big), \qquad \text{cut at } t \iff d_t \gt \theta \tag{217.4} \]
Read the symbols
\(h_t(b)\)
the fraction of pixels of frame \(t\) whose color falls in bin \(b\)
\(\sum_b \min(\cdot,\cdot)\)
histogram intersection: 1 for two frames with the same colors, 0 for two frames with no color in common
\(\theta\)
the cut threshold

On our spliced clip the two real cuts give \(d_t = 0.629\) and \(0.597\). Inside a shot, even with the moving fisheye camera, \(d_t\) is at most 0.035 (mean 0.016). [Certain, our run] Any threshold between 0.035 and 0.597 finds exactly the two cuts. The margin is wide here because the two sports have very different colors; two shots of the same field are much closer, and then a learned shot detector such as TransNet V2 is the safer choice. [Likely]

Algorithm 2: shot-aware tracking
  1. input: frames, the prompted frame 0, cut threshold \(\theta\), same-scene threshold \(\delta\)
  2. compute \(d_t\) for every frame (formula 217.4); cuts \(\leftarrow \{t : d_t \gt \theta\}\); split the video into shots
  3. for each shot, compare the histogram of its first frame with frame 0: keep the shot if the distance is below \(\delta\) // same camera view
  4. run Algorithm 1 on the kept frames only, in order; give every other frame an empty mask
  5. at a kept shot that starts after a gap, run the detector again and check each track against a detection (box IoU); re-prompt when they disagree

With \(\theta = 0.3\) and \(\delta = 0.3\): the judo shot is 0.631 away from frame 0 and is dropped; the boxing shot after it is 0.149 away and is kept. [Certain, our run] SAM 2 then sees exactly the 87 boxing frames of Part 2, so its masks are the ones from Part 2, and every judo frame gets an empty mask. Figure 3 compares the two modes.

red-helmet boxerblue-helmet boxerrefereedetected cut
Figure 3. Boxing frames 0–43, the judo shot, then boxing frames 44–86 (121 frames). Top: the cut score of formula (217.4). Middle: \(\mathcal{J}\) per frame for each boxing track (during the judo shot the correct mask is empty, so \(\mathcal{J} = 1\) for an empty prediction and \(\mathcal{J} = 0\) for any mask). Bottom: the object score logits of the naive run. In shot-aware mode, grey marks show the frames that SAM 2 never sees.
How to read Figure 3

In naive mode, follow the orange line through the yellow band: the blue-helmet boxer’s track has \(\mathcal{J} = 0\) for the whole judo shot, and its object score (bottom) stays well above 0. Its \(\mathcal{J}\&\mathcal{F}\) falls from 0.941 to 0.672. Switch to shot-aware: all three tracks are correct in the judo shot. Now move \(\theta\) above 0.597: only the first cut is found, the judo shot and the boxing shot after it look like one shot, the whole second half is dropped, and every player is missing for 43 frames (\(\mathcal{J}\&\mathcal{F}\) 0.625). Above 0.629 no cut is found, and you are back to the naive result. A threshold that is too high is as bad as no cut detection at all.

Worked example: the cut experiment in numbers

Naive run, \(\mathcal{J}\&\mathcal{F}\) over frames 1 to 119: red-helmet boxer 0.930, blue-helmet boxer 0.672, referee 0.887, mean 0.830. Shot-aware run: 0.956, 0.958, 0.947, mean 0.954. [Certain, our run]

Look closely at the shot-aware 0.954: it is higher than the 0.935 of the same masks on the boxing clip alone (Part 2). Nothing got better. The 34 judo frames are scored as perfect, because an empty prediction for an empty truth gives \(\mathcal{J} = 1\) and \(\mathcal{F} = 1\). Frames where the object is absent are easy points. When you compare two systems, compare them on the same frames, and also report the score on frames where the object is visible.

Prompt quality: jittered boxes and a single click

Detectors are not perfect, so we moved each corner of each prompt box by a random amount, up to a fraction \(a\) of the box width or height (two random seeds per level), and ran the whole clip again. We also replaced each box by one positive click at the box center. [Certain, our run]

Prompt on frame 0Boxing \(\mathcal{J}\&\mathcal{F}\) (seed 0 / seed 1)Judo \(\mathcal{J}\&\mathcal{F}\) (seed 0 / seed 1)
Detector box, as in Part 20.9350.874
Box corners moved up to \(a = 0.1\)0.477 / 0.8970.825 / 0.847
\(a = 0.2\)0.626 / 0.8820.650 / 0.849
\(a = 0.3\)0.437 / 0.7740.384 / 0.806
One positive click at the box center0.5720.862

All values are clip \(\mathcal{J}\&\mathcal{F}\), averaged over the players. [Certain, our run] Three lessons come out of this table.

  • Small box errors can be fatal, and it depends on luck. On boxing, a 10% jitter gave 0.897 with one seed and 0.477 with the other. In the bad case the red-helmet boxer’s box lost 24 pixels at the bottom and 13 at the right, and his first mask had \(\mathcal{J} = 0.203\); his track then scored 0.188 for the whole clip. [Certain, our run] Two seeds per level are too few to give a smooth curve; they are enough to show that the risk is real.
  • The first mask decides the track. Over all 40 tracks in this section (5 players × 8 prompt settings), the \(\mathcal{J}\) of the first mask and the \(\mathcal{J}\&\mathcal{F}\) of the whole track have a correlation of 0.974. Every track whose first mask had \(\mathcal{J}\) below 0.7 ended at \(\mathcal{J}\&\mathcal{F}\) 0.809 or lower; every track whose first mask had \(\mathcal{J}\) of 0.85 or more ended at 0.813 or higher. [Certain, our run] SAM 2 does not repair a bad start by itself: the prompted frame stays in memory and keeps telling it what the object looks like.
  • A click is ambiguous: part or whole? The center of a person’s box is often on the shirt. For the red-helmet boxer and the referee, SAM 2 segmented only the shirt (first-mask \(\mathcal{J}\) 0.232 and 0.416), and the tracks followed the shirts: \(\mathcal{J}\&\mathcal{F}\) 0.296 and 0.483. The blue-helmet boxer’s click gave the whole body (0.938), and on judo the clicks worked (0.862 for the clip). [Certain, our run] A box tells SAM 2 the extent of the object; a single click does not.

The practical rule: before you propagate, check every first mask. Without labels, use SAM 2’s predicted IoU for the prompt frame (the official image predictor returns it) and a simple sanity check, such as “a person mask should fill a good part of the person’s box”. If a mask fails, add a positive click on a missed body part or a negative click on the wrong person, and segment frame 0 again.

Part 4: J&F from a few labeled frames

DAVIS gives a human mask for every frame. Your own sports data will not: drawing a careful mask of a player takes minutes, so you will label a few frames per clip. How many do you need? Treat it as sampling. Let \(s_f\) be the per-frame score \(\tfrac12(\mathcal{J}_f + \mathcal{F}_f)\), averaged over players, for the \(n\) scorable frames. If you label a set \(S\) of \(k\) frames, your estimate and its standard error are:

\[ \hat m_S = \frac{1}{k} \sum_{f \in S} s_f, \qquad \operatorname{SE}(\hat m_S) = \frac{\sigma}{\sqrt{k}} \sqrt{\frac{n - k}{n - 1}} \tag{217.5} \]
Read the symbols
\(s_f\)
the \(\mathcal{J}\&\mathcal{F}\) of frame \(f\), averaged over the players
\(n\)
the number of frames you could label (32 for judo, 85 for boxing)
\(\sigma\)
the standard deviation of \(s_f\) over all \(n\) frames: how much the quality changes from frame to frame
\(\sqrt{(n-k)/(n-1)}\)
the finite-population correction: labels are drawn without repetition, and when \(k = n\) the error is exactly 0

A range of about \(\pm 1.645\,\operatorname{SE}\) holds the true value 90% of the time when the frames are chosen at random. The formula says two things. The error shrinks only as \(1/\sqrt{k}\): to halve it you need 4 times more labels. And it scales with \(\sigma\): a clip whose quality changes a lot from frame to frame (a judo throw) needs more labels than a smooth one (boxing).

Worked example: how many judo frames to label?

Judo, all 32 scorable frames: true \(\mathcal{J}\&\mathcal{F} = 0.874\), \(\sigma = 0.0558\). With \(k = 5\) random frames: \(\operatorname{SE} = 0.0558 / \sqrt{5} \times \sqrt{27/31} = 0.0233\), so the 90% range is about \(\pm 0.038\), a width of 0.077. In 400 random draws, 90% of the estimates fell in [0.838, 0.910], a width of 0.072: the formula is close. With 5 evenly spaced frames the estimate was 0.871, within 0.004 of the truth. [Certain, our run]

Boxing has a smaller \(\sigma = 0.0200\), so 5 random frames already give a 90% width of 0.027. [Certain, our run] In our 400 draws, the judo clip needed 17 random frames to get the 90% width below 0.03; boxing needed 5. [Certain, our run] Spreading the labels evenly over the clip helps, because it covers all phases of the action. But it can be unlucky too: with \(k = 1\) the single middle judo frame lands in the throw, and the estimate is 0.773, off by 0.101.

true \(\mathcal{J}\&\mathcal{F}\) (all frames)\(k\) evenly spaced frames90% of 400 random choices of \(k\) frames
Figure 4. Estimating the clip’s \(\mathcal{J}\&\mathcal{F}\) (formula 217.3) from only \(k\) labeled frames, using our real per-frame \(\mathcal{J}\) and \(\mathcal{F}\). The shaded band is computed in your browser from 400 random choices of \(k\) frames (seeded, so it is the same every time); the green line uses \(k\) evenly spaced frames.
How to read Figure 4

The band is wide on the left and closes to zero at \(k = n\), as formula (217.5) says. Compare the two clips at \(k = 5\): judo’s band is almost three times wider than boxing’s, because judo’s per-frame quality swings during the throw. The green line is often closer to the truth than the band’s edges, but it jumps around: a single evenly spaced choice can land on a hard frame. In practice, label a few frames spread over the clip, plus the frames you expect to be hard (occlusions, fast motion, right after cuts), and report those separately.

Algorithm 3: an evaluation plan for your own sports clips
  1. split the footage into shots (formula 217.4); sample shots, not only frames, so that one easy shot does not dominate
  2. in each sampled shot, label \(k\) frames spread over the shot (5 to 10), plus every frame within 5 frames after a cut back
  3. also label frames where the player is absent, but score them separately (they are “free” \(\mathcal{J} = 1\) points)
  4. compute \(\mathcal{J}\), \(\mathcal{F}\) and \(\mathcal{J}\&\mathcal{F}\) per player with the official tolerance; compute \(\sigma\) and the SE of formula (217.5)
  5. report: \(\mathcal{J}\&\mathcal{F}\) with a 90% range, \(\mathcal{J}\&\mathcal{F}\) on visible frames only, the number of frames with a mask on the wrong person, and the worst frames

The code

This is the code we ran, shortened only by removing logging. It needs torch, transformers 5.x, opencv-python, pillow and the DAVIS 2017 480p folder. Inference is deterministic for a fixed input (no sampling anywhere); the random box jitter uses a fixed NumPy seed.

import math, numpy as np, cv2, torch
from PIL import Image
from transformers import (AutoImageProcessor, AutoModelForObjectDetection,
                          Sam2VideoModel, Sam2VideoProcessor)
DEV = "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu")
ROOT = "DAVIS/"
def frame(seq, i): return Image.open(ROOT + f"JPEGImages/480p/{seq}/{i:05d}.jpg").convert("RGB")
def label(seq, i): return np.array(Image.open(ROOT + f"Annotations/480p/{seq}/{i:05d}.png"))

# 1) detector boxes -> prompts, formula (217.1): the k largest person boxes with score >= s_min
det_proc = AutoImageProcessor.from_pretrained("PekingU/rtdetr_r50vd")
det = AutoModelForObjectDetection.from_pretrained("PekingU/rtdetr_r50vd").eval()
def player_boxes(img, k, s_min=0.5):
    with torch.no_grad():
        out = det(**det_proc(images=img, return_tensors="pt"))
    r = det_proc.post_process_object_detection(out, target_sizes=[img.size[::-1]], threshold=s_min)[0]
    boxes = [b.tolist() for l, b in zip(r["labels"], r["boxes"]) if det.config.id2label[int(l)] == "person"]
    return sorted(boxes, key=lambda b: -(b[2] - b[0]) * (b[3] - b[1]))[:k]

# 2) SAM 2: one object id per box on frame 0, then propagate (Algorithm 1)
sam = Sam2VideoModel.from_pretrained("facebook/sam2.1-hiera-tiny").to(DEV).eval()
proc = Sam2VideoProcessor.from_pretrained("facebook/sam2.1-hiera-tiny")
def track(frames, boxes):
    sess = proc.init_video_session(video=[np.array(f) for f in frames], inference_device=DEV, dtype=torch.float32)
    ids = list(range(1, len(boxes) + 1))
    proc.add_inputs_to_inference_session(sess, frame_idx=0, obj_ids=list(ids), input_boxes=[boxes])   # pass a copy: the session empties this list
    masks = {i: [None] * len(frames) for i in ids}; scores = {i: [None] * len(frames) for i in ids}
    with torch.no_grad():
        sam(inference_session=sess, frame_idx=0)               # segment the prompted frame first
        for out in sam.propagate_in_video_iterator(sess):
            m = proc.post_process_masks([out.pred_masks], original_sizes=[[sess.video_height, sess.video_width]],
                                        binarize=True)[0]
            for j, oid in enumerate(out.object_ids):
                masks[oid][out.frame_idx] = m[j, 0].cpu().numpy().astype(bool)
                scores[oid][out.frame_idx] = float(out.object_score_logits.reshape(-1)[j])   # l_t, formula (217.2)
    return masks, scores

# 3) DAVIS metrics, formula (217.3): J = IoU, F = boundary F-measure with ceil(0.008 * diagonal) pixels
def J(p, g):
    u = (p | g).sum(); return 1.0 if u == 0 else (p & g).sum() / u
def bmap(m):                                                 # boundary pixels, as seg2bmap in the DAVIS code
    e = np.zeros_like(m); s = np.zeros_like(m); se = np.zeros_like(m)
    e[:, :-1] = m[:, 1:]; s[:-1, :] = m[1:, :]; se[:-1, :-1] = m[1:, 1:]
    b = (m ^ e) | (m ^ s) | (m ^ se); b[-1, :] = m[-1, :] ^ e[-1, :]; b[:, -1] = m[:, -1] ^ s[:, -1]; b[-1, -1] = 0
    return b
def F(p, g):
    r = math.ceil(0.008 * math.hypot(*p.shape)); yy, xx = np.mgrid[-r:r + 1, -r:r + 1]
    disk = (xx**2 + yy**2 <= r * r).astype(np.uint8)
    bp, bg = bmap(p), bmap(g); nP, nG = bp.sum(), bg.sum()
    if nP == 0 or nG == 0: return float(nP == nG)
    prec = (bp & cv2.dilate(bg.astype(np.uint8), disk).astype(bool)).sum() / nP
    rec = (bg & cv2.dilate(bp.astype(np.uint8), disk).astype(bool)).sum() / nG
    return 0.0 if prec + rec == 0 else 2 * prec * rec / (prec + rec)

# 4) cut score, formula (217.4): 1 - histogram intersection of 8x8x8 color bins
def hist(img):
    a = (np.asarray(img) // 32).reshape(-1, 3).astype(int)
    h = np.bincount(a[:, 0] * 64 + a[:, 1] * 8 + a[:, 2], minlength=512).astype(float)
    return h / h.sum()
def cut_scores(frames):
    H = [hist(f) for f in frames]
    return [0.0] + [1 - np.minimum(H[t - 1], H[t]).sum() for t in range(1, len(H))]

# run one clip
seq, k = "judo", 2
frames = [frame(seq, i) for i in range(34)]
boxes = player_boxes(frames[0], k)
masks, scores = track(frames, boxes)
g0 = label(seq, 0)                                            # match each track to a labeled player by IoU on frame 0
match = {o: max(set(np.unique(g0)) - {0}, key=lambda L: J(masks[o][0], g0 == L)) for o in masks}
Js = {o: [J(masks[o][f], label(seq, f) == match[o]) for f in range(1, 33)] for o in masks}
Fs = {o: [F(masks[o][f], label(seq, f) == match[o]) for f in range(1, 33)] for o in masks}
jf = (np.mean([np.mean(v) for v in Js.values()]) + np.mean([np.mean(v) for v in Fs.values()])) / 2
print(f"J&F = {jf:.3f}")          # our run: 0.874

Three details in this code are easy to get wrong. The sam(inference_session=sess, frame_idx=0) call before propagation is needed; without it the Hugging Face iterator raises “Cannot determine the starting frame index”. [Certain, our run] The session keeps a reference to the obj_ids list you pass and removes ids from it once their prompt has been processed, so if you reuse that list later you get an empty list and no error; pass a copy. [Certain, our run, transformers 5.23] And the official sam2 package has the same loop under other names: init_state, add_new_points_or_box and propagate_in_video. [Certain]

The results sheet

SettingValue
Hardware, softwareApple M5 Pro, MPS, float32; torch 2.15.0, transformers 5.23.0
DataDAVIS 2017 480p: boxing-fisheye (87 frames, 3 objects), judo (34 frames, 2 objects)
Detector, prompt ruleRT-DETR R50 (COCO), class “person”, \(s_{\min} = 0.5\), the \(k\) largest boxes
TrackerSAM 2.1 Hiera-Tiny, boxes on frame 0, forward propagation, default memory (prompted + 6 recent)
MetricDAVIS \(\mathcal{J}\) and \(\mathcal{F}\), 8-pixel boundary tolerance, frames 1 to \(T-2\), mean over objects
\(\mathcal{J}\&\mathcal{F}\), boxing / judo0.935 / 0.874 (B+: 0.937 / 0.867)
Cut test, naive / shot-aware0.830 / 0.954; one track on the wrong person for 34 frames in the naive run
Cut detector marginreal cuts 0.597 and 0.629, largest within-shot score 0.035
Prompt qualitybox jitter of 10% gave boxing \(\mathcal{J}\&\mathcal{F}\) 0.477 (one seed of two); a center click gave 0.572 (boxing) and 0.862 (judo); first-mask \(\mathcal{J}\) vs track \(\mathcal{J}\&\mathcal{F}\) correlation 0.974 over 40 tracks
Labels needed (90% width under 0.03)boxing: 5 random frames; judo: 17

Where this is used

ApplicationPipelineWhat breaks first
Player tracking for sports analyticsdetector → SAM 2 per player → field calibration → speed and distanceidentity after cuts and in crowds (Part 3); a track on the wrong person gives silently wrong statistics
Broadcast graphics (highlighting a player, drawing behind players)a click or box from an operator → SAM 2 masks → compositingboundary quality (\(\mathcal{F}\)) on fast motion and blur
Highlight and key-moment clipstracks + events from trajectories (session 207)tracks that drift during the very action that matters (the judo throw)
Labeling data for training a faster modeldetector + SAM 2 as a pseudo-labeler, humans fix a few frames, as in SAM 2’s own data engineerrors at the prompt frame copied into every frame
Object-level video search and Video-RAGmasklets stored per shot as index entries (session 210)shot boundaries; the same person in two shots must be linked by re-identification, not by SAM 2’s memory
Privacy (blurring faces or bystanders)face or person detector → SAM 2 → blur every framerecall: one missed frame shows the face; objects coming into view after frame 0 need new prompts

Common mistakes

  • Choosing players by detector score. On our judo clip the three highest-scoring people are bystanders. Use what you know about the scene: size, position on the field, team colors.
  • Not checking the prompt frame. If the first mask is wrong, every later frame copies it. Look at frame 0 (or at SAM 2’s predicted IoU) before propagating hundreds of frames.
  • Running SAM 2 across camera cuts. The memory assumes one continuous shot. In our test one track followed a judo player for 34 frames with high confidence. Detect cuts first and track shot by shot.
  • Trusting the object score as a cut detector. It turned off two of three tracks after the cut, and kept the third on the wrong person with a logit of 8. Motion cues help inside a shot (SAMURAI adds motion cues to SAM 2’s memory selection for this reason [Certain], with a Kalman filter that predicts the box [Likely]), but a cut breaks motion too.
  • Letting absent frames inflate the score. Empty prediction on empty truth is \(\mathcal{J} = 1\) and \(\mathcal{F} = 1\). A clip with many absent frames looks better than it is. Report visible-frame scores too.
  • Scoring the prompt frame. The DAVIS protocol skips the first frame (and the last). Including frame 0 rewards a good prompt, not good tracking.
  • Calling a 0.005 difference a win. With 5 objects in 2 clips, Tiny and B+ were within 0.007 of each other in both directions. You need many more objects, and a confidence interval, to rank models.
  • Labeling only easy frames. Few labeled frames give a noisy estimate (formula 217.5), and frames picked by convenience are biased. Spread them, add the hard frames on purpose, and report both.

Interview questions

Design a pipeline that gives you a mask for every player in a match video.core
Split the video into shots with a shot-boundary detector. In each shot, run a person detector on the first frame (or a few key frames), keep the players with a scene rule (on the field, large enough, team colors), and give each box to SAM 2 as a prompt with its own object id. Propagate through the shot. Run the detector again every N frames and after every cut; match detections to tracks by box IoU, the association step of box trackers such as ByteTrack; start new tracks for new players and re-prompt tracks that disagree with the detector. Link tracks across shots with a re-identification model (appearance, jersey number). Measure \(\mathcal{J}\&\mathcal{F}\) on a few labeled frames per shot, plus the number of frames with a mask on the wrong person.
Why can’t you take the top-k detections by score as the players?basic
The detector’s score is “how sure am I that this is a person”, not “is this a player”. Bystanders, coaches and spectators are people too, and they are often easier to detect: standing still, not occluded. In our judo clip the three top-scoring boxes (0.939, 0.933, 0.932) were bystanders; the players scored 0.925 and 0.904. You need a second rule that uses scene knowledge, such as size, position on the field, or team colors.
What happens when SAM 2 tracks through a camera cut? Why?core
After the cut, memory attention still looks for the pixels that best match the stored memories. If something in the new shot looks similar (another person in light clothes), the track can jump onto it, and the recent-frame memories then reinforce the wrong target frame after frame. In our test one boxer’s track followed a judo player for the whole 34-frame shot with an object score logit rising from 3.7 to 8.1. The occlusion head was trained to detect a hidden object, not a change of camera, so it does not reliably catch cuts. Detect cuts separately and track shot by shot.
How did the tracks recover after the cut back, and when would they not?advanced
The prompted frame stays in SAM 2’s memory bank (it does not drop out of the FIFO of recent frames), so at the cut back the boxers could be matched against it again, and \(\mathcal{J}\) was 0.94, 0.92 and 0.86 on the first frame back. Recovery would fail when the view at the cut back looks very different from the prompted frame (a different camera angle, a close-up, a long time later with players in new positions), or when a similar-looking person is closer in appearance than the true player. Then you need re-detection and re-identification, not memory.
Write the DAVIS \(\mathcal{J}\&\mathcal{F}\) for a clip with several players. Which frames are scored?basic
For each object, average \(\mathcal{J}\) (mask IoU) over the scored frames and average \(\mathcal{F}\) (boundary F-measure, tolerance \(\lceil 0.008 \times \text{diagonal} \rceil\) pixels) over the same frames. Average each over objects, then \(\mathcal{J}\&\mathcal{F}\) is the mean of the two. The official code skips the first frame (it has the given annotation or prompt) and the last frame. Absent objects with an empty prediction score \(\mathcal{J} = \mathcal{F} = 1\).
You can afford to label only 5 frames per clip. How good is your \(\mathcal{J}\&\mathcal{F}\) estimate?core
Treat it as sampling without replacement: \(\operatorname{SE} = \sigma/\sqrt{k}\,\sqrt{(n-k)/(n-1)}\), where \(\sigma\) is the frame-to-frame standard deviation of \(\mathcal{J}\&\mathcal{F}\). For our judo clip (\(\sigma = 0.0558\), \(n = 32\)), \(k = 5\) gives SE 0.023 and a 90% range of about \(\pm 0.038\); random draws gave a width of 0.072. For boxing (\(\sigma = 0.020\)) the same 5 frames give a width of 0.027. So the answer depends on how much quality varies within the clip; estimate \(\sigma\) from a pilot, and add the hard frames on purpose.
Box prompt, click prompt or mask prompt: which would you use for players, and why?core
A box from a detector is the cheapest automatic prompt and fixes the extent of the object, which a single click does not: in our boxing run, a click at the box center gave only the shirt for two of three people, and their tracks scored 0.296 and 0.483. Its weakness is overlap: two players in contact share much of a box (our judo boxes overlap with IoU 0.40), and a bad box gives a bad first mask that the tracker copies forward. A mask prompt is the strongest but needs a segmenter or a human. In practice: box from the detector, check the first mask, and add a positive or negative click when it is wrong.
Hiera-B+ scored 0.937 and Tiny 0.935 on boxing, but B+ was worse on judo. What do you conclude?advanced
Nothing about which model is better: 5 objects in 2 clips cannot separate differences of a few thousandths, and the per-frame errors are dominated by the same hard frames (occlusion, blur) for both. You would need many more clips and objects, a paired comparison on the same frames, and a confidence interval (for example, a bootstrap over objects). What you can conclude is that Tiny is good enough for these clips at less than half the parameters and lower latency.
How would you detect camera cuts, and how do you pick the threshold?basic
Compare color histograms of neighboring frames: \(d_t = 1 - \sum_b \min(h_{t-1}(b), h_t(b))\), and call a cut when \(d_t\) is above a threshold. Pick the threshold from data: measure \(d_t\) inside shots (camera motion, flashes) and at labeled cuts, and put the threshold in the gap. In our clip the gap was 0.035 to 0.597, so 0.3 is safe. For gradual transitions (fades, wipes) or two shots of the same field, use a learned detector such as TransNet V2.
Why did \(\mathcal{J}\) drop at boxing frame 26 while \(\mathcal{F}\) stayed high?advanced
The referee was partly behind a boxer. The predicted mask missed some of his visible pixels in thin gaps, so the area overlap (\(\mathcal{J}\)) fell to 0.630. But \(\mathcal{F}\) measures the outline within an 8-pixel tolerance, and most of his outline was still where it should be. \(\mathcal{J}\) punishes missing area, \(\mathcal{F}\) punishes wrong outlines; occlusion with thin visible slivers hurts \(\mathcal{J}\) much more than \(\mathcal{F}\).

Exercises

  1. In Figure 1, choose Boxing and “Every box above threshold”. What is the lowest threshold that gives exactly the 3 people in the ring? Why does no threshold work for judo?
  2. A track has object score logit \(\ell_t = 8.1\). What visibility probability does SAM 2 assign? What about \(\ell_t = -1.3\)?
  3. Using the results table, compute the boxing clip \(\mathcal{J}\&\mathcal{F}\) from the per-object \(\mathcal{J}\) and \(\mathcal{F}\) values.
  4. In the naive cut run, the blue-helmet boxer has \(\mathcal{J} = 0\) on 34 of the 119 scored frames. If its \(\mathcal{J}\) and \(\mathcal{F}\) on all other scored frames were 1, what would its \(\mathcal{J}\&\mathcal{F}\) be? Compare with the measured 0.672.
  5. For the boxing clip (\(\sigma = 0.0200\), \(n = 85\)), compute the SE of formula (217.5) for \(k = 10\) and the 90% range width.
  6. Run Algorithm 2 with a gap: boxing frames 0–43, then boxing frames 60–86 (16 frames skipped). Predict whether the tracks recover, then compare with our run below.
Answerscheck
1. The weakest of the three in-ring people scores 0.916 and the best spectator 0.730, so any threshold from 0.74 to 0.91 works (0.74 in the slider’s 0.01 steps). For judo, bystanders score 0.939, 0.933 and 0.932, higher than both players (0.925, 0.904), so every threshold that keeps the players also keeps bystanders. 2. \(\sigma(8.1) = 1/(1 + e^{-8.1}) = 0.9997\); \(\sigma(-1.3) = 0.214\), so that mask is dropped. 3. Mean \(\mathcal{J} = (0.913 + 0.908 + 0.884)/3 = 0.902\); mean \(\mathcal{F} = 0.969\); \(\mathcal{J}\&\mathcal{F} = (0.9017 + 0.9689)/2 = 0.935\) (keep 4 digits, or the rounding gives 0.936). 4. \(\mathcal{J}\) and \(\mathcal{F}\) would each be \(85/119 = 0.714\), so \(\mathcal{J}\&\mathcal{F} = 0.714\). The measured 0.672 is lower because the other frames are not perfect either. 5. \(\operatorname{SE} = 0.0200/\sqrt{10} \times \sqrt{75/84} = 0.00598\); the 90% width is \(2 \times 1.645 \times 0.00598 = 0.0197\), close to the 0.020 we measured from random draws. 6. In our run all three tracks recovered at once. With 16 frames skipped, \(\mathcal{J}\) on the first frame after the jump was 0.936, 0.902 and 0.942, and the clip \(\mathcal{J}\&\mathcal{F}\) was 0.934; with 30 frames skipped, 0.872, 0.915 and 0.799, and 0.936. [Certain, our run] (These clip scores use different frames from the 0.935 of Part 2, so do not compare them closely.) It works here because the camera view is the same and the players still look like they did in the prompted frame 0, which never leaves the memory. Coming back from a different camera angle, or after minutes of play, you should expect failures: re-detect and check each track against a detection, as in step 5 of Algorithm 2.

Key takeaways

  • The pipeline is detector → choose players → box prompts → SAM 2 propagation → \(\mathcal{J}\&\mathcal{F}\). Choosing players is a real step: by score alone, our judo prompts landed on bystanders (formula 217.1).
  • On two DAVIS sports clips, SAM 2.1 Tiny with detector boxes reached \(\mathcal{J}\&\mathcal{F}\) 0.935 (boxing) and 0.874 (judo) and never lost a player; the hard frames were occlusions by other people and the blurred legs of a thrown player.
  • Across a camera cut, one track followed the wrong person for the whole 34-frame shot with a high object score (formula 217.2 did not save it). A histogram cut detector (217.4) with shot-by-shot tracking fixed it.
  • Absent frames are free points in \(\mathcal{J}\&\mathcal{F}\) (217.3); report visible-frame scores and wrong-person frames too.
  • With \(k\) labeled frames, the error of your \(\mathcal{J}\&\mathcal{F}\) estimate is about \(\sigma/\sqrt{k}\) (217.5): 5 frames were enough for boxing, 17 for the judo throw.

Key formulas

\[ \mathcal{P} = \operatorname{top}_k\big(\{d : s_d \ge s_{\min}\}, \operatorname{area}\big), \qquad \hat M_t = \varnothing \text{ if } \ell_t \le 0, \qquad d_t = 1 - \sum_b \min\big(h_{t-1}(b), h_t(b)\big) \]
\[ \mathcal{J}\&\mathcal{F} = \tfrac12\big(\overline{\mathcal{J}} + \overline{\mathcal{F}}\big), \qquad \operatorname{SE}(\hat m_S) = \frac{\sigma}{\sqrt{k}}\sqrt{\frac{n-k}{n-1}} \]

References for this session

  1. Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
  2. Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
  3. Pont-Tuset et al. (2017). The 2017 DAVIS Challenge on Video Object Segmentation
  4. Cheng & Schwing (2022). XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
  5. Zhao, Lv, Xu, Wei, Wang, Dang, Liu & Chen (2023). DETRs Beat YOLOs on Real-time Object Detection (RT-DETR)
  6. Meta FAIR (2024). SAM 2 official repository: checkpoints, SAM 2.1 speed and accuracy table, video predictor
  7. Hugging Face (2025). transformers: SAM 2 Video (Sam2VideoModel, Sam2VideoProcessor) and the facebook/sam2.1-hiera-tiny checkpoint
  8. DAVIS challenge organizers (2017). DAVIS 2017 evaluation code (J and F measures)
  9. Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
  10. Zhang, Sun, Jiang, Yu, Weng, Yuan, Luo, Liu & Wang (2021). ByteTrack: Multi-Object Tracking by Associating Every Detection Box
  11. Yang, Huang, Chai, Jiang & Hwang (2024). SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory