Video object segmentation and tracking: from VOS to SAM 2
You draw a mask on an object in the first frame of a video, and the model must follow that object, pixel by pixel, through motion, changes of look and occlusion. This session builds the idea step by step: the task and its metrics (J and F), memory-based matching as attention over past frames (STM, XMem), SAM 2 with its streaming memory and point, box and mask prompts, and referring segmentation from a sentence.
What you will learn
- What semi-supervised video object segmentation (VOS) is, and how it differs from tracking boxes, video instance segmentation and referring segmentation
- Why following the previous mask drifts and gets lost, and why matching against a memory of past frames works better
- The memory read of STM and its successors as plain attention: query pixels, memory keys, mask values, with numbers
- How XMem keeps memory bounded on long videos with sensory, working and long-term memory, and how much memory that saves
- How SAM 2 works as a streaming model: memory attention, a memory bank of prompted and recent frames, object pointers, an occlusion head, and prompts on any frame
- How the metrics J (region IoU), F (boundary accuracy), J&F, recall and decay are computed, and what they hide
- How referring VOS segments an object from a sentence, either end to end or with a detector plus SAM 2
Why this matters
You click on a football player in the first frame of a match video. A good system then gives you that player’s exact outline in every later frame, while he runs, turns, gets smaller, and disappears behind other players for a second. This is video object segmentation, and it is now a one-click tool.
In session 168 we met SAM, which turns a click into a mask on one image, and had a first look at SAM 2. In session 203 we gave a whole clip one label. This session sits between the two: we follow one object through time, at the level of pixels. The problem is harder than it looks. The object changes its look, other objects look similar, and when it is hidden there is nothing to follow at all.
The solution used by all strong systems since 2019 is one idea that you already know from transformers: keep a memory of past frames with their masks, and let every pixel of the new frame attend to that memory. STM introduced it, XMem made it work on long videos, and SAM 2 turned it into a promptable model trained on 35.5 million masks. [Certain]
Real uses: rotoscoping in video editing (cutting a person out of a shot), sports analytics, annotation tools that create training data for self-driving cars, medical video, robotics, and object-level search in video archives. In this course, it is also the base of session 217.
The task: a mask in frame 1, follow the object
A video is a list of frames \(I_1, \dots, I_T\). In semi-supervised VOS, someone gives the true mask \(M_1\) of one or more objects in the first frame. The model must output the masks \(\hat M_2, \dots, \hat M_T\) of the same objects in all later frames. “Semi-supervised” only means that one frame is labeled at test time; the model itself was trained earlier on other videos. Formally, at each step the model may use the current frame and everything it has seen so far:
- Read the symbols
- \(I_t\)
- frame \(t\) of the video, an RGB image
- \(M_1\)
- the given mask of the first frame: for each pixel, which object it belongs to (or background)
- \(\hat M_s\)
- the model’s own predicted mask for frame \(s\); there is no ground truth after frame 1, so the model must trust its own past answers
- \(f_\theta\)
- the network, with weights \(\theta\) learned on training videos; it does not train on the test video
Formula (204.1) says the model is online (also called streaming or causal): frame \(t\) is processed with frames \(1, \dots, t-1\) only, never with future frames. That is what lets it run on a live camera. Notice also the hidden danger: past predictions \(\hat M_s\) feed into the future. One wrong mask can be copied forward, frame after frame. This is called drift.
The same idea appears in several related tasks. It helps to keep them apart:
| Task | Input at test time | Output |
|---|---|---|
| Semi-supervised VOS | mask(s) in the first frame | masks of the same objects in every frame |
| Interactive VOS / promptable visual segmentation | clicks, boxes or masks on any frames, added over time | masks in every frame, refined after each new prompt |
| Unsupervised VOS | nothing | masks of the main moving objects |
| Referring VOS | a sentence, such as “the dog jumping over the fence” | the mask of the object the sentence describes, in every frame |
| Video instance segmentation | a fixed list of classes | masks, class labels and identities of all objects of those classes |
| Single-object tracking | a box in the first frame | a box in every frame |
| Multi-object tracking | a detector for a class (for example “person”) | boxes with stable identities over time |
Masks are much richer than boxes: a box around a running person contains a lot of background, and a box tells you nothing about which pixels are the person. But they are also more expensive to compute and to label. DAVIS 2017, the classic benchmark, has only 150 sequences with 10,459 annotated frames and 376 objects, about 70 frames per sequence, evaluated at 480p. [Certain] This tiny size shaped the whole field: models had to learn general “matching” skills, not memorize objects.
Three ways to follow an object
Suppose each pixel of each frame has a feature vector: a short list of numbers produced by an image encoder that describes what the pixel looks like (its color, texture and what object part it is). How can we use the features to find the object in frame \(t\)?
- Propagate from the previous frame. Take \(\hat M_{t-1}\), and move it a little to fit frame \(t\). Early methods (MaskTrack, 2017) worked like this. [Likely] It is good at smooth motion. But if the object is fully hidden for one frame, \(\hat M_{t-1}\) is empty, and an empty mask propagates to an empty mask forever. The object is lost.
- Match against the first frame. For each pixel of frame \(t\), find the most similar pixels in frame 1, and copy their label (object or background). This never gets lost after an occlusion, because frame 1 is always there. But objects change: the player turns around, the light changes. After a while, nothing in frame 1 looks like the object any more.
- Match against a memory. Keep several past frames with their masks: the first frame (true mask, reliable but old) and some recent frames (predicted masks, fresh but possibly wrong). Match against all of them at once. This combines the strengths of both.
Option 3 is the memory network idea of STM (Space-Time Memory networks, Oh et al., 2019). [Certain] Each memory frame is stored as a set of keys (features that are used for matching) and values (features that carry the mask information). Each pixel of the current frame produces a query. Then the read is exactly attention, as in session 17:
- Read the symbols
- \(j\)
- one pixel of the current frame (the query side); there are \(HW\) of them
- \(i\)
- one pixel of one memory frame; the sum runs over all pixels of all memory frames together
- \(\mathbf{q}_j, \mathbf{k}_i\)
- query and key vectors (for example 64 numbers each), made from image features only
- \(\mathbf{v}_i\)
- the value of memory pixel \(i\): features that encode both what it looked like and whether it was inside the mask
- \(\tau\)
- a temperature; smaller \(\tau\) makes each query attend to fewer, closer matches (in a transformer \(\tau = \sqrt{d}\))
- \(\hat{\mathbf{v}}_j\)
- the read-out for pixel \(j\); a decoder turns it into the mask probability
This is space-time attention: the softmax runs over every position in every memory frame, so a pixel can match a pixel anywhere, even far away and many frames ago. There is no assumption that the object moved only a little. That is why memory methods recover after occlusions and fast motion.
In the simplest case, the value is just the mask label: \(v_i = 1\) inside the object and \(0\) outside. Then \(\hat v_j\) is a weighted vote: the fraction of attention that pixel \(j\) gives to object pixels. Figure 1 runs this read one line at a time, and Figure 2 runs it on every pixel of a whole toy video.
Take 2-D unit features described by an angle (think of it as a hue). The memory has four pixels: two object pixels at \(0^\circ\) and \(10^\circ\) (\(v = 1\)), and two background pixels at \(150^\circ\) and \(220^\circ\) (\(v = 0\)). Use \(\tau = 0.1\). For unit vectors, \(\mathbf{q}^\top \mathbf{k} = \cos(\text{angle difference})\).
A query pixel at \(40^\circ\): scores \(\cos(40^\circ)/0.1 = 7.66\), \(\cos(30^\circ)/0.1 = 8.66\), \(\cos(110^\circ)/0.1 = -3.42\), \(\cos(180^\circ)/0.1 = -10.0\). The softmax gives weights \(0.269, 0.731, 0.000, 0.000\). The read-out is \(0.269 + 0.731 = 1.000\): object.
A query pixel at \(80^\circ\), halfway between object and background: scores \(1.74, 3.42, 3.42, -7.66\); weights \(0.085, 0.458, 0.458, 0.000\); read-out \(0.542\). It is barely object. If the object keeps changing color in this direction, it will soon be read as background. This is why the memory must contain recent views of the object.
- for each memory pixel \(i\): \(s_i \leftarrow \mathbf{q}^\top \mathbf{k}_i / \tau = \cos(\theta_q - \theta_i)/\tau\)
- \(m \leftarrow \max_i s_i\) // subtracted so that exp never overflows
- \(e_i \leftarrow \exp(s_i - m)\)
- \(Z \leftarrow \sum_i e_i\)
- \(A_i \leftarrow e_i / Z\) // the softmax of (204.2)
- \(\hat v \leftarrow \sum_i A_i v_i\)
- object if \(\hat v \gt 0.5\), else background
How to read Figure 1
Press Play. The first four steps measure the angle from the query arrow to each key and turn it into a score. Then the scores go through the softmax, one line at a time, and the weights \(A_i\) of the object keys are added up into \(\hat v\). At \(80^\circ\) the query is \(70^\circ\) from the object key at \(10^\circ\) and also \(70^\circ\) from the background key at \(150^\circ\): two equal scores, so they get equal weights (0.458 each), and the result 0.542 is barely “object”. Choose \(40^\circ\): almost all weight goes to the two object keys. Choose \(120^\circ\) (Exercise 1): it goes to the \(150^\circ\) background key. Now raise \(\tau\) to 1: the weights become flat and every key gets a share, so the vote is soft. A small \(\tau\) is what lets one pixel pick its few true matches among thousands of memory positions.
How to read Figure 2
“Play frames” moves the frame slider through the video, so you can watch the predicted mask follow (or lose) the block. Start with “Previous frame only”. Everything is perfect until the object is fully hidden at frame 22; the predicted mask becomes empty, and an empty memory can only produce empty masks. J stays at 1 during the occlusion (an empty prediction for an empty truth is correct), then drops to 0 forever. Now choose “First frame only”: the object is found again after the pillar, but its color drifts away from frame 1, and in the last frames J falls. With “First + 6 recent”, both problems are fixed. Then move the color-change slider to 2.6: the object comes back from behind the pillar looking too different from frame 1, and all 6 recent frames show only the pillar. Raise \(N\) to 7 or 8 so that the memory reaches back to a frame before the occlusion: it is recovered. That is the argument for a long memory, and the problem XMem solves.
What is in a key and a value
In STM, both the memory and the query frame go through encoders. The memory encoder sees the frame and its mask, so its values know “this was the object”. The query encoder sees only the new frame. A decoder takes the read-out \(\hat{\mathbf{v}}_j\) together with the query frame’s own features and predicts the mask at full resolution. Features live on a grid 16 times smaller than the image: a 480 × 864 frame becomes a 30 × 54 grid, 1,620 positions. With several objects, the same frame is matched once per object (or with shared keys and one value per object).
Later work improved the read without changing the idea. STCN (Cheng et al., 2021) computes keys from the image only, so the keys of a memory frame do not depend on which object is being tracked, and can be shared by all objects. [Certain] It also replaces the dot product by the negative squared distance:
Inside a softmax over \(i\), the term \(\lVert \mathbf{q}_j \rVert^2\) is the same for every memory pixel and cancels. The term \(-\lVert \mathbf{k}_i \rVert^2\) does not: with a plain dot product, a memory key with a large norm wins for many queries, and a few keys take most of the attention. The L2 form removes that bias, so more of the memory is actually used. XMem (second line) adds two learned factors: a shrinkage term \(s_i \ge 1\) per memory key (how confident and sharp this key is) and a selection term \(e_{cj} \in [0, 1]\) per query and channel (which feature channels matter for this query). With all \(s\) and \(e\) equal to 1 it is the L2 similarity again. [Certain]
The memory problem on long videos: XMem
STM adds a new memory frame every few frames (every 5th frame is common) and never deletes one. [Likely] The cost of formula (204.2) is proportional to the memory size, and so is the GPU memory. On a 70-frame DAVIS clip this is fine. On a 10-minute video it is not.
Assume 30 × 54 = 1,620 key positions per memory frame, one memory frame every 5 video frames, keys with 64 numbers and values with 512 numbers, stored in 16-bit floats (the key and value sizes of XMem). [Certain] A 10-minute video at 24 frames per second has 14,400 frames, so 2,880 memory frames (the first frame plus one every 5 frames): 2,880 × 1,620 = 4,665,600 memory positions. Storage: \(4{,}665{,}600 \times (64 + 512) \times 2\) bytes = 5.37 GB, for one object. Compute: each of the 1,620 query positions compares with all of them, \(1{,}620 \times 4{,}665{,}600 \times 64 \approx 4.8 \times 10^{11}\) multiply-adds per frame, just for the similarities. The model gets slower with every frame, and eventually runs out of memory.
XMem (Cheng and Schwing, 2022) fixes this with three memory stores, named after a model of human memory (Atkinson–Shiffrin). [Certain]
- Sensory memory: a small hidden state (64 channels) per position, updated by a GRU (a simple recurrent cell) at every frame. It carries short-term, smooth information, like where the object just was.
- Working memory: high-resolution keys and values of recent memory frames, exactly as in STM. A new memory frame is added every \(r = 5\) frames. Its size is kept between \(T_{\min} = 5\) and \(T_{\max} = 10\) frames.
- Long-term memory: a compact set of prototypes, at most \(LT_{\max} = 10{,}000\) elements.
When the working memory reaches \(T_{\max}\) frames, XMem keeps the first frame (it has the true mask) and the most recent \(T_{\min} - 1\) frames. The other \(T_{\max} - T_{\min} = 5\) frames are consolidated: from their \(5 \times 1{,}620 = 8{,}100\) positions, the 128 most-used ones become prototype keys. “Used” means: they received a lot of attention weight in past reads. Each prototype’s value is then a softmax-weighted average of the candidates’ values (memory potentiation), so information from all 8,100 positions is kept in compressed form. [Certain] That is a compression of \(8{,}100 / 128 \approx 63\) times. When the long-term memory is full, the least-used prototypes are removed. The memory read (204.2) runs over working and long-term memory together.
The bound does not depend on the video length. XMem reports that it never uses more than about 1.4 GB of GPU memory, and on the Long-time Video benchmark its J&F barely changes when the videos are made 3 times longer (89.8 at 1×, 90.0 at 3×), while STCN drops from 87.3 to 84.6. [Certain]
- input: frames \(I_1, \dots, I_T\), first mask \(M_1\); settings \(r, T_{\min}, T_{\max}, P, LT_{\max}\)
- encode \((I_1, M_1)\) into keys and values; working memory \(\mathcal{W} \leftarrow \{\text{frame } 1\}\), long-term \(\mathcal{L} \leftarrow \varnothing\), sensory state \(h \leftarrow 0\)
- for \(t = 2, \dots, T\):
- \(\mathbf{q} \leftarrow\) key encoder\((I_t)\) // HW query vectors
- \(\hat{\mathbf{v}} \leftarrow\) memory read (204.2) with similarity (204.3) over \(\mathcal{W} \cup \mathcal{L}\)
- \(\hat M_t \leftarrow\) decoder\((\hat{\mathbf{v}}, h, \text{features of } I_t)\); update \(h\) with the GRU
- if \(t \bmod r = 0\): encode \((I_t, \hat M_t)\) and add it to \(\mathcal{W}\)
- if \(\lvert \mathcal{W} \rvert = T_{\max}\): pick the \(P\) most-used keys of the \(T_{\max} - T_{\min}\) middle frames, compute their values by potentiation, add them to \(\mathcal{L}\); remove those frames from \(\mathcal{W}\)
- if \(\lvert \mathcal{L} \rvert \gt LT_{\max}\): remove the least-used prototypes
- return \(\hat M_2, \dots, \hat M_T\)
How to read Figure 3
The red line grows without end: one more memory frame every \(r\) frames. The green XMem line is a saw-tooth (the working memory fills to 10 frames, then 5 are consolidated into the long-term memory). It rises slowly until the long-term memory reaches 10,000 prototypes, after about 83 seconds with \(r = 5\), and then stays in a narrow band between 18,100 and 26,200 positions. SAM 2 is flat from the 7th frame on: it simply forgets old frames that were not prompted. Both bounded designs stay near 20,000 to 30,000 positions, which is why they run at a constant speed on hour-long videos. The price of SAM 2’s choice is visible in Figure 2: a short window can miss an object that was hidden for longer than \(N\) frames.
A newer model from the same group, Cutie (2023), adds object-level memory: a small set of object queries that read from the pixel memory and summarize the target, which helps in crowded scenes with similar-looking objects. [Likely]
SAM 2: promptable segmentation with a streaming memory
Memory networks like XMem need a full mask in frame 1. In practice nobody wants to draw a mask; people want to click. And when the model makes a mistake in frame 80, the user wants to click again in frame 80, not start over. SAM 2 (Ravi et al., 2024) unifies the two ideas. It is SAM (a promptable image segmenter) plus a streaming memory (a memory-based VOS model). Its task is called promptable visual segmentation: prompts can be positive or negative clicks, boxes, or masks, given on any frame, and the model returns a masklet (the object’s mask in every frame). [Certain] On a single image the memory is empty, and SAM 2 is just an image segmenter, about 6 times faster than SAM (130.1 against 21.7 frames per second on an A100). [Certain]
Per frame, SAM 2 does the following (Figure 4):
- The image encoder (Hiera, a hierarchical vision transformer pretrained with MAE; sizes T, S, B+, L; input 1024 × 1024) runs once per frame and gives the frame features \(F_t\). Its cost is shared by all tracked objects. [Certain]
- Memory attention (\(L = 4\) transformer blocks by default) conditions \(F_t\) on the memory bank: self-attention, then cross-attention to the memories, then an MLP. It uses 2-D rotary position encodings (RoPE). [Certain]
- The memory bank is a FIFO (first in, first out) queue of the \(N = 6\) most recent frames, another FIFO queue of up to \(M\) prompted frames, and object pointers. Spatial memories are projected to 64 channels. An object pointer is a 256-number summary of the object (from the mask decoder’s output token), split into 4 tokens of 64 numbers for the cross-attention. [Certain]
- The prompt encoder and mask decoder are as in SAM: prompts become tokens, and the decoder outputs mask logits plus a predicted IoU. On frames without a prompt the decoder runs with no prompt tokens and relies on the memory-conditioned features.
- An occlusion head (an extra output token and an MLP) predicts whether the object is visible in this frame at all. If not, the output mask is empty, and a learned “occluded” embedding is added to that frame’s memory. [Certain]
- The memory encoder fuses the predicted mask with \(F_t\) and writes the result into the bank.
- Read the symbols
- \(F_t\)
- image features of frame \(t\) from the encoder (64 × 64 positions for a 1024 input) [Likely]
- \(\mathcal{M}_s\)
- the spatial memory of frame \(s\): its features fused with its predicted mask, 64 channels per position
- \(\mathcal{P}_t\)
- the frames before \(t\) where the user gave a prompt; they do not drop out with the recent queue
- \(\mathbf{o}_s\)
- object pointers: one compact vector per memory frame that summarizes the object
- \(F'_t\)
- features that now carry “which object we are following”; the mask decoder reads them
Compare with formula (204.2): the cross-attention inside MemAttn is the same memory read, now inside a transformer block with learned projections and position encodings. The first frame of STM and XMem (always kept) becomes the set of prompted frames \(\mathcal{P}_t\) in SAM 2.
How to read Figure 4
On a normal frame, the only information about the object comes from the memory bank on the left: memory attention turns “frame features” into “frame features that know the target”. A click or a box adds prompt tokens to the mask decoder, the same way as in SAM. A mask prompt is different: it already gives the object’s pixels on that frame, so its main job is to become a strong prompted memory. In every case the result is written back into the memory bank, and a prompted frame stays in the bank even when it is old. With “Object hidden”, the occlusion head says “not visible”: the output is empty and the memory is marked as occluded, so later frames do not learn that the object is now the pillar.
How SAM 2 is trained
SAM 2 is trained on images and videos together, on sequences of 8 frames. The first prompt is the true mask (50% of the time), a positive click (25%) or a box (25%), and up to 2 frames get corrective clicks sampled from the error regions of the model’s own prediction. [Certain] This simulates a user. The training video data, SA-V, was built with a data engine like SA-1B: annotators used SAM 2 itself, and each round produced a better model. SA-V has 50.9K videos (about 196 hours) with 642.6K masklets and 35.5M masks; 42.5% of the manual masklets disappear at some point, which teaches occlusion. [Certain]
| Model (semi-supervised VOS, J&F) | MOSE val | DAVIS 2017 val | YouTube-VOS 2019 val |
|---|---|---|---|
| XMem | 59.6 | 86.0 | 85.6 |
| Cutie-base+ | 71.7 | 88.1 | 87.5 |
| SAM 2 (Hiera-B+), 43.8 FPS | 76.6 | 90.2 | 88.6 |
| SAM 2 (Hiera-L), 30.2 FPS | 77.9 | 90.7 | 89.3 |
The numbers are from the SAM 2 paper (FPS on one A100). [Certain] Note how close the models are on DAVIS, a small and old benchmark, and how far apart on MOSE, a benchmark of complex scenes with crowds, occlusions and objects that disappear. This is a general lesson: when benchmarks saturate, the differences move to harder data.
- input: video frames, a stream of user prompts (frame index, clicks / box / mask)
- for \(t = 1, \dots, T\): \(F_t \leftarrow\) image encoder\((I_t)\) // can be cached and shared by all objects
- on the first prompt at frame \(p\): \(\hat M_p \leftarrow\) mask decoder\((F_p, \text{prompt tokens})\); write \(\mathcal{M}_p\) as a prompted memory
- for each later frame \(t\): \(F'_t \leftarrow\) MemAttn\((F_t; \mathcal{B}_t)\) (204.5); \(\hat M_t, \text{visible}_t \leftarrow\) mask decoder\((F'_t)\)
- if not visible\(_t\): \(\hat M_t \leftarrow \varnothing\), mark the memory as occluded
- write \(\mathcal{M}_t\) into the recent FIFO (keep the last \(N = 6\))
- when the user corrects frame \(c\) with a click: recompute \(\hat M_c\) with the new prompt, add \(\mathcal{M}_c\) to the prompted set, re-propagate from \(c\) (the video can also be processed backward)
- return the masklet \(\hat M_1, \dots, \hat M_T\)
Watching the memory bank work
Algorithm 2 hides the important details in two short lines: what the occlusion head decides, and what is written into the bank. In SAM 2 the occlusion head is a learned output token followed by a small MLP. [Certain] To watch the mechanism, we replace it with the simplest stand-in that behaves the same way: the object score of a frame is the strongest read-out of any of its pixels. If no pixel of the new frame matches the object memories well, the object is declared hidden. Writing this together with the FIFO update of the recent memories:
- Read the symbols
- \(\hat v_{tj}\)
- the memory read-out (204.2) for pixel \(j\) of frame \(t\): the share of its attention that lands on object pixels of the memory
- \(s_t\)
- the object score of frame \(t\); in SAM 2 a learned head outputs this score, here it is the largest read-out
- \(\hat M_t\)
- the output mask: the pixels whose read-out is above one half, or nothing if the object is hidden
- \(\mathcal{R}_t\)
- the recent memories (a FIFO queue of length \(N\)); the prompted memories are kept outside this queue and never drop out
Figure 5 runs SAM 2’s loop on a toy video of 20 frames with 16 × 8 pixels. A block moves right and changes color by a few degrees per frame. It passes behind a pillar and is fully hidden in frames 10 to 13. The user gives one click on frame 1; a tiny stand-in for the prompt decoder grows a region from the click over neighbors of similar color (within \(25^\circ\)), which gives the 9-pixel mask of the block. From then on, every frame runs lines 4 to 8 of the algorithm box, with the real memory read (204.2), \(\tau = 0.1\), over all \(128\) pixels of every frame in the bank.
Default settings: \(N = 6\), color change \(4^\circ\) per frame. At frame 14 the bank holds the prompted frame 1 and the recent frames 8 to 13: 7 frames × 128 pixels = 896 keys. Frames 10 to 13 are marked occluded (their stored masks are empty). Only one column of the block is visible, right of the pillar. Take its middle pixel, at position (11, 4), with hue \(68.1^\circ\). Its attention is split as follows: frame 9 gets 0.492, frame 8 gets 0.288, the prompted frame 1 gets 0.058, and the four hidden frames together get about 0.16. The read-out is \(\hat v = 4.2982 / 5.9195 = 0.726\): object. The object score is \(s_{14} = 0.962 \gt 0.5\), so the frame is visible, and the mask has exactly the 3 true pixels.
Now set \(N = 4\). The bank at frame 14 holds frame 1 and only the four hidden frames 10 to 13; nothing recent shows the block. The same pixel reads \(\hat v = 0.131\), the object score is 0.751, and the mask has 1 pixel (\(\mathcal{J} = 0.333\)). From frame 15 on the block is declared hidden and stays lost: mean J over frames 2 to 20 drops from 1.000 to 0.649.
Finally keep \(N = 6\) but switch the occlusion head off: the decoder must always output its best guess, \(\{ j : \hat v_{tj} \ge 0.5\, s_t \}\). In frame 10 that is 19 pixels, mostly on the pillar; in frames 11 to 13 it is the whole pillar (40 pixels). These wrong masks are written into memory as “object”, so from frame 15 on the pillar stays in the mask next to the real block (\(\mathcal{J} = 6/46 = 0.130\) in frame 15). Mean J: 0.526. One bad write poisons every later frame. That is why SAM 2 marks hidden frames instead of writing a guess.
- input: frames \(I_1, \dots, I_{20}\), one click on frame 1
- \(\hat M_1 \leftarrow\) prompt decoder\((F_1, \text{click})\); write \(\mathcal{M}_1\) as a prompted memory
- for \(t = 2, \dots, 20\):
- \(F_t \leftarrow\) image encoder\((I_t)\)
- \(\hat v_{tj} \leftarrow \sum_i \softmax_i(\mathbf{q}_j^\top \mathbf{k}_i / \tau)\, v_i\) over the bank \(\mathcal{B}_t\) (204.2)
- \(s_t \leftarrow \max_j \hat v_{tj}\); visible \(\leftarrow s_t \gt 0.5\) (204.6)no occlusion head: \(s_t \leftarrow \max_j \hat v_{tj}\) only sets a threshold
- \(\hat M_t \leftarrow \{ j : \hat v_{tj} \gt 0.5 \}\) if visible, else \(\varnothing\) and mark occluded\(\hat M_t \leftarrow \{ j : \hat v_{tj} \ge 0.5\, s_t \}\) // always a best guess
- write \(\mathcal{M}_t\) into the recent FIFO; keep the last \(N\)
How to read Figure 5
Press Play and watch one frame take five steps: encode (line 4), read the memory (line 5, the read-out map fills in and the attention lines appear), decide visibility (line 6), output the mask (line 7), and write the new memory into the FIFO (line 8, the new slot is marked “new” and the oldest recent memory is dropped). During frames 10 to 13 the object score falls to 0.000, the mask is empty, and red “occluded” marks appear in the bank. At frame 14 the attention of the query pixel goes mostly to frames 8 and 9, the last frames before the pillar. Then try the presets. “Short memory” has no pre-occlusion frame left at frame 14 and loses the block. “No occlusion head” writes the pillar into memory as the object, and the error never goes away. Raise the color change to 6: even \(N = 6\) cannot bridge the gap, because the block now looks too different from both frame 1 and frame 9.
Measuring VOS: J, F, recall and decay
A VOS benchmark compares the predicted mask \(\hat M\) with the true mask \(G\) of each object in each frame. Two measures are used, both from DAVIS. [Certain] The region similarity \(\mathcal{J}\) is the intersection over union (IoU, also called the Jaccard index):
The second line matters: when the object is hidden and the model correctly outputs nothing, the score is 1, not undefined. (The official DAVIS code returns 1 when the union is empty.) [Certain]
IoU counts pixels, so it is dominated by the inside of large objects and barely notices a messy outline. The boundary accuracy \(\mathcal{F}\) looks only at the contours. Let \(B(\hat M)\) and \(B(G)\) be the boundary pixels of the two masks, and let \(d\) be a small distance tolerance:
- Read the symbols
- \(P_c\)
- contour precision: the fraction of the predicted outline that lies close to the true outline
- \(R_c\)
- contour recall: the fraction of the true outline that the prediction comes close to
- \(d\)
- the tolerance; the official code uses 0.008 of the image diagonal, rounded up: for 854 × 480 that is \(\lceil 0.008 \times 979.7 \rceil = 8\) pixels [Certain]
- \(\mathcal{F}\)
- the harmonic mean of the two (the F-measure), between 0 and 1
The benchmark score averages over objects, not over videos: first over the frames of each object, then over all objects \(\mathcal{O}\). The final number is the mean of the two:
Here \(\mathcal{F}_o\) is the set of scored frames of object \(o\) (the given first frame is not scored; the official code also skips the last frame). [Certain] A video with 3 objects counts 3 times. DAVIS also reports two summary statistics per object, from its per-frame scores \(\mathcal{J}_f\):
Recall counts how often the object is “basically found”; decay measures drift: a large positive decay means the model was good at the start and got worse. Both are in Figure 2’s read-out. YouTube-VOS reports J and F separately for categories seen in training and unseen ones, and an overall score that averages the four. [Likely]
Take the 38-pixel object of Figure 6 and predict exactly the same shape, shifted one pixel to the right. The intersection has 32 pixels and the union 44, so \(\mathcal{J} = 32/44 = 0.727\). Both outlines have 18 boundary pixels. With tolerance \(d = 0\), only 8 of the 18 predicted boundary pixels lie on the true boundary: \(P_c = R_c = 0.444\), \(\mathcal{F} = 0.444\). With \(d = 1\), every boundary pixel is within one pixel of the other outline: \(\mathcal{F} = 1.000\).
For a square object of side \(s\) shifted by one pixel, \(\mathcal{J} = (s - 1)/(s + 1)\): 0.714 for \(s = 6\), 0.905 for \(s = 20\), 0.980 for \(s = 100\). The same one-pixel error costs a small object far more. That is why benchmark scores are lower on datasets with many small objects, and why you should report results by object size.
How to read Figure 6
The start is the one-pixel shift from the worked example. Move \(d\) from 1 to 0 and watch F fall from 1.000 to 0.444 while J stays at 0.727: F is about outlines, J about area. Press “Perfect”, then “Add a thin spike”: J falls only to 0.927, because the spike has few pixels, but the outline is now longer and partly wrong. Try removing one corner pixel, or making the outline jagged by clicking cells along the edge: J changes little, F changes a lot at \(d = 0\). Press “Clear”: an empty prediction gives \(\mathcal{J} = 0\) and \(\mathcal{F} = 0\).
Referring segmentation: from a sentence to a masklet
Clicks need a person. Many applications start from language instead: “segment the dog that jumps over the fence”, “track the player with number 10”. This is referring video object segmentation (RVOS). The sentence may describe appearance (“the man in the red shirt”), position (“the car on the left”) or motion (“the bird that flies away first”), and the last kind can only be solved by looking at several frames.
The standard benchmarks: Ref-YouTube-VOS (3,978 videos with about 15K expressions) and Ref-DAVIS17 (90 videos, 1,544 expressions about 205 objects), both scored with J, F and J&F. [Certain] MeViS (2023) focuses on motion expressions, where a single frame is not enough to know which object is meant. [Certain]
There are two families of methods.
End to end. ReferFormer (2022) treats the sentence as a set of queries. A few object queries (5 by default) are conditioned on the sentence embedding and shared across all frames. Each query attends to the video features and becomes a dynamic kernel: a weight vector that is dotted with every pixel feature to produce a mask, the same pattern as SAM’s mask decoder. [Certain] Because the same query is used in every frame, the masks of one query form a track automatically. In formula form, for query \(n\) and frame \(t\):
- Read the symbols
- \(\mathbf{z}_n\)
- learned object query \(n\)
- \(\mathbf{e}_{\text{text}}\)
- the sentence embedding from a text encoder
- \(\mathbf{w}_{n,t}\)
- the dynamic kernel of query \(n\) in frame \(t\) (computed by the transformer \(g\))
- \(E_t(x, y)\)
- the pixel feature at \((x, y)\) in frame \(t\)
- \(p_{t,n}\)
- a score for “query \(n\) is the referred object in frame \(t\)”; the best query on average gives the answer
On the Ref-YouTube-VOS validation server, ReferFormer reports J&F 62.4 with a Swin-L backbone, 64.9 with Video-Swin-B and joint training on RefCOCO images. [Certain] Note how much lower this is than the 85 to 90 of semi-supervised VOS: understanding the sentence is the hard part, not the tracking.
Detect, then track. A modular pipeline uses an open-vocabulary grounding model (Grounding DINO, or a VLM that outputs boxes) to find the referred object in one or a few key frames, then gives the box to SAM 2 as a prompt. This is how many practical systems work (for example the “Grounded SAM 2” open-source project). [Likely] Recent models merge the two: Sa2VA connects a multimodal LLM to SAM 2. The LLM reads the video and the question and generates tokens that guide SAM 2 in producing the masks [Certain]; concretely, a special segmentation token whose hidden state is used as a prompt for SAM 2’s decoder, which then tracks the object. [Likely] SAM 3 (2025) adds short noun phrases (“yellow school bus”) as prompts directly, returning all matching instances with identities. [Certain]
- input: frames \(I_1, \dots, I_T\), sentence \(x\), stride \(k\), threshold \(\tau\)
- for key frames \(t \in \{1, 1 + k, 1 + 2k, \dots\}\): \((\mathbf{b}_t, p_t) \leftarrow\) grounding model\((I_t, x)\) // best box and its score
- \(t^\ast \leftarrow \argmax_t p_t\); if \(p_{t^\ast} \lt \tau\): return “object not found”
- prompt SAM 2 with box \(\mathbf{b}_{t^\ast}\) on frame \(t^\ast\)
- propagate forward from \(t^\ast\) to \(T\), and backward from \(t^\ast\) to 1 (Algorithm 2)
- optional check: on other key frames, if SAM 2’s mask and the grounding box disagree strongly (low IoU), add the box as a new prompt
- return the masklet
For motion expressions, a single key frame is not enough: “the dog that starts running” is just “a dog” in most frames. Then you need a model that looks at the whole clip, such as a session 205 style model that picks the frame and the object, or an end-to-end RVOS model.
The PyTorch version
The code below runs the experiment of Figure 2 with real tensors: a toy video, the memory read (204.2), three memory designs, and the J and F measures (204.7), (204.8). It needs only PyTorch and runs in about a second on a CPU. Because PyTorch’s random numbers differ from the figure’s, the “first frame only” score is a little different from Figure 2.
import torch import torch.nn.functional as F torch.manual_seed(0) # 1) a toy video: T frames of H x W pixels; each pixel has a 2-D unit "feature" (its hue) T, H, W = 40, 12, 20 def make_video(drift=2.0, noise=12.0): feats, masks = [], [] for t in range(T): hue = torch.where(torch.arange(H)[:, None] < 6, 220.0, 150.0).expand(H, W).clone() # sky, grass m = torch.zeros(H, W) x0 = round(1 + t * 14 / (T - 1)) m[4:8, x0:x0 + 4] = 1 # the object: a 4 x 4 block moving right hue[m.bool()] = drift * t # its color changes slowly over time m[:, 9:14] = 0; hue[:, 9:14] = 270.0 # a pillar in front of everything a = torch.deg2rad(hue + (torch.rand(H, W) - 0.5) * 2 * noise) feats.append(torch.stack([a.cos(), a.sin()], -1).reshape(H * W, 2)) masks.append(m.reshape(H * W)) return feats, masks # 2) memory read, formula (204.2): softmax over all memory pixels, then average their mask values def memory_read(q, K, V, tau=0.1): A = F.softmax(q @ K.T / tau, dim=1) # (HW, memory size); each row sums to 1 return A @ V # (HW,) soft mask in [0, 1] def track(feats, first_mask, keep_first=True, n_recent=6): preds = [first_mask] for t in range(1, T): mem = [0] if keep_first else [] mem += [s for s in range(max(0, t - n_recent), t) if s not in mem] K = torch.cat([feats[s] for s in mem]) V = torch.cat([preds[s] for s in mem]) # past PREDICTED masks, not the truth preds.append((memory_read(feats[t], K, V) > 0.5).float()) return preds # 3) region measure J, formula (204.7); J = 1 when both masks are empty def jaccard(p, g): inter, union = (p * g).sum(), ((p + g) > 0).sum() return 1.0 if union == 0 else (inter / union).item() feats, masks = make_video() for name, kf, n in [("previous frame only", False, 1), ("first frame only", True, 0), ("first + 6 recent", True, 6)]: preds = track(feats, masks[0], kf, n) J = [jaccard(preds[t], masks[t]) for t in range(1, T)] print(f"{name:20s} mean J = {sum(J) / len(J):.3f} J at last frame = {J[-1]:.2f}") # previous frame only mean J = 0.667 J at last frame = 0.00 (lost after the pillar) # first frame only mean J = 0.863 J at last frame = 0.12 (the color drifted away) # first + 6 recent mean J = 1.000 J at last frame = 1.00 # 4) boundary measure F, formula (204.8), with a tolerance of d pixels def boundary(m): # mask pixels with a 4-neighbour outside the mask cross = torch.tensor([[0., 1, 0], [1, 1, 1], [0, 1, 0]])[None, None] full = F.conv2d(F.pad(m[None, None], (1, 1, 1, 1)), cross)[0, 0] == 5 return m.bool() & ~full def f_measure(p, g, d=1.0): bp, bg = boundary(p).nonzero().float(), boundary(g).nonzero().float() if len(bp) == 0 or len(bg) == 0: return float(len(bp) == len(bg)) dist = torch.cdist(bp, bg) # distances between all boundary pixels prec = (dist.min(1).values <= d).float().mean() # predicted boundary near the true one rec = (dist.min(0).values <= d).float().mean() # true boundary near the predicted one return 0.0 if prec + rec == 0 else (2 * prec * rec / (prec + rec)).item() g = torch.zeros(10, 14); g[2:8, 3:10] = 1; g[2, 3] = g[2, 9] = g[7, 3] = g[7, 9] = 0 p = torch.roll(g, 1, dims=1) # the same mask, one pixel to the right print(f"J = {jaccard(p, g):.3f} F(d=0) = {f_measure(p, g, 0):.3f} F(d=1) = {f_measure(p, g, 1):.3f}") # J = 0.727 F(d=0) = 0.444 F(d=1) = 1.000 (the worked example)
To use the real SAM 2, the official sam2 package has a video predictor with the same loop as Algorithm 2: you initialize a state on a folder of frames, add clicks or a box for an object id on some frame, and call a propagation function that yields the masks frame by frame. [Likely] The lab session does this on sports video.
Where this is used
| Application | Setup | What matters |
|---|---|---|
| Video editing and rotoscoping | clicks on one frame, SAM 2 propagates; corrections on bad frames | boundary quality (F), few clicks, handling hair and motion blur |
| Sports analytics | detector finds players, SAM 2 tracks each one; numbers read with OCR or a VLM | identity switches between similar players, occlusions in crowds (session 217) |
| Data labeling for driving and robotics | annotator labels a few frames, the model fills in the rest (a data engine like SA-V) | speed of the human loop; label quality checks |
| Medical video (endoscopy, ultrasound) | mask of an organ or tool in one frame, tracked through the clip [Likely] | domain shift from natural images; often needs fine-tuning |
| Video retrieval and Video-RAG | object masklets become object-level entries in an index: “all shots where this car appears” (session 210) | long-video memory, re-identification after scene cuts |
| Key-moment detection | track the ball or a player, detect events from its trajectory (session 207) | stable tracks; the occlusion signal is itself useful |
| Referring segmentation in assistants | “highlight the person who drops the bag”: a Video-LLM chooses the object, SAM 2 tracks it | understanding motion words; refusing when the object is not there |
| Privacy and compliance | track faces or license plates and blur them in every frame | recall: one missed frame shows the face |
Common mistakes
- Running an image segmenter on each frame independently. SAM on frame 50 does not know which object you clicked in frame 1, and the masks flicker. You need a memory (or at least an association step).
- Ignoring occlusion. If a hidden frame’s wrong mask (for example on the occluder) goes into memory, the model starts following the occluder. Use the occlusion score, and do not write low-confidence frames as strong memories.
- Unbounded memory on long videos. Keeping every frame works in a 3-second demo and runs out of GPU memory on a 20-minute video (formula 204.4, Figure 3).
- Too short a memory. The opposite mistake: a window of \(N\) frames cannot bring back an object that was hidden longer than \(N\) frames unless the prompted frame still looks like it (Figure 2 at drift 2.6).
- Averaging J over videos instead of objects. DAVIS averages over objects (formula 204.9). A video with 5 objects must weigh 5 times more. Your numbers will not match the leaderboard otherwise.
- Reporting only J. J barely changes for bad outlines on large objects. Report J, F and J&F, and use the official evaluation code (its boundary tolerance and frame selection matter).
- Comparing models at different resolutions or with different first-frame inputs. A mask prompt is much easier than a single click; 480p is easier than full resolution for speed but loses thin parts.
- Assuming objects keep their identity across scene cuts. Memory matching assumes one continuous shot. After a cut, detect again and re-identify, or the model will latch onto something else.
Interview questions
What is semi-supervised video object segmentation, and why is it called semi-supervised?basic
Why does “propagate the previous mask” fail, and how does memory-based matching fix it?basic
Write the STM memory read as a formula and explain each part.core
Why do STCN and XMem use a negative squared L2 distance instead of a dot product?advanced
How does XMem keep memory bounded? Give the numbers.core
Walk through what SAM 2 does on one frame of a video.core
How does SAM 2 handle an object that disappears and comes back?core
Why not simply run SAM on every frame and link the masks by IoU?basic
Define J and F. When do they disagree?core
How would you build a system that segments “the man in the red shirt who sits down” in a video?advanced
SAM 2 is 90.7 J&F on DAVIS 2017 but 77.9 on MOSE. What does that tell you?advanced
Exercises
- In the worked memory read, what is the read-out for a query at \(120^\circ\)? Is the pixel object or background?
- XMem settings: \(r = 5\), \(T_{\min} = 5\), \(T_{\max} = 10\), \(P = 128\), \(LT_{\max} = 10{,}000\). After how many consolidations is the long-term memory full, and about how many seconds of 24 FPS video is that? Check with Figure 3.
- A square object of 10 × 10 pixels is predicted shifted by 2 pixels to the right. Compute J.
- A predicted outline has 40 boundary pixels, 30 of them within \(d\) of the true outline; the true outline has 50 boundary pixels, 35 of them within \(d\) of the prediction. Compute F.
- A benchmark has video A with 1 object (\(\mathcal{J} = 0.9\)) and video B with 3 objects (\(\mathcal{J} = 0.6, 0.7, 0.8\)). Compute the DAVIS-style mean J and the (wrong) per-video mean.
- In Figure 2, set the color change to 2.6 per frame and keep the first frame. What is the smallest \(N\) that recovers the object after the pillar? Explain why in terms of which frames are in memory at frame 28.
Answerscheck
Key takeaways
- Semi-supervised VOS (204.1): given the mask in frame 1, predict the object’s mask in every later frame, online, using only past frames and the model’s own past predictions.
- Propagating the previous mask drifts and gets lost after occlusion. Memory networks match every pixel of the new frame against stored frames with attention (204.2): queries from the new frame, keys and mask-carrying values from memory.
- Memory must be both reliable (the prompted first frame) and fresh (recent frames). XMem bounds memory on long videos with working and long-term (prototype) memory: at most about 26,200 positions per object (204.4).
- SAM 2 = SAM + streaming memory (204.5): memory attention over prompted frames, the 6 most recent frames and object pointers; prompts (clicks, boxes, masks) on any frame; an occlusion head; trained on SA-V (50.9K videos, 35.5M masks).
- Metrics: J is mask IoU (204.7), F is boundary F-measure with a tolerance (204.8); J&F is averaged over objects (204.9); recall and decay (204.10) show how often the object is found and how much the model drifts.
- Referring VOS replaces the first-frame mask with a sentence: end-to-end text-conditioned queries (204.11), or a grounding model / Video-LLM that prompts SAM 2.
Key formulas
References for this session
- Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
- Cheng & Schwing (2022). XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
- Pont-Tuset et al. (2017). The 2017 DAVIS Challenge on Video Object Segmentation
- Kirillov et al. (2023). Segment Anything
- Oh, Lee, Xu & Kim (2019). Video Object Segmentation using Space-Time Memory Networks
- Cheng, Tai & Tang (2021). Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation
- Cheng, Oh, Price, Lee & Schwing (2023). Putting the Object Back into Video Object Segmentation
- Xu et al. (2018). YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
- Ding et al. (2023). MOSE: A New Dataset for Video Object Segmentation in Complex Scenes
- Perazzi et al. (2016). A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation
- Wu, Jiang, Sun, Yuan & Luo (2022). Language as Queries for Referring Video Object Segmentation
- Ding et al. (2023). MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions
- Yuan et al. (2025). Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos
- Carion et al. (2025). SAM 3: Segment Anything with Concepts
- Yang et al. (2024). SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory