LLM & VLM courseBlog
204Track 8: Video and multimodal systems · session 4 of 21

Video object segmentation and tracking: from VOS to SAM 2

You draw a mask on an object in the first frame of a video, and the model must follow that object, pixel by pixel, through motion, changes of look and occlusion. This session builds the idea step by step: the task and its metrics (J and F), memory-based matching as attention over past frames (STM, XMem), SAM 2 with its streaming memory and point, box and mask prompts, and referring segmentation from a sentence.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 16 to 17 (attention), session 168 (SAM and open-vocabulary segmentation), session 201 (video as data)

What you will learn

  • What semi-supervised video object segmentation (VOS) is, and how it differs from tracking boxes, video instance segmentation and referring segmentation
  • Why following the previous mask drifts and gets lost, and why matching against a memory of past frames works better
  • The memory read of STM and its successors as plain attention: query pixels, memory keys, mask values, with numbers
  • How XMem keeps memory bounded on long videos with sensory, working and long-term memory, and how much memory that saves
  • How SAM 2 works as a streaming model: memory attention, a memory bank of prompted and recent frames, object pointers, an occlusion head, and prompts on any frame
  • How the metrics J (region IoU), F (boundary accuracy), J&F, recall and decay are computed, and what they hide
  • How referring VOS segments an object from a sentence, either end to end or with a detector plus SAM 2

Why this matters

You click on a football player in the first frame of a match video. A good system then gives you that player’s exact outline in every later frame, while he runs, turns, gets smaller, and disappears behind other players for a second. This is video object segmentation, and it is now a one-click tool.

In session 168 we met SAM, which turns a click into a mask on one image, and had a first look at SAM 2. In session 203 we gave a whole clip one label. This session sits between the two: we follow one object through time, at the level of pixels. The problem is harder than it looks. The object changes its look, other objects look similar, and when it is hidden there is nothing to follow at all.

The solution used by all strong systems since 2019 is one idea that you already know from transformers: keep a memory of past frames with their masks, and let every pixel of the new frame attend to that memory. STM introduced it, XMem made it work on long videos, and SAM 2 turned it into a promptable model trained on 35.5 million masks. [Certain]

Real uses: rotoscoping in video editing (cutting a person out of a shot), sports analytics, annotation tools that create training data for self-driving cars, medical video, robotics, and object-level search in video archives. In this course, it is also the base of session 217.

The task: a mask in frame 1, follow the object

A video is a list of frames \(I_1, \dots, I_T\). In semi-supervised VOS, someone gives the true mask \(M_1\) of one or more objects in the first frame. The model must output the masks \(\hat M_2, \dots, \hat M_T\) of the same objects in all later frames. “Semi-supervised” only means that one frame is labeled at test time; the model itself was trained earlier on other videos. Formally, at each step the model may use the current frame and everything it has seen so far:

\[ \hat M_t = f_\theta\big(I_t;\ \{(I_s, \hat M_s)\}_{s=1}^{t-1}\big), \qquad \hat M_1 = M_1 \tag{204.1} \]
Read the symbols
\(I_t\)
frame \(t\) of the video, an RGB image
\(M_1\)
the given mask of the first frame: for each pixel, which object it belongs to (or background)
\(\hat M_s\)
the model’s own predicted mask for frame \(s\); there is no ground truth after frame 1, so the model must trust its own past answers
\(f_\theta\)
the network, with weights \(\theta\) learned on training videos; it does not train on the test video

Formula (204.1) says the model is online (also called streaming or causal): frame \(t\) is processed with frames \(1, \dots, t-1\) only, never with future frames. That is what lets it run on a live camera. Notice also the hidden danger: past predictions \(\hat M_s\) feed into the future. One wrong mask can be copied forward, frame after frame. This is called drift.

The same idea appears in several related tasks. It helps to keep them apart:

TaskInput at test timeOutput
Semi-supervised VOSmask(s) in the first framemasks of the same objects in every frame
Interactive VOS / promptable visual segmentationclicks, boxes or masks on any frames, added over timemasks in every frame, refined after each new prompt
Unsupervised VOSnothingmasks of the main moving objects
Referring VOSa sentence, such as “the dog jumping over the fence”the mask of the object the sentence describes, in every frame
Video instance segmentationa fixed list of classesmasks, class labels and identities of all objects of those classes
Single-object trackinga box in the first framea box in every frame
Multi-object trackinga detector for a class (for example “person”)boxes with stable identities over time

Masks are much richer than boxes: a box around a running person contains a lot of background, and a box tells you nothing about which pixels are the person. But they are also more expensive to compute and to label. DAVIS 2017, the classic benchmark, has only 150 sequences with 10,459 annotated frames and 376 objects, about 70 frames per sequence, evaluated at 480p. [Certain] This tiny size shaped the whole field: models had to learn general “matching” skills, not memorize objects.

Three ways to follow an object

Suppose each pixel of each frame has a feature vector: a short list of numbers produced by an image encoder that describes what the pixel looks like (its color, texture and what object part it is). How can we use the features to find the object in frame \(t\)?

  1. Propagate from the previous frame. Take \(\hat M_{t-1}\), and move it a little to fit frame \(t\). Early methods (MaskTrack, 2017) worked like this. [Likely] It is good at smooth motion. But if the object is fully hidden for one frame, \(\hat M_{t-1}\) is empty, and an empty mask propagates to an empty mask forever. The object is lost.
  2. Match against the first frame. For each pixel of frame \(t\), find the most similar pixels in frame 1, and copy their label (object or background). This never gets lost after an occlusion, because frame 1 is always there. But objects change: the player turns around, the light changes. After a while, nothing in frame 1 looks like the object any more.
  3. Match against a memory. Keep several past frames with their masks: the first frame (true mask, reliable but old) and some recent frames (predicted masks, fresh but possibly wrong). Match against all of them at once. This combines the strengths of both.

Option 3 is the memory network idea of STM (Space-Time Memory networks, Oh et al., 2019). [Certain] Each memory frame is stored as a set of keys (features that are used for matching) and values (features that carry the mask information). Each pixel of the current frame produces a query. Then the read is exactly attention, as in session 17:

\[ \begin{gathered} A_{ji} = \frac{\exp\big(\mathbf{q}_j^\top \mathbf{k}_i / \tau\big)}{\sum_{i'} \exp\big(\mathbf{q}_j^\top \mathbf{k}_{i'} / \tau\big)} \\ \hat{\mathbf{v}}_j = \sum_i A_{ji}\, \mathbf{v}_i \end{gathered} \tag{204.2} \]
Read the symbols
\(j\)
one pixel of the current frame (the query side); there are \(HW\) of them
\(i\)
one pixel of one memory frame; the sum runs over all pixels of all memory frames together
\(\mathbf{q}_j, \mathbf{k}_i\)
query and key vectors (for example 64 numbers each), made from image features only
\(\mathbf{v}_i\)
the value of memory pixel \(i\): features that encode both what it looked like and whether it was inside the mask
\(\tau\)
a temperature; smaller \(\tau\) makes each query attend to fewer, closer matches (in a transformer \(\tau = \sqrt{d}\))
\(\hat{\mathbf{v}}_j\)
the read-out for pixel \(j\); a decoder turns it into the mask probability

This is space-time attention: the softmax runs over every position in every memory frame, so a pixel can match a pixel anywhere, even far away and many frames ago. There is no assumption that the object moved only a little. That is why memory methods recover after occlusions and fast motion.

In the simplest case, the value is just the mask label: \(v_i = 1\) inside the object and \(0\) outside. Then \(\hat v_j\) is a weighted vote: the fraction of attention that pixel \(j\) gives to object pixels. Figure 1 runs this read one line at a time, and Figure 2 runs it on every pixel of a whole toy video.

Worked example: one memory read

Take 2-D unit features described by an angle (think of it as a hue). The memory has four pixels: two object pixels at \(0^\circ\) and \(10^\circ\) (\(v = 1\)), and two background pixels at \(150^\circ\) and \(220^\circ\) (\(v = 0\)). Use \(\tau = 0.1\). For unit vectors, \(\mathbf{q}^\top \mathbf{k} = \cos(\text{angle difference})\).

A query pixel at \(40^\circ\): scores \(\cos(40^\circ)/0.1 = 7.66\), \(\cos(30^\circ)/0.1 = 8.66\), \(\cos(110^\circ)/0.1 = -3.42\), \(\cos(180^\circ)/0.1 = -10.0\). The softmax gives weights \(0.269, 0.731, 0.000, 0.000\). The read-out is \(0.269 + 0.731 = 1.000\): object.

A query pixel at \(80^\circ\), halfway between object and background: scores \(1.74, 3.42, 3.42, -7.66\); weights \(0.085, 0.458, 0.458, 0.000\); read-out \(0.542\). It is barely object. If the object keeps changing color in this direction, it will soon be read as background. This is why the memory must contain recent views of the object.

query \(\mathbf{q}_j\)memory keys \(\mathbf{k}_i\)value \(v_i = 1\) (object)current step
  1. for each memory pixel \(i\): \(s_i \leftarrow \mathbf{q}^\top \mathbf{k}_i / \tau = \cos(\theta_q - \theta_i)/\tau\)
  2. \(m \leftarrow \max_i s_i\) // subtracted so that exp never overflows
  3. \(e_i \leftarrow \exp(s_i - m)\)
  4. \(Z \leftarrow \sum_i e_i\)
  5. \(A_i \leftarrow e_i / Z\) // the softmax of (204.2)
  6. \(\hat v \leftarrow \sum_i A_i v_i\)
  7. object if \(\hat v \gt 0.5\), else background
Figure 1. The memory read (204.2) of the worked example, computed in your browser one line at a time. Four memory pixels sit on a hue wheel: two object pixels (\(0^\circ\), \(10^\circ\), \(v = 1\)) and two background pixels (\(150^\circ\), \(220^\circ\), \(v = 0\)). All features are unit vectors, so \(\mathbf{q}^\top \mathbf{k}\) is the cosine of the angle between them. The table fills in one column per step; the box under it shows the formula of the current line with the real numbers.
How to read Figure 1

Press Play. The first four steps measure the angle from the query arrow to each key and turn it into a score. Then the scores go through the softmax, one line at a time, and the weights \(A_i\) of the object keys are added up into \(\hat v\). At \(80^\circ\) the query is \(70^\circ\) from the object key at \(10^\circ\) and also \(70^\circ\) from the background key at \(150^\circ\): two equal scores, so they get equal weights (0.458 each), and the result 0.542 is barely “object”. Choose \(40^\circ\): almost all weight goes to the two object keys. Choose \(120^\circ\) (Exercise 1): it goes to the \(150^\circ\) background key. Now raise \(\tau\) to 1: the weights become flat and every key gets a share, so the vote is soft. A small \(\tau\) is what lets one pixel pick its few true matches among thousands of memory positions.

true mask (outline)predicted mask (fill)frames in memory nowobject fully hidden
Figure 2. A toy video of 40 frames, 20 × 12 pixels. A small block moves right, its color slowly changes, and it passes behind a pillar (fully hidden in frames 22 to 27). Every pixel’s feature is its color (a 2-D unit vector, plus seeded noise). Each frame is segmented with the memory read of formula (204.2), \(\tau = 0.1\), using the first frame (true mask) and the \(N\) most recent frames (predicted masks). The bottom plot is J (formula 204.7) for frames 2 to 40.
How to read Figure 2

“Play frames” moves the frame slider through the video, so you can watch the predicted mask follow (or lose) the block. Start with “Previous frame only”. Everything is perfect until the object is fully hidden at frame 22; the predicted mask becomes empty, and an empty memory can only produce empty masks. J stays at 1 during the occlusion (an empty prediction for an empty truth is correct), then drops to 0 forever. Now choose “First frame only”: the object is found again after the pillar, but its color drifts away from frame 1, and in the last frames J falls. With “First + 6 recent”, both problems are fixed. Then move the color-change slider to 2.6: the object comes back from behind the pillar looking too different from frame 1, and all 6 recent frames show only the pillar. Raise \(N\) to 7 or 8 so that the memory reaches back to a frame before the occlusion: it is recovered. That is the argument for a long memory, and the problem XMem solves.

What is in a key and a value

In STM, both the memory and the query frame go through encoders. The memory encoder sees the frame and its mask, so its values know “this was the object”. The query encoder sees only the new frame. A decoder takes the read-out \(\hat{\mathbf{v}}_j\) together with the query frame’s own features and predicts the mask at full resolution. Features live on a grid 16 times smaller than the image: a 480 × 864 frame becomes a 30 × 54 grid, 1,620 positions. With several objects, the same frame is matched once per object (or with shared keys and one value per object).

Later work improved the read without changing the idea. STCN (Cheng et al., 2021) computes keys from the image only, so the keys of a memory frame do not depend on which object is being tracked, and can be shared by all objects. [Certain] It also replaces the dot product by the negative squared distance:

\[ \begin{gathered} S_{ij}^{\text{L2}} = -\lVert \mathbf{k}_i - \mathbf{q}_j \rVert^2 = 2\,\mathbf{k}_i^\top \mathbf{q}_j - \lVert \mathbf{k}_i \rVert^2 - \lVert \mathbf{q}_j \rVert^2 \\ S_{ij}^{\text{XMem}} = -s_i \sum_{c} e_{cj}\, (k_{ci} - q_{cj})^2 \end{gathered} \tag{204.3} \]

Inside a softmax over \(i\), the term \(\lVert \mathbf{q}_j \rVert^2\) is the same for every memory pixel and cancels. The term \(-\lVert \mathbf{k}_i \rVert^2\) does not: with a plain dot product, a memory key with a large norm wins for many queries, and a few keys take most of the attention. The L2 form removes that bias, so more of the memory is actually used. XMem (second line) adds two learned factors: a shrinkage term \(s_i \ge 1\) per memory key (how confident and sharp this key is) and a selection term \(e_{cj} \in [0, 1]\) per query and channel (which feature channels matter for this query). With all \(s\) and \(e\) equal to 1 it is the L2 similarity again. [Certain]

The memory problem on long videos: XMem

STM adds a new memory frame every few frames (every 5th frame is common) and never deletes one. [Likely] The cost of formula (204.2) is proportional to the memory size, and so is the GPU memory. On a 70-frame DAVIS clip this is fine. On a 10-minute video it is not.

Worked example: how big does an STM-style memory get?

Assume 30 × 54 = 1,620 key positions per memory frame, one memory frame every 5 video frames, keys with 64 numbers and values with 512 numbers, stored in 16-bit floats (the key and value sizes of XMem). [Certain] A 10-minute video at 24 frames per second has 14,400 frames, so 2,880 memory frames (the first frame plus one every 5 frames): 2,880 × 1,620 = 4,665,600 memory positions. Storage: \(4{,}665{,}600 \times (64 + 512) \times 2\) bytes = 5.37 GB, for one object. Compute: each of the 1,620 query positions compares with all of them, \(1{,}620 \times 4{,}665{,}600 \times 64 \approx 4.8 \times 10^{11}\) multiply-adds per frame, just for the similarities. The model gets slower with every frame, and eventually runs out of memory.

XMem (Cheng and Schwing, 2022) fixes this with three memory stores, named after a model of human memory (Atkinson–Shiffrin). [Certain]

  • Sensory memory: a small hidden state (64 channels) per position, updated by a GRU (a simple recurrent cell) at every frame. It carries short-term, smooth information, like where the object just was.
  • Working memory: high-resolution keys and values of recent memory frames, exactly as in STM. A new memory frame is added every \(r = 5\) frames. Its size is kept between \(T_{\min} = 5\) and \(T_{\max} = 10\) frames.
  • Long-term memory: a compact set of prototypes, at most \(LT_{\max} = 10{,}000\) elements.

When the working memory reaches \(T_{\max}\) frames, XMem keeps the first frame (it has the true mask) and the most recent \(T_{\min} - 1\) frames. The other \(T_{\max} - T_{\min} = 5\) frames are consolidated: from their \(5 \times 1{,}620 = 8{,}100\) positions, the 128 most-used ones become prototype keys. “Used” means: they received a lot of attention weight in past reads. Each prototype’s value is then a softmax-weighted average of the candidates’ values (memory potentiation), so information from all 8,100 positions is kept in compressed form. [Certain] That is a compression of \(8{,}100 / 128 \approx 63\) times. When the long-term memory is full, the least-used prototypes are removed. The memory read (204.2) runs over working and long-term memory together.

\[ \lvert \text{memory} \rvert \le T_{\max} \cdot HW + LT_{\max} = 10 \times 1{,}620 + 10{,}000 = 26{,}200 \tag{204.4} \]

The bound does not depend on the video length. XMem reports that it never uses more than about 1.4 GB of GPU memory, and on the Long-time Video benchmark its J&F barely changes when the videos are made 3 times longer (89.8 at 1×, 90.0 at 3×), while STCN drops from 87.3 to 84.6. [Certain]

Algorithm 1: memory-based semi-supervised VOS (XMem style)
  1. input: frames \(I_1, \dots, I_T\), first mask \(M_1\); settings \(r, T_{\min}, T_{\max}, P, LT_{\max}\)
  2. encode \((I_1, M_1)\) into keys and values; working memory \(\mathcal{W} \leftarrow \{\text{frame } 1\}\), long-term \(\mathcal{L} \leftarrow \varnothing\), sensory state \(h \leftarrow 0\)
  3. for \(t = 2, \dots, T\):
  4.   \(\mathbf{q} \leftarrow\) key encoder\((I_t)\) // HW query vectors
  5.   \(\hat{\mathbf{v}} \leftarrow\) memory read (204.2) with similarity (204.3) over \(\mathcal{W} \cup \mathcal{L}\)
  6.   \(\hat M_t \leftarrow\) decoder\((\hat{\mathbf{v}}, h, \text{features of } I_t)\); update \(h\) with the GRU
  7.   if \(t \bmod r = 0\): encode \((I_t, \hat M_t)\) and add it to \(\mathcal{W}\)
  8.   if \(\lvert \mathcal{W} \rvert = T_{\max}\): pick the \(P\) most-used keys of the \(T_{\max} - T_{\min}\) middle frames, compute their values by potentiation, add them to \(\mathcal{L}\); remove those frames from \(\mathcal{W}\)
  9.   if \(\lvert \mathcal{L} \rvert \gt LT_{\max}\): remove the least-used prototypes
  10. return \(\hat M_2, \dots, \hat M_T\)
STM style: keep every memory frameXMem: working + long-termSAM 2: prompted + 6 recent frames
Figure 3. Memory size per object (number of key positions, log scale) for a 24 frames-per-second video. STM style and XMem use 1,620 positions per memory frame (a 30 × 54 feature grid), XMem with \(T_{\min} = 5\), \(T_{\max} = 10\), \(P = 128\), \(LT_{\max} = 10{,}000\). SAM 2 is drawn with one prompted frame and \(N = 6\) recent frames of 64 × 64 = 4,096 positions each (1024 × 1024 input at stride 16). [Likely]
How to read Figure 3

The red line grows without end: one more memory frame every \(r\) frames. The green XMem line is a saw-tooth (the working memory fills to 10 frames, then 5 are consolidated into the long-term memory). It rises slowly until the long-term memory reaches 10,000 prototypes, after about 83 seconds with \(r = 5\), and then stays in a narrow band between 18,100 and 26,200 positions. SAM 2 is flat from the 7th frame on: it simply forgets old frames that were not prompted. Both bounded designs stay near 20,000 to 30,000 positions, which is why they run at a constant speed on hour-long videos. The price of SAM 2’s choice is visible in Figure 2: a short window can miss an object that was hidden for longer than \(N\) frames.

A newer model from the same group, Cutie (2023), adds object-level memory: a small set of object queries that read from the pixel memory and summarize the target, which helps in crowded scenes with similar-looking objects. [Likely]

SAM 2: promptable segmentation with a streaming memory

Memory networks like XMem need a full mask in frame 1. In practice nobody wants to draw a mask; people want to click. And when the model makes a mistake in frame 80, the user wants to click again in frame 80, not start over. SAM 2 (Ravi et al., 2024) unifies the two ideas. It is SAM (a promptable image segmenter) plus a streaming memory (a memory-based VOS model). Its task is called promptable visual segmentation: prompts can be positive or negative clicks, boxes, or masks, given on any frame, and the model returns a masklet (the object’s mask in every frame). [Certain] On a single image the memory is empty, and SAM 2 is just an image segmenter, about 6 times faster than SAM (130.1 against 21.7 frames per second on an A100). [Certain]

Per frame, SAM 2 does the following (Figure 4):

  • The image encoder (Hiera, a hierarchical vision transformer pretrained with MAE; sizes T, S, B+, L; input 1024 × 1024) runs once per frame and gives the frame features \(F_t\). Its cost is shared by all tracked objects. [Certain]
  • Memory attention (\(L = 4\) transformer blocks by default) conditions \(F_t\) on the memory bank: self-attention, then cross-attention to the memories, then an MLP. It uses 2-D rotary position encodings (RoPE). [Certain]
  • The memory bank is a FIFO (first in, first out) queue of the \(N = 6\) most recent frames, another FIFO queue of up to \(M\) prompted frames, and object pointers. Spatial memories are projected to 64 channels. An object pointer is a 256-number summary of the object (from the mask decoder’s output token), split into 4 tokens of 64 numbers for the cross-attention. [Certain]
  • The prompt encoder and mask decoder are as in SAM: prompts become tokens, and the decoder outputs mask logits plus a predicted IoU. On frames without a prompt the decoder runs with no prompt tokens and relies on the memory-conditioned features.
  • An occlusion head (an extra output token and an MLP) predicts whether the object is visible in this frame at all. If not, the output mask is empty, and a learned “occluded” embedding is added to that frame’s memory. [Certain]
  • The memory encoder fuses the predicted mask with \(F_t\) and writes the result into the bank.
\[ \begin{gathered} F'_t = \mathrm{MemAttn}\big(F_t;\ \mathcal{B}_t\big) \\ \mathcal{B}_t = \{\mathcal{M}_p\}_{p \in \mathcal{P}_t} \cup \{\mathcal{M}_{t-1}, \dots, \mathcal{M}_{t-N}\} \cup \{\mathbf{o}_s\} \end{gathered} \tag{204.5} \]
Read the symbols
\(F_t\)
image features of frame \(t\) from the encoder (64 × 64 positions for a 1024 input) [Likely]
\(\mathcal{M}_s\)
the spatial memory of frame \(s\): its features fused with its predicted mask, 64 channels per position
\(\mathcal{P}_t\)
the frames before \(t\) where the user gave a prompt; they do not drop out with the recent queue
\(\mathbf{o}_s\)
object pointers: one compact vector per memory frame that summarizes the object
\(F'_t\)
features that now carry “which object we are following”; the mask decoder reads them

Compare with formula (204.2): the cross-attention inside MemAttn is the same memory read, now inside a transformer block with learned projections and position encodings. The first frame of STM and XMem (always kept) becomes the set of prompted frames \(\mathcal{P}_t\) in SAM 2.

Figure 4. What SAM 2 does for one frame. Choose the kind of input on this frame; the active path is highlighted. The image encoder and memory attention run on every frame; the prompt encoder only when the user gives a prompt on this frame.
How to read Figure 4

On a normal frame, the only information about the object comes from the memory bank on the left: memory attention turns “frame features” into “frame features that know the target”. A click or a box adds prompt tokens to the mask decoder, the same way as in SAM. A mask prompt is different: it already gives the object’s pixels on that frame, so its main job is to become a strong prompted memory. In every case the result is written back into the memory bank, and a prompted frame stays in the bank even when it is old. With “Object hidden”, the occlusion head says “not visible”: the output is empty and the memory is marked as occluded, so later frames do not learn that the object is now the pillar.

How SAM 2 is trained

SAM 2 is trained on images and videos together, on sequences of 8 frames. The first prompt is the true mask (50% of the time), a positive click (25%) or a box (25%), and up to 2 frames get corrective clicks sampled from the error regions of the model’s own prediction. [Certain] This simulates a user. The training video data, SA-V, was built with a data engine like SA-1B: annotators used SAM 2 itself, and each round produced a better model. SA-V has 50.9K videos (about 196 hours) with 642.6K masklets and 35.5M masks; 42.5% of the manual masklets disappear at some point, which teaches occlusion. [Certain]

Model (semi-supervised VOS, J&F)MOSE valDAVIS 2017 valYouTube-VOS 2019 val
XMem59.686.085.6
Cutie-base+71.788.187.5
SAM 2 (Hiera-B+), 43.8 FPS76.690.288.6
SAM 2 (Hiera-L), 30.2 FPS77.990.789.3

The numbers are from the SAM 2 paper (FPS on one A100). [Certain] Note how close the models are on DAVIS, a small and old benchmark, and how far apart on MOSE, a benchmark of complex scenes with crowds, occlusions and objects that disappear. This is a general lesson: when benchmarks saturate, the differences move to harder data.

Algorithm 2: interactive video segmentation with SAM 2
  1. input: video frames, a stream of user prompts (frame index, clicks / box / mask)
  2. for \(t = 1, \dots, T\): \(F_t \leftarrow\) image encoder\((I_t)\) // can be cached and shared by all objects
  3. on the first prompt at frame \(p\): \(\hat M_p \leftarrow\) mask decoder\((F_p, \text{prompt tokens})\); write \(\mathcal{M}_p\) as a prompted memory
  4. for each later frame \(t\): \(F'_t \leftarrow\) MemAttn\((F_t; \mathcal{B}_t)\) (204.5); \(\hat M_t, \text{visible}_t \leftarrow\) mask decoder\((F'_t)\)
  5.   if not visible\(_t\): \(\hat M_t \leftarrow \varnothing\), mark the memory as occluded
  6.   write \(\mathcal{M}_t\) into the recent FIFO (keep the last \(N = 6\))
  7. when the user corrects frame \(c\) with a click: recompute \(\hat M_c\) with the new prompt, add \(\mathcal{M}_c\) to the prompted set, re-propagate from \(c\) (the video can also be processed backward)
  8. return the masklet \(\hat M_1, \dots, \hat M_T\)

Watching the memory bank work

Algorithm 2 hides the important details in two short lines: what the occlusion head decides, and what is written into the bank. In SAM 2 the occlusion head is a learned output token followed by a small MLP. [Certain] To watch the mechanism, we replace it with the simplest stand-in that behaves the same way: the object score of a frame is the strongest read-out of any of its pixels. If no pixel of the new frame matches the object memories well, the object is declared hidden. Writing this together with the FIFO update of the recent memories:

\[ \begin{gathered} s_t = \max_j \hat v_{tj}, \qquad \text{visible}_t = \mathbb{1}\big[s_t \gt 0.5\big] \\ \hat M_t = \begin{cases} \{\, j : \hat v_{tj} \gt 0.5 \,\} & \text{if visible}_t \\ \varnothing, \ \text{memory marked occluded} & \text{otherwise} \end{cases} \\ \mathcal{R}_{t+1} = \text{the last } N \text{ entries of } \big(\mathcal{R}_t,\ \mathcal{M}_t\big) \end{gathered} \tag{204.6} \]
Read the symbols
\(\hat v_{tj}\)
the memory read-out (204.2) for pixel \(j\) of frame \(t\): the share of its attention that lands on object pixels of the memory
\(s_t\)
the object score of frame \(t\); in SAM 2 a learned head outputs this score, here it is the largest read-out
\(\hat M_t\)
the output mask: the pixels whose read-out is above one half, or nothing if the object is hidden
\(\mathcal{R}_t\)
the recent memories (a FIFO queue of length \(N\)); the prompted memories are kept outside this queue and never drop out

Figure 5 runs SAM 2’s loop on a toy video of 20 frames with 16 × 8 pixels. A block moves right and changes color by a few degrees per frame. It passes behind a pillar and is fully hidden in frames 10 to 13. The user gives one click on frame 1; a tiny stand-in for the prompt decoder grows a region from the click over neighbors of similar color (within \(25^\circ\)), which gives the 9-pixel mask of the block. From then on, every frame runs lines 4 to 8 of the algorithm box, with the real memory read (204.2), \(\tau = 0.1\), over all \(128\) pixels of every frame in the bank.

Worked example: frame 14, the block comes back

Default settings: \(N = 6\), color change \(4^\circ\) per frame. At frame 14 the bank holds the prompted frame 1 and the recent frames 8 to 13: 7 frames × 128 pixels = 896 keys. Frames 10 to 13 are marked occluded (their stored masks are empty). Only one column of the block is visible, right of the pillar. Take its middle pixel, at position (11, 4), with hue \(68.1^\circ\). Its attention is split as follows: frame 9 gets 0.492, frame 8 gets 0.288, the prompted frame 1 gets 0.058, and the four hidden frames together get about 0.16. The read-out is \(\hat v = 4.2982 / 5.9195 = 0.726\): object. The object score is \(s_{14} = 0.962 \gt 0.5\), so the frame is visible, and the mask has exactly the 3 true pixels.

Now set \(N = 4\). The bank at frame 14 holds frame 1 and only the four hidden frames 10 to 13; nothing recent shows the block. The same pixel reads \(\hat v = 0.131\), the object score is 0.751, and the mask has 1 pixel (\(\mathcal{J} = 0.333\)). From frame 15 on the block is declared hidden and stays lost: mean J over frames 2 to 20 drops from 1.000 to 0.649.

Finally keep \(N = 6\) but switch the occlusion head off: the decoder must always output its best guess, \(\{ j : \hat v_{tj} \ge 0.5\, s_t \}\). In frame 10 that is 19 pixels, mostly on the pillar; in frames 11 to 13 it is the whole pillar (40 pixels). These wrong masks are written into memory as “object”, so from frame 15 on the pillar stays in the mask next to the real block (\(\mathcal{J} = 6/46 = 0.130\) in frame 15). Mean J: 0.526. One bad write poisons every later frame. That is why SAM 2 marks hidden frames instead of writing a guess.

query pixel, read-out and predicted masktrue mask (outline)prompted memoryrecent memory (stored mask)occluded / dropped
  1. input: frames \(I_1, \dots, I_{20}\), one click on frame 1
  2. \(\hat M_1 \leftarrow\) prompt decoder\((F_1, \text{click})\); write \(\mathcal{M}_1\) as a prompted memory
  3. for \(t = 2, \dots, 20\):
  4. \(F_t \leftarrow\) image encoder\((I_t)\)
  5. \(\hat v_{tj} \leftarrow \sum_i \softmax_i(\mathbf{q}_j^\top \mathbf{k}_i / \tau)\, v_i\) over the bank \(\mathcal{B}_t\) (204.2)
  6. \(s_t \leftarrow \max_j \hat v_{tj}\); visible \(\leftarrow s_t \gt 0.5\) (204.6)
  7. \(\hat M_t \leftarrow \{ j : \hat v_{tj} \gt 0.5 \}\) if visible, else \(\varnothing\) and mark occluded
  8. write \(\mathcal{M}_t\) into the recent FIFO; keep the last \(N\)
Figure 5. The SAM 2 memory loop computed in your browser. Top: the 20 frames; under each, a bar for its J once it has been processed, and a mark for its role in the current bank. Middle: the current frame (pixel colors are the features), the read-out \(\hat v_j\) of every pixel (204.2), and the output mask with the true outline (204.6). Bottom: the memory bank (prompted frame, then the \(N\) recent frames, oldest first). Lines and bars show how the attention of the query pixel is split over the memory frames. Click a pixel of the current frame to choose the query; click a frame in the strip to jump to it.
How to read Figure 5

Press Play and watch one frame take five steps: encode (line 4), read the memory (line 5, the read-out map fills in and the attention lines appear), decide visibility (line 6), output the mask (line 7), and write the new memory into the FIFO (line 8, the new slot is marked “new” and the oldest recent memory is dropped). During frames 10 to 13 the object score falls to 0.000, the mask is empty, and red “occluded” marks appear in the bank. At frame 14 the attention of the query pixel goes mostly to frames 8 and 9, the last frames before the pillar. Then try the presets. “Short memory” has no pre-occlusion frame left at frame 14 and loses the block. “No occlusion head” writes the pillar into memory as the object, and the error never goes away. Raise the color change to 6: even \(N = 6\) cannot bridge the gap, because the block now looks too different from both frame 1 and frame 9.

Measuring VOS: J, F, recall and decay

A VOS benchmark compares the predicted mask \(\hat M\) with the true mask \(G\) of each object in each frame. Two measures are used, both from DAVIS. [Certain] The region similarity \(\mathcal{J}\) is the intersection over union (IoU, also called the Jaccard index):

\[ \mathcal{J}(\hat M, G) = \frac{\lvert \hat M \cap G \rvert}{\lvert \hat M \cup G \rvert}, \qquad \mathcal{J} = 1 \text{ if } \hat M \cup G = \varnothing \tag{204.7} \]

The second line matters: when the object is hidden and the model correctly outputs nothing, the score is 1, not undefined. (The official DAVIS code returns 1 when the union is empty.) [Certain]

IoU counts pixels, so it is dominated by the inside of large objects and barely notices a messy outline. The boundary accuracy \(\mathcal{F}\) looks only at the contours. Let \(B(\hat M)\) and \(B(G)\) be the boundary pixels of the two masks, and let \(d\) be a small distance tolerance:

\[ \begin{gathered} P_c = \frac{\lvert \{ x \in B(\hat M) : \operatorname{dist}(x, B(G)) \le d \} \rvert}{\lvert B(\hat M) \rvert}, \quad R_c = \frac{\lvert \{ x \in B(G) : \operatorname{dist}(x, B(\hat M)) \le d \} \rvert}{\lvert B(G) \rvert} \\ \mathcal{F} = \frac{2 P_c R_c}{P_c + R_c} \end{gathered} \tag{204.8} \]
Read the symbols
\(P_c\)
contour precision: the fraction of the predicted outline that lies close to the true outline
\(R_c\)
contour recall: the fraction of the true outline that the prediction comes close to
\(d\)
the tolerance; the official code uses 0.008 of the image diagonal, rounded up: for 854 × 480 that is \(\lceil 0.008 \times 979.7 \rceil = 8\) pixels [Certain]
\(\mathcal{F}\)
the harmonic mean of the two (the F-measure), between 0 and 1

The benchmark score averages over objects, not over videos: first over the frames of each object, then over all objects \(\mathcal{O}\). The final number is the mean of the two:

\[ \begin{gathered} m(\mathcal{J}) = \frac{1}{\lvert \mathcal{O} \rvert} \sum_{o \in \mathcal{O}} \frac{1}{\lvert \mathcal{F}_o \rvert} \sum_{f \in \mathcal{F}_o} \mathcal{J}\big(\hat M_f^{o}, G_f^{o}\big) \\ \mathcal{J}\&\mathcal{F} = \tfrac12 \big( m(\mathcal{J}) + m(\mathcal{F}) \big) \end{gathered} \tag{204.9} \]

Here \(\mathcal{F}_o\) is the set of scored frames of object \(o\) (the given first frame is not scored; the official code also skips the last frame). [Certain] A video with 3 objects counts 3 times. DAVIS also reports two summary statistics per object, from its per-frame scores \(\mathcal{J}_f\):

\[ \begin{gathered} \text{Recall} = \frac{1}{\lvert \mathcal{F}_o \rvert} \sum_f \mathbb{1}\big[\mathcal{J}_f \gt 0.5\big] \\ \text{Decay} = \operatorname{mean}(\mathcal{J} \text{ on the first quarter}) - \operatorname{mean}(\mathcal{J} \text{ on the last quarter}) \end{gathered} \tag{204.10} \]

Recall counts how often the object is “basically found”; decay measures drift: a large positive decay means the model was good at the start and got worse. Both are in Figure 2’s read-out. YouTube-VOS reports J and F separately for categories seen in training and unseen ones, and an overall score that averages the four. [Likely]

Worked example: J punishes small objects, F forgives small shifts

Take the 38-pixel object of Figure 6 and predict exactly the same shape, shifted one pixel to the right. The intersection has 32 pixels and the union 44, so \(\mathcal{J} = 32/44 = 0.727\). Both outlines have 18 boundary pixels. With tolerance \(d = 0\), only 8 of the 18 predicted boundary pixels lie on the true boundary: \(P_c = R_c = 0.444\), \(\mathcal{F} = 0.444\). With \(d = 1\), every boundary pixel is within one pixel of the other outline: \(\mathcal{F} = 1.000\).

For a square object of side \(s\) shifted by one pixel, \(\mathcal{J} = (s - 1)/(s + 1)\): 0.714 for \(s = 6\), 0.905 for \(s = 20\), 0.980 for \(s = 100\). The same one-pixel error costs a small object far more. That is why benchmark scores are lower on datasets with many small objects, and why you should report results by object size.

true mask (outline)predicted mask (click cells to toggle)boundary pixel within dboundary pixel not within d
Figure 6. J and F on a 14 × 10 grid. The true mask (orange outline) has 38 pixels. Click any cell to add it to or remove it from the prediction. Boundary pixels are mask pixels with at least one of their 4 neighbors outside the mask; the tolerance uses straight-line distance, as the disk-shaped dilation in the DAVIS code.
How to read Figure 6

The start is the one-pixel shift from the worked example. Move \(d\) from 1 to 0 and watch F fall from 1.000 to 0.444 while J stays at 0.727: F is about outlines, J about area. Press “Perfect”, then “Add a thin spike”: J falls only to 0.927, because the spike has few pixels, but the outline is now longer and partly wrong. Try removing one corner pixel, or making the outline jagged by clicking cells along the edge: J changes little, F changes a lot at \(d = 0\). Press “Clear”: an empty prediction gives \(\mathcal{J} = 0\) and \(\mathcal{F} = 0\).

Referring segmentation: from a sentence to a masklet

Clicks need a person. Many applications start from language instead: “segment the dog that jumps over the fence”, “track the player with number 10”. This is referring video object segmentation (RVOS). The sentence may describe appearance (“the man in the red shirt”), position (“the car on the left”) or motion (“the bird that flies away first”), and the last kind can only be solved by looking at several frames.

The standard benchmarks: Ref-YouTube-VOS (3,978 videos with about 15K expressions) and Ref-DAVIS17 (90 videos, 1,544 expressions about 205 objects), both scored with J, F and J&F. [Certain] MeViS (2023) focuses on motion expressions, where a single frame is not enough to know which object is meant. [Certain]

There are two families of methods.

End to end. ReferFormer (2022) treats the sentence as a set of queries. A few object queries (5 by default) are conditioned on the sentence embedding and shared across all frames. Each query attends to the video features and becomes a dynamic kernel: a weight vector that is dotted with every pixel feature to produce a mask, the same pattern as SAM’s mask decoder. [Certain] Because the same query is used in every frame, the masks of one query form a track automatically. In formula form, for query \(n\) and frame \(t\):

\[ \begin{gathered} \mathbf{w}_{n} = g\big(\mathbf{z}_n,\ \mathbf{e}_{\text{text}},\ \{F_t\}\big), \qquad \hat M_{t,n}(x, y) = \sigma\big(\mathbf{w}_{n,t}^\top E_t(x, y)\big) \\ n^\ast = \argmax_n \ \tfrac{1}{T} \textstyle\sum_t p_{t,n} \end{gathered} \tag{204.11} \]
Read the symbols
\(\mathbf{z}_n\)
learned object query \(n\)
\(\mathbf{e}_{\text{text}}\)
the sentence embedding from a text encoder
\(\mathbf{w}_{n,t}\)
the dynamic kernel of query \(n\) in frame \(t\) (computed by the transformer \(g\))
\(E_t(x, y)\)
the pixel feature at \((x, y)\) in frame \(t\)
\(p_{t,n}\)
a score for “query \(n\) is the referred object in frame \(t\)”; the best query on average gives the answer

On the Ref-YouTube-VOS validation server, ReferFormer reports J&F 62.4 with a Swin-L backbone, 64.9 with Video-Swin-B and joint training on RefCOCO images. [Certain] Note how much lower this is than the 85 to 90 of semi-supervised VOS: understanding the sentence is the hard part, not the tracking.

Detect, then track. A modular pipeline uses an open-vocabulary grounding model (Grounding DINO, or a VLM that outputs boxes) to find the referred object in one or a few key frames, then gives the box to SAM 2 as a prompt. This is how many practical systems work (for example the “Grounded SAM 2” open-source project). [Likely] Recent models merge the two: Sa2VA connects a multimodal LLM to SAM 2. The LLM reads the video and the question and generates tokens that guide SAM 2 in producing the masks [Certain]; concretely, a special segmentation token whose hidden state is used as a prompt for SAM 2’s decoder, which then tracks the object. [Likely] SAM 3 (2025) adds short noun phrases (“yellow school bus”) as prompts directly, returning all matching instances with identities. [Certain]

Algorithm 3: referring VOS with a grounding model plus SAM 2
  1. input: frames \(I_1, \dots, I_T\), sentence \(x\), stride \(k\), threshold \(\tau\)
  2. for key frames \(t \in \{1, 1 + k, 1 + 2k, \dots\}\): \((\mathbf{b}_t, p_t) \leftarrow\) grounding model\((I_t, x)\) // best box and its score
  3. \(t^\ast \leftarrow \argmax_t p_t\); if \(p_{t^\ast} \lt \tau\): return “object not found”
  4. prompt SAM 2 with box \(\mathbf{b}_{t^\ast}\) on frame \(t^\ast\)
  5. propagate forward from \(t^\ast\) to \(T\), and backward from \(t^\ast\) to 1 (Algorithm 2)
  6. optional check: on other key frames, if SAM 2’s mask and the grounding box disagree strongly (low IoU), add the box as a new prompt
  7. return the masklet

For motion expressions, a single key frame is not enough: “the dog that starts running” is just “a dog” in most frames. Then you need a model that looks at the whole clip, such as a session 205 style model that picks the frame and the object, or an end-to-end RVOS model.

The PyTorch version

The code below runs the experiment of Figure 2 with real tensors: a toy video, the memory read (204.2), three memory designs, and the J and F measures (204.7), (204.8). It needs only PyTorch and runs in about a second on a CPU. Because PyTorch’s random numbers differ from the figure’s, the “first frame only” score is a little different from Figure 2.

import torch
import torch.nn.functional as F
torch.manual_seed(0)

# 1) a toy video: T frames of H x W pixels; each pixel has a 2-D unit "feature" (its hue)
T, H, W = 40, 12, 20
def make_video(drift=2.0, noise=12.0):
    feats, masks = [], []
    for t in range(T):
        hue = torch.where(torch.arange(H)[:, None] < 6, 220.0, 150.0).expand(H, W).clone()  # sky, grass
        m = torch.zeros(H, W)
        x0 = round(1 + t * 14 / (T - 1))
        m[4:8, x0:x0 + 4] = 1                      # the object: a 4 x 4 block moving right
        hue[m.bool()] = drift * t                  # its color changes slowly over time
        m[:, 9:14] = 0; hue[:, 9:14] = 270.0       # a pillar in front of everything
        a = torch.deg2rad(hue + (torch.rand(H, W) - 0.5) * 2 * noise)
        feats.append(torch.stack([a.cos(), a.sin()], -1).reshape(H * W, 2))
        masks.append(m.reshape(H * W))
    return feats, masks

# 2) memory read, formula (204.2): softmax over all memory pixels, then average their mask values
def memory_read(q, K, V, tau=0.1):
    A = F.softmax(q @ K.T / tau, dim=1)            # (HW, memory size); each row sums to 1
    return A @ V                                    # (HW,) soft mask in [0, 1]

def track(feats, first_mask, keep_first=True, n_recent=6):
    preds = [first_mask]
    for t in range(1, T):
        mem = [0] if keep_first else []
        mem += [s for s in range(max(0, t - n_recent), t) if s not in mem]
        K = torch.cat([feats[s] for s in mem])
        V = torch.cat([preds[s] for s in mem])     # past PREDICTED masks, not the truth
        preds.append((memory_read(feats[t], K, V) > 0.5).float())
    return preds

# 3) region measure J, formula (204.7); J = 1 when both masks are empty
def jaccard(p, g):
    inter, union = (p * g).sum(), ((p + g) > 0).sum()
    return 1.0 if union == 0 else (inter / union).item()

feats, masks = make_video()
for name, kf, n in [("previous frame only", False, 1), ("first frame only", True, 0), ("first + 6 recent", True, 6)]:
    preds = track(feats, masks[0], kf, n)
    J = [jaccard(preds[t], masks[t]) for t in range(1, T)]
    print(f"{name:20s} mean J = {sum(J) / len(J):.3f}   J at last frame = {J[-1]:.2f}")
# previous frame only  mean J = 0.667   J at last frame = 0.00   (lost after the pillar)
# first frame only     mean J = 0.863   J at last frame = 0.12   (the color drifted away)
# first + 6 recent     mean J = 1.000   J at last frame = 1.00

# 4) boundary measure F, formula (204.8), with a tolerance of d pixels
def boundary(m):                                   # mask pixels with a 4-neighbour outside the mask
    cross = torch.tensor([[0., 1, 0], [1, 1, 1], [0, 1, 0]])[None, None]
    full = F.conv2d(F.pad(m[None, None], (1, 1, 1, 1)), cross)[0, 0] == 5
    return m.bool() & ~full

def f_measure(p, g, d=1.0):
    bp, bg = boundary(p).nonzero().float(), boundary(g).nonzero().float()
    if len(bp) == 0 or len(bg) == 0:
        return float(len(bp) == len(bg))
    dist = torch.cdist(bp, bg)                     # distances between all boundary pixels
    prec = (dist.min(1).values <= d).float().mean()  # predicted boundary near the true one
    rec = (dist.min(0).values <= d).float().mean()   # true boundary near the predicted one
    return 0.0 if prec + rec == 0 else (2 * prec * rec / (prec + rec)).item()

g = torch.zeros(10, 14); g[2:8, 3:10] = 1; g[2, 3] = g[2, 9] = g[7, 3] = g[7, 9] = 0
p = torch.roll(g, 1, dims=1)                       # the same mask, one pixel to the right
print(f"J = {jaccard(p, g):.3f}   F(d=0) = {f_measure(p, g, 0):.3f}   F(d=1) = {f_measure(p, g, 1):.3f}")
# J = 0.727   F(d=0) = 0.444   F(d=1) = 1.000   (the worked example)

To use the real SAM 2, the official sam2 package has a video predictor with the same loop as Algorithm 2: you initialize a state on a folder of frames, add clicks or a box for an object id on some frame, and call a propagation function that yields the masks frame by frame. [Likely] The lab session does this on sports video.

Where this is used

ApplicationSetupWhat matters
Video editing and rotoscopingclicks on one frame, SAM 2 propagates; corrections on bad framesboundary quality (F), few clicks, handling hair and motion blur
Sports analyticsdetector finds players, SAM 2 tracks each one; numbers read with OCR or a VLMidentity switches between similar players, occlusions in crowds (session 217)
Data labeling for driving and roboticsannotator labels a few frames, the model fills in the rest (a data engine like SA-V)speed of the human loop; label quality checks
Medical video (endoscopy, ultrasound)mask of an organ or tool in one frame, tracked through the clip [Likely]domain shift from natural images; often needs fine-tuning
Video retrieval and Video-RAGobject masklets become object-level entries in an index: “all shots where this car appears” (session 210)long-video memory, re-identification after scene cuts
Key-moment detectiontrack the ball or a player, detect events from its trajectory (session 207)stable tracks; the occlusion signal is itself useful
Referring segmentation in assistants“highlight the person who drops the bag”: a Video-LLM chooses the object, SAM 2 tracks itunderstanding motion words; refusing when the object is not there
Privacy and compliancetrack faces or license plates and blur them in every framerecall: one missed frame shows the face

Common mistakes

  • Running an image segmenter on each frame independently. SAM on frame 50 does not know which object you clicked in frame 1, and the masks flicker. You need a memory (or at least an association step).
  • Ignoring occlusion. If a hidden frame’s wrong mask (for example on the occluder) goes into memory, the model starts following the occluder. Use the occlusion score, and do not write low-confidence frames as strong memories.
  • Unbounded memory on long videos. Keeping every frame works in a 3-second demo and runs out of GPU memory on a 20-minute video (formula 204.4, Figure 3).
  • Too short a memory. The opposite mistake: a window of \(N\) frames cannot bring back an object that was hidden longer than \(N\) frames unless the prompted frame still looks like it (Figure 2 at drift 2.6).
  • Averaging J over videos instead of objects. DAVIS averages over objects (formula 204.9). A video with 5 objects must weigh 5 times more. Your numbers will not match the leaderboard otherwise.
  • Reporting only J. J barely changes for bad outlines on large objects. Report J, F and J&F, and use the official evaluation code (its boundary tolerance and frame selection matter).
  • Comparing models at different resolutions or with different first-frame inputs. A mask prompt is much easier than a single click; 480p is easier than full resolution for speed but loses thin parts.
  • Assuming objects keep their identity across scene cuts. Memory matching assumes one continuous shot. After a cut, detect again and re-identify, or the model will latch onto something else.

Interview questions

What is semi-supervised video object segmentation, and why is it called semi-supervised?basic
Given a video and the true mask of one or more objects in the first frame, output the masks of those same objects in all later frames. It is “semi-supervised” only because one frame’s annotation is given at test time; the network is trained beforehand on other videos and is not trained on the test video. Unsupervised VOS gets no mask; interactive VOS gets clicks over time; referring VOS gets a sentence.
Why does “propagate the previous mask” fail, and how does memory-based matching fix it?basic
Propagation copies its own errors forward (drift), and after a full occlusion the previous mask is empty, so there is nothing to propagate: the object is lost forever. Memory matching compares each pixel of the new frame with pixels of several stored frames, including the first frame with the true mask, by attention over all positions. The first frame lets the model re-find the object after occlusion; the recent frames handle changes of appearance.
Write the STM memory read as a formula and explain each part.core
For query pixel \(j\) of the current frame and memory pixels \(i\) from all memory frames: \(A_{ji} = \softmax_i(\mathbf{q}_j^\top \mathbf{k}_i / \tau)\) and \(\hat{\mathbf{v}}_j = \sum_i A_{ji} \mathbf{v}_i\). Keys and queries come from image features and are used for matching; values come from the memory encoder, which also saw the mask, so they carry “object or not” information. The softmax runs over space and time together, so there is no limit on how far an object can move. A decoder turns \(\hat{\mathbf{v}}_j\) plus the current frame’s features into the mask. It is cross-attention with the current frame as queries and the memory as keys and values.
Why do STCN and XMem use a negative squared L2 distance instead of a dot product?advanced
\(-\lVert \mathbf{k} - \mathbf{q} \rVert^2 = 2\mathbf{k}^\top\mathbf{q} - \lVert \mathbf{k} \rVert^2 - \lVert \mathbf{q} \rVert^2\). The \(\lVert \mathbf{q} \rVert^2\) term is constant over memory items and cancels in the softmax, but \(-\lVert \mathbf{k} \rVert^2\) penalizes large-norm keys. With a dot product, a few keys with large norms win for many queries and most of the memory is never used. L2 gives better memory coverage. XMem adds a per-key shrinkage factor (confidence) and a per-query channel selection, which reduce to plain L2 when set to 1.
How does XMem keep memory bounded? Give the numbers.core
It has three stores. Sensory memory: a GRU hidden state updated every frame. Working memory: high-resolution keys and values of recent memory frames, added every 5 frames, kept between 5 and 10 frames. When it reaches 10, the first frame and the 4 most recent stay, and the other 5 frames (8,100 positions at 1,620 each) are consolidated into 128 prototypes chosen by usage (accumulated attention), with values averaged by similarity (potentiation). Long-term memory holds at most 10,000 prototypes, removing the least used. So memory is at most \(10 \times 1{,}620 + 10{,}000 = 26{,}200\) positions per object, independent of video length.
Walk through what SAM 2 does on one frame of a video.core
(1) The Hiera image encoder computes frame features once. (2) Memory attention, 4 transformer blocks with self-attention, cross-attention to the memory bank and an MLP, conditions the features on the target. The bank has the 6 most recent frames’ memories, the memories of prompted frames, and object pointer tokens. (3) If the user gave clicks or a box on this frame, the prompt encoder adds tokens; a mask prompt is embedded and added to the features. (4) The SAM-style mask decoder outputs the mask, a predicted IoU, and an occlusion score. (5) The memory encoder fuses the predicted mask with the frame features and writes the result into the bank (FIFO for recent frames; prompted frames are kept).
How does SAM 2 handle an object that disappears and comes back?core
An occlusion head predicts whether the object is visible in the current frame. If not, SAM 2 outputs an empty mask and adds a learned occlusion embedding to that frame’s memory, so the memory does not teach “the target now looks like the occluder”. When the object comes back, the prompted frames (always in the bank) and the object pointers still describe it, so memory attention finds it again. If it was hidden longer than the recent window and has changed a lot, recovery can fail; the user then adds a click, which becomes a new prompted memory. SA-V was built with many disappearances (42.5% of manual masklets) to train this.
Why not simply run SAM on every frame and link the masks by IoU?basic
SAM has no memory, so on each frame you would need a new prompt, and the prompt location must come from somewhere: the previous mask, which brings back drift and loss after occlusion. Linking by IoU fails with fast motion and when similar objects cross. Per-frame masks also flicker, because each frame is decided independently. SAM 2 conditions every frame on a memory of the target, so one prompt is enough for the whole video, and it is faster because the decoder is light and the encoder runs once per frame.
Define J and F. When do they disagree?core
J is the IoU of predicted and true masks (1 if both are empty). F is the harmonic mean of contour precision and recall: the fraction of the predicted boundary within a tolerance \(d\) of the true boundary, and vice versa (DAVIS uses \(d\) = 0.008 of the image diagonal). They disagree when errors are small in area but large in outline (a thin spike, a jagged edge, hair) — F drops, J does not — or when a small object is shifted by a pixel — J drops a lot, F at a tolerance of a few pixels does not. J&F is their mean, averaged over objects.
How would you build a system that segments “the man in the red shirt who sits down” in a video?advanced
“Red shirt” is appearance, “sits down” is motion, so one frame is not enough. Option 1: run a grounding model on several key frames to get candidate boxes for “man in red shirt”, track every candidate with SAM 2, then decide which track sits down using a Video-LLM on the candidate tracks, or simple trajectory features (the person’s height drops). Option 2: an end-to-end model that reads the whole clip with the sentence (ReferFormer-style queries, or a Sa2VA-style LLM that emits a segmentation token for SAM 2). Evaluate with J&F on Ref-YouTube-VOS or MeViS, and test “no such person” cases so the system can answer “not found” instead of always returning a mask.
SAM 2 is 90.7 J&F on DAVIS 2017 but 77.9 on MOSE. What does that tell you?advanced
DAVIS 2017 is small (30 validation videos), short (about 70 frames), and has few heavy occlusions; it is close to saturated, and differences between strong models are only a few points. MOSE has crowded scenes, similar objects and objects that disappear, which is what breaks memory matching. The gap between models is also larger there (XMem 59.6, Cutie-base+ 71.7, SAM 2 77.9). So: choose evaluation data that looks like your application, and expect much lower numbers on long, crowded videos than on DAVIS.

Exercises

  1. In the worked memory read, what is the read-out for a query at \(120^\circ\)? Is the pixel object or background?
  2. XMem settings: \(r = 5\), \(T_{\min} = 5\), \(T_{\max} = 10\), \(P = 128\), \(LT_{\max} = 10{,}000\). After how many consolidations is the long-term memory full, and about how many seconds of 24 FPS video is that? Check with Figure 3.
  3. A square object of 10 × 10 pixels is predicted shifted by 2 pixels to the right. Compute J.
  4. A predicted outline has 40 boundary pixels, 30 of them within \(d\) of the true outline; the true outline has 50 boundary pixels, 35 of them within \(d\) of the prediction. Compute F.
  5. A benchmark has video A with 1 object (\(\mathcal{J} = 0.9\)) and video B with 3 objects (\(\mathcal{J} = 0.6, 0.7, 0.8\)). Compute the DAVIS-style mean J and the (wrong) per-video mean.
  6. In Figure 2, set the color change to 2.6 per frame and keep the first frame. What is the smallest \(N\) that recovers the object after the pillar? Explain why in terms of which frames are in memory at frame 28.
Answerscheck
1. Scores \(\cos(120^\circ)/0.1 = -5.0\), \(\cos(110^\circ)/0.1 = -3.42\), \(\cos(30^\circ)/0.1 = 8.66\), \(\cos(100^\circ)/0.1 = -1.74\); the weight on the 150° background pixel is 1.000 (to 3 digits), so the read-out is 0.000: background. 2. Each consolidation adds 128 prototypes: \(\lceil 10{,}000/128 \rceil = 79\) consolidations. One consolidation happens every 5 memory frames = 25 video frames, after a start of 9 memory frames (45 frames), so the cap is reached at about frame 1,996, which is 83 seconds. 3. Intersection \(10 \times 8 = 80\), union \(10 \times 12 = 120\), \(\mathcal{J} = 0.667\) (formula \((s-2)/(s+2)\)). 4. \(P_c = 30/40 = 0.75\), \(R_c = 35/50 = 0.70\), \(\mathcal{F} = 2 \times 0.75 \times 0.70 / 1.45 = 0.724\). 5. Per object: \((0.9 + 0.6 + 0.7 + 0.8)/4 = 0.75\). Per video: \((0.9 + 0.7)/2 = 0.80\). The per-video mean hides that the multi-object video is harder. 6. \(N = 7\). The object is fully hidden in frames 22 to 27. At frame 28, \(N = 6\) holds frames 22 to 27: only pillar and background, no object pixels. The first frame’s object (hue near 0°) is too far from the returning object (about 70°), which is now closer to the grass (150°). With \(N = 7\), frame 21 is also in memory, where 4 object pixels are still visible with a similar color, so the read-out finds the object again. Mean J jumps from 0.708 to 0.979.

Key takeaways

  • Semi-supervised VOS (204.1): given the mask in frame 1, predict the object’s mask in every later frame, online, using only past frames and the model’s own past predictions.
  • Propagating the previous mask drifts and gets lost after occlusion. Memory networks match every pixel of the new frame against stored frames with attention (204.2): queries from the new frame, keys and mask-carrying values from memory.
  • Memory must be both reliable (the prompted first frame) and fresh (recent frames). XMem bounds memory on long videos with working and long-term (prototype) memory: at most about 26,200 positions per object (204.4).
  • SAM 2 = SAM + streaming memory (204.5): memory attention over prompted frames, the 6 most recent frames and object pointers; prompts (clicks, boxes, masks) on any frame; an occlusion head; trained on SA-V (50.9K videos, 35.5M masks).
  • Metrics: J is mask IoU (204.7), F is boundary F-measure with a tolerance (204.8); J&F is averaged over objects (204.9); recall and decay (204.10) show how often the object is found and how much the model drifts.
  • Referring VOS replaces the first-frame mask with a sentence: end-to-end text-conditioned queries (204.11), or a grounding model / Video-LLM that prompts SAM 2.

Key formulas

\[ \hat{\mathbf{v}}_j = \sum_i \softmax_i\!\left(\frac{\mathbf{q}_j^\top \mathbf{k}_i}{\tau}\right) \mathbf{v}_i, \qquad \mathcal{J} = \frac{\lvert \hat M \cap G \rvert}{\lvert \hat M \cup G \rvert}, \qquad \mathcal{F} = \frac{2 P_c R_c}{P_c + R_c} \]
\[ \lvert \text{memory}_{\text{XMem}} \rvert \le T_{\max} HW + LT_{\max}, \qquad F'_t = \mathrm{MemAttn}\big(F_t;\ \{\mathcal{M}_p\} \cup \{\mathcal{M}_{t-1..t-N}\} \cup \{\mathbf{o}_s\}\big) \]

References for this session

  1. Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
  2. Cheng & Schwing (2022). XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
  3. Pont-Tuset et al. (2017). The 2017 DAVIS Challenge on Video Object Segmentation
  4. Kirillov et al. (2023). Segment Anything
  5. Oh, Lee, Xu & Kim (2019). Video Object Segmentation using Space-Time Memory Networks
  6. Cheng, Tai & Tang (2021). Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation
  7. Cheng, Oh, Price, Lee & Schwing (2023). Putting the Object Back into Video Object Segmentation
  8. Xu et al. (2018). YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
  9. Ding et al. (2023). MOSE: A New Dataset for Video Object Segmentation in Complex Scenes
  10. Perazzi et al. (2016). A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation
  11. Wu, Jiang, Sun, Yuan & Luo (2022). Language as Queries for Referring Video Object Segmentation
  12. Ding et al. (2023). MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions
  13. Yuan et al. (2025). Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos
  14. Carion et al. (2025). SAM 3: Segment Anything with Concepts
  15. Yang et al. (2024). SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory