LLM & VLM courseBlog
168Track 7: Vision and VLMs · session 6 of 38

Open-vocabulary detection and segmentation

A normal detector can only find the classes it was trained on. Open-vocabulary models find anything you can name: they compare each region of the image with the text of your query. This session builds the idea step by step: region–text scores, box losses and matching, OWL-ViT and Grounding DINO for boxes, SAM and SAM 2 for masks in images and video, and the simple trick that lets a VLM write boxes and points as text.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 164 and 166 (ViT, CLIP); session 167 (SigLIP) helps

What you will learn

  • What grounding means, and how a closed-set classifier layer becomes an open vocabulary by using text embeddings as class weights
  • How OWL-ViT scores every image token against every query (cosine, learned shift and scale, sigmoid), and why sigmoid and not softmax
  • Box losses, matching and clean-up: IoU, GIoU, L1, the Hungarian assignment used by DETR-style detectors, and greedy NMS, with numbers
  • How Grounding DINO fuses text and image early, selects 900 queries with the text, and why it beats late fusion on phrases
  • How SAM turns a click or a box into three candidate masks with a 50 ms decoder, and how SAM 2 adds a memory bank for video
  • How VLMs write boxes and points as text (PaliGemma, Qwen2-VL, Qwen2.5-VL, Kosmos-2, Molmo), and how to quantize and parse them

Why this matters

A classic object detector is trained on a fixed list of classes, for example the 80 classes of COCO. Ask it for “a forklift” or “the red cup on the left” and it has no answer. Open-vocabulary models remove the fixed list: you type what you want, and they return boxes or masks for it.

The last two sessions built the key tool. CLIP and SigLIP map an image and a sentence into the same vector space, so a dot product says how well they match (session 166, session 167). But they match the whole image with the whole sentence. This session pushes the same idea down to regions: which box, or which pixels, match which words? That link between words and places in the image is called grounding.

Three families of models do this today, and you will meet all three in real systems:

  • Open-vocabulary detectors (OWL-ViT, OWLv2, Grounding DINO, GLIP, YOLO-World): text in, boxes with scores out.
  • Promptable segmenters (SAM, SAM 2, SAM 3): a click, a box or a short phrase in, a pixel mask out, for images and for video.
  • Vision-language models (PaliGemma, Qwen2-VL, Qwen2.5-VL, Molmo): they write boxes and points as text tokens, inside a normal answer.

Production pipelines often chain them: a detector finds “every person carrying a box”, SAM 2 turns each box into a mask and tracks it through the video, and a VLM answers questions about the result. Auto-labeling tools, robot perception, video editing and content moderation are built this way. [Likely]

From a fixed class list to any text

A detector outputs a set of boxes, and for each box a class score. A box is four numbers: either corners \((x_1, y_1, x_2, y_2)\) or centre and size \((c_x, c_y, w, h)\). The class score comes from a final linear layer. If \(\mathbf{r}_i \in \R^d\) is the feature vector of region \(i\), and \(W \in \R^{C \times d}\) holds one row per class, the logits are \(W \mathbf{r}_i\). The list of classes is frozen into \(W\): there are exactly \(C\) rows, and each one was learned from boxes labeled with that class.

The open-vocabulary trick is to replace the rows of \(W\) with text embeddings. Write each class name (or any phrase) as text, run it through a text encoder such as CLIP’s, and use the resulting vector \(\mathbf{t}_j\) as the class weight. Now the “class list” is whatever text you give at test time. If the region features live in the same space as CLIP text embeddings, a region of a forklift will score high against the text “forklift”, even if no forklift box was ever in the detection training data. This is exactly CLIP zero-shot classification (session 166), applied to each region instead of to the whole image.

Words to know

  • Grounding: connecting a piece of text to a place in the image (a box, a mask, or a point). Phrase grounding gives a box for each noun phrase of a caption. Referring expression comprehension gives the one box that a phrase such as “the man in the blue shirt” describes (benchmarks: RefCOCO, RefCOCO+, RefCOCOg).
  • Open-vocabulary detection: detection where the test classes can be new. The standard test is LVIS, a dataset with about 1,200 classes split by frequency. The rare classes (seen in very few training images) are held out from box training, and the score on them, APrare, measures how well the model generalizes to classes it never saw with boxes. [Certain]
  • AP (average precision): the standard detection metric. A predicted box counts as correct if its overlap with a true box (IoU, defined below) is above a threshold; AP averages precision over recall levels, classes and (for COCO-style AP) IoU thresholds from 0.5 to 0.95.
  • Open-set or zero-shot detection: the same idea; papers use the words loosely.

The first successful open-vocabulary detectors (for example ViLD) kept a classic two-stage detector and distilled CLIP: they trained the region features to imitate the CLIP image embedding of each cropped region. Later models trained the whole detector end to end on image–text and grounding data. We will look at two of them in detail: OWL-ViT (simple, late fusion) and Grounding DINO (deep, early fusion).

Scoring a region against a text query

Take one region feature \(\mathbf{r}_i\) and one query embedding \(\mathbf{t}_j\). OWL-ViT normalizes both to length 1, takes the cosine similarity, and then applies a learned shift and scale that are predicted from the region itself:

\[ \begin{gathered} p_{ij} = \sigma\big( s_i \, (\cos(\mathbf{r}_i, \mathbf{t}_j) + b_i) \big) \\ \cos(\mathbf{r}, \mathbf{t}) = \frac{\mathbf{r}^\top \mathbf{t}}{\lVert \mathbf{r} \rVert \, \lVert \mathbf{t} \rVert} \end{gathered} \tag{168.1} \]
Read the symbols
\(\mathbf{r}_i\)
the embedding of image region (or image token) \(i\), projected into the text space
\(\mathbf{t}_j\)
the text embedding of query \(j\), for example “a photo of a cat”
\(b_i, s_i\)
a shift and a positive scale, each computed by a small linear layer from the region feature; in the Hugging Face code \(s_i = \mathrm{ELU}(\cdot) + 1\), so it is always positive [Certain]
\(\sigma\)
the sigmoid, \(\sigma(z) = 1/(1 + e^{-z})\); it turns each logit into an independent probability
\(p_{ij}\)
the probability that region \(i\) shows the thing described by query \(j\)

Two choices here matter a lot.

Sigmoid, not softmax. A softmax over the queries forces the scores of one region to add up to 1. That is wrong for detection, for two reasons. First, most regions are background and should match no query; with a softmax, a region of empty wall still gives probability 1 to some query if you only asked for one. Second, one region can match several queries at once (“dog” and “pet”). A sigmoid asks a separate yes/no question for every pair, which is the same idea as the SigLIP loss (session 167). Try both in Figure 1.

Shift and scale per region. Cosine similarities between images and text are small numbers, often 0.1 to 0.35 for CLIP-like models, and their typical level differs from region to region. [Likely] The learned \(b_i\) moves the decision point, and \(s_i\) sharpens it, so that a fixed threshold such as 0.1 or 0.3 works across images.

Worked example: one region, three queries

Say a region has cosine similarities 0.31 with “cat”, 0.12 with “dog” and 0.02 with “sofa”, and the head predicts \(b = -0.22\), \(s = 12\) for it (toy values, chosen to look like real ones).

“cat”: \(12 \times (0.31 - 0.22) = 1.08\), and \(\sigma(1.08) = 0.746\). “dog”: \(12 \times (0.12 - 0.22) = -1.2\), \(\sigma = 0.231\). “sofa”: \(12 \times (-0.20) = -2.4\), \(\sigma = 0.083\).

With a threshold of 0.5, only “cat” is kept. The three probabilities add up to 1.06, not 1: each is its own yes/no answer. With a softmax over the same three logits, the region would get 0.958 for “cat” even if the cosine for “cat” were only slightly higher than the others; and if you asked only for “sofa”, the softmax would give 1.0 to “sofa”.

Figure 1. Text queries against six candidate regions of a toy scene, scored with formula (168.1) (\(b = -0.22\), \(s = 12\) for every region). The cosine similarities are hand-made toy values, not outputs of a real model. Boxes above the threshold are drawn with their best query and score.
How to read Figure 1

Click query chips to add or remove them. With the default queries, three boxes pass 0.5: “cat”, “dog” and “red ball”. Add “pet”: the cat and the dog now match two queries each, which a sigmoid allows. Add “blue ball” and see that “red ball” does not fire on the blue ball: attributes are part of the text embedding. Now select only “giraffe” and turn on the softmax: every region becomes a giraffe with score 1.000, because a softmax over one query is always 1. With the sigmoid, nothing passes the threshold, which is the right answer.

Figure 1 hides the arithmetic. Figure 2 opens it up. It runs formula (168.1) one (region, query) pair at a time, on four regions and three queries with tiny 6-dimensional embeddings. A real model does the same with a few hundred dimensions and hundreds of regions, all at once as one matrix product; the loop here is only to let you see each number. The text embeddings are normalized once and reused for every region: this is why late fusion is cheap when you have many queries.

region embedding \(\mathbf{r}\)text embedding \(\mathbf{t}\)product \(r_d \times t_d\) (adds up to the dot product)
  1. for each query \(j\): \(\hat{\mathbf{t}}_j \leftarrow \mathbf{t}_j / \lVert \mathbf{t}_j \rVert\) // once, cached
  2. for each region \(i\):
  3. \(\hat{\mathbf{r}}_i \leftarrow \mathbf{r}_i / \lVert \mathbf{r}_i \rVert\)
  4. for each query \(j\):
  5. \(c \leftarrow \hat{\mathbf{r}}_i^\top \hat{\mathbf{t}}_j\) // cosine
  6. \(z \leftarrow s\,(c + b)\); \(p \leftarrow \sigma(z)\) // (168.1)
  7. if \(p \ge \tau\): output (box \(i\), query \(j\), \(p\))
  8. return all outputs
Figure 2. Formula (168.1) computed in the browser, pair by pair. Four regions (R1 cat, R2 dog, R3 sofa, R4 empty wall) and three queries (“cat”, “pet”, “sofa”) have hand-made 6-dimensional embeddings; one shift \(b\) and one scale \(s\) are used for all regions (a real model predicts them per region). The table fills with \(p\) and the cosine; green cells pass the threshold.
How to read Figure 2

Press Play or Step. The highlighted line of the algorithm box is the one that runs; the box next to it shows the formula with the real numbers. The bars show the two vectors and their product in each dimension; the green products add up to the dot product. With the defaults (\(\tau = 0.5\), \(b = -0.40\), \(s = 10\)), R1 and “cat” give \(\mathbf{r} \cdot \mathbf{t} = 0.930\), norms 1.020 and 0.995, cosine 0.917, \(z = 10 \times (0.917 - 0.40) = 5.165\) and \(p = 0.994\). Four pairs pass: R1 with “cat” and “pet”, R2 with “pet”, and R3 with “sofa”. The empty wall passes nothing; its best score is 0.170 for “sofa”. Lower \(\tau\) to 0.2: R2 now also passes for “cat” (\(p = 0.240\)), a false positive. Set \(s\) to 1: every \(p\) now lies between 0.349 and 0.634. The same four pairs still pass 0.5, because \(p \ge 0.5\) exactly when \(c \ge -b\), whatever \(s\) is. But the margins are tiny: at \(\tau = 0.4\), 10 of the 12 pairs pass. A large learned scale is what makes one fixed threshold safe.

Where the boxes come from, and how to train them

A score is only half of a detection. The model must also say where. Modern open-vocabulary detectors follow DETR: the model outputs a fixed set of predictions (one per image token in OWL-ViT, 900 “queries” in Grounding DINO), each with a box and class scores. During training, we must decide which prediction is responsible for which ground-truth object. To do that, and to train the boxes, we need a way to compare two boxes.

IoU and GIoU

The intersection over union of a predicted box \(\hat{B}\) and a true box \(B\) is the area they share divided by the area they cover together:

\[ \begin{aligned} \mathrm{IoU}(B, \hat{B}) &= \frac{|B \cap \hat{B}|}{|B \cup \hat{B}|} \\ &= \frac{|B \cap \hat{B}|}{|B| + |\hat{B}| - |B \cap \hat{B}|} \end{aligned} \tag{168.2} \]

IoU is 1 for a perfect box and 0 when the boxes do not touch. That second fact is a problem for training: as soon as the boxes are apart, IoU is 0 no matter how far apart they are, so its gradient is 0 and it cannot pull the prediction closer. The generalized IoU fixes this with the smallest box \(C\) that encloses both:

\[ \mathrm{GIoU} = \mathrm{IoU} - \frac{|C| - |B \cup \hat{B}|}{|C|} \tag{168.3} \]

The extra term is the fraction of the enclosing box that is empty. It grows as the boxes move apart, so GIoU keeps falling (down to −1) even when IoU is stuck at 0. GIoU equals IoU exactly when the union already fills \(C\), for example when one box contains the other. Explore it in Figure 3.

Worked example: IoU and GIoU

True box \((10, 10, 50, 50)\), predicted box \((20, 20, 60, 60)\). Each has area \(40 \times 40 = 1600\). They share \(30 \times 30 = 900\). Union \(= 1600 + 1600 - 900 = 2300\). IoU \(= 900/2300 = 0.391\). The enclosing box is \((10, 10, 60, 60)\), area 2500, of which \(2500 - 2300 = 200\) is empty. GIoU \(= 0.391 - 200/2500 = 0.311\).

Now move the prediction to \((60, 10, 100, 50)\): IoU \(= 0\), GIoU \(= -0.111\). Move it further to \((110, 10, 150, 50)\): IoU is still 0, but GIoU \(= -0.429\). GIoU knows the second box is worse; IoU does not.

true box Bpredicted boxenclosing box CIoUGIoU
Figure 3. IoU (168.2) and GIoU (168.3) for a true box of 160 × 140 and a predicted box you can move and resize. The lower plot shows both measures as the prediction slides horizontally at the current size and vertical shift; the dot marks the current position.
How to read Figure 3

Slide the prediction far to the right. In the plot, the IoU curve becomes a flat line at 0 (no gradient, nothing to learn from), while the GIoU curve keeps going down. That slope is what pulls a far-away box toward its target early in training. Then shrink the prediction until it sits inside the true box: the enclosing box equals the true box, the empty-area term is 0, and GIoU equals IoU.

Score threshold and NMS: from raw outputs to a short list

At test time the detector scores every prediction against every query: 576 predictions for OWL-ViT B/32 at 768 px, 900 for Grounding DINO. Two simple filters turn this into a short list. First, a score threshold \(\tau\) drops the predictions the model is not sure about. Second, a detector often gives several near-identical boxes for one object: slightly shifted, slightly larger. Non-maximum suppression (NMS) keeps the best box of each group and removes the others. “Non-maximum” means: every box that is not the local maximum of the score is suppressed. It uses IoU (168.2) to decide which boxes belong to the same object:

\[ \begin{gathered} \mathcal{D} = \{\, k : p_k \ge \tau \,\}, \text{ sorted by } p_k \text{, highest first} \\ k \in \mathcal{K} \iff \mathrm{IoU}(\hat{\mathbf{b}}_m, \hat{\mathbf{b}}_k) \le \lambda \ \text{ for every } m \in \mathcal{K} \text{ before } k \end{gathered} \tag{168.4} \]
Read the symbols
\(p_k, \hat{\mathbf{b}}_k\)
the score (for one query) and the box of prediction \(k\)
\(\tau\)
the score threshold; lower means more recall and more false positives
\(\mathcal{D}\)
the candidates that pass the threshold, in order of score
\(\lambda\)
the IoU threshold of NMS; 0.5 is a common default [Likely]
\(\mathcal{K}\)
the kept boxes; a box is kept only if it does not overlap too much with any box kept before it

Three details matter. NMS is greedy: only kept boxes can suppress others, and the order is the score order. It runs per query (class-aware NMS): a “cat” box should not delete a “pet” box on the same animal, unless you want one label per object. And in the worst case it computes an IoU for every pair, so its cost grows with the square of the number of candidates; this is one reason to apply \(\tau\) first. [Certain]

NMS has one weakness that Figure 4 shows: crowds. Two real objects that overlap more than \(\lambda\) look like duplicates, and the weaker one is deleted. A softer variant, Soft-NMS, lowers the score of an overlapping box instead of removing it. [Certain] DETR-style models learn not to produce duplicates (next subsection), so they often run without NMS; many practical pipelines still add it as a cheap safety step. [Likely]

kept box (current one is thick)box being comparedintersectionsuppressedwaiting or below \(\tau\)
  1. \(\mathcal{D} \leftarrow\) boxes with \(p \ge \tau\), sorted by \(p\) // score threshold
  2. \(\mathcal{K} \leftarrow \emptyset\)
  3. while \(\mathcal{D}\) is not empty:
  4. \(m \leftarrow\) first box of \(\mathcal{D}\); move \(m\) to \(\mathcal{K}\)
  5. for each box \(b\) still in \(\mathcal{D}\):
  6. \(u \leftarrow \mathrm{IoU}(m, b)\) // (168.2)
  7. if \(u \gt \lambda\): remove \(b\) from \(\mathcal{D}\) // suppressed
  8. return \(\mathcal{K}\)
Figure 4. Greedy NMS (168.4) run step by step on seven candidate boxes for the query “cat” (A to G, toy scores). A, B and D sit on the left cat; C and E sit on the right cat; F is a weak false positive on the dog; G covers only part of the left cat. Every IoU is computed live from the box corners, in pixels of the 640 × 300 scene.
How to read Figure 4

Press Play or Step and follow the highlighted line. With \(\tau = 0.3\), G (0.18) is dropped at line 1. A (0.91) is kept first. Its IoU with B is \(19880 / (24335 + 23250 - 19880) = 0.718\), above \(\lambda = 0.5\), so B is suppressed; D (0.689) is suppressed too. The right cat C has IoU 0.283 with A and survives. C then suppresses its own duplicate E (0.719). The result is A, C and F: one box per cat, plus the dog box, which only a higher \(\tau\) (0.5) removes. Now set \(\lambda = 0.2\): A suppresses C (0.283 \(\gt\) 0.2), but E has IoU 0.191 with A and stays, so the right cat keeps its worse box. This is the crowd problem. Set \(\lambda = 0.7\): D (0.689) is no longer a “duplicate” and the left cat gets two boxes.

Matching predictions to objects

Say an image has 2 objects and the model makes 3 predictions. Which prediction should learn to be the cat? DETR answers with a one-to-one matching. For every pair (object \(i\), prediction \(k\)), compute a cost that is low when the prediction has the right class and the right box:

\[ \begin{aligned} \mathcal{C}_{ik} &= -\lambda_{\text{cls}}\, \hat{p}_k(c_i) + \lambda_{L1} \lVert \mathbf{b}_i - \hat{\mathbf{b}}_k \rVert_1 \\ &\quad + \lambda_{\text{giou}} \big(1 - \mathrm{GIoU}(\mathbf{b}_i, \hat{\mathbf{b}}_k)\big) \\ \hat{\pi} &= \argmin_{\pi} \textstyle\sum_{i} \mathcal{C}_{i\, \pi(i)} \end{aligned} \tag{168.5} \]
Read the symbols
\(c_i, \mathbf{b}_i\)
the class and the box (normalized to [0, 1]) of ground-truth object \(i\)
\(\hat{p}_k(c_i)\)
prediction \(k\)’s probability for that class; in open-vocabulary models, the class is the text query of the object
\(\pi\)
an assignment: each object gets a different prediction; predictions left over are matched to “no object”
\(\lambda\)
weights; Grounding DINO uses 2.0, 5.0 and 2.0 for class, L1 and GIoU in the matching cost [Certain]

The best assignment is found with the Hungarian algorithm (in Python: linear_sum_assignment from scipy.optimize), which solves this in polynomial time. Then the loss is computed only on the matched pairs, and every unmatched prediction is trained to say “no object” for every query:

\[ \begin{aligned} \mathcal{L} = \lambda_{\text{cls}} \mathcal{L}_{\text{focal}} &+ \textstyle\sum_{i} \lambda_{L1} \lVert \mathbf{b}_i - \hat{\mathbf{b}}_{\hat\pi(i)} \rVert_1 \\ &+ \textstyle\sum_{i} \lambda_{\text{giou}} \big(1 - \mathrm{GIoU}(\mathbf{b}_i, \hat{\mathbf{b}}_{\hat\pi(i)})\big) \end{aligned} \tag{168.6} \]

The class part is a sigmoid focal loss over all (prediction, query) pairs. Almost all of those pairs are negatives (background), and plain cross-entropy would be dominated by thousands of easy negatives. The focal loss multiplies each term by \((1 - p_t)^\gamma\), where \(p_t\) is the probability given to the correct answer:

\[ \begin{gathered} \mathcal{L}_{\text{focal}} = -\sum_{k, j} \alpha_t \, (1 - p_t)^{\gamma} \log p_t \\ p_t = \begin{cases} p_{kj} & \text{if } y_{kj} = 1 \\ 1 - p_{kj} & \text{if } y_{kj} = 0 \end{cases} \end{gathered} \tag{168.7} \]

With the usual \(\gamma = 2\), an easy example with \(p_t = 0.9\) has its loss cut from 0.105 to \(0.01 \times 0.105 = 0.0011\), while a hard example with \(p_t = 0.3\) keeps most of its loss (1.204 becomes 0.590). [Certain]

Worked example: matching 3 predictions to 2 objects

Objects (normalized boxes): cat \((0.10, 0.30, 0.40, 0.80)\) and dog \((0.55, 0.40, 0.90, 0.85)\). Predictions: P0 has \(p(\text{cat}) = 0.7\), box \((0.12, 0.32, 0.42, 0.78)\); P1 has \(p(\text{dog}) = 0.6\), box \((0.50, 0.42, 0.88, 0.90)\); P2 has \(p(\text{cat}) = 0.3, p(\text{dog}) = 0.25\), box \((0.15, 0.30, 0.45, 0.70)\). With weights 1, 5 and 2 in formula (168.5):

CostP0P1P2
cat0.0927.8791.581
dog7.7700.6877.905

There are 6 possible assignments. The best is cat→P0, dog→P1, total 0.779; the next best (cat→P2, dog→P1) costs 2.267. P2 is a near-duplicate of the cat box. It is matched to “no object”, so the loss teaches it to give low scores. This is why DETR-style models do not need non-maximum suppression: duplicates are punished during training.

Algorithm 1: one training step of a DETR-style open-vocabulary detector
  1. input: image, its objects \(\{(c_i, \mathbf{b}_i)\}\), a text prompt that contains the object names plus some negative names
  2. \(\mathbf{t}_1, \dots, \mathbf{t}_Q \leftarrow\) text encoder on the prompt // one embedding per query (or per token)
  3. \(\{(\hat{\mathbf{b}}_k, \mathbf{r}_k)\}_{k=1}^{K} \leftarrow\) detector on the image (and the text, if it fuses early)
  4. \(p_{kj} \leftarrow \sigma(\text{logit}(\mathbf{r}_k, \mathbf{t}_j))\) for all \(k, j\) // formula (168.1)
  5. build cost \(\mathcal{C}_{ik}\) for all objects \(i\) and predictions \(k\) // formula (168.5)
  6. \(\hat\pi \leftarrow\) Hungarian algorithm on \(\mathcal{C}\) // one prediction per object
  7. targets \(y_{kj} \leftarrow 1\) if prediction \(k\) is matched to an object whose query is \(j\), else 0
  8. \(\mathcal{L} \leftarrow\) focal loss on all \(y_{kj}\) + L1 and GIoU on matched boxes // formulas (168.6), (168.7)
  9. backpropagate \(\mathcal{L}\) and update the weights

OWL-ViT and OWLv2: a ViT that detects

OWL-ViT (“Vision Transformer for Open-World Localization”, Minderer et al., 2022) is the simplest strong open-vocabulary detector. Start from a CLIP-style image–text model (a ViT and a text transformer trained contrastively). Then make three small changes [Certain]:

  1. Remove the final pooling of the image tower. Keep all output tokens: each token will become one prediction.
  2. On each token, add a linear class head that projects it into the text space, giving \(\mathbf{r}_i\) for formula (168.1).
  3. On each token, add a small MLP box head that predicts \((c_x, c_y, w, h)\).

Then fine-tune everything end to end with the matching loss of Algorithm 1 on detection data. The text side is unchanged: a query is embedded once, exactly as in CLIP. This is late fusion: the image and the text meet only in the final dot product.

How many predictions? One per image token. The OWL-ViT B/32 checkpoint reads 768 × 768 images with patches of 32, so it makes \((768/32)^2 = 24^2 = 576\) predictions per image. The OWLv2 B/16 checkpoint reads 960 × 960 with patches of 16: \(60^2 = 3600\) predictions. [Certain] (Values from the Hugging Face config files.)

A small but useful detail: the box head does not start from nothing. It adds a fixed bias so that, before training, every token predicts a box at its own patch position, with the size of one patch. The bias is the inverse sigmoid (logit) of that prior box, because the head’s output goes through a sigmoid. [Certain] So token \(i\) only needs to learn an offset from its own location, which makes training faster and more stable.

Image-conditioned (one-shot) detection

Since the query is only a vector in the shared space, it does not have to come from text. Take the embedding \(\mathbf{r}_k\) of a box in a different image (for example, a photo of one specific product) and use it as the query. OWL-ViT then finds similar objects in new images. This is useful when the thing is hard to name, like a particular part or logo. [Certain]

OWLv2 and self-training

Human-labeled boxes are expensive; image–text pairs from the web are cheap. OWLv2 (Minderer et al., 2023) uses an existing OWL-ViT to put pseudo-boxes on billions of web images, using words from their captions as queries, keeps the confident ones, and trains a new detector on them. This recipe, called OWL-ST, scales detection training to over 1 billion examples. With an L/14 model it raised LVIS APrare from 31.2% to 44.6%, a 43% relative gain, without any human boxes for those rare classes. [Certain]

Grounding DINO: fuse the text early

OWL-ViT computes image features without looking at the text. That is fast (encode the image once, then try any number of queries), but the image features cannot adapt to what you asked. Grounding DINO (Liu et al., 2023) takes the opposite path: it mixes text and image inside the detector, at three places [Certain]:

  1. Feature enhancer (6 layers). Image features (from a Swin transformer) and text features (from BERT) attend to each other with cross-attention, in both directions.
  2. Language-guided query selection. The detector decoder starts from \(N_q = 900\) queries. Instead of fixed learned queries, it picks the 900 image positions that best match any text token:
\[ \begin{gathered} \mathbf{s} = \max_{j}\, \big( X_I X_T^\top \big)_{:,\, j} \\ \mathcal{I} = \operatorname{TopK}_{N_q}(\mathbf{s}) \end{gathered} \tag{168.8} \]
Read the symbols
\(X_I \in \R^{N_I \times d}\)
the image features after the enhancer, one row per position (all scales together)
\(X_T \in \R^{N_T \times d}\)
the text features, one row per token; at most 256 tokens [Certain]
\(\max_j\)
for each image position, its best match over all text tokens
\(\mathcal{I}\)
the indices of the \(N_q = 900\) best image positions; they initialize the decoder queries
  1. Cross-modality decoder (6 layers). Each query attends to the image features and to the text features, then predicts a box. Its class logits are dot products with every text token, so a 900-query model with a 256-token prompt produces \(900 \times 256 = 230{,}400\) logits per image.

The prompt is one string with all the names, for example cat . remote control . sofa . (the public code uses “ . ” as the separator [Likely]). A naive text encoder would let “cat” attend to “sofa”, which mixes unrelated names. Grounding DINO adds attention masks so that tokens only attend within their own phrase. The paper calls this a sub-sentence level text representation. [Certain]

Results: 52.5 AP on COCO with no COCO training data (zero-shot), and a mean of 26.1 AP on the ODinW benchmark of 35 real-world datasets, zero-shot. [Certain] Early fusion makes it good at long referring phrases (“the second cup from the left”), at the cost of running the whole detector again for every new prompt.

Two related models are covered in session 174. GLIP (2021) first turned detection into phrase grounding: it computes region–word alignment scores \(S = O P^\top\) between region features \(O\) and word features \(P\), and trains on detection, grounding and caption data together. YOLO-World (2024) brings the idea to a real-time YOLO detector: 35.4 AP on LVIS at 52.0 FPS on a V100 GPU. [Certain] Its trick for speed is that a fixed vocabulary can be encoded once and folded into the weights (“prompt, then detect”).

image pathtext or prompt pathfusion and outputs
Figure 5. Three designs. OWL-ViT fuses late: text meets image only in the final dot product, so the image is encoded once for any number of queries. Grounding DINO fuses early, in the encoder, in query selection and in the decoder. SAM is not a text model: a heavy encoder runs once per image, and a light decoder turns each prompt (point, box, mask) into masks.

SAM: promptable segmentation

A box is coarse. Many tasks need the exact pixels: cutting an object out of a photo, measuring an area, editing a video. Segment Anything (SAM, Kirillov et al., 2023) defines a new task, promptable segmentation: given an image and a prompt (one or more points, a box, or a rough mask), return a valid mask for the thing the prompt points at. SAM does not use class names at all. It answers “which object is here?”, not “where is the cat?”.

Architecture: heavy once, light many times

SAM has three parts, sized for interactive use [Certain]:

  • Image encoder. A ViT-H/16 pretrained with MAE (session 165), on a 1024 × 1024 input. It outputs a \(64 \times 64\) grid of 256-dim features (\(1024/16 = 64\)). This is the expensive part, and it runs once per image.
  • Prompt encoder. Points and boxes become positional encodings plus a learned “positive point”, “negative point” or “box corner” embedding. A mask prompt is downsampled by convolutions and added to the image features.
  • Mask decoder. Two transformer layers with 256-dim tokens. The prompt tokens and a few learned output tokens attend to the image features, and the image features attend back (two-way attention). Then the image features are upsampled 4× by two transposed convolutions, to \(256 \times 256\) with 32 channels.

Each mask is a dot product between the upsampled features and one output token, passed through a small MLP:

\[ \begin{gathered} M_k(x, y) = \mathbf{w}_k^\top \, E_{\uparrow}(x, y) \\ \mathbf{w}_k = \mathrm{MLP}_k(\mathbf{o}_k) \in \R^{32} \end{gathered} \tag{168.9} \]
Read the symbols
\(E_{\uparrow}(x, y)\)
the upsampled image feature at pixel \((x, y)\) of the 256 × 256 output grid, 32 numbers
\(\mathbf{o}_k\)
output token \(k\) after the two decoder layers
\(\mathbf{w}_k\)
a “dynamic classifier” for this prompt: a weight vector that says which features mean “inside the mask”
\(M_k(x, y)\)
the mask logit; \(M_k \gt 0\) means inside. The 256 × 256 mask is upsampled to the image size at the end.

This is the same pattern as formula (168.1): a vector from the prompt side, dotted with a vector from the image side. Here it is done for each of the \(256 \times 256 = 65{,}536\) output pixels, which costs only \(65{,}536 \times 32 \approx 2.1\) million multiply-adds per mask. With the image embedding already computed, the prompt encoder and mask decoder run in about 50 ms on a CPU in a web browser, and the decoder needs less than 1% of the encoder’s compute. [Certain] That is what makes clicking feel instant.

Ambiguity: three masks for one click

One click on a shirt logo could mean the logo, the shirt, or the whole person. All three are valid. If the model had to output one mask, training would push it toward an average of the three, which is no good mask at all. SAM outputs 3 masks for a single prompt (roughly: sub-part, part, whole object) and trains with only the best of them [Certain]:

\[ \begin{aligned} \mathcal{L}_{\text{SAM}} = \min_{k \in \{1,2,3\}} \Big[ & 20\, \mathcal{L}_{\text{focal}}(M_k, M^\ast) \\ & + \mathcal{L}_{\text{dice}}(M_k, M^\ast) \Big] + \mathcal{L}_{\text{IoU}} \end{aligned} \tag{168.10} \]

Here \(M^\ast\) is the true mask, the focal-to-dice ratio is 20:1, and \(\mathcal{L}_{\text{IoU}}\) is a mean-squared error that trains a small head to predict the IoU of each output mask with the truth. [Certain] At test time this predicted IoU ranks the three masks. When the prompt has several points and is no longer ambiguous, one mask is enough. The dice loss measures overlap directly, so it is not swamped by the many background pixels:

\[ \mathcal{L}_{\text{dice}} = 1 - \frac{2 \sum_{x} p_x m_x}{\sum_x p_x + \sum_x m_x} \tag{168.11} \]

where \(p_x = \sigma(M(x))\) is the predicted probability at pixel \(x\) and \(m_x \in \{0, 1\}\) the true mask. For example, with predictions \((0.9, 0.8, 0.3, 0.1)\) on pixels whose truth is \((1, 1, 1, 0)\): the overlap is \(0.9 + 0.8 + 0.3 = 2.0\), the sums are 2.1 and 3, so the dice loss is \(1 - 4/5.1 = 0.216\).

Figure 6. A toy version of SAM’s behavior on a 40 × 26 grid image. Each pixel belongs to up to three nested regions (for example logo ⊂ mug body ⊂ whole mug). One click gives up to three candidate masks; more clicks give a single mask. The regions are hand-made; a real SAM computes the masks with formula (168.9).
How to read Figure 6

The start state is one positive click on the logo. You get three candidates: the logo, the mug body, and the whole mug with its handle. Click a thumbnail to choose (a real SAM ranks them by predicted IoU). Now press Reset, click the mug body away from the logo, then click the handle: only the whole mug contains both points, so one mask is returned. Switch to “Negative click” and click the logo: the logo is cut out of the mask. Negative points are how users fix a mask that is too big. Click the mug and a leaf: no region holds both, so you get the union of two objects.

The data engine and SA-1B

No dataset with a billion masks existed, so the SAM team built one with the model in the loop. First, annotators corrected SAM’s masks by clicking (assisted manual stage). Then SAM proposed masks and annotators added the missing ones (semi-automatic). Finally SAM ran alone on a grid of points over each image (fully automatic). The result, SA-1B, has 1.1 billion masks on 11 million licensed images: about 100 masks per image. [Certain] During training, each mask is used in 11 rounds of simulated interaction, where new clicks are sampled in the error regions of the previous prediction. [Certain]

SAM 2: segmentation in video with a memory

In a video, you want to click on an object once and get its mask in every frame, even when it turns, gets smaller, or is hidden for a while. Running SAM on every frame independently cannot do this: SAM has no idea which object you meant in frame 1. SAM 2 (Ravi et al., 2024) adds a streaming memory [Certain]. Think of it like this:

The parts are:

  • An image encoder (Hiera, a hierarchical ViT pretrained with MAE) runs once per frame, as frames arrive.
  • Memory attention (4 transformer layers) conditions the current frame’s features on the past: self-attention, then cross-attention to a memory bank, then an MLP.
  • The memory bank holds spatial memories of up to \(N\) recent frames (\(N = 6\) by default) and of the frames where the user gave prompts, plus object pointers: compact vectors (from the mask decoder’s output tokens) that summarize the object.
  • After the mask decoder predicts the mask of the current frame, a memory encoder fuses that mask with the frame’s features and writes a new memory into the bank (first in, first out for the recent part).
  • An occlusion head predicts whether the object is visible in the current frame at all, so the model can output “no mask” while the object is hidden, and pick it up again later.
\[ \begin{aligned} F_t' &= \mathrm{MemAttn}\big(F_t;\ \mathcal{S}_t\big) \\ \mathcal{S}_t &= \{\mathcal{M}_{p}\}_{p \in \mathcal{P}} \cup \{\mathcal{M}_{t-1}, \dots, \mathcal{M}_{t-N}\} \\ &\quad \cup \{\text{pointers}\} \end{aligned} \tag{168.12} \]
Read the symbols
\(F_t\)
the image features of frame \(t\) (from the encoder)
\(\mathcal{M}_s\)
the memory of frame \(s\): its features fused with its predicted mask
\(\mathcal{P}\)
the set of frames where the user clicked; these memories are kept, they do not drop out of the bank
\(F_t'\)
features that now “know” which object we are following; the SAM-style mask decoder reads them

On a single image the memory is empty, and SAM 2 is just a better, faster SAM: 130.1 FPS against SAM’s 21.7 on an A100, about 6 times faster. [Certain] For video, the paper reports better accuracy with 3 times fewer user interactions than earlier methods. The Hiera-B+ model runs at 43.8 FPS and Hiera-L at 30.2 FPS on one A100. [Certain] The training data, SA-V, has 50.9K videos with 642.6K masklets (object tracks), 35.5M masks in total; about 55 masks per masklet. [Certain]

prompted frame (kept)recent frame memory (FIFO, \(N = 6\))occluded frame
Figure 7. SAM 2 processing a 12-frame toy video. The user clicks the ball in frame 1. The ball goes behind a pillar in frames 6 and 7, and the user adds a correction click in frame 9. The memory bank always holds the prompted frames plus the 6 most recent frames.
How to read Figure 7

Step through the frames. Until frame 7, the recent part of the bank fills up. At frame 8, frame 1 is older than 6 frames, but it stays because it was prompted. During the occlusion the occlusion head says “not visible” and the output mask is empty; those memories are marked as occluded. When the ball comes back in frame 8, the pointer and the prompted memory let the model find the same ball again. At frame 9 the correction click adds frame 9 to the kept set.

SAM 3: back to words

SAM 3 (Carion et al., 2025) closes the loop with open-vocabulary detection. Its task is promptable concept segmentation: give a short noun phrase such as “yellow school bus”, an image example, or both, and it returns masks and identities for every matching instance, in images and in video. It adds a detector that shares the backbone with the SAM 2 style tracker, and a presence head that first decides whether the concept is in the image at all, separately from where it is. [Certain] The authors report about twice the accuracy of earlier systems on this task. [Certain]

Algorithm 2: “find, segment and track every X” with a detector and SAM 2
  1. input: video frames \(I_1, \dots, I_T\), text query “X”, score threshold \(\tau\)
  2. \(\{(\hat{\mathbf{b}}_k, p_k)\} \leftarrow\) open-vocabulary detector on \(I_1\) with query “X” // OWLv2 or Grounding DINO
  3. keep boxes with \(p_k \gt \tau\); remove duplicates with NMS if the detector needs it
  4. for each kept box \(k\): add \(\hat{\mathbf{b}}_k\) as a box prompt for object \(k\) on frame 1 of SAM 2
  5. for \(t = 1, \dots, T\):
  6.   \(F_t \leftarrow\) image encoder on \(I_t\) // once per frame, shared by all objects
  7.   for each object \(k\): \(F'_{t,k} \leftarrow\) memory attention (168.12); mask \(\leftarrow\) decoder; write memory
  8.   every few frames, run the detector again to start tracks for new objects // objects can enter later
  9. return one mask track (masklet) per object

How VLMs output boxes and points as text

A VLM only produces tokens. To output a box, it must write the box as text. There are two main ways, and both start the same way: turn each coordinate into an integer in a fixed range. With \(K\) bins and image size \(S\) (width for \(x\), height for \(y\)):

\[ \begin{gathered} k = \min\!\Big(K - 1,\ \Big\lfloor \frac{x}{S}\, K \Big\rfloor\Big) \\ \hat{x} = \frac{k + 0.5}{K}\, S, \qquad |x - \hat{x}| \le \frac{S}{2K} \end{gathered} \tag{168.13} \]

Then either (a) add \(K\) new special tokens to the vocabulary, one per bin, so a box costs exactly 4 tokens; or (b) write the integers as plain digits with ordinary tokens. Normalizing by \(S\) makes the numbers independent of the image size; using absolute pixels avoids the normalization but ties the numbers to the resolution the model saw. Real systems use different choices [Certain]:

ModelHow a box or point is writtenRange
Pix2Seq (2021)Each coordinate is a quantized token; a detection is a token sequencebins
Kosmos-2Image cut into a 32 × 32 grid: 1,024 location tokens. A box is the top-left and bottom-right cell: <p> a campfire </p><box><loc4><loc1007></box>32 per axis
PaliGemma1,024 tokens <loc0000> … <loc1023>, order \(y_{\min}, x_{\min}, y_{\max}, x_{\max}\), then the label. Prompt: detect cat ; dog. Masks use 128 extra <seg…> tokens from a VQ-VAE1,024 per axis
Qwen2-VLPlain digits, normalized to [0, 1000): <|object_ref_start|>the eyes on a giraffe<|object_ref_end|><|box_start|>(176,106),(232,160)<|box_end|>[0, 1000)
Qwen2.5-VLAbsolute pixel coordinates of the input image, often inside JSONpixels
MolmoPoints as XML-like text in percent: <point x="10.0" y="10.0" alt="…">…</point>; counting by pointing at each object0 to 100

The model learns these formats from training data that was converted from detection and grounding datasets. Nothing in the architecture “knows” about boxes: the coordinates are just tokens that the language model learns to predict with cross-entropy, one after another (session 191).

Worked example: one box, four formats

Image 640 × 480 pixels; box \((x_1, y_1, x_2, y_2) = (100, 50, 300, 400)\).

PaliGemma (\(K = 1024\), \(y\) first): \(\lfloor 50/480 \cdot 1024 \rfloor = 106\), \(\lfloor 100/640 \cdot 1024 \rfloor = 160\), \(\lfloor 400/480 \cdot 1024 \rfloor = 853\), \(\lfloor 300/640 \cdot 1024 \rfloor = 480\). Output: <loc0106><loc0160><loc0853><loc0480> dog. Decoding at bin centres gives \((100.3, 49.9, 300.3, 400.1)\): the error bound (168.13) is \(640/2048 = 0.31\) px in \(x\).

Qwen2-VL ([0, 1000)): \(100/640 \cdot 1000 = 156.25 \to 156\), \(50/480 \cdot 1000 = 104.2 \to 104\), \(468.75 \to 468\), \(833.3 \to 833\). Output: (156,104),(468,833).

Kosmos-2 (32 × 32 grid): the top-left corner falls in row 3, column 5; the bottom-right in row 26, column 15. Each cell is 20 × 15 px, so corners can be off by up to 10 px in \(x\) and 7.5 px in \(y\). Coarse, but only 2 tokens per box.

Molmo (a point at the box centre, in percent): \(x = 200/640 = 31.25\%\), \(y = 225/480 = 46.875\%\).

Figure 8. A box on a 640 × 480 image written as text in four formats (formula 168.13), then parsed back. The solid box is the true box; the dashed box is what you get by decoding the text at bin centres (for Qwen2-VL, decoding uses \(k \cdot S / 1000\) as is common in practice [Likely]; Qwen2.5-VL is shown in a JSON form with a bbox_2d key, a common output style [Likely]). The grid lines show the Kosmos-2 cells.
How to read Figure 8

With PaliGemma and Qwen2-VL the decoded box sits on top of the true box: the error is below one pixel, much smaller than the error of the model itself. Switch to Kosmos-2 and move a slider slowly: the decoded corner jumps in steps of 20 px (x) or 15 px (y), one grid cell at a time. Notice also the cost: 4 special tokens for PaliGemma and 2 for Kosmos-2, but about 19 characters of digits and punctuation for the Qwen2-VL box, which is usually more than a dozen tokens because many tokenizers split numbers into single digits. [Likely]

The PyTorch version

The code below is small and deterministic (runs on CPU in a second). It builds an OWL-ViT-style head (formula 168.1 with the box prior), IoU and GIoU (168.2, 168.3), the matching (168.5) on the worked example, the detection loss (168.6, 168.7), a SAM-style mask decoder output with the min-over-3 loss (168.9 to 168.11), and the box-to-text formats (168.13). The printed values match the worked examples above.

import itertools, re
import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) An OWL-ViT-style open-vocabulary head: one box and one embedding per image token
class OpenVocabHead(nn.Module):
    def __init__(self, d_img=768, d_emb=512, grid=24):
        super().__init__()
        self.cls_proj = nn.Linear(d_img, d_emb)          # token -> joint image-text space
        self.shift, self.scale = nn.Linear(d_img, 1), nn.Linear(d_img, 1)
        self.box_mlp = nn.Sequential(nn.Linear(d_img, d_img), nn.GELU(), nn.Linear(d_img, 4))
        ys, xs = torch.meshgrid(torch.arange(grid), torch.arange(grid), indexing="ij")
        ctr = torch.stack([(xs + 0.5) / grid, (ys + 0.5) / grid], -1).reshape(-1, 2)
        prior = torch.cat([ctr, torch.full_like(ctr, 1 / grid)], -1)   # each token starts at its own patch
        self.register_buffer("box_bias", torch.logit(prior))
    def forward(self, tokens, text_emb):                 # tokens (N, d_img), text_emb (Q, d_emb)
        r = F.normalize(self.cls_proj(tokens), dim=-1)
        t = F.normalize(text_emb, dim=-1)
        scale = F.elu(self.scale(tokens)) + 1            # always > 0
        logits = (r @ t.T + self.shift(tokens)) * scale  # formula (168.1), before the sigmoid
        boxes = torch.sigmoid(self.box_mlp(tokens) + self.box_bias)   # (cx, cy, w, h) in [0, 1]
        return logits, boxes

head = OpenVocabHead()
tokens, text = torch.randn(576, 768), torch.randn(3, 512)   # 24 x 24 tokens; 3 queries, e.g. "cat", "dog", "sofa"
logits, boxes = head(tokens, text)
print(logits.shape, boxes.shape)                     # torch.Size([576, 3]) torch.Size([576, 4])

# 2) Boxes: IoU and GIoU, formulas (168.2) and (168.3)
def xyxy(b):
    cx, cy, w, h = b.unbind(-1)
    return torch.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], -1)
def iou_giou(a, b):                                  # a (n, 4), b (m, 4), xyxy -> (n, m) each
    area = lambda x: (x[:, 2] - x[:, 0]) * (x[:, 3] - x[:, 1])
    lt, rb = torch.max(a[:, None, :2], b[None, :, :2]), torch.min(a[:, None, 2:], b[None, :, 2:])
    inter = (rb - lt).clamp(min=0).prod(-1)
    union = area(a)[:, None] + area(b)[None, :] - inter
    lt_c, rb_c = torch.min(a[:, None, :2], b[None, :, :2]), torch.max(a[:, None, 2:], b[None, :, 2:])
    hull = (rb_c - lt_c).prod(-1)
    iou = inter / union
    return iou, iou - (hull - union) / hull
gt = torch.tensor([[10., 10, 50, 50]]); pr = torch.tensor([[20., 20, 60, 60], [60., 10, 100, 50]])
iou, giou = iou_giou(gt, pr)
print(iou.round(decimals=3), giou.round(decimals=3)) # [[0.391, 0.]] [[0.311, -0.111]]

# 3) Matching, formula (168.5): brute force here; real code uses scipy linear_sum_assignment
def match(prob, pboxes, labels, gboxes, w_cls=1.0, w_l1=5.0, w_giou=2.0):
    _, giou = iou_giou(gboxes, pboxes)
    cost = (-w_cls * prob[:, labels].T + w_l1 * torch.cdist(gboxes, pboxes, p=1) + w_giou * (1 - giou))
    G, P = cost.shape
    best = min(itertools.permutations(range(P), G), key=lambda pm: sum(cost[i, pm[i]] for i in range(G)))
    return list(best), cost
prob = torch.tensor([[0.70, 0.10], [0.20, 0.60], [0.30, 0.25]])         # 3 predictions x 2 classes
pboxes = torch.tensor([[.12, .32, .42, .78], [.50, .42, .88, .90], [.15, .30, .45, .70]])
gboxes = torch.tensor([[.10, .30, .40, .80], [.55, .40, .90, .85]]); labels = torch.tensor([0, 1])
assign, cost = match(prob, pboxes, labels, gboxes)
print(assign, cost.round(decimals=3))                # [0, 1]; prediction 2 is matched to "no object"

# 4) Detection loss on the matched pairs: sigmoid focal + L1 + GIoU, formula (168.6)
def sigmoid_focal(logit, target, alpha=0.25, gamma=2.0):
    p = torch.sigmoid(logit)
    ce = F.binary_cross_entropy_with_logits(logit, target, reduction="none")
    p_t = p * target + (1 - p) * (1 - target)
    a_t = alpha * target + (1 - alpha) * (1 - target)
    return (a_t * (1 - p_t) ** gamma * ce).sum()
target = torch.zeros_like(prob); target[assign, labels] = 1.0           # unmatched rows stay all-zero
mp, mg = pboxes[assign], gboxes
loss = (sigmoid_focal(torch.logit(prob), target) / len(labels)
        + 5.0 * (mp - mg).abs().sum(-1).mean() + 2.0 * (1 - iou_giou(mg, mp)[1].diag()).mean())
print(round(loss.item(), 4))

# 5) A SAM-style mask: dot product of upsampled image features with a mask token, formula (168.8)
feat = torch.randn(32, 256, 256)                     # upsampled image embedding (C=32, 256 x 256)
mask_tokens = torch.randn(3, 32)                     # 3 output tokens after the decoder MLPs
mask_logits = torch.einsum("kc,chw->khw", mask_tokens, feat)            # (3, 256, 256)
gt_mask = (torch.rand(256, 256) > 0.5).float()
def dice_loss(logit, t):
    p = torch.sigmoid(logit).flatten(1); t = t.flatten()[None]
    return 1 - (2 * (p * t).sum(-1) + 1) / (p.sum(-1) + t.sum(-1) + 1)
per_mask = 20 * torch.stack([sigmoid_focal(m, gt_mask) / gt_mask.numel() for m in mask_logits]) + dice_loss(mask_logits, gt_mask)
print(mask_logits.shape, per_mask.argmin().item())   # only the best of the 3 masks gets the gradient, formula (168.10)

# 6) Boxes as text, formula (168.12): PaliGemma-style and Qwen2-VL-style
def q(v, size, k): return min(k - 1, int(v / size * k))
def to_paligemma(box, W, H, label):
    x1, y1, x2, y2 = box
    return "".join(f"<loc{q(v, s, 1024):04d}>" for v, s in [(y1, H), (x1, W), (y2, H), (x2, W)]) + " " + label
def to_qwen2vl(box, W, H, label):
    x1, y1, x2, y2 = box
    return (f"<|object_ref_start|>{label}<|object_ref_end|><|box_start|>"
            f"({q(x1, W, 1000)},{q(y1, H, 1000)}),({q(x2, W, 1000)},{q(y2, H, 1000)})<|box_end|>")
def parse_paligemma(s, W, H):
    ys1, xs1, ys2, xs2 = [(int(k) + 0.5) / 1024 for k in re.findall(r"<loc(\d{4})>", s)[:4]]
    return [round(v, 1) for v in (xs1 * W, ys1 * H, xs2 * W, ys2 * H)]
s = to_paligemma((100, 50, 300, 400), 640, 480, "dog"); print(s)        # <loc0106><loc0160><loc0853><loc0480> dog
print(parse_paligemma(s, 640, 480))                  # [100.3, 49.9, 300.3, 400.1]
print(to_qwen2vl((100, 50, 300, 400), 640, 480, "dog"))                 # ...(156,104),(468,833)...

Where this is used

ApplicationTypical toolWhy this tool
Auto-labeling a new detection datasetGrounding DINO or OWLv2 for boxes, then SAM for masks, then human reviewNo training needed for a new class; people only fix errors
Detect a rare or custom object (a product, a defect)OWL-ViT image-conditioned query, or a fine-tuned open-vocabulary detectorThe thing is hard to name; one example image is enough to query
Real-time detection with a fixed custom vocabulary (robots, cameras)YOLO-World with the vocabulary folded into the weightsText encoding is removed from the runtime path
Photo and video editing (cut-out, background removal, effects)SAM for clicks on images; SAM 2 to propagate the mask through the videoInteractive: 50 ms per prompt after one encoder pass
Video object segmentation and trackingSAM 2 with a box or click on the first frame (session 204)Memory bank handles appearance change and short occlusions
“Find every moment a red car appears” in long videoOpen-vocabulary detector on sampled frames, then tracking; feeds key-moment detection (session 207)Text query at search time, no retraining
Video-RAG and visual question answering with evidenceA VLM that returns boxes as text (Qwen2.5-VL), or detector crops passed to a VLM (session 210)Answers can point at the evidence; crops raise effective resolution
Counting (“how many people in the queue?”)Molmo-style pointing, or detection plus countingPointing at each item is more reliable than guessing a number [Likely]
GUI agents and document AIVLMs that output click points or element boxes as textThe agent must say where to click (session 154)

Common mistakes

  • Using a softmax over the queries. A region of background then always “matches” something. Use the model’s sigmoid scores and a threshold, as in Figure 1.
  • Mixing box formats. \((c_x, c_y, w, h)\) vs \((x_1, y_1, x_2, y_2)\), normalized vs pixels, and PaliGemma’s \(y\)-first order. Write one conversion function and test it on a known box, like the worked example.
  • Drawing boxes on the wrong image size. The model saw a resized or padded image. OWL-ViT and OWLv2 pad to a square; their normalized boxes refer to the padded square, not to the original rectangle. [Likely]
  • Bad prompts for Grounding DINO. Separate names with “ . ”, keep the total under 256 tokens, and lower-case the text. Long lists of class names split across several calls change the scores, because the names attend inside the model. [Likely]
  • One global threshold for all queries. Scores are not calibrated across queries: “person” may need 0.3 and “screwdriver” 0.15. Tune per class on a few labeled images.
  • Expecting SAM to know classes. SAM masks have no labels and a single click is ambiguous. Use box prompts from a detector, or several points, and check the predicted IoU.
  • Running SAM 2 per frame without memory (for example, calling the image predictor in a loop). You lose identity and occlusion handling. Use the video predictor, which keeps the memory bank.
  • Re-encoding the image for every prompt. SAM’s encoder is the expensive part; cache the image embedding and run only the decoder per click.

Interview questions

How can a detector recognize a class it never saw boxes for?basic
Replace the fixed classification layer with text embeddings. Each region feature is projected into the space of a text encoder that was trained contrastively on image–text pairs (CLIP-style), and the class score is the similarity between the region and the embedding of the class name. Knowledge of what “forklift” looks like comes from the image–text pretraining, not from detection labels. The box head is class-agnostic, so it generalizes to new objects too. Evaluation: AP on LVIS rare classes held out from box training.
Why do open-vocabulary detectors use a sigmoid per query instead of a softmax over queries?core
A softmax forces each region’s scores to sum to 1 over the queries you gave. Background regions then still give high probability to some query, and the score of one query depends on which other queries you asked for. A region can also match several queries at once (“dog”, “pet”). A sigmoid gives an independent yes/no probability per (region, query) pair, so “nothing here” is possible and scores do not change when you add queries (for late-fusion models like OWL-ViT). Training uses a sigmoid focal loss.
What is GIoU and why is it used as a loss instead of IoU?core
GIoU = IoU − (|C| − |union|)/|C|, where C is the smallest box enclosing both. IoU is 0 for any non-overlapping pair, so its gradient is 0 and it cannot pull a distant prediction closer. GIoU keeps decreasing as boxes move apart (down to −1), so it gives a useful gradient everywhere, and it is scale-invariant unlike L1. DETR-style detectors combine L1 and GIoU. Example: boxes (10,10,50,50) and (20,20,60,60): IoU 0.391, GIoU 0.311.
Explain Hungarian matching in DETR-style detectors. Why does it remove the need for NMS?core
The model outputs a fixed set of K predictions. For each ground-truth object and each prediction, a cost combines class probability, L1 box distance and GIoU. The Hungarian algorithm finds the one-to-one assignment with the lowest total cost; unmatched predictions are trained to predict “no object”. Because only one prediction is rewarded per object, duplicates are explicitly pushed to low scores during training, so the model learns not to produce them and NMS is not needed (in practice some pipelines still apply it for safety).
Compare OWL-ViT and Grounding DINO. When would you pick each?advanced
OWL-ViT fuses late: the image is encoded once into one box and one embedding per token, and any number of text (or image) queries are scored with a dot product. This is cheap for many queries and supports image-conditioned queries. Grounding DINO fuses early: cross-attention between text and image in the encoder, language-guided query selection (top 900 positions by max similarity to any text token) and a cross-modality decoder. It is stronger on long referring expressions and gets 52.5 zero-shot COCO AP, but must rerun the detector for each new prompt, and its prompt is limited to 256 tokens. Pick OWL-style for large vocabularies or image queries, Grounding DINO for phrase grounding and auto-labeling quality.
How does SAM handle an ambiguous point prompt?core
It predicts three masks from three output tokens (roughly sub-part, part, whole) and, during training, backpropagates only the loss of the best one (minimum over the three). A small head predicts each mask’s IoU with the truth, trained with MSE, and at test time the predicted IoU ranks the masks. With several points or a box the prompt is less ambiguous and a single mask output is used. Without this, the model would learn to average valid answers into a blurry mask.
Why is SAM fast enough to feel interactive?basic
The cost is split. The heavy ViT-H image encoder runs once per image and produces a 64 × 64 × 256 embedding. Each prompt only runs the prompt encoder and a two-layer mask decoder, which needs under 1% of the encoder compute: about 50 ms on a CPU in a browser. Masks are dot products between a 32-dim prompt-specific vector and 256 × 256 upsampled features, about 2 million multiply-adds per mask.
How does SAM 2 keep track of an object through a video, including occlusion?advanced
Each frame’s features pass through memory attention (4 layers) that cross-attends to a memory bank: spatial memories of the prompted frames (always kept) and of the \(N\) most recent frames (FIFO, \(N = 6\) by default), plus object pointer vectors. After decoding the mask, a memory encoder fuses the mask with the frame features and writes a new memory. An occlusion head predicts whether the object is visible; when it is not, the output is empty and the memory is marked occluded, and the prompted memory and pointers let the model re-find the same object later. On single images the memory is empty and it behaves like an improved SAM, about 6× faster.
How does a VLM output a bounding box? Compare special location tokens with plain digits.core
The box is written as text and learned with ordinary next-token cross-entropy. Coordinates are quantized into \(K\) bins. Special tokens (PaliGemma: 1,024 <loc> tokens, y first; Kosmos-2: 32 × 32 grid cells) cost exactly 4 (or 2) tokens per box and have a clear one-token-per-value meaning, but need new embeddings. Plain digits (Qwen2-VL normalized to [0, 1000); Qwen2.5-VL absolute pixels) reuse the existing vocabulary and read naturally in JSON, but take more tokens and the model must learn number formatting. Quantization error is at most \(S/(2K)\): 0.31 px for 640 px and \(K = 1024\).
Design a system that finds and segments every “person wearing a helmet” in a factory video stream.advanced
Decode frames at a low rate (for example 2–5 FPS). Run an open-vocabulary detector (Grounding DINO for the phrase, or OWLv2 / YOLO-World with “person” and “helmet” and a box-in-box rule) on keyframes. Tune per-class thresholds on a small labeled set. Feed kept boxes as prompts to SAM 2’s video predictor to get masklets and identities across frames, caching the image encoder output per frame. Re-run the detector every few frames to start new tracks and to drop lost ones. Evaluate on labeled clips with AP for detection and J&F for masks; watch latency per stream on the target GPU. Optionally ask a VLM to verify hard cases with crops.
What is OWLv2’s self-training and why does it help on rare classes?advanced
An existing detector labels web image–text pairs with pseudo-boxes, using words from the captions as queries, and the confident ones are kept. A new detector is trained on this much larger set (over 1 billion examples), then optionally fine-tuned on human labels. Rare classes appear many more times in web data than in labeled detection sets, so the detector sees boxes for them without human annotation. With L/14, LVIS AP-rare went from 31.2% to 44.6%. Risks: confirmation bias from the teacher’s errors, which filtering and a large label space reduce.

Exercises

  1. A region has cosine similarities 0.26 with “pet” and 0.18 with “red ball”, with \(b = -0.22\) and \(s = 12\). Compute both probabilities with formula (168.1). Which pass a threshold of 0.5?
  2. Compute IoU and GIoU for the true box \((0, 0, 10, 10)\) and the prediction \((5, 0, 15, 10)\).
  3. A prediction box lies completely inside the true box. Show that GIoU equals IoU.
  4. How many predictions does an OWL-ViT-style model make on a 1008 × 1008 input with patch 14? How many class logits for 20 queries?
  5. Compute the dice loss for predictions \((1, 1, 0, 0)\) against the truth \((1, 0, 1, 0)\).
  6. Write the PaliGemma tokens for the box \((x_1, y_1, x_2, y_2) = (320, 240, 640, 480)\) on a 640 × 480 image.
  7. A SAM 2 user clicks frames 1 and 20 of a 30-frame video. With \(N = 6\), which memories are in the bank when frame 25 is processed?
  8. Run NMS (168.4) with \(\lambda = 0.5\) on A \((0, 0, 10, 10)\) with score 0.9, B \((1, 0, 11, 10)\) with 0.8, and C \((6, 0, 16, 10)\) with 0.7. Which boxes are kept? What changes with \(\lambda = 0.2\)?
Answerscheck
1. “pet”: \(12 \times 0.04 = 0.48\), \(\sigma = 0.618\), passes. “red ball”: \(12 \times (-0.04) = -0.48\), \(\sigma = 0.382\), does not. 2. Intersection \(5 \times 10 = 50\), union \(100 + 100 - 50 = 150\), IoU \(= 0.333\). Enclosing box \((0, 0, 15, 10)\), area 150 = union, so GIoU \(= 0.333\) too. 3. The enclosing box of a box inside another is the outer box, which is the union, so \(|C| - |B \cup \hat B| = 0\). 4. \((1008/14)^2 = 72^2 = 5184\) predictions, and \(5184 \times 20 = 103{,}680\) logits. 5. Overlap 1, sums 2 and 2: \(1 - 2/4 = 0.5\). 6. \(y_1\): \(\lfloor 240/480 \cdot 1024 \rfloor = 512\); \(x_1\): 512; \(y_2\): \(\lfloor 1024 \rfloor = 1024\) is clipped to 1023; \(x_2\): 1023. Tokens: <loc0512><loc0512><loc1023><loc1023>. 7. The prompted frames 1 and 20 (always kept) and the recent frames 19 to 24; frame 20 is both prompted and recent. 8. A is kept first. IoU(A, B) \(= 90 / (100 + 100 - 90) = 0.818 \gt 0.5\): B is suppressed. IoU(A, C) \(= 40 / 160 = 0.25 \le 0.5\): C is kept. Result: A and C. With \(\lambda = 0.2\), \(0.25 \gt 0.2\), so C is suppressed too and only A is left; if C is a different object, it is lost.

Key takeaways

  • Open vocabulary means: class weights are text embeddings. A region is scored against any phrase with a cosine, a learned shift and scale, and a sigmoid (168.1); sigmoid, not softmax, so “nothing here” and multiple matches are possible.
  • Boxes are trained with L1 plus GIoU (168.3), which keeps a gradient when boxes do not overlap, after a one-to-one Hungarian matching (168.5); duplicates learn to say “no object”, so NMS is not needed. Other detectors use a score threshold and greedy NMS (168.4), which removes near-copies but can delete a real object in a crowd.
  • OWL-ViT fuses late (one box and embedding per ViT token, 576 for B/32 at 768 px); OWLv2 scales it with self-training to LVIS AP-rare 44.6%. Grounding DINO fuses early, picks 900 queries with the text (168.8), and reaches 52.5 zero-shot COCO AP.
  • SAM: a heavy encoder once per image, a 50 ms decoder per prompt; masks are dot products (168.9); three masks plus a predicted IoU resolve ambiguity (168.10). SAM 2 adds a memory bank of prompted and 6 recent frames (168.12) and an occlusion head for video.
  • VLMs write boxes as text: quantize to \(K\) bins (168.13), then special tokens (PaliGemma, Kosmos-2) or digits (Qwen2-VL normalized to 1000, Qwen2.5-VL in pixels); points work the same way (Molmo).

Key formulas

\[ \begin{gathered} p_{ij} = \sigma\big(s_i(\cos(\mathbf{r}_i, \mathbf{t}_j) + b_i)\big) \\ \mathrm{GIoU} = \mathrm{IoU} - \frac{|C| - |B \cup \hat B|}{|C|} \\ M_k(x, y) = \mathbf{w}_k^\top E_{\uparrow}(x, y) \\ \mathcal{L}_{\text{SAM}} = \min_k \big[20\,\mathcal{L}_{\text{focal}} + \mathcal{L}_{\text{dice}}\big] + \mathcal{L}_{\text{IoU}} \\ k = \Big\lfloor \frac{x}{S} K \Big\rfloor, \qquad |x - \hat x| \le \frac{S}{2K} \end{gathered} \]

References for this session

  1. Li et al. (2021). Grounded Language-Image Pre-training
  2. Cheng et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection
  3. Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers
  4. Liu et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
  5. Kirillov et al. (2023). Segment Anything
  6. Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
  7. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
  8. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  9. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  10. Lin et al. (2017). Focal Loss for Dense Object Detection
  11. Li et al. (2021). Grounded Language-Image Pre-training
  12. Cheng et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection
  13. Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers
  14. Liu et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
  15. Kirillov et al. (2023). Segment Anything
  16. Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
  17. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  18. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  19. Bai et al. (2025). Qwen2.5-VL Technical Report
  20. Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
  21. He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
  22. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
  23. Bodla et al. (2017). Soft-NMS: Improving Object Detection With One Line of Code