Open-vocabulary detection and segmentation
A normal detector can only find the classes it was trained on. Open-vocabulary models find anything you can name: they compare each region of the image with the text of your query. This session builds the idea step by step: region–text scores, box losses and matching, OWL-ViT and Grounding DINO for boxes, SAM and SAM 2 for masks in images and video, and the simple trick that lets a VLM write boxes and points as text.
What you will learn
- What grounding means, and how a closed-set classifier layer becomes an open vocabulary by using text embeddings as class weights
- How OWL-ViT scores every image token against every query (cosine, learned shift and scale, sigmoid), and why sigmoid and not softmax
- Box losses, matching and clean-up: IoU, GIoU, L1, the Hungarian assignment used by DETR-style detectors, and greedy NMS, with numbers
- How Grounding DINO fuses text and image early, selects 900 queries with the text, and why it beats late fusion on phrases
- How SAM turns a click or a box into three candidate masks with a 50 ms decoder, and how SAM 2 adds a memory bank for video
- How VLMs write boxes and points as text (PaliGemma, Qwen2-VL, Qwen2.5-VL, Kosmos-2, Molmo), and how to quantize and parse them
Why this matters
A classic object detector is trained on a fixed list of classes, for example the 80 classes of COCO. Ask it for “a forklift” or “the red cup on the left” and it has no answer. Open-vocabulary models remove the fixed list: you type what you want, and they return boxes or masks for it.
The last two sessions built the key tool. CLIP and SigLIP map an image and a sentence into the same vector space, so a dot product says how well they match (session 166, session 167). But they match the whole image with the whole sentence. This session pushes the same idea down to regions: which box, or which pixels, match which words? That link between words and places in the image is called grounding.
Three families of models do this today, and you will meet all three in real systems:
- Open-vocabulary detectors (OWL-ViT, OWLv2, Grounding DINO, GLIP, YOLO-World): text in, boxes with scores out.
- Promptable segmenters (SAM, SAM 2, SAM 3): a click, a box or a short phrase in, a pixel mask out, for images and for video.
- Vision-language models (PaliGemma, Qwen2-VL, Qwen2.5-VL, Molmo): they write boxes and points as text tokens, inside a normal answer.
Production pipelines often chain them: a detector finds “every person carrying a box”, SAM 2 turns each box into a mask and tracks it through the video, and a VLM answers questions about the result. Auto-labeling tools, robot perception, video editing and content moderation are built this way. [Likely]
From a fixed class list to any text
A detector outputs a set of boxes, and for each box a class score. A box is four numbers: either corners \((x_1, y_1, x_2, y_2)\) or centre and size \((c_x, c_y, w, h)\). The class score comes from a final linear layer. If \(\mathbf{r}_i \in \R^d\) is the feature vector of region \(i\), and \(W \in \R^{C \times d}\) holds one row per class, the logits are \(W \mathbf{r}_i\). The list of classes is frozen into \(W\): there are exactly \(C\) rows, and each one was learned from boxes labeled with that class.
The open-vocabulary trick is to replace the rows of \(W\) with text embeddings. Write each class name (or any phrase) as text, run it through a text encoder such as CLIP’s, and use the resulting vector \(\mathbf{t}_j\) as the class weight. Now the “class list” is whatever text you give at test time. If the region features live in the same space as CLIP text embeddings, a region of a forklift will score high against the text “forklift”, even if no forklift box was ever in the detection training data. This is exactly CLIP zero-shot classification (session 166), applied to each region instead of to the whole image.
Words to know
- Grounding: connecting a piece of text to a place in the image (a box, a mask, or a point). Phrase grounding gives a box for each noun phrase of a caption. Referring expression comprehension gives the one box that a phrase such as “the man in the blue shirt” describes (benchmarks: RefCOCO, RefCOCO+, RefCOCOg).
- Open-vocabulary detection: detection where the test classes can be new. The standard test is LVIS, a dataset with about 1,200 classes split by frequency. The rare classes (seen in very few training images) are held out from box training, and the score on them, APrare, measures how well the model generalizes to classes it never saw with boxes. [Certain]
- AP (average precision): the standard detection metric. A predicted box counts as correct if its overlap with a true box (IoU, defined below) is above a threshold; AP averages precision over recall levels, classes and (for COCO-style AP) IoU thresholds from 0.5 to 0.95.
- Open-set or zero-shot detection: the same idea; papers use the words loosely.
The first successful open-vocabulary detectors (for example ViLD) kept a classic two-stage detector and distilled CLIP: they trained the region features to imitate the CLIP image embedding of each cropped region. Later models trained the whole detector end to end on image–text and grounding data. We will look at two of them in detail: OWL-ViT (simple, late fusion) and Grounding DINO (deep, early fusion).
Scoring a region against a text query
Take one region feature \(\mathbf{r}_i\) and one query embedding \(\mathbf{t}_j\). OWL-ViT normalizes both to length 1, takes the cosine similarity, and then applies a learned shift and scale that are predicted from the region itself:
- Read the symbols
- \(\mathbf{r}_i\)
- the embedding of image region (or image token) \(i\), projected into the text space
- \(\mathbf{t}_j\)
- the text embedding of query \(j\), for example “a photo of a cat”
- \(b_i, s_i\)
- a shift and a positive scale, each computed by a small linear layer from the region feature; in the Hugging Face code \(s_i = \mathrm{ELU}(\cdot) + 1\), so it is always positive [Certain]
- \(\sigma\)
- the sigmoid, \(\sigma(z) = 1/(1 + e^{-z})\); it turns each logit into an independent probability
- \(p_{ij}\)
- the probability that region \(i\) shows the thing described by query \(j\)
Two choices here matter a lot.
Sigmoid, not softmax. A softmax over the queries forces the scores of one region to add up to 1. That is wrong for detection, for two reasons. First, most regions are background and should match no query; with a softmax, a region of empty wall still gives probability 1 to some query if you only asked for one. Second, one region can match several queries at once (“dog” and “pet”). A sigmoid asks a separate yes/no question for every pair, which is the same idea as the SigLIP loss (session 167). Try both in Figure 1.
Shift and scale per region. Cosine similarities between images and text are small numbers, often 0.1 to 0.35 for CLIP-like models, and their typical level differs from region to region. [Likely] The learned \(b_i\) moves the decision point, and \(s_i\) sharpens it, so that a fixed threshold such as 0.1 or 0.3 works across images.
Say a region has cosine similarities 0.31 with “cat”, 0.12 with “dog” and 0.02 with “sofa”, and the head predicts \(b = -0.22\), \(s = 12\) for it (toy values, chosen to look like real ones).
“cat”: \(12 \times (0.31 - 0.22) = 1.08\), and \(\sigma(1.08) = 0.746\). “dog”: \(12 \times (0.12 - 0.22) = -1.2\), \(\sigma = 0.231\). “sofa”: \(12 \times (-0.20) = -2.4\), \(\sigma = 0.083\).
With a threshold of 0.5, only “cat” is kept. The three probabilities add up to 1.06, not 1: each is its own yes/no answer. With a softmax over the same three logits, the region would get 0.958 for “cat” even if the cosine for “cat” were only slightly higher than the others; and if you asked only for “sofa”, the softmax would give 1.0 to “sofa”.
How to read Figure 1
Click query chips to add or remove them. With the default queries, three boxes pass 0.5: “cat”, “dog” and “red ball”. Add “pet”: the cat and the dog now match two queries each, which a sigmoid allows. Add “blue ball” and see that “red ball” does not fire on the blue ball: attributes are part of the text embedding. Now select only “giraffe” and turn on the softmax: every region becomes a giraffe with score 1.000, because a softmax over one query is always 1. With the sigmoid, nothing passes the threshold, which is the right answer.
Figure 1 hides the arithmetic. Figure 2 opens it up. It runs formula (168.1) one (region, query) pair at a time, on four regions and three queries with tiny 6-dimensional embeddings. A real model does the same with a few hundred dimensions and hundreds of regions, all at once as one matrix product; the loop here is only to let you see each number. The text embeddings are normalized once and reused for every region: this is why late fusion is cheap when you have many queries.
- for each query \(j\): \(\hat{\mathbf{t}}_j \leftarrow \mathbf{t}_j / \lVert \mathbf{t}_j \rVert\) // once, cached
- for each region \(i\):
- \(\hat{\mathbf{r}}_i \leftarrow \mathbf{r}_i / \lVert \mathbf{r}_i \rVert\)
- for each query \(j\):
- \(c \leftarrow \hat{\mathbf{r}}_i^\top \hat{\mathbf{t}}_j\) // cosine
- \(z \leftarrow s\,(c + b)\); \(p \leftarrow \sigma(z)\) // (168.1)
- if \(p \ge \tau\): output (box \(i\), query \(j\), \(p\))
- return all outputs
How to read Figure 2
Press Play or Step. The highlighted line of the algorithm box is the one that runs; the box next to it shows the formula with the real numbers. The bars show the two vectors and their product in each dimension; the green products add up to the dot product. With the defaults (\(\tau = 0.5\), \(b = -0.40\), \(s = 10\)), R1 and “cat” give \(\mathbf{r} \cdot \mathbf{t} = 0.930\), norms 1.020 and 0.995, cosine 0.917, \(z = 10 \times (0.917 - 0.40) = 5.165\) and \(p = 0.994\). Four pairs pass: R1 with “cat” and “pet”, R2 with “pet”, and R3 with “sofa”. The empty wall passes nothing; its best score is 0.170 for “sofa”. Lower \(\tau\) to 0.2: R2 now also passes for “cat” (\(p = 0.240\)), a false positive. Set \(s\) to 1: every \(p\) now lies between 0.349 and 0.634. The same four pairs still pass 0.5, because \(p \ge 0.5\) exactly when \(c \ge -b\), whatever \(s\) is. But the margins are tiny: at \(\tau = 0.4\), 10 of the 12 pairs pass. A large learned scale is what makes one fixed threshold safe.
Where the boxes come from, and how to train them
A score is only half of a detection. The model must also say where. Modern open-vocabulary detectors follow DETR: the model outputs a fixed set of predictions (one per image token in OWL-ViT, 900 “queries” in Grounding DINO), each with a box and class scores. During training, we must decide which prediction is responsible for which ground-truth object. To do that, and to train the boxes, we need a way to compare two boxes.
IoU and GIoU
The intersection over union of a predicted box \(\hat{B}\) and a true box \(B\) is the area they share divided by the area they cover together:
IoU is 1 for a perfect box and 0 when the boxes do not touch. That second fact is a problem for training: as soon as the boxes are apart, IoU is 0 no matter how far apart they are, so its gradient is 0 and it cannot pull the prediction closer. The generalized IoU fixes this with the smallest box \(C\) that encloses both:
The extra term is the fraction of the enclosing box that is empty. It grows as the boxes move apart, so GIoU keeps falling (down to −1) even when IoU is stuck at 0. GIoU equals IoU exactly when the union already fills \(C\), for example when one box contains the other. Explore it in Figure 3.
True box \((10, 10, 50, 50)\), predicted box \((20, 20, 60, 60)\). Each has area \(40 \times 40 = 1600\). They share \(30 \times 30 = 900\). Union \(= 1600 + 1600 - 900 = 2300\). IoU \(= 900/2300 = 0.391\). The enclosing box is \((10, 10, 60, 60)\), area 2500, of which \(2500 - 2300 = 200\) is empty. GIoU \(= 0.391 - 200/2500 = 0.311\).
Now move the prediction to \((60, 10, 100, 50)\): IoU \(= 0\), GIoU \(= -0.111\). Move it further to \((110, 10, 150, 50)\): IoU is still 0, but GIoU \(= -0.429\). GIoU knows the second box is worse; IoU does not.
How to read Figure 3
Slide the prediction far to the right. In the plot, the IoU curve becomes a flat line at 0 (no gradient, nothing to learn from), while the GIoU curve keeps going down. That slope is what pulls a far-away box toward its target early in training. Then shrink the prediction until it sits inside the true box: the enclosing box equals the true box, the empty-area term is 0, and GIoU equals IoU.
Score threshold and NMS: from raw outputs to a short list
At test time the detector scores every prediction against every query: 576 predictions for OWL-ViT B/32 at 768 px, 900 for Grounding DINO. Two simple filters turn this into a short list. First, a score threshold \(\tau\) drops the predictions the model is not sure about. Second, a detector often gives several near-identical boxes for one object: slightly shifted, slightly larger. Non-maximum suppression (NMS) keeps the best box of each group and removes the others. “Non-maximum” means: every box that is not the local maximum of the score is suppressed. It uses IoU (168.2) to decide which boxes belong to the same object:
- Read the symbols
- \(p_k, \hat{\mathbf{b}}_k\)
- the score (for one query) and the box of prediction \(k\)
- \(\tau\)
- the score threshold; lower means more recall and more false positives
- \(\mathcal{D}\)
- the candidates that pass the threshold, in order of score
- \(\lambda\)
- the IoU threshold of NMS; 0.5 is a common default [Likely]
- \(\mathcal{K}\)
- the kept boxes; a box is kept only if it does not overlap too much with any box kept before it
Three details matter. NMS is greedy: only kept boxes can suppress others, and the order is the score order. It runs per query (class-aware NMS): a “cat” box should not delete a “pet” box on the same animal, unless you want one label per object. And in the worst case it computes an IoU for every pair, so its cost grows with the square of the number of candidates; this is one reason to apply \(\tau\) first. [Certain]
NMS has one weakness that Figure 4 shows: crowds. Two real objects that overlap more than \(\lambda\) look like duplicates, and the weaker one is deleted. A softer variant, Soft-NMS, lowers the score of an overlapping box instead of removing it. [Certain] DETR-style models learn not to produce duplicates (next subsection), so they often run without NMS; many practical pipelines still add it as a cheap safety step. [Likely]
- \(\mathcal{D} \leftarrow\) boxes with \(p \ge \tau\), sorted by \(p\) // score threshold
- \(\mathcal{K} \leftarrow \emptyset\)
- while \(\mathcal{D}\) is not empty:
- \(m \leftarrow\) first box of \(\mathcal{D}\); move \(m\) to \(\mathcal{K}\)
- for each box \(b\) still in \(\mathcal{D}\):
- \(u \leftarrow \mathrm{IoU}(m, b)\) // (168.2)
- if \(u \gt \lambda\): remove \(b\) from \(\mathcal{D}\) // suppressed
- return \(\mathcal{K}\)
How to read Figure 4
Press Play or Step and follow the highlighted line. With \(\tau = 0.3\), G (0.18) is dropped at line 1. A (0.91) is kept first. Its IoU with B is \(19880 / (24335 + 23250 - 19880) = 0.718\), above \(\lambda = 0.5\), so B is suppressed; D (0.689) is suppressed too. The right cat C has IoU 0.283 with A and survives. C then suppresses its own duplicate E (0.719). The result is A, C and F: one box per cat, plus the dog box, which only a higher \(\tau\) (0.5) removes. Now set \(\lambda = 0.2\): A suppresses C (0.283 \(\gt\) 0.2), but E has IoU 0.191 with A and stays, so the right cat keeps its worse box. This is the crowd problem. Set \(\lambda = 0.7\): D (0.689) is no longer a “duplicate” and the left cat gets two boxes.
Matching predictions to objects
Say an image has 2 objects and the model makes 3 predictions. Which prediction should learn to be the cat? DETR answers with a one-to-one matching. For every pair (object \(i\), prediction \(k\)), compute a cost that is low when the prediction has the right class and the right box:
- Read the symbols
- \(c_i, \mathbf{b}_i\)
- the class and the box (normalized to [0, 1]) of ground-truth object \(i\)
- \(\hat{p}_k(c_i)\)
- prediction \(k\)’s probability for that class; in open-vocabulary models, the class is the text query of the object
- \(\pi\)
- an assignment: each object gets a different prediction; predictions left over are matched to “no object”
- \(\lambda\)
- weights; Grounding DINO uses 2.0, 5.0 and 2.0 for class, L1 and GIoU in the matching cost [Certain]
The best assignment is found with the Hungarian algorithm (in Python: linear_sum_assignment from scipy.optimize), which solves this in polynomial time. Then the loss is computed only on the matched pairs, and every unmatched prediction is trained to say “no object” for every query:
The class part is a sigmoid focal loss over all (prediction, query) pairs. Almost all of those pairs are negatives (background), and plain cross-entropy would be dominated by thousands of easy negatives. The focal loss multiplies each term by \((1 - p_t)^\gamma\), where \(p_t\) is the probability given to the correct answer:
With the usual \(\gamma = 2\), an easy example with \(p_t = 0.9\) has its loss cut from 0.105 to \(0.01 \times 0.105 = 0.0011\), while a hard example with \(p_t = 0.3\) keeps most of its loss (1.204 becomes 0.590). [Certain]
Objects (normalized boxes): cat \((0.10, 0.30, 0.40, 0.80)\) and dog \((0.55, 0.40, 0.90, 0.85)\). Predictions: P0 has \(p(\text{cat}) = 0.7\), box \((0.12, 0.32, 0.42, 0.78)\); P1 has \(p(\text{dog}) = 0.6\), box \((0.50, 0.42, 0.88, 0.90)\); P2 has \(p(\text{cat}) = 0.3, p(\text{dog}) = 0.25\), box \((0.15, 0.30, 0.45, 0.70)\). With weights 1, 5 and 2 in formula (168.5):
| Cost | P0 | P1 | P2 |
|---|---|---|---|
| cat | 0.092 | 7.879 | 1.581 |
| dog | 7.770 | 0.687 | 7.905 |
There are 6 possible assignments. The best is cat→P0, dog→P1, total 0.779; the next best (cat→P2, dog→P1) costs 2.267. P2 is a near-duplicate of the cat box. It is matched to “no object”, so the loss teaches it to give low scores. This is why DETR-style models do not need non-maximum suppression: duplicates are punished during training.
- input: image, its objects \(\{(c_i, \mathbf{b}_i)\}\), a text prompt that contains the object names plus some negative names
- \(\mathbf{t}_1, \dots, \mathbf{t}_Q \leftarrow\) text encoder on the prompt // one embedding per query (or per token)
- \(\{(\hat{\mathbf{b}}_k, \mathbf{r}_k)\}_{k=1}^{K} \leftarrow\) detector on the image (and the text, if it fuses early)
- \(p_{kj} \leftarrow \sigma(\text{logit}(\mathbf{r}_k, \mathbf{t}_j))\) for all \(k, j\) // formula (168.1)
- build cost \(\mathcal{C}_{ik}\) for all objects \(i\) and predictions \(k\) // formula (168.5)
- \(\hat\pi \leftarrow\) Hungarian algorithm on \(\mathcal{C}\) // one prediction per object
- targets \(y_{kj} \leftarrow 1\) if prediction \(k\) is matched to an object whose query is \(j\), else 0
- \(\mathcal{L} \leftarrow\) focal loss on all \(y_{kj}\) + L1 and GIoU on matched boxes // formulas (168.6), (168.7)
- backpropagate \(\mathcal{L}\) and update the weights
OWL-ViT and OWLv2: a ViT that detects
OWL-ViT (“Vision Transformer for Open-World Localization”, Minderer et al., 2022) is the simplest strong open-vocabulary detector. Start from a CLIP-style image–text model (a ViT and a text transformer trained contrastively). Then make three small changes [Certain]:
- Remove the final pooling of the image tower. Keep all output tokens: each token will become one prediction.
- On each token, add a linear class head that projects it into the text space, giving \(\mathbf{r}_i\) for formula (168.1).
- On each token, add a small MLP box head that predicts \((c_x, c_y, w, h)\).
Then fine-tune everything end to end with the matching loss of Algorithm 1 on detection data. The text side is unchanged: a query is embedded once, exactly as in CLIP. This is late fusion: the image and the text meet only in the final dot product.
How many predictions? One per image token. The OWL-ViT B/32 checkpoint reads 768 × 768 images with patches of 32, so it makes \((768/32)^2 = 24^2 = 576\) predictions per image. The OWLv2 B/16 checkpoint reads 960 × 960 with patches of 16: \(60^2 = 3600\) predictions. [Certain] (Values from the Hugging Face config files.)
A small but useful detail: the box head does not start from nothing. It adds a fixed bias so that, before training, every token predicts a box at its own patch position, with the size of one patch. The bias is the inverse sigmoid (logit) of that prior box, because the head’s output goes through a sigmoid. [Certain] So token \(i\) only needs to learn an offset from its own location, which makes training faster and more stable.
Image-conditioned (one-shot) detection
Since the query is only a vector in the shared space, it does not have to come from text. Take the embedding \(\mathbf{r}_k\) of a box in a different image (for example, a photo of one specific product) and use it as the query. OWL-ViT then finds similar objects in new images. This is useful when the thing is hard to name, like a particular part or logo. [Certain]
OWLv2 and self-training
Human-labeled boxes are expensive; image–text pairs from the web are cheap. OWLv2 (Minderer et al., 2023) uses an existing OWL-ViT to put pseudo-boxes on billions of web images, using words from their captions as queries, keeps the confident ones, and trains a new detector on them. This recipe, called OWL-ST, scales detection training to over 1 billion examples. With an L/14 model it raised LVIS APrare from 31.2% to 44.6%, a 43% relative gain, without any human boxes for those rare classes. [Certain]
Grounding DINO: fuse the text early
OWL-ViT computes image features without looking at the text. That is fast (encode the image once, then try any number of queries), but the image features cannot adapt to what you asked. Grounding DINO (Liu et al., 2023) takes the opposite path: it mixes text and image inside the detector, at three places [Certain]:
- Feature enhancer (6 layers). Image features (from a Swin transformer) and text features (from BERT) attend to each other with cross-attention, in both directions.
- Language-guided query selection. The detector decoder starts from \(N_q = 900\) queries. Instead of fixed learned queries, it picks the 900 image positions that best match any text token:
- Read the symbols
- \(X_I \in \R^{N_I \times d}\)
- the image features after the enhancer, one row per position (all scales together)
- \(X_T \in \R^{N_T \times d}\)
- the text features, one row per token; at most 256 tokens [Certain]
- \(\max_j\)
- for each image position, its best match over all text tokens
- \(\mathcal{I}\)
- the indices of the \(N_q = 900\) best image positions; they initialize the decoder queries
- Cross-modality decoder (6 layers). Each query attends to the image features and to the text features, then predicts a box. Its class logits are dot products with every text token, so a 900-query model with a 256-token prompt produces \(900 \times 256 = 230{,}400\) logits per image.
The prompt is one string with all the names, for example cat . remote control . sofa . (the public code uses “ . ” as the separator [Likely]). A naive text encoder would let “cat” attend to “sofa”, which mixes unrelated names. Grounding DINO adds attention masks so that tokens only attend within their own phrase. The paper calls this a sub-sentence level text representation. [Certain]
Results: 52.5 AP on COCO with no COCO training data (zero-shot), and a mean of 26.1 AP on the ODinW benchmark of 35 real-world datasets, zero-shot. [Certain] Early fusion makes it good at long referring phrases (“the second cup from the left”), at the cost of running the whole detector again for every new prompt.
Two related models are covered in session 174. GLIP (2021) first turned detection into phrase grounding: it computes region–word alignment scores \(S = O P^\top\) between region features \(O\) and word features \(P\), and trains on detection, grounding and caption data together. YOLO-World (2024) brings the idea to a real-time YOLO detector: 35.4 AP on LVIS at 52.0 FPS on a V100 GPU. [Certain] Its trick for speed is that a fixed vocabulary can be encoded once and folded into the weights (“prompt, then detect”).
SAM: promptable segmentation
A box is coarse. Many tasks need the exact pixels: cutting an object out of a photo, measuring an area, editing a video. Segment Anything (SAM, Kirillov et al., 2023) defines a new task, promptable segmentation: given an image and a prompt (one or more points, a box, or a rough mask), return a valid mask for the thing the prompt points at. SAM does not use class names at all. It answers “which object is here?”, not “where is the cat?”.
Architecture: heavy once, light many times
SAM has three parts, sized for interactive use [Certain]:
- Image encoder. A ViT-H/16 pretrained with MAE (session 165), on a 1024 × 1024 input. It outputs a \(64 \times 64\) grid of 256-dim features (\(1024/16 = 64\)). This is the expensive part, and it runs once per image.
- Prompt encoder. Points and boxes become positional encodings plus a learned “positive point”, “negative point” or “box corner” embedding. A mask prompt is downsampled by convolutions and added to the image features.
- Mask decoder. Two transformer layers with 256-dim tokens. The prompt tokens and a few learned output tokens attend to the image features, and the image features attend back (two-way attention). Then the image features are upsampled 4× by two transposed convolutions, to \(256 \times 256\) with 32 channels.
Each mask is a dot product between the upsampled features and one output token, passed through a small MLP:
- Read the symbols
- \(E_{\uparrow}(x, y)\)
- the upsampled image feature at pixel \((x, y)\) of the 256 × 256 output grid, 32 numbers
- \(\mathbf{o}_k\)
- output token \(k\) after the two decoder layers
- \(\mathbf{w}_k\)
- a “dynamic classifier” for this prompt: a weight vector that says which features mean “inside the mask”
- \(M_k(x, y)\)
- the mask logit; \(M_k \gt 0\) means inside. The 256 × 256 mask is upsampled to the image size at the end.
This is the same pattern as formula (168.1): a vector from the prompt side, dotted with a vector from the image side. Here it is done for each of the \(256 \times 256 = 65{,}536\) output pixels, which costs only \(65{,}536 \times 32 \approx 2.1\) million multiply-adds per mask. With the image embedding already computed, the prompt encoder and mask decoder run in about 50 ms on a CPU in a web browser, and the decoder needs less than 1% of the encoder’s compute. [Certain] That is what makes clicking feel instant.
Ambiguity: three masks for one click
One click on a shirt logo could mean the logo, the shirt, or the whole person. All three are valid. If the model had to output one mask, training would push it toward an average of the three, which is no good mask at all. SAM outputs 3 masks for a single prompt (roughly: sub-part, part, whole object) and trains with only the best of them [Certain]:
Here \(M^\ast\) is the true mask, the focal-to-dice ratio is 20:1, and \(\mathcal{L}_{\text{IoU}}\) is a mean-squared error that trains a small head to predict the IoU of each output mask with the truth. [Certain] At test time this predicted IoU ranks the three masks. When the prompt has several points and is no longer ambiguous, one mask is enough. The dice loss measures overlap directly, so it is not swamped by the many background pixels:
where \(p_x = \sigma(M(x))\) is the predicted probability at pixel \(x\) and \(m_x \in \{0, 1\}\) the true mask. For example, with predictions \((0.9, 0.8, 0.3, 0.1)\) on pixels whose truth is \((1, 1, 1, 0)\): the overlap is \(0.9 + 0.8 + 0.3 = 2.0\), the sums are 2.1 and 3, so the dice loss is \(1 - 4/5.1 = 0.216\).
How to read Figure 6
The start state is one positive click on the logo. You get three candidates: the logo, the mug body, and the whole mug with its handle. Click a thumbnail to choose (a real SAM ranks them by predicted IoU). Now press Reset, click the mug body away from the logo, then click the handle: only the whole mug contains both points, so one mask is returned. Switch to “Negative click” and click the logo: the logo is cut out of the mask. Negative points are how users fix a mask that is too big. Click the mug and a leaf: no region holds both, so you get the union of two objects.
The data engine and SA-1B
No dataset with a billion masks existed, so the SAM team built one with the model in the loop. First, annotators corrected SAM’s masks by clicking (assisted manual stage). Then SAM proposed masks and annotators added the missing ones (semi-automatic). Finally SAM ran alone on a grid of points over each image (fully automatic). The result, SA-1B, has 1.1 billion masks on 11 million licensed images: about 100 masks per image. [Certain] During training, each mask is used in 11 rounds of simulated interaction, where new clicks are sampled in the error regions of the previous prediction. [Certain]
SAM 2: segmentation in video with a memory
In a video, you want to click on an object once and get its mask in every frame, even when it turns, gets smaller, or is hidden for a while. Running SAM on every frame independently cannot do this: SAM has no idea which object you meant in frame 1. SAM 2 (Ravi et al., 2024) adds a streaming memory [Certain]. Think of it like this:
The parts are:
- An image encoder (Hiera, a hierarchical ViT pretrained with MAE) runs once per frame, as frames arrive.
- Memory attention (4 transformer layers) conditions the current frame’s features on the past: self-attention, then cross-attention to a memory bank, then an MLP.
- The memory bank holds spatial memories of up to \(N\) recent frames (\(N = 6\) by default) and of the frames where the user gave prompts, plus object pointers: compact vectors (from the mask decoder’s output tokens) that summarize the object.
- After the mask decoder predicts the mask of the current frame, a memory encoder fuses that mask with the frame’s features and writes a new memory into the bank (first in, first out for the recent part).
- An occlusion head predicts whether the object is visible in the current frame at all, so the model can output “no mask” while the object is hidden, and pick it up again later.
- Read the symbols
- \(F_t\)
- the image features of frame \(t\) (from the encoder)
- \(\mathcal{M}_s\)
- the memory of frame \(s\): its features fused with its predicted mask
- \(\mathcal{P}\)
- the set of frames where the user clicked; these memories are kept, they do not drop out of the bank
- \(F_t'\)
- features that now “know” which object we are following; the SAM-style mask decoder reads them
On a single image the memory is empty, and SAM 2 is just a better, faster SAM: 130.1 FPS against SAM’s 21.7 on an A100, about 6 times faster. [Certain] For video, the paper reports better accuracy with 3 times fewer user interactions than earlier methods. The Hiera-B+ model runs at 43.8 FPS and Hiera-L at 30.2 FPS on one A100. [Certain] The training data, SA-V, has 50.9K videos with 642.6K masklets (object tracks), 35.5M masks in total; about 55 masks per masklet. [Certain]
How to read Figure 7
Step through the frames. Until frame 7, the recent part of the bank fills up. At frame 8, frame 1 is older than 6 frames, but it stays because it was prompted. During the occlusion the occlusion head says “not visible” and the output mask is empty; those memories are marked as occluded. When the ball comes back in frame 8, the pointer and the prompted memory let the model find the same ball again. At frame 9 the correction click adds frame 9 to the kept set.
SAM 3: back to words
SAM 3 (Carion et al., 2025) closes the loop with open-vocabulary detection. Its task is promptable concept segmentation: give a short noun phrase such as “yellow school bus”, an image example, or both, and it returns masks and identities for every matching instance, in images and in video. It adds a detector that shares the backbone with the SAM 2 style tracker, and a presence head that first decides whether the concept is in the image at all, separately from where it is. [Certain] The authors report about twice the accuracy of earlier systems on this task. [Certain]
- input: video frames \(I_1, \dots, I_T\), text query “X”, score threshold \(\tau\)
- \(\{(\hat{\mathbf{b}}_k, p_k)\} \leftarrow\) open-vocabulary detector on \(I_1\) with query “X” // OWLv2 or Grounding DINO
- keep boxes with \(p_k \gt \tau\); remove duplicates with NMS if the detector needs it
- for each kept box \(k\): add \(\hat{\mathbf{b}}_k\) as a box prompt for object \(k\) on frame 1 of SAM 2
- for \(t = 1, \dots, T\):
- \(F_t \leftarrow\) image encoder on \(I_t\) // once per frame, shared by all objects
- for each object \(k\): \(F'_{t,k} \leftarrow\) memory attention (168.12); mask \(\leftarrow\) decoder; write memory
- every few frames, run the detector again to start tracks for new objects // objects can enter later
- return one mask track (masklet) per object
How VLMs output boxes and points as text
A VLM only produces tokens. To output a box, it must write the box as text. There are two main ways, and both start the same way: turn each coordinate into an integer in a fixed range. With \(K\) bins and image size \(S\) (width for \(x\), height for \(y\)):
Then either (a) add \(K\) new special tokens to the vocabulary, one per bin, so a box costs exactly 4 tokens; or (b) write the integers as plain digits with ordinary tokens. Normalizing by \(S\) makes the numbers independent of the image size; using absolute pixels avoids the normalization but ties the numbers to the resolution the model saw. Real systems use different choices [Certain]:
| Model | How a box or point is written | Range |
|---|---|---|
| Pix2Seq (2021) | Each coordinate is a quantized token; a detection is a token sequence | bins |
| Kosmos-2 | Image cut into a 32 × 32 grid: 1,024 location tokens. A box is the top-left and bottom-right cell: <p> a campfire </p><box><loc4><loc1007></box> | 32 per axis |
| PaliGemma | 1,024 tokens <loc0000> … <loc1023>, order \(y_{\min}, x_{\min}, y_{\max}, x_{\max}\), then the label. Prompt: detect cat ; dog. Masks use 128 extra <seg…> tokens from a VQ-VAE | 1,024 per axis |
| Qwen2-VL | Plain digits, normalized to [0, 1000): <|object_ref_start|>the eyes on a giraffe<|object_ref_end|><|box_start|>(176,106),(232,160)<|box_end|> | [0, 1000) |
| Qwen2.5-VL | Absolute pixel coordinates of the input image, often inside JSON | pixels |
| Molmo | Points as XML-like text in percent: <point x="10.0" y="10.0" alt="…">…</point>; counting by pointing at each object | 0 to 100 |
The model learns these formats from training data that was converted from detection and grounding datasets. Nothing in the architecture “knows” about boxes: the coordinates are just tokens that the language model learns to predict with cross-entropy, one after another (session 191).
Image 640 × 480 pixels; box \((x_1, y_1, x_2, y_2) = (100, 50, 300, 400)\).
PaliGemma (\(K = 1024\), \(y\) first): \(\lfloor 50/480 \cdot 1024 \rfloor = 106\), \(\lfloor 100/640 \cdot 1024 \rfloor = 160\), \(\lfloor 400/480 \cdot 1024 \rfloor = 853\), \(\lfloor 300/640 \cdot 1024 \rfloor = 480\). Output: <loc0106><loc0160><loc0853><loc0480> dog. Decoding at bin centres gives \((100.3, 49.9, 300.3, 400.1)\): the error bound (168.13) is \(640/2048 = 0.31\) px in \(x\).
Qwen2-VL ([0, 1000)): \(100/640 \cdot 1000 = 156.25 \to 156\), \(50/480 \cdot 1000 = 104.2 \to 104\), \(468.75 \to 468\), \(833.3 \to 833\). Output: (156,104),(468,833).
Kosmos-2 (32 × 32 grid): the top-left corner falls in row 3, column 5; the bottom-right in row 26, column 15. Each cell is 20 × 15 px, so corners can be off by up to 10 px in \(x\) and 7.5 px in \(y\). Coarse, but only 2 tokens per box.
Molmo (a point at the box centre, in percent): \(x = 200/640 = 31.25\%\), \(y = 225/480 = 46.875\%\).
bbox_2d key, a common output style [Likely]). The grid lines show the Kosmos-2 cells.How to read Figure 8
With PaliGemma and Qwen2-VL the decoded box sits on top of the true box: the error is below one pixel, much smaller than the error of the model itself. Switch to Kosmos-2 and move a slider slowly: the decoded corner jumps in steps of 20 px (x) or 15 px (y), one grid cell at a time. Notice also the cost: 4 special tokens for PaliGemma and 2 for Kosmos-2, but about 19 characters of digits and punctuation for the Qwen2-VL box, which is usually more than a dozen tokens because many tokenizers split numbers into single digits. [Likely]
The PyTorch version
The code below is small and deterministic (runs on CPU in a second). It builds an OWL-ViT-style head (formula 168.1 with the box prior), IoU and GIoU (168.2, 168.3), the matching (168.5) on the worked example, the detection loss (168.6, 168.7), a SAM-style mask decoder output with the min-over-3 loss (168.9 to 168.11), and the box-to-text formats (168.13). The printed values match the worked examples above.
import itertools, re import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) An OWL-ViT-style open-vocabulary head: one box and one embedding per image token class OpenVocabHead(nn.Module): def __init__(self, d_img=768, d_emb=512, grid=24): super().__init__() self.cls_proj = nn.Linear(d_img, d_emb) # token -> joint image-text space self.shift, self.scale = nn.Linear(d_img, 1), nn.Linear(d_img, 1) self.box_mlp = nn.Sequential(nn.Linear(d_img, d_img), nn.GELU(), nn.Linear(d_img, 4)) ys, xs = torch.meshgrid(torch.arange(grid), torch.arange(grid), indexing="ij") ctr = torch.stack([(xs + 0.5) / grid, (ys + 0.5) / grid], -1).reshape(-1, 2) prior = torch.cat([ctr, torch.full_like(ctr, 1 / grid)], -1) # each token starts at its own patch self.register_buffer("box_bias", torch.logit(prior)) def forward(self, tokens, text_emb): # tokens (N, d_img), text_emb (Q, d_emb) r = F.normalize(self.cls_proj(tokens), dim=-1) t = F.normalize(text_emb, dim=-1) scale = F.elu(self.scale(tokens)) + 1 # always > 0 logits = (r @ t.T + self.shift(tokens)) * scale # formula (168.1), before the sigmoid boxes = torch.sigmoid(self.box_mlp(tokens) + self.box_bias) # (cx, cy, w, h) in [0, 1] return logits, boxes head = OpenVocabHead() tokens, text = torch.randn(576, 768), torch.randn(3, 512) # 24 x 24 tokens; 3 queries, e.g. "cat", "dog", "sofa" logits, boxes = head(tokens, text) print(logits.shape, boxes.shape) # torch.Size([576, 3]) torch.Size([576, 4]) # 2) Boxes: IoU and GIoU, formulas (168.2) and (168.3) def xyxy(b): cx, cy, w, h = b.unbind(-1) return torch.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], -1) def iou_giou(a, b): # a (n, 4), b (m, 4), xyxy -> (n, m) each area = lambda x: (x[:, 2] - x[:, 0]) * (x[:, 3] - x[:, 1]) lt, rb = torch.max(a[:, None, :2], b[None, :, :2]), torch.min(a[:, None, 2:], b[None, :, 2:]) inter = (rb - lt).clamp(min=0).prod(-1) union = area(a)[:, None] + area(b)[None, :] - inter lt_c, rb_c = torch.min(a[:, None, :2], b[None, :, :2]), torch.max(a[:, None, 2:], b[None, :, 2:]) hull = (rb_c - lt_c).prod(-1) iou = inter / union return iou, iou - (hull - union) / hull gt = torch.tensor([[10., 10, 50, 50]]); pr = torch.tensor([[20., 20, 60, 60], [60., 10, 100, 50]]) iou, giou = iou_giou(gt, pr) print(iou.round(decimals=3), giou.round(decimals=3)) # [[0.391, 0.]] [[0.311, -0.111]] # 3) Matching, formula (168.5): brute force here; real code uses scipy linear_sum_assignment def match(prob, pboxes, labels, gboxes, w_cls=1.0, w_l1=5.0, w_giou=2.0): _, giou = iou_giou(gboxes, pboxes) cost = (-w_cls * prob[:, labels].T + w_l1 * torch.cdist(gboxes, pboxes, p=1) + w_giou * (1 - giou)) G, P = cost.shape best = min(itertools.permutations(range(P), G), key=lambda pm: sum(cost[i, pm[i]] for i in range(G))) return list(best), cost prob = torch.tensor([[0.70, 0.10], [0.20, 0.60], [0.30, 0.25]]) # 3 predictions x 2 classes pboxes = torch.tensor([[.12, .32, .42, .78], [.50, .42, .88, .90], [.15, .30, .45, .70]]) gboxes = torch.tensor([[.10, .30, .40, .80], [.55, .40, .90, .85]]); labels = torch.tensor([0, 1]) assign, cost = match(prob, pboxes, labels, gboxes) print(assign, cost.round(decimals=3)) # [0, 1]; prediction 2 is matched to "no object" # 4) Detection loss on the matched pairs: sigmoid focal + L1 + GIoU, formula (168.6) def sigmoid_focal(logit, target, alpha=0.25, gamma=2.0): p = torch.sigmoid(logit) ce = F.binary_cross_entropy_with_logits(logit, target, reduction="none") p_t = p * target + (1 - p) * (1 - target) a_t = alpha * target + (1 - alpha) * (1 - target) return (a_t * (1 - p_t) ** gamma * ce).sum() target = torch.zeros_like(prob); target[assign, labels] = 1.0 # unmatched rows stay all-zero mp, mg = pboxes[assign], gboxes loss = (sigmoid_focal(torch.logit(prob), target) / len(labels) + 5.0 * (mp - mg).abs().sum(-1).mean() + 2.0 * (1 - iou_giou(mg, mp)[1].diag()).mean()) print(round(loss.item(), 4)) # 5) A SAM-style mask: dot product of upsampled image features with a mask token, formula (168.8) feat = torch.randn(32, 256, 256) # upsampled image embedding (C=32, 256 x 256) mask_tokens = torch.randn(3, 32) # 3 output tokens after the decoder MLPs mask_logits = torch.einsum("kc,chw->khw", mask_tokens, feat) # (3, 256, 256) gt_mask = (torch.rand(256, 256) > 0.5).float() def dice_loss(logit, t): p = torch.sigmoid(logit).flatten(1); t = t.flatten()[None] return 1 - (2 * (p * t).sum(-1) + 1) / (p.sum(-1) + t.sum(-1) + 1) per_mask = 20 * torch.stack([sigmoid_focal(m, gt_mask) / gt_mask.numel() for m in mask_logits]) + dice_loss(mask_logits, gt_mask) print(mask_logits.shape, per_mask.argmin().item()) # only the best of the 3 masks gets the gradient, formula (168.10) # 6) Boxes as text, formula (168.12): PaliGemma-style and Qwen2-VL-style def q(v, size, k): return min(k - 1, int(v / size * k)) def to_paligemma(box, W, H, label): x1, y1, x2, y2 = box return "".join(f"<loc{q(v, s, 1024):04d}>" for v, s in [(y1, H), (x1, W), (y2, H), (x2, W)]) + " " + label def to_qwen2vl(box, W, H, label): x1, y1, x2, y2 = box return (f"<|object_ref_start|>{label}<|object_ref_end|><|box_start|>" f"({q(x1, W, 1000)},{q(y1, H, 1000)}),({q(x2, W, 1000)},{q(y2, H, 1000)})<|box_end|>") def parse_paligemma(s, W, H): ys1, xs1, ys2, xs2 = [(int(k) + 0.5) / 1024 for k in re.findall(r"<loc(\d{4})>", s)[:4]] return [round(v, 1) for v in (xs1 * W, ys1 * H, xs2 * W, ys2 * H)] s = to_paligemma((100, 50, 300, 400), 640, 480, "dog"); print(s) # <loc0106><loc0160><loc0853><loc0480> dog print(parse_paligemma(s, 640, 480)) # [100.3, 49.9, 300.3, 400.1] print(to_qwen2vl((100, 50, 300, 400), 640, 480, "dog")) # ...(156,104),(468,833)...
Where this is used
| Application | Typical tool | Why this tool |
|---|---|---|
| Auto-labeling a new detection dataset | Grounding DINO or OWLv2 for boxes, then SAM for masks, then human review | No training needed for a new class; people only fix errors |
| Detect a rare or custom object (a product, a defect) | OWL-ViT image-conditioned query, or a fine-tuned open-vocabulary detector | The thing is hard to name; one example image is enough to query |
| Real-time detection with a fixed custom vocabulary (robots, cameras) | YOLO-World with the vocabulary folded into the weights | Text encoding is removed from the runtime path |
| Photo and video editing (cut-out, background removal, effects) | SAM for clicks on images; SAM 2 to propagate the mask through the video | Interactive: 50 ms per prompt after one encoder pass |
| Video object segmentation and tracking | SAM 2 with a box or click on the first frame (session 204) | Memory bank handles appearance change and short occlusions |
| “Find every moment a red car appears” in long video | Open-vocabulary detector on sampled frames, then tracking; feeds key-moment detection (session 207) | Text query at search time, no retraining |
| Video-RAG and visual question answering with evidence | A VLM that returns boxes as text (Qwen2.5-VL), or detector crops passed to a VLM (session 210) | Answers can point at the evidence; crops raise effective resolution |
| Counting (“how many people in the queue?”) | Molmo-style pointing, or detection plus counting | Pointing at each item is more reliable than guessing a number [Likely] |
| GUI agents and document AI | VLMs that output click points or element boxes as text | The agent must say where to click (session 154) |
Common mistakes
- Using a softmax over the queries. A region of background then always “matches” something. Use the model’s sigmoid scores and a threshold, as in Figure 1.
- Mixing box formats. \((c_x, c_y, w, h)\) vs \((x_1, y_1, x_2, y_2)\), normalized vs pixels, and PaliGemma’s \(y\)-first order. Write one conversion function and test it on a known box, like the worked example.
- Drawing boxes on the wrong image size. The model saw a resized or padded image. OWL-ViT and OWLv2 pad to a square; their normalized boxes refer to the padded square, not to the original rectangle. [Likely]
- Bad prompts for Grounding DINO. Separate names with “ . ”, keep the total under 256 tokens, and lower-case the text. Long lists of class names split across several calls change the scores, because the names attend inside the model. [Likely]
- One global threshold for all queries. Scores are not calibrated across queries: “person” may need 0.3 and “screwdriver” 0.15. Tune per class on a few labeled images.
- Expecting SAM to know classes. SAM masks have no labels and a single click is ambiguous. Use box prompts from a detector, or several points, and check the predicted IoU.
- Running SAM 2 per frame without memory (for example, calling the image predictor in a loop). You lose identity and occlusion handling. Use the video predictor, which keeps the memory bank.
- Re-encoding the image for every prompt. SAM’s encoder is the expensive part; cache the image embedding and run only the decoder per click.
Interview questions
How can a detector recognize a class it never saw boxes for?basic
Why do open-vocabulary detectors use a sigmoid per query instead of a softmax over queries?core
What is GIoU and why is it used as a loss instead of IoU?core
Explain Hungarian matching in DETR-style detectors. Why does it remove the need for NMS?core
Compare OWL-ViT and Grounding DINO. When would you pick each?advanced
How does SAM handle an ambiguous point prompt?core
Why is SAM fast enough to feel interactive?basic
How does SAM 2 keep track of an object through a video, including occlusion?advanced
How does a VLM output a bounding box? Compare special location tokens with plain digits.core
Design a system that finds and segments every “person wearing a helmet” in a factory video stream.advanced
What is OWLv2’s self-training and why does it help on rare classes?advanced
Exercises
- A region has cosine similarities 0.26 with “pet” and 0.18 with “red ball”, with \(b = -0.22\) and \(s = 12\). Compute both probabilities with formula (168.1). Which pass a threshold of 0.5?
- Compute IoU and GIoU for the true box \((0, 0, 10, 10)\) and the prediction \((5, 0, 15, 10)\).
- A prediction box lies completely inside the true box. Show that GIoU equals IoU.
- How many predictions does an OWL-ViT-style model make on a 1008 × 1008 input with patch 14? How many class logits for 20 queries?
- Compute the dice loss for predictions \((1, 1, 0, 0)\) against the truth \((1, 0, 1, 0)\).
- Write the PaliGemma tokens for the box \((x_1, y_1, x_2, y_2) = (320, 240, 640, 480)\) on a 640 × 480 image.
- A SAM 2 user clicks frames 1 and 20 of a 30-frame video. With \(N = 6\), which memories are in the bank when frame 25 is processed?
- Run NMS (168.4) with \(\lambda = 0.5\) on A \((0, 0, 10, 10)\) with score 0.9, B \((1, 0, 11, 10)\) with 0.8, and C \((6, 0, 16, 10)\) with 0.7. Which boxes are kept? What changes with \(\lambda = 0.2\)?
Answerscheck
<loc0512><loc0512><loc1023><loc1023>. 7. The prompted frames 1 and 20 (always kept) and the recent frames 19 to 24; frame 20 is both prompted and recent. 8. A is kept first. IoU(A, B) \(= 90 / (100 + 100 - 90) = 0.818 \gt 0.5\): B is suppressed. IoU(A, C) \(= 40 / 160 = 0.25 \le 0.5\): C is kept. Result: A and C. With \(\lambda = 0.2\), \(0.25 \gt 0.2\), so C is suppressed too and only A is left; if C is a different object, it is lost.Key takeaways
- Open vocabulary means: class weights are text embeddings. A region is scored against any phrase with a cosine, a learned shift and scale, and a sigmoid (168.1); sigmoid, not softmax, so “nothing here” and multiple matches are possible.
- Boxes are trained with L1 plus GIoU (168.3), which keeps a gradient when boxes do not overlap, after a one-to-one Hungarian matching (168.5); duplicates learn to say “no object”, so NMS is not needed. Other detectors use a score threshold and greedy NMS (168.4), which removes near-copies but can delete a real object in a crowd.
- OWL-ViT fuses late (one box and embedding per ViT token, 576 for B/32 at 768 px); OWLv2 scales it with self-training to LVIS AP-rare 44.6%. Grounding DINO fuses early, picks 900 queries with the text (168.8), and reaches 52.5 zero-shot COCO AP.
- SAM: a heavy encoder once per image, a 50 ms decoder per prompt; masks are dot products (168.9); three masks plus a predicted IoU resolve ambiguity (168.10). SAM 2 adds a memory bank of prompted and 6 recent frames (168.12) and an occlusion head for video.
- VLMs write boxes as text: quantize to \(K\) bins (168.13), then special tokens (PaliGemma, Kosmos-2) or digits (Qwen2-VL normalized to 1000, Qwen2.5-VL in pixels); points work the same way (Molmo).
Key formulas
References for this session
- Li et al. (2021). Grounded Language-Image Pre-training
- Cheng et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection
- Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers
- Liu et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- Kirillov et al. (2023). Segment Anything
- Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Lin et al. (2017). Focal Loss for Dense Object Detection
- Li et al. (2021). Grounded Language-Image Pre-training
- Cheng et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection
- Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers
- Liu et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- Kirillov et al. (2023). Segment Anything
- Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
- He et al. (2021). Masked Autoencoders Are Scalable Vision Learners
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
- Bodla et al. (2017). Soft-NMS: Improving Object Detection With One Line of Code