Grounded pretraining: GLIP and YOLO-World
GLIP turned object detection into phrase grounding: instead of a fixed class layer, each region is scored against the words of a text prompt, so detection data, grounding data and web captions can train one model. YOLO-World brought the idea to a real-time YOLO detector by encoding the vocabulary once and folding it into the weights. This session builds both step by step, with the losses, the fusion layers, the pseudo-labeling pipeline, and a clear answer to when a detector beats a VLM.
What you will learn
- How GLIP rewrites a detector’s class layer as region–word alignment scores \(S = O P^\top\), and how sub-word tokens are handled in the loss and at test time
- What language-aware deep fusion (X-MHA) does, why it helps grounding and prompt tuning, and why it costs one pass per prompt chunk (31 passes for LVIS)
- How grounding data and self-training on web captions raise rare-class accuracy, with the real numbers from the GLIP paper
- How YOLO-World adds open vocabulary to YOLOv8: a frozen CLIP text encoder, a scaled cosine head, max-sigmoid gates and image-pooling attention
- How YOLO-World is trained (online vocabulary, region–text loss, pseudo-labels filtered with CLIP) and deployed (prompt-then-detect, vocabulary as 1 × 1 convolution weights)
- When a specialist detector beats a VLM: speed, dense calibrated boxes and small objects, computed from token counts and feature strides
Why this matters
A normal detector has a fixed class list baked into its last layer. GLIP showed that you can replace that layer with words: detection becomes phrase grounding, the task of finding the box for each phrase in a sentence. YOLO-World then made the same idea fast enough for real-time video.
In session 168 we met the basic trick: score each region against the embedding of a text query instead of against a learned class row. That session looked at OWL-ViT, Grounding DINO and SAM. This session goes deeper into the two models that turned the trick into a pretraining recipe:
- GLIP (Grounded Language-Image Pre-training, Li et al., 2021): one loss for detection data and grounding data, deep fusion of image and text inside the detector, and self-training on millions of web image–caption pairs. It reached 49.8 AP on COCO without seeing a single COCO image. [Certain]
- YOLO-World (Cheng et al., 2024): a YOLOv8 detector with a frozen CLIP text encoder and light fusion layers. It reaches 35.4 AP on LVIS at 52.0 frames per second on one V100 GPU, zero-shot. [Certain] Its key idea is prompt, then detect: encode the vocabulary once, fold it into the weights, then run a plain fast detector.
Both ideas still matter in 2026. Auto-labeling pipelines use GLIP-style or Grounding-DINO-style models to put boxes on raw data. Robots, drones, retail cameras and video analytics use YOLO-World-style models because they run on small GPUs at video rate. And interviews ask a recurring design question: when do you use a specialist detector, and when do you just ask a VLM? The last part of this session answers it with numbers: speed, box quality and small objects.
Detection as phrase grounding
Two tasks look different but are the same at heart. Object detection: given an image, find boxes and give each one a class from a fixed list (COCO has 80 classes). Phrase grounding: given an image and a sentence such as “a man walks his dog on the beach”, find the box for each phrase (“a man”, “his dog”, “the beach”). Both tasks localize things and align them with a concept. GLIP’s first contribution is to write both with one formula.
The classic classifier
A detector first turns the image into \(N\) region features. Each one goes to a box regressor and to a classifier. The classifier is one linear layer:
- Read the symbols
- \(O\)
- the region (or box, or anchor) features: one row of size \(d\) for each of the \(N\) candidate regions
- \(W \in \R^{c \times d}\)
- the classifier weights: one learned row per class, \(c\) classes in total
- \(S_{\text{cls}}\)
- the class logits: entry \((i, j)\) says how much region \(i\) looks like class \(j\)
- \(T \in \{0, 1\}^{N \times c}\)
- the targets: which region should get which class, after the detector’s matching step (anchor rules or Hungarian matching)
- \(\mathcal{L}_{\text{loc}}\)
- the box loss (and, in some detectors, a centerness or objectness loss)
The classes live in \(W\). If you want a new class, you need new rows, and new rows need labeled boxes.
Replace the class rows with words
GLIP writes the class list as a sentence, the prompt, and runs it through a text encoder (BERT-base in GLIP). For COCO the prompt is simply all 80 names joined with periods: person. bicycle. car. … toothbrush. The paper found this format works better with BERT than the friendlier “Detect: person, bicycle, car, …”. [Certain] Then the logits are dot products between regions and words:
- Read the symbols
- \(P\)
- the contextual token features from the text encoder, one row per token; \(P\) plays the role that \(W\) played before
- \(M\)
- the number of tokens in the prompt (including periods and special tokens)
- \(S_{\text{ground}}\)
- the region–word alignment scores; entry \((i, m)\) says how well region \(i\) matches token \(m\)
Everything else in the detector stays the same: same backbone, same box head, same matching. The paper checked this. A DyHead detector with a Swin-Tiny backbone gets 49.4 AP on COCO both as a classic detector and after the rewrite into a grounding model. [Certain] So nothing is lost by the rewrite, and something big is gained: the “class list” is now any text.
Tokens are not phrases
There is one technical gap. The targets \(T\) are per phrase (\(N \times c\)), but the logits are per token (\(N \times M\)), and \(M\) is always larger than \(c\). The paper lists four reasons [Certain]: a phrase can have several words (“traffic light”); one word can split into several sub-word tokens (BERT splits “toothbrush” into tooth and ##brush); the prompt adds tokens such as periods and [CLS]; and a no-object token may be added at the end.
The fix is simple. Expand the targets to tokens, and fold tokens back into phrases at test time:
- Read the symbols
- \(T' \in \{0,1\}^{N \times M}\)
- the expanded target: every token of a positive phrase is positive; added tokens are negative for every region
- \(\tau_c\)
- the set of token positions that belong to phrase \(c\)
- \(\sigma\)
- the sigmoid; the loss is a binary (focal) loss on every (region, token) pair, as in one-stage detectors
- \(p_{i,c}\)
- the probability that region \(i\) shows phrase \(c\): the average of its token probabilities [Certain]
The prompt person. traffic light. toothbrush. becomes \(M = 10\) BERT tokens: [CLS] person . traffic light . tooth ##brush . [SEP] (checked with the real bert-base-uncased tokenizer). The full 80-class COCO prompt is 195 tokens, which fits in GLIP’s limit of 256. [Certain]
Take a region that shows a traffic light, with toy logits \(-1.5\) for person, \(1.4\) for traffic, \(2.2\) for light, \(-3.0\) and \(-2.0\) for tooth and ##brush, and around \(-4\) to \(-5\) for the other tokens. The sigmoids are 0.182, 0.802, 0.900, 0.047 and 0.119.
Phrase probabilities by formula (174.3): person 0.182; traffic light \((0.802 + 0.900)/2 = 0.851\); toothbrush \((0.047 + 0.119)/2 = 0.083\). The expanded target row is \(T' = (0,0,0,1,1,0,0,0,0,0)\). The binary cross-entropy summed over the 10 tokens is 0.752; the largest terms are traffic (0.220, a positive with only 0.802) and person (0.201, a negative with 0.182). With a focal loss (\(\alpha = 0.25\), \(\gamma = 2\)) the same row costs only 0.0089, because every token is already on the right side.
How to read Figure 1
Dark cells are high probabilities. The “traffic light” region lights up two columns, because the phrase has two tokens; the “toothbrush” region lights up tooth and ##brush. Periods and special tokens stay dark for every row: they are trained as negatives. Click the background row: all its phrase probabilities are low, which is the right answer for a region with no object, and possible only because each cell is its own sigmoid. Tick the box to see the target \(T'\) that the loss compares against.
- input: image, its boxes and class names; the full class list of the dataset
- \(C_{\text{pos}} \leftarrow\) classes present in the image; sample some negative classes; shuffle // GLIP keeps up to 85 names per prompt
- prompt \(\leftarrow\) “name1. name2. … .”; record which tokens belong to which name
- \(O^0 \leftarrow\) image backbone; \(P^0 \leftarrow\) BERT on the prompt
- \(O, P \leftarrow\) deep fusion layers on \((O^0, P^0)\) // formulas (174.4), (174.5)
- \(S \leftarrow O P^\top\) // (174.2)
- \(T \leftarrow\) the detector’s usual assignment of regions to ground-truth boxes; \(T' \leftarrow\) expand to tokens // (174.3)
- \(\mathcal{L} \leftarrow \mathrm{focal}(S; T') + \mathcal{L}_{\text{loc}}\); backpropagate
The random sampling in step 2 is not only a fix for long class lists. Objects365 has 365 classes, which do not fit in 256 tokens, so GLIP keeps all positive names and fills the prompt with random negatives: with probability 0.5 up to 85 names, otherwise a random number of them. [Certain] Shuffling the order every time is also a data augmentation: the model cannot memorize “position 7 means dog”.
Deep fusion: let the image read the prompt
Formula (174.2) is late fusion: the image and the text are encoded apart and meet only in the final dot product. GLIP adds deep fusion: in the last layers of the detector, image features and word features attend to each other. The paper calls this language-aware deep fusion. [Certain]
The fusion block is a cross-modality multi-head attention (X-MHA). One attention matrix between regions and tokens is used in both directions:
- Read the symbols
- \(A\)
- attention scores between each region (rows) and each token (columns); the softmax runs over tokens for the image side and over regions for the text side
- \(W^{(q,\cdot)}, W^{(v,\cdot)}, W^{(\text{out},\cdot)}\)
- learned query, value and output projections, one set for the image (I) and one for the language (L)
- \(O_{t2i}\)
- “text to image” context: for each region, a mix of the token features it attends to
- \(P_{i2t}\)
- “image to text” context: for each token, a mix of the region features it attends to
The contexts are added back, and each side then runs its own normal layer. GLIP stacks this over the \(L\) modules of the DyHead detection head, with new BERT layers on the text side:
Remove the two context terms and you are back to a late-fusion model. Note that there is no separate key projection: one score matrix \(A\) serves both directions, which is cheaper than two separate cross-attentions. You saw the same kind of co-attention between two streams in session 172.
Why fuse? Two reasons, both measured. First, grounding of real sentences needs it: “the man holding the umbrella” is about a relation, and a region feature computed without the sentence cannot know which man you mean. Second, the image features become conditioned on the prompt. That is what makes prompt tuning work (below). In the ablation ladder on COCO zero-shot, the rewrite alone (GLIP-T A) gives 42.9 AP, adding deep fusion (GLIP-T B) gives 44.9, and adding human-labeled grounding data (GLIP-T C) gives 46.7. [Certain]
The price of fusion
Deep fusion has a cost that grows with the vocabulary. Because the image features depend on the prompt, you must run the fused layers again for every prompt. And a prompt is limited in length: GLIP caps it at 256 tokens. [Certain] So a large vocabulary must be split into chunks:
- Read the symbols
- \(C\)
- the number of class names you want to detect
- \(k\)
- how many names you put in one prompt (GLIP used \(k = 40\) for LVIS) [Certain]
- \(n_{\text{pass}}\)
- how many times the fused layers must run on the same image
LVIS has \(C = 1203\) classes. With 40 names per prompt, \(n_{\text{pass}} = \lceil 1203/40 \rceil = \lceil 30.08 \rceil = 31\) passes per image. The YOLO-World paper measured GLIP-T at 0.12 FPS for this zero-shot LVIS setting on a V100, against 52.0 FPS for YOLO-World-L: a factor of about 433. [Certain] Part of the gap is the bigger backbone; most of it is the 31 passes.
Even on COCO with a single prompt, fusion is not free. GLIP’s own measurement: GLIP-T runs at 4.84 FPS without fusion and 2.52 FPS with it on a P100; GLIP-L at 0.54 vs 0.32 FPS. [Certain]
Scaling up with grounding data
The unified formula has a big practical effect: detection data and grounding data now train the same loss. That matters because grounding data has a much richer vocabulary. The largest detection datasets have at most about 2,000 classes, while Flickr30K Entities alone contains 44,518 unique phrases and Visual Genome captions 110,689. [Certain]
GLIP uses three kinds of data [Certain]:
| Kind | Name in the paper | Content |
|---|---|---|
| Detection (human boxes, fixed classes) | Objects365 (O365); FourODs | O365: 0.66M images, 365 classes. FourODs: O365, OpenImages, Visual Genome and ImageNet boxes, 2.66M images |
| Grounding (human boxes, free phrases) | GoldG | 0.8M examples from Flickr30K, VG Caption and GQA, curated by MDETR, with COCO images removed |
| Web image–caption pairs (no boxes) | Cap4M; Cap24M | 4M and 24M pairs from the web; boxes come from a teacher model |
The third row is the interesting one. Web captions have no boxes. GLIP uses self-training: train a teacher GLIP on the human-labeled data, use it to put boxes on the noun phrases of web captions (found with an NLP parser), then train a student on gold data plus these pseudo-labels. In total GLIP-L trains on 27M grounding examples: 3M human-annotated and 24M pseudo-labeled. [Certain]
Why can the student beat the teacher? The paper’s answer: the teacher uses the sentence context to make an educated guess. It may not know the word “vaccine”, but in “a nurse holds a vaccine in a small vial” it can find the vial. The guess becomes a supervised label for the student, which then learns the new word. [Likely] (This is the authors’ explanation, not a proof.)
The gains show up most on rare classes. On LVIS minival, the rare-class score APr goes from 13.5 (GLIP-T B, Objects365 only) to 17.7 with gold grounding data (+4.2) and to 20.8 with web data added (+3.1). [Certain] Web data barely moves COCO (46.7 vs 46.3), because COCO’s 80 classes are already covered by Objects365. Rare things need rich language; common things do not.
Two more training tricks: images from caption data get up to 19 negative captions from other images mixed into the prompt, so the model must first find the right caption and then ground it; and the BERT encoder uses a learning rate of \(10^{-5}\) while the rest uses \(10^{-4}\). [Certain]
Results
| Model | Backbone | Pretraining data | COCO zero-shot AP | LVIS minival AP / APr |
|---|---|---|---|---|
| GLIP-T (C) | Swin-T | O365, GoldG | 46.7 | 24.9 / 17.7 |
| GLIP-T | Swin-T | O365, GoldG, Cap4M | 46.3 | 26.0 / 20.8 |
| GLIP-L | Swin-L | FourODs, GoldG, Cap24M | 49.8 | 37.3 / 28.2 |
All numbers are zero-shot: no COCO or LVIS training images. [Certain] For scale: a Faster R-CNN with a ResNet-101 backbone trained on COCO gets 42.0 AP. Fine-tuned on COCO, GLIP-L reaches 60.8 AP on val2017 and 61.5 on test-dev. On Flickr30K phrase grounding it reaches 87.1 Recall@1 on test. [Certain]
Prompts as a cheap way to adapt
Because the detector reads the prompt, you can adapt it in two cheap ways. Manual prompts: describe the class better. On an aquarium dataset, writing “stingray, which is flat and round” raised AP50 for stingray from 4.6 to 9.7, with no training. [Certain] Prompt tuning: keep the whole model frozen, run BERT once on the prompt to get \(P^0\), then throw BERT away and train only \(P^0\) on the new task’s labels. For GLIP-T and GLIP-L, prompt tuning almost matches full fine-tuning on the 13 datasets of the ODinW benchmark, and the gap shrinks as the model grows. [Certain] This only works because of deep fusion: in a late-fusion model, tuning \(P^0\) is just tuning a linear classifier. (Prompt tuning for LLMs is the same idea: see session 56.)
The successors: GLIPv2 (2022) adds understanding tasks such as VQA and captioning on top of the same grounding backbone [Likely], and Grounding DINO (2023) moves grounded pretraining into a DETR-style detector, with fusion in the encoder, the query selection and the decoder (session 168). [Certain]
YOLO-World: a real-time open-vocabulary detector
GLIP and Grounding DINO are accurate but heavy: Swin backbones, deep fusion, a full pass per prompt. They run at a few frames per second at best. Many real systems need 30 FPS or more on modest hardware. YOLO-World asks: can a YOLO detector, the standard choice for real-time detection, learn open vocabulary? [Certain]
The model has three parts [Certain]:
- A YOLOv8 detector. A convolutional backbone (Darknet-style) makes multi-scale feature maps \(C_3, C_4, C_5\); a path aggregation network (PAN) mixes them top-down and bottom-up into \(P_3, P_4, P_5\) at strides 8, 16 and 32; a decoupled head predicts, at every position, a box and an object embedding \(\mathbf{e}_k \in \R^D\) instead of class logits.
- A CLIP text encoder turns each vocabulary word or phrase into an embedding \(\mathbf{w}_j\), giving \(W \in \R^{C \times D}\). The public configs use the text tower of CLIP ViT-B/32, so \(D = 512\). [Certain] It is frozen during pretraining.
- RepVL-PAN, the “re-parameterizable vision-language PAN”: the PAN plus two light fusion steps, one from text to image and one from image to text.
The class score is a scaled cosine, as in CLIP:
- Read the symbols
- \(\mathbf{e}_k\)
- the object embedding predicted at position \(k\) of the feature maps
- \(\mathbf{w}_j\)
- the CLIP text embedding of vocabulary entry \(j\)
- \(\alpha, \beta\)
- a learned scale and shift. In the public code \(\alpha = \exp(\text{logit\_scale})\) starts at \(1/0.07 = 14.29\), the same start value as CLIP’s temperature, and \(\beta\) starts at 0 [Certain]
- \(s_{k,j}\)
- the logit that position \(k\) shows thing \(j\); a sigmoid turns it into a probability
The paper notes that both the L2 normalization and the affine \(\alpha, \beta\) are important for stable training. [Certain] Cosines are small numbers (between −1 and 1), so without a scale of about 14 the sigmoid could never get close to 0 or 1. For example, a cosine of 0.32 gives a logit of \(14.29 \times 0.32 = 4.57\) and a probability of 0.990; a cosine of 0.05 gives 0.714 and 0.671. The learned \(\beta\) moves the threshold: with \(\beta = -3\), the same cosines give 0.828 and 0.092.
Text to image: max-sigmoid attention
Real cross-attention between thousands of pixels and the vocabulary would be too slow. YOLO-World uses a much cheaper gate, inside the text-guided CSPLayer (T-CSPLayer, a YOLOv8 C2f block with text guidance added). For every pixel of feature map \(X_l\), take its best dot product with any word, squash it with a sigmoid, and scale the pixel by it:
- Read the symbols
- \(X_l \in \R^{H \times W \times D}\)
- the image feature map at pyramid level \(l \in \{3, 4, 5\}\)
- \(X_l W_j^\top\)
- for each pixel, its dot product with word \(j\): an \(H \times W\) map per word
- \(\max_j\)
- keep, for each pixel, only the best-matching word
- \(\sigma(\cdot)\)
- the gate, between 0 and 1: near 1 where some vocabulary word matches, lower where nothing does
This is not a softmax over pixels or over words. It does not mix features; it only re-weights them, one number per pixel. The cost is one \(D \times C\) product per pixel, the same as a \(1 \times 1\) convolution with \(C\) output channels. Explore it in Figure 2.
How to read Figure 2
Click words to add or remove them from the vocabulary; click a cell to see its dot product with every chosen word. With “man” and “dog”, the man and dog cells open almost fully (gate 0.90 to 0.92). The woman cells open partly (about 0.81) because the toy “woman” vector is close to “man”. Car cells sit at exactly 0.500: their dot product with both words is 0, and \(\sigma(0) = 0.5\). Background cells close (about 0.13). Add “car” and the car opens. The gate never reaches 0, so nothing is deleted: it is a soft hint to the later layers about what the user asked for.
Image to text: image-pooling attention
The other direction makes the text embeddings image-aware. Again cheaply: max-pool each of the three feature maps down to a 3 × 3 grid, which gives \(3 \times 9 = 27\) image tokens, and let the words attend to them [Certain]:
The queries are the \(C\) words; the keys and values are only 27 pooled tokens, whatever the image size. So the cost is tiny: \(C \times 27\) attention scores. The updated \(W'\) is then used in the head, formula (174.7). The words can adapt to the image (for example, “apple” on a desk of electronics versus in a fruit bowl). [Likely]
Training YOLO-World
YOLO-World treats every annotation as a region–text pair \((B_i, t_i)\): the text can be a class name, a noun phrase or a short description. [Certain] Training data is mixed the same way as in GLIP:
| Dataset | Type | Images | Annotations |
|---|---|---|---|
| Objects365 v1 | detection, 365 classes | 609k | 9,621k |
| GQA | grounding | 621k | 3,681k |
| Flickr30K | grounding | 149k | 641k |
| CC3M† (pseudo-labeled) | image–text | 246k | 821k |
(Values from Table 1 of the paper; COCO images are removed from GQA and Flickr30K. [Certain])
Online vocabulary. YOLO trains on mosaics of 4 images. For each mosaic, the vocabulary is all positive nouns from the 4 images plus random negative nouns from the same dataset, at most \(M = 80\) entries. [Certain] As in GLIP, the random negatives teach the model to say “no” to words that are not in the image.
Assignment and loss. Positions are matched to ground-truth objects with YOLOv8’s task-aligned assignment (TAL, from the TOOD detector): a position is a good match when both its class score and its box IoU are high. Each matched position gets the index of its text as the label. The loss is
- Read the symbols
- \(\mathcal{L}_{\text{con}}\)
- the region–text contrastive loss: a cross-entropy between the similarities (174.7) and the assignments. The public code uses a sigmoid binary cross-entropy over the vocabulary, inherited from YOLOv8 [Likely]
- \(\mathcal{L}_{\text{iou}}\)
- an IoU-based box loss (CIoU in YOLOv8)
- \(\mathcal{L}_{\text{dfl}}\)
- distribution focal loss: YOLOv8 predicts each box side as a small distribution over distances, and this loss trains it
- \(\lambda_I\)
- turns the box losses off for pseudo-labeled web images, whose boxes are too noisy to regress to; their region–text pairs still train \(\mathcal{L}_{\text{con}}\) [Certain]
Pseudo-labels from captions
To use web image–caption pairs, YOLO-World builds region–text pairs automatically, in three steps [Certain]: (1) extract nouns from the caption with a simple n-gram method; (2) run GLIP-L with those nouns as the prompt to get boxes with confidences \(c_i\); (3) re-score and filter with CLIP. For step 3, each box is cropped and scored against its noun with CLIP, giving \(s^r_i\), and the whole image is scored against the caption, giving \(s^{\text{img}}\). Then:
- Read the symbols
- \(c_i\)
- GLIP-L’s confidence for box \(i\)
- \(s^r_i\)
- CLIP’s score for the crop of box \(i\) against its noun
- \(\tilde{c}_i\)
- the new confidence: the geometric mean of the two opinions, so both models must agree
- \(s^{\text{region}}\)
- the mean \(s^r\) of the boxes that survived
- \(s\)
- the image-level score; images whose caption and boxes do not fit well are dropped entirely
The thresholds (0.5 for NMS, 0.3 for boxes and images) were set by looking at part of the results; the pipeline has no human in the loop. [Certain] Step through it in Figure 4. Note the chain of teachers: GLIP was trained with self-training, and YOLO-World learns partly from GLIP’s boxes. This is a form of knowledge distillation from a slow model into a fast one (session 59).
How to read Figure 4
Press Next to move through the six steps. Step 4 removes the second “dog” box: it overlaps the first one with IoU 0.784, above 0.5. Step 5 removes the small “man” box: its rescored confidence \(\sqrt{0.30 \times 0.12} = 0.190\) is below 0.3, even though GLIP alone gave it 0.30. Step 6 keeps the image with \(s = 0.306\). Tick the checkbox and replay step 6: with a weak caption match the image score falls to 0.228 and the whole image is dropped.
What each part buys
The ablations in the paper (YOLO-World-L, zero-shot LVIS minival AP) [Certain]:
- Data: Objects365 alone gives 23.5 AP. Adding GQA gives 31.9, a gain of 8.4, from the rich phrases. Adding GoldG instead gives 32.5, and adding CC3M† on top 33.0.
- Text encoder: a frozen CLIP text encoder gives 22.4 AP; a frozen BERT-base only 14.6. Fine-tuning BERT helps it (18.3), but fine-tuning CLIP hurts (19.3): with only 365 classes of text, the encoder forgets its general knowledge. Language that was learned with images is the key ingredient.
- RepVL-PAN: on Objects365 it raises AP from 22.4 to 23.5; with GQA from 29.7 to 31.9. Fusion helps more when the text is rich.
Prompt, then detect: re-parameterization
Here is the idea that makes YOLO-World fast at deployment. In most use cases the vocabulary does not change from frame to frame: a warehouse camera always looks for “person, forklift, pallet, helmet”. So encode the vocabulary once, offline, and turn it into ordinary weights. The paper calls this prompt-then-detect with an offline vocabulary. [Certain]
Look at where \(W\) is used. In the max-sigmoid gate (174.8) and in the head (174.7), the image features are multiplied by \(W^\top\) at every pixel. A matrix product applied at every pixel is a \(1 \times 1\) convolution. Reshape \(W \in \R^{C \times D}\) to \(C \times D \times 1 \times 1\) and use it as the kernel:
After this rewrite the deployed model is a normal CNN. It has no text encoder, no tokenizer, no attention over text, and it can be exported to ONNX or TensorRT like any YOLO. The paper counts YOLO-World-L at 48M parameters re-parameterized versus 110M with the text encoder attached, and the speed at 52.0 FPS versus 17.6 FPS in the original form (V100, no TensorRT). [Certain] To change the vocabulary, you re-run the text encoder once and swap the weights.
- offline, once per vocabulary: names \(\leftarrow\) [“person”, “forklift”, …]
- \(W \leftarrow \mathrm{CLIPText}(\text{names})\) // \(C \times D\), here \(D = 512\)
- \(W' \leftarrow\) image-pooling update if kept, otherwise \(W' = W\) // see the note below
- write \(W\) into the \(1 \times 1\) kernels of every T-CSPLayer gate, and \(W'\) (normalized) into the head // (174.12)
- export the network (ONNX / TensorRT); the text encoder is not shipped
- online, per frame: boxes, \(s_{k,j} \leftarrow\) CNN(frame) // one pass, 8,400 positions at 640 × 640
- keep \(\sigma(s_{k,j}) \gt\) threshold; NMS per class // YOLO-World still uses NMS
A note on step 3: image-pooling attention depends on the image, so strictly it cannot be folded into fixed weights; the appendix gives a simplified form for deployment, and for fine-tuning on COCO the authors remove the T-CSPLayers and image-pooling attention altogether, because 80 fixed classes need little vision–language interaction. [Certain] Fine-tuned this way, YOLO-World-L gets 53.3 AP on COCO at 156 FPS (V100 with TensorRT), next to YOLOv8-L trained from scratch at 52.9 AP and 159 FPS. [Certain] So pretraining gives the open vocabulary and a small accuracy gain, at almost no speed cost.
Zero-shot results
| Model | Params | Pretraining data | FPS (V100) | LVIS minival AP / APr |
|---|---|---|---|---|
| GLIP-T | 232M | O365, GoldG, Cap4M | 0.12 | 26.0 / 20.8 |
| Grounding DINO-T | 172M | O365, GoldG, Cap4M | 1.5 | 27.4 / 18.1 |
| DetCLIP-T | 155M | O365, GoldG | 2.3 | 34.4 / 26.9 |
| YOLO-World-S | 13M | O365, GoldG | 74.1 | 26.2 / 19.1 |
| YOLO-World-M | 29M | O365, GoldG | 58.1 | 31.0 / 23.8 |
| YOLO-World-L | 48M | O365, GoldG, CC3M† | 52.0 | 35.4 / 27.6 |
(Table 2 of the YOLO-World paper, “fixed AP” on LVIS minival, up to 1,000 predictions per image, FPS without TensorRT. [Certain] The LVIS numbers for GLIP here are the same as in GLIP’s own table.) The 13M-parameter YOLO-World-S matches GLIP-T’s AP while being more than 600 times faster in this setting. Note what is not in the table: GLIP-L and Grounding DINO-L are still more accurate, at far higher cost. YOLO-World trades a little accuracy for a lot of speed.
A follow-up, YOLOE (2025), extends the same family to text prompts, visual prompts and a prompt-free mode in one real-time model. [Likely]
Where these beat VLMs: speed, boxes, small objects
A modern VLM such as Qwen2.5-VL can also output boxes: you ask “find every person”, and it writes coordinates as text (session 168). So why keep a detector? Three reasons, each with a mechanism you can compute.
1. Speed: one pass versus one token at a time. A detector scores every position and every word in one forward pass: 8,400 positions at 640 × 640 for YOLOv8 (\(80^2 + 40^2 + 20^2\)). [Certain] A VLM writes boxes autoregressively. With the real Qwen2-VL tokenizer, one box with its label, <|object_ref_start|>dog<|object_ref_end|><|box_start|>(156,104),(468,833)<|box_end|>, is 22 tokens, because every digit is its own token; the Qwen2.5-VL JSON form {"bbox_2d": [100, 50, 300, 400], "label": "dog"} is 31 tokens. [Certain] Fifty objects cost about 1,100 generated tokens. Output time grows with the number of objects; a detector’s time does not.
2. Boxes: dense and calibrated. A detector gives a score for every (box, word) pair, so you can pick a threshold, trade precision for recall, and compute AP. A VLM returns one answer list. It may stop early, skip objects in a crowded scene, or repeat one. [Likely] There is no natural score to rank by, so standard detection metrics and threshold tuning are harder.
3. Small objects: stride 8 versus 28 pixels per token. YOLO’s finest map \(P_3\) has stride 8: a 16-pixel object covers \(2 \times 2 = 4\) cells. Qwen2-VL cuts the image into 14-pixel patches and merges each 2 × 2 group into one token, so one LLM token covers 28 × 28 pixels. [Certain] The same 16-pixel object covers about a third of one token. Its details are mixed with its neighbours before the language model ever sees them. (VLMs can raise the input resolution, but the token count grows with the square of the side.)
How to read Figure 5
At the defaults (16-pixel objects, 50 of them, 25 ms per token), the object covers 4 detector cells but 0.33 VLM tokens, and the VLM needs 1,100 output tokens, about 27.5 s, against 19.2 ms. Drag the object size up to 112: now it spans 16 tokens and the resolution argument goes away. Drag the number of objects down to 1: the VLM needs 22 tokens, 0.55 s, still far slower but fine for a chat answer. The detector bar never moves.
Where VLMs win. Long, compositional queries (“the cup the woman is about to pick up”), reasoning over the scene, counting with explanation, and answering questions that are not about boxes at all. A common design is to combine them: a fast detector proposes boxes on every frame, and a VLM looks at a few crops when a hard question needs reasoning (session 182). [Likely]
A store wants to count 30 product types on shelves, at 10 frames per second per camera, on one GPU for 4 cameras: 40 frames per second in total. YOLO-World-L at 52 FPS on a V100 fits, with the 30 names folded into the weights. [Certain for the 52 FPS] A VLM writing 60 boxes per frame at 22 tokens each needs 1,320 tokens per frame; even at a fast 10 ms per token that is 13.2 s per frame, about 530 times too slow for 40 frames per second. The VLM is useful once per hour, to answer “which shelf looks wrong, and why?”.
The PyTorch version
The code below is small and deterministic (CPU, about a second). It builds the GLIP target expansion and phrase scores (174.3) on the worked example, a sigmoid focal loss, one X-MHA fusion block (174.4), YOLO-World’s contrastive head (174.7), the max-sigmoid gate (174.8), the 27 pooled tokens of image-pooling attention (174.9), a check that the vocabulary as a \(1 \times 1\) convolution gives the same numbers (174.12), and the pseudo-label filter (174.11) on the example of Figure 4. The printed values match the text.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) GLIP: detection as phrase grounding (formulas 174.2 to 174.4) tokens = ["[CLS]", "person", ".", "traffic", "light", ".", "tooth", "##brush", ".", "[SEP]"] phrase_tokens = {"person": [1], "traffic light": [3, 4], "toothbrush": [6, 7]} def expand_target(T, phrase_tokens, M): """T (N, c) region-to-class targets -> T' (N, M) region-to-token targets.""" Tp = torch.zeros(T.shape[0], M) for j, idx in enumerate(phrase_tokens.values()): Tp[:, idx] = T[:, j:j + 1] # every sub-word of a positive phrase is positive return Tp # ".", [CLS], [SEP] stay 0 (negative) def phrase_probs(S, phrase_tokens): P = torch.sigmoid(S) # one yes/no probability per (region, token) return torch.stack([P[:, idx].mean(-1) for idx in phrase_tokens.values()], -1) S = torch.tensor([[-4.0, -1.5, -5.0, 1.4, 2.2, -5.0, -3.0, -2.0, -5.0, -4.5]]) # one region: a traffic light T = torch.tensor([[0., 1., 0.]]) Tp = expand_target(T, phrase_tokens, len(tokens)) print(Tp.int().tolist()) # [[0, 0, 0, 1, 1, 0, 0, 0, 0, 0]] print(phrase_probs(S, phrase_tokens).round(decimals=3)) # tensor([[0.1820, 0.8510, 0.0830]]) print(round(F.binary_cross_entropy_with_logits(S, Tp, reduction="sum").item(), 3)) # 0.752 def sigmoid_focal(S, Tp, alpha=0.25, gamma=2.0): p = torch.sigmoid(S) ce = F.binary_cross_entropy_with_logits(S, Tp, reduction="none") p_t = p * Tp + (1 - p) * (1 - Tp) a_t = alpha * Tp + (1 - alpha) * (1 - Tp) return (a_t * (1 - p_t) ** gamma * ce).sum() print(round(sigmoid_focal(S, Tp).item(), 4)) # 0.0089: easy tokens count very little # 2) GLIP deep fusion: one cross-modality attention block (formula 174.5) class XMHA(nn.Module): def __init__(self, d=64): super().__init__() self.qi, self.ql = nn.Linear(d, d), nn.Linear(d, d) self.vi, self.vl = nn.Linear(d, d), nn.Linear(d, d) self.oi, self.ol = nn.Linear(d, d), nn.Linear(d, d) def forward(self, O, P): # O (N, d) regions, P (M, d) tokens attn = self.qi(O) @ self.ql(P).T / O.shape[-1] ** 0.5 # (N, M), shared by both directions O_t2i = self.oi(attn.softmax(-1) @ self.vl(P)) # text -> image context P_i2t = self.ol(attn.T.softmax(-1) @ self.vi(O)) # image -> text context return O + O_t2i, P + P_i2t O, P = torch.randn(100, 64), torch.randn(10, 64) O2, P2 = XMHA()(O, P) print(tuple(O2.shape), tuple(P2.shape), tuple((O2 @ P2.T).shape)) # (100, 64) (10, 64) (100, 10) # 3) YOLO-World pieces (formulas 174.7 to 174.9) class TextContrastiveHead(nn.Module): def __init__(self): super().__init__() self.logit_scale = nn.Parameter(torch.tensor(math.log(1 / 0.07))) # alpha = 14.29 at init self.bias = nn.Parameter(torch.zeros(())) # beta = 0 at init def forward(self, e, w): # e (B, D, H, W) object embeddings, w (C, D) e, w = F.normalize(e, dim=1), F.normalize(w, dim=-1) return torch.einsum("bdhw,cd->bchw", e, w) * self.logit_scale.exp() + self.bias def max_sigmoid(X, W): # X (B, D, H, W), W (C, D) gate = torch.einsum("bdhw,cd->bchw", X, W).amax(1, keepdim=True).sigmoid() return X * gate # each pixel scaled by its best word match def image_pooling_tokens(feats): # feats: 3 levels (B, D, H_l, W_l) return torch.cat([F.adaptive_max_pool2d(f, 3).flatten(2) for f in feats], 2).transpose(1, 2) D, C = 512, 4 W = torch.randn(C, D) # text embeddings of 4 words feats = [torch.randn(1, D, s, s) for s in (80, 40, 20)] # P3, P4, P5 at 640 x 640 Xt = image_pooling_tokens(feats); print(tuple(Xt.shape)) # (1, 27, 512) mha = nn.MultiheadAttention(D, 8, batch_first=True) W2 = W[None] + mha(W[None], Xt, Xt)[0]; print(tuple(W2.shape)) # (1, 4, 512) image-aware text print(sum(f.shape[2] * f.shape[3] for f in feats)) # 8400 candidate positions # Re-parameterization: the vocabulary becomes the weight of a 1 x 1 convolution conv = nn.Conv2d(D, C, 1, bias=False) with torch.no_grad(): conv.weight.copy_(W[:, :, None, None]) X3 = feats[0] diff = (conv(X3) - torch.einsum("bdhw,cd->bchw", X3, W)).abs().max().item() print(diff < 1e-3) # True: same numbers, no text encoder gated = max_sigmoid(X3, W); print(tuple(gated.shape)) # (1, 512, 80, 80) head = TextContrastiveHead() print(round(head.logit_scale.exp().item(), 2), tuple(head(X3, W).shape)) # 14.29 (1, 4, 80, 80) # 4) Pseudo-label filtering for image-text data (formula 174.11) def iou(a, b): ix = max(0, min(a[2], b[2]) - max(a[0], b[0])); iy = max(0, min(a[3], b[3]) - max(a[1], b[1])) inter = ix * iy; area = lambda r: (r[2] - r[0]) * (r[3] - r[1]) return inter / (area(a) + area(b) - inter) def filter_pseudo(props, s_img, nms_thr=0.5, box_thr=0.3, img_thr=0.3): for p in props: p["score"] = math.sqrt(p["c"] * p["s_r"]) # rescore with CLIP kept = [] for p in sorted(props, key=lambda p: -p["score"]): # NMS within each noun if all(q["noun"] != p["noun"] or iou(q["box"], p["box"]) <= nms_thr for q in kept): kept.append(p) kept = [p for p in kept if p["score"] > box_thr] if not kept: return [], 0.0 s_region = sum(p["s_r"] for p in kept) / len(kept) s = math.sqrt(s_img * s_region) return (kept if s > img_thr else []), s props = [dict(noun="man", c=0.82, s_r=0.30, box=(40, 60, 200, 400)), dict(noun="dog", c=0.74, s_r=0.28, box=(220, 250, 380, 400)), dict(noun="dog", c=0.41, s_r=0.27, box=(230, 240, 390, 410)), dict(noun="beach", c=0.52, s_r=0.20, box=(0, 300, 640, 480)), dict(noun="man", c=0.30, s_r=0.12, box=(500, 100, 560, 200))] kept, s = filter_pseudo(props, s_img=0.36) print([(p["noun"], round(p["score"], 3)) for p in kept], round(s, 3)) # [('man', 0.496), ('dog', 0.455), ('beach', 0.322)] 0.306
Things to try: replace mean by amax in phrase_probs and see how one confident sub-word dominates; set logit_scale to 0 (so \(\alpha = 1\)) and see that the sigmoids of cosines all stay between 0.27 and 0.73; give the pseudo-label filter s_img=0.20 and see the image dropped.
Where this is used
| Application | Typical tool | Why this tool |
|---|---|---|
| Real-time detection with a custom fixed vocabulary (robots, drones, retail, safety cameras) | YOLO-World with the vocabulary folded into the weights, exported to TensorRT | Plain CNN at video rate; new classes without new boxes |
| Auto-labeling a new detection dataset | GLIP-L or Grounding DINO as a labeler, CLIP re-scoring, human review | The slow model is accurate; run it once, offline |
| Training a small fast detector for a new domain | Pseudo-labels from a grounded model, then fine-tune a YOLO-World or YOLO | Distills an open-vocabulary teacher into a cheap student |
| Phrase grounding and referring expressions (“the man holding the umbrella”) | GLIP, Grounding DINO (deep fusion) | Region features must depend on the sentence |
| Video key-moment search (“when does a forklift appear?”) | YOLO-World on sampled frames, then tracking; results feed session 207 | Fast enough to scan hours of video; text query at search time |
| Video object tracking and segmentation | Detector boxes as prompts for SAM 2 (session 204) | The detector finds and names, SAM 2 gives masks and identities |
| Video-RAG and VQA with evidence | Detector crops and tags stored with each frame; a VLM answers (session 210) | Cheap tags make frames searchable; the VLM only reads a few |
| Few-shot adaptation to a new domain (medical, aerial, industrial) | GLIP prompt tuning, or fine-tuning YOLO-World on a few hundred boxes | Few parameters to train; one model serves many tasks |
Common mistakes
- Using a softmax over the vocabulary. Both GLIP and YOLO-World score each (region, word) pair with its own sigmoid. A softmax forces every background box to pick some word, and makes scores change when you add words.
- Forgetting the sub-word tokens. In a GLIP-style loss, every token of a positive phrase is positive and the periods are negative. Scoring a phrase by only its first token, or training periods as positives, quietly hurts multi-word and rare classes.
- One giant prompt. Deep-fusion models have a length limit (256 tokens for GLIP); text beyond it is cut off and those classes silently disappear. Split into chunks, and remember that the scores of a class can change with the other names in its chunk.
- Fine-tuning the CLIP text encoder on a small label set. In YOLO-World’s ablation this dropped zero-shot LVIS AP from 22.4 to 19.3. Freeze it, or use a very small learning rate (the paper uses 0.01× for fine-tuning).
- Re-encoding the vocabulary every frame. With a fixed vocabulary, run the text encoder once and re-parameterize. Keeping the text encoder in the online path costs YOLO-World-L about 3 times in speed (17.6 vs 52.0 FPS).
- Trusting zero-shot scores across classes. “person” and “screwdriver” need different thresholds. Tune per class on a small labeled set, as with any open-vocabulary detector.
- Training box regression on pseudo-boxes. Web pseudo-labels are good enough for “what”, not for exact “where”. YOLO-World turns the box loss off (\(\lambda_I = 0\)) for them.
- Using a VLM where a detector is needed. Many objects, small objects, or a frame-rate budget: use a detector, and keep the VLM for the few questions that need reasoning.
Interview questions
How does GLIP turn object detection into phrase grounding?basic
Why is unifying detection and grounding useful for pretraining?core
What is language-aware deep fusion, and what does it cost?core
How can a GLIP student trained on pseudo-labels beat its teacher?advanced
How does YOLO-World stay real-time while accepting text?core
Explain max-sigmoid attention. Why not normal cross-attention?core
Why does YOLO-World use a frozen CLIP text encoder and not BERT?basic
Why are both an L2 normalization and a learned scale needed in the YOLO-World head?core
Walk through YOLO-World’s pseudo-labeling pipeline. What can go wrong?advanced
A product team asks: “Why not just use a VLM to detect objects in our video stream?” What do you answer?advanced
What is prompt tuning in GLIP, and why does it fail for late-fusion models?advanced
Exercises
- The 80-name COCO prompt is 195 BERT tokens. (a) How many tokens per class is that on average? (b) At that rate, how many 256-token prompts would the 1,203 LVIS names need at minimum? Why might GLIP still use 40 names per prompt (31 prompts)?
- In Figure 1, the toothbrush region has logits 1.8 for
toothand 0.9 for##brush. Compute its “toothbrush” probability with formula (174.3). - A YOLO-World position has cosine 0.25 with “forklift”. Compute the probability with \(\alpha = 14.29\) and \(\beta = 0\), then with \(\beta = -3\).
- A pixel’s dot products with three vocabulary words are −0.4, 1.2 and 0.3. What is its max-sigmoid gate? What is it if the vocabulary has only the first word?
- How many candidate positions does a YOLOv8-style head have at 1280 × 1280 input (strides 8, 16, 32)? How many stride-8 cells wide is a 12-pixel object, and how many 28-pixel VLM tokens?
- Two pseudo-boxes: (a) GLIP confidence 0.6, CLIP crop score 0.25; (b) GLIP confidence 0.9, CLIP crop score 0.09. Which ones survive the 0.3 threshold of formula (174.11)? What does (b) suggest about the box?
- A VLM must output 30 boxes at 22 tokens each, decoding at 20 ms per token. How long does that take, and how many YOLO-World-L frames (52 FPS) fit in the same time?
Answerscheck
Key takeaways
- GLIP rewrites detection as grounding: the class weights \(W\) become token features \(P\) of a text prompt, \(S = O P^\top\) (174.2). Targets expand to sub-word tokens; phrase scores are token averages (174.3). Detection and grounding data then train one loss.
- Deep fusion (174.4, 174.5) makes image features depend on the prompt: better grounding and prompt tuning, but one fused pass per prompt chunk: 31 passes for LVIS, 0.12 FPS for GLIP-T.
- Self-training on web captions (27M examples for GLIP-L) mainly helps rare classes: LVIS AP-rare 13.5 → 17.7 → 20.8 for GLIP-T variants. Zero-shot COCO: 49.8 AP for GLIP-L.
- YOLO-World: YOLOv8 + frozen CLIP text encoder + RepVL-PAN. Scaled cosine head (174.7), max-sigmoid gate (174.8), 27-token image-pooling attention (174.9), box losses off for pseudo-labels (174.10).
- Prompt, then detect: the vocabulary becomes \(1 \times 1\) convolution weights (174.12), so the deployed model is a plain CNN: 35.4 zero-shot LVIS AP at 52.0 FPS on a V100.
- Detectors beat VLMs on speed (one pass vs 22 tokens per box), dense calibrated boxes, and small objects (stride 8 vs 28 px per token). VLMs win on reasoning and long queries; combine them.
Key formulas
References for this session
- Li et al. (2021). Grounded Language-Image Pre-training
- Cheng et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection
- Liu et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- Li et al. (2021). Grounded Language-Image Pre-training
- Cheng et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection
- Liu et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers
- Lin et al. (2017). Focal Loss for Dense Object Detection
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Bai et al. (2025). Qwen2.5-VL Technical Report