SigLIP and SigLIP 2
SigLIP keeps CLIP’s two towers but changes the loss: every image–caption pair in the batch becomes its own yes/no question, scored with a sigmoid. There is no softmax over the batch, so the loss is cheaper to split across devices and works better with small batches. SigLIP 2 adds a captioning decoder, self-distillation and masked prediction, many languages and a native-aspect-ratio variant, and it is the image encoder inside many open VLMs.
What you will learn
- The sigmoid loss: one binary classification per image–caption pair, its gradient, and how it differs from CLIP’s softmax
- Why SigLIP needs a learned bias, and why −10 is close to the best start −ln(N − 1)
- The chunked ring implementation: only a b × b block of logits in memory, with collective permutes instead of all-gathers
- What the batch-size study found: sigmoid wins below 16k, both saturate around 32k
- Sigmoid scores at inference: multi-label answers and “none of these”
- SigLIP 2: LocCa decoder, SILC and TIPS losses, multilingual data, ACID distillation and NaFlex resizing
- Why PaliGemma, LLaVA-OneVision, Idefics2, SmolVLM and Gemma 3 use SigLIP-shaped encoders, and how resolution sets the visual token count
Why this matters
CLIP (session 166) taught us to put images and captions into one shared space with a softmax over the whole batch. SigLIP (Sigmoid Loss for Language–Image Pre-training, Zhai et al., 2023) changes one thing: the loss. Instead of asking “which of these \(N\) captions belongs to this image?”, it asks, for every image–caption pair separately, “do these two belong together, yes or no?”. That small change removes the batch-wide normalization, makes the loss cheaper to compute across many GPUs, and works better when the batch is small.
SigLIP 2 (Tschannen et al., 2025) keeps the same architecture and the same sigmoid loss, and adds three training signals that CLIP-style models lacked: a small captioning decoder that also predicts boxes, self-distillation and masked prediction for better per-patch features, and data curation for the small models. It is trained on many languages and comes in a variant that keeps the image’s own aspect ratio (NaFlex).
Why should an LLM engineer care? Because the image encoder inside many open vision-language models (VLMs) is a SigLIP-family model. PaliGemma uses SigLIP So400m/14 [Certain]; LLaVA-OneVision, Idefics2 and SmolVLM use SigLIP-shaped encoders, and Gemma 3 uses a 400M SigLIP-style encoder at 896 pixels (their config.json files, checked below) [Certain]. When you choose, fine-tune or serve a VLM (session 182, session 177), you are choosing a SigLIP checkpoint, a resolution and a patch size, and that choice fixes the number of visual tokens your language model must read.
From “pick one” to “yes or no”
Start from the CLIP setup. A batch has \(N\) pairs \((x_i, y_i)\): image \(x_i\) and its caption \(y_i\). Each tower produces a vector, and the vectors are scaled to length 1 (L2 normalization):
- Read the symbols
- \(f, g\)
- the image tower (a ViT) and the text tower (a transformer), each ending in a pooling head
- \(\mathbf{u}_i^\top \mathbf{v}_j\)
- the cosine similarity of image \(i\) and caption \(j\), a number between −1 and 1
- \(t = e^{t'}\)
- the temperature scale; the model learns \(t'\), so \(t\) stays positive. SigLIP starts at \(t = 10\)
- \(b\)
- a learned bias, one number for the whole model. This is new compared with CLIP. SigLIP starts at \(b = -10\)
- \(\ell_{ij}\)
- the logit of the pair: a large positive value means “these match”
CLIP turns each row (and each column) of the \(N \times N\) logit matrix into a softmax: “among these \(N\) captions, pick the one for image \(i\)”. The probability of the right caption depends on all the other captions in the batch, because they share the denominator.
SigLIP treats each cell as its own binary classification. The label is \(z_{ij} = +1\) when \(i = j\) (a true pair) and \(z_{ij} = -1\) otherwise. The probability that a pair matches is the sigmoid \(\sigma(\ell) = 1/(1 + e^{-\ell})\), and the loss is the usual logistic (binary cross-entropy) loss, summed over all \(N^2\) cells and divided by \(N\):
- Read the symbols
- \(z_{ij}\)
- +1 for the \(N\) true pairs on the diagonal, −1 for the \(N^2 - N\) other cells (the negatives)
- \(\sigma(z\ell)\)
- the probability the model gives to the correct answer for this cell: \(\sigma(\ell)\) for a true pair, \(1 - \sigma(\ell) = \sigma(-\ell)\) for a negative
- \(1/N\)
- the sum is divided by the batch size, not by \(N^2\), as in the paper
Two things follow at once. First, there is no denominator that couples the cells: each term \(\log\sigma(z_{ij}\ell_{ij})\) depends on one image and one caption only. Second, the loss is symmetric by construction. CLIP must average an image-to-text softmax and a text-to-image softmax; SigLIP has one term per pair, and it is the same in both directions.
Figure 1 shows both losses on the same small batch. Click any cell and move its cosine: watch which loss terms change.
How to read Figure 1
Click a cell to select it, then move the third slider. In SigLIP only that one cell’s term changes, so “terms changed” is always 1. In CLIP, the cell sits inside the softmax of its row (image → text) and of its column (text → image), so two of the \(2N = 8\) CLIP terms change, and if the selected cell is a negative, the right answers of its row and column become less likely. Now try the bias slider at \(t = 20\): with \(b = 0\) most negatives get \(\sigma \gt 0.5\), and the SigLIP loss is large; around \(b = -4\) the negatives are near 0 and the true pairs near 1. The dog/puppy cells stay uncertain: they are real near-duplicates that the batch labels as negatives.
Use the cosine matrix from the CLIP session (dog, cat, car): rows are images, columns captions, \(\begin{pmatrix} 0.30 & 0.10 & 0.05 \\ 0.12 & 0.28 & 0.15 \\ 0.02 & 0.20 & 0.25 \end{pmatrix}\), with \(t = 10\) and \(b = -2\).
Logits \(\ell = 10\cos - 2\): the diagonal is \(1.0, 0.8, 0.5\); the off-diagonal cells are \(-1.0, -1.5, -0.8, -0.5, -1.8, 0.0\).
Diagonal terms \(\log(1 + e^{-\ell})\): \(0.313, 0.371, 0.474\). Negative terms \(\log(1 + e^{\ell})\): \(0.313, 0.201, 0.371, 0.474, 0.153, 0.693\). The worst cell is image 3 with caption 2 (logit 0: the model says “50% match” to a wrong pair).
Sum of all nine terms: \(3.364\); divided by \(N = 3\): \(\mathcal{L}_{\text{SigLIP}} = 1.121\). (CLIP’s loss with scale 10 on the same matrix is 0.368. The two numbers are on different scales and cannot be compared directly: SigLIP sums \(N^2\) terms, CLIP averages \(2N\).)
Figure 2 runs both losses on this worked example, one term at a time. Watch the size of each step: a SigLIP step reads one cell, a CLIP step reads a whole row or column.
- \(\ell_{ij} \leftarrow t \cos_{ij} + b\) for all \(N^2\) cells // formula (167.1)
- \(\text{acc} \leftarrow 0\)
- for \(i = 1..N\), \(j = 1..N\):
- \(z \leftarrow +1\) if \(i = j\), else \(-1\)
- \(\text{acc} \leftarrow \text{acc} + \log(1 + e^{-z \ell_{ij}})\) // this cell only
- return \(\text{acc} / N\) // formula (167.2)
- \(\ell_{ij} \leftarrow t \cos_{ij}\) // formula (166.2); a shared b would cancel
- for \(i = 1..N\): // image → text
- \(\text{acc} \leftarrow \text{acc} + \operatorname{lse}(\ell_{i,1..N}) - \ell_{ii}\) // whole row
- for \(j = 1..N\): // text → image
- \(\text{acc} \leftarrow \text{acc} + \operatorname{lse}(\ell_{1..N,j}) - \ell_{jj}\) // whole column
- return \(\text{acc} / (2N)\) // formula (166.5)
How to read Figure 2
Press Play, or Step through by hand. With SigLIP on the 3 × 3 example (\(t = 10\), \(b = -2\)), the first cell gives \(\ell_{11} = 1.00\) and the term \(\log(1 + e^{-1}) = 0.313\). The tallest block in the bar comes from cell (3, 2): \(\ell = 0.00\), term \(\ln 2 = 0.693\). The sum of the nine blocks is 3.364, and the last step divides by \(N = 3\): 1.121, the number of the worked example. Now switch to CLIP. Row 3 needs all three logits (0.20, 2.00, 2.50) at once: \(\operatorname{lse} = 3.035\), term \(0.535\), \(P_{33} = 0.586\). The six terms add to 2.210 and the loss is 0.368. Move the bias slider: the SigLIP bar changes, the CLIP numbers do not. Switch to 4 × 4 to see the dog/puppy hard negatives.
What the gradient does
For one cell, the derivative of \(\log(1 + e^{-z\ell})\) with respect to \(\ell\) is simple. Write the 0/1 label \(y_{ij} = (1 + z_{ij})/2\), so \(y = 1\) on the diagonal and 0 elsewhere:
The temperature is learned through \(t'\), with \(t = e^{t'}\), so \(\partial \ell_{ij} / \partial t' = t \cos_{ij}\). Its gradient is the same error, weighted by each cell’s cosine:
- Read the symbols
- \(\sigma(\ell_{ij}) - y_{ij}\)
- the error of one cell: positive when a negative is too likely, negative when a true pair is too unlikely
- \(\mathbf{u}_i^\top \mathbf{v}_j\)
- the cosine of the cell. A larger \(t\) stretches the logits away from \(b\): it helps when true pairs have larger cosines than negatives
“Predicted probability minus label”: the same form as logistic regression. A true pair is pulled up with force \(1 - \sigma(\ell_{ii})\); a negative is pushed down with force \(\sigma(\ell_{ij})\). Compare CLIP’s gradient \(\tfrac{1}{2N}(P_{ij} + Q_{ij} - 2\delta_{ij})\) (formula 166.6): there \(P_{ij}\) is a softmax over the row, so the push on one negative depends on every other caption in the row. In SigLIP the push depends on that cell alone. An easy negative (\(\sigma \approx 0\)) gets almost no gradient in both losses; a hard negative gets a large one in both. In the worked example, the cell with logit 0 gets the largest push: \((0.5 - 0)/3 = 0.167\).
Probabilities that do not have to sum to 1
The two losses produce different kinds of numbers at test time. With CLIP, zero-shot classification is a softmax over the class prompts (formula 166.9): the probabilities sum to 1, so the model must pick one class even when none fits. With SigLIP, \(\sigma(t\, \mathbf{u}^\top \mathbf{v}_k + b)\) is a separate “yes” probability for each prompt \(k\). Several prompts can be likely at once (a photo with a dog and a ball), and all of them can be unlikely (“none of these”). This is what you want for multi-label tagging, for filtering, and for detection-style scoring where each region–text pair needs its own score.
How to read Figure 3
With all five captions, both methods rank the dog captions first. Now untick the two dog captions. The softmax still gives about 0.67 to “a cat on a sofa”, because its three probabilities must sum to 1. The sigmoid scores do not move: the cat caption stays at 0.12, below 0.5, so the model is saying “no match here”. The bias sets the threshold: \(\sigma(\ell) = 0.5\) exactly when \(t\cos = -b\), that is at cosine \(-b/t = 0.25\) with the default sliders.
A practical warning: the sigmoid score is only as calibrated as the training data. A threshold of 0.5 is a good start, but tune it on your own data before you use it for filtering [Likely].
The bias term and the start of training
In a batch of \(N\) pairs there are \(N\) positives and \(N^2 - N\) negatives. With \(N = 16{,}384\) that is 268 million negatives for 16 thousand positives [Certain, Section 4.8 of the paper]. At the start of training the embeddings are random, so all cosines are close to 0 and every logit is close to \(b\). Put \(\ell_{ij} = b\) into (167.2), using \(\operatorname{sp}(a) = \log(1 + e^{a})\) (the softplus):
- Read the symbols
- \(\mathcal{L}_0(b)\)
- the loss at initialization (all cosines 0), per row: one positive term and \(N - 1\) negative terms
- \(b^\star\)
- the bias that is best before the model has learned anything: it says “a random pair matches with probability \(1/N\)”, the true base rate
With \(N = 32{,}768\) (SigLIP’s usual batch): \(b^\star = -\ln 32{,}767 = -10.40\). The paper’s choice \(b = -10\) is very close to this prior.
At \(b = -10\): \(\sigma(-10) = 4.54 \times 10^{-5}\). The positive term is \(\operatorname{sp}(10) = 10.00\), the negatives add \(32{,}767 \times 4.54\times10^{-5} = 1.49\), so \(\mathcal{L}_0 = 11.49\), and the gradient on \(b\) is only \(+0.49\).
At \(b = 0\): every term is \(\ln 2 = 0.693\), so \(\mathcal{L}_0 = 32{,}768 \times 0.693 = 22{,}713\), and the gradient on \(b\) is \(+16{,}383\). Almost all of it comes from the negatives. The first optimizer steps would spend their effort pushing every pair apart (correcting the base rate) instead of learning which pairs match. That is the “massive over-correction” the paper avoids with \(b = -10\).
How to read Figure 4
The green curve (one positive per row) wants \(b\) large; the orange curve (\(N - 1\) negatives) wants \(b\) very negative. Their sum has its minimum at \(b^\star\), which moves left by \(\ln 2 = 0.69\) every time you double the batch. Note that the temperature does not appear: at initialization the cosines are near 0, so \(t\) multiplies almost nothing. At \(b = 0\) the loss and the gradient on \(b\) are thousands of times larger than at \(b^\star\).
The paper’s ablation (base model, batch 8k, 900M examples) shows that this matters [Certain, their Table 4]. Zero-shot ImageNet accuracy was 63.0% with \(b = -10\), \(t' = \log 10\); 62.0% with no bias at all; 61.7% with \(b = 0\); and only 53.7% with \(b = 0\) together with \(t' = \log 1\) (\(t = 1\)).
Figure 5 trains only these two numbers. The cosines are frozen (the towers do not learn), so we can watch what the loss asks of \(t\) and \(b\) alone. Each Adam step runs the four lines of the algorithm: logits (167.1), loss (167.2), gradients (167.3, 167.4) and the update. Adam uses \(\beta_2 = 0.95\), the paper’s setting.
- \(t \leftarrow e^{t'}\); \(\ell_{ij} \leftarrow t \cos_{ij} + b\) // formula (167.1)
- \(\mathcal{L} \leftarrow \frac{1}{N} \sum_{i,j} \log(1 + e^{-z_{ij} \ell_{ij}})\) // formula (167.2)
- \(g_b \leftarrow \frac1N \sum (\sigma(\ell_{ij}) - y_{ij})\); \(g_{t'} \leftarrow \frac{t}{N} \sum (\sigma(\ell_{ij}) - y_{ij}) \cos_{ij}\) // (167.3), (167.4)
- Adam step on \(b\) and \(t'\) with \(\beta_1 = 0.9\), \(\beta_2 = 0.95\), learning rate \(\eta\)
How to read Figure 5
Press Play with the defaults (\(N = 16\), start \(b = -10\), \(t = 10\)). At step 0 all the dark cells are on the diagonal: every true pair has a logit near \(-7\), so the loss is 7.182 and almost all of it is the positive part. \(\partial \mathcal{L} / \partial b = -0.998\): the bias goes up and \(t\) grows. After 25 steps the loss is 0.770 and \(t = 48.9\); after 200 steps it is 0.405, with \(t = 55.1\) and \(b = -12.0\). Now set the start bias to 0 and press Reset and Play. The start loss is 14.65 and the dark cells are the negatives. The first steps push \(b\) down and even shrink \(t\) (to 2.15 at step 25), because that pushes all logits toward \(b\). After 50 steps the loss is 2.768, against 0.501 with \(b = -10\). Choose \(N = 32\): with \(b = 0\) the start loss doubles to 29.88, and after 50 steps it is still 4.072. A bigger batch has more negatives per row, so a bad start bias costs more steps.
No softmax over the batch: the chunked implementation
Contrastive training uses data parallelism: the global batch \(|B|\) is split over \(D\) devices, each holding \(b_d = |B|/D\) pairs. (We write \(b_d\) to keep it apart from the bias \(b\).) To compute CLIP’s softmax for one image, a device needs the logits against all \(|B|\) captions, because the normalizer is a sum over the whole row. The usual way is to all-gather every embedding onto every device. A naive implementation then builds the full \(|B| \times |B|\) matrix; a careful one builds \(b_d \times |B|\) logits per device.
The sigmoid loss is a plain sum of independent cell terms, so it can be cut into \(D \times D\) blocks and summed in any order:
The outer sum runs on each device in parallel (sum over \(d_i\)). The middle sum is a ring: at each step every device passes its captions to its neighbour (a collective permute), computes the \(b_d \times b_d\) block of logits against the captions it holds now, adds the block’s loss, and throws the block away. After \(D\) steps every image has met every caption, and one final cross-device sum gives the loss. At no time does a device hold more than \(b_d \times b_d\) logits. The paper reports that \(D\) collective permutes are typically faster than two all-gathers [Certain, Section 3.3].
How to read Figure 6
Step 1: each device holds its own images and captions. Step 2: each device computes its diagonal block, which contains its positives. Steps 3 and 4: captions move one device along the ring, and each device computes a block of pure negatives. The bright block is the only thing in memory; finished blocks are faded. The last step adds the three per-device sums. With the default real numbers, \(|B| = 32{,}768\) on 256 devices, each block is \(128 \times 128\): 65.5 kB, compared with 16.8 MB for one device’s \(128 \times 32{,}768\) slice and 4.29 GB for the full matrix.
- input: image embeddings \(F \in \R^{n \times d}\), text embeddings \(G \in \R^{n \times d}\), parameters \(t', b\)
- \(U \leftarrow\) normalize rows of \(F\); \(\;V \leftarrow\) normalize rows of \(G\)
- \(L \leftarrow e^{t'}\, U V^\top + b\) // n × n logits, formula (167.1)
- \(Z \leftarrow 2I - \mathbf{1}\mathbf{1}^\top\) // \(+1\) on the diagonal, \(-1\) elsewhere
- return \(-\tfrac{1}{n} \sum_{i,j} \log\sigma(Z_{ij} L_{ij})\) // formula (167.2); use a stable log-sigmoid
- input: local images \(U_k\) and captions \(V_k\) (\(b_d\) each), \(t, b\)
- \(\text{acc} \leftarrow 0\); \(\;V \leftarrow V_k\)
- for step \(s = 0, \dots, D - 1\):
- \(L \leftarrow t\, U_k V^\top + b\) // only \(b_d \times b_d\) in memory
- labels: \(Z \leftarrow 2I - \mathbf{1}\mathbf{1}^\top\) if \(s = 0\) (own captions), else \(Z \leftarrow -\mathbf{1}\mathbf{1}^\top\)
- \(\text{acc} \leftarrow \text{acc} - \sum \log\sigma(Z \odot L)\)
- \(V \leftarrow\) captions received from the neighbour device // collective permute
- return \(\tfrac{1}{|B|}\,\text{all-reduce-sum}(\text{acc})\) // formula (167.6)
Does SigLIP still need a huge batch?
CLIP used a batch of 32,768, and the CLIP session showed why softmax training likes big batches: the bound \(I \ge \ln N - \mathcal{L}\) (formula 166.8) cannot certify more than \(\ln N\) nats of shared information. Removing the batch-wide softmax lets us ask the question directly. The SigLIP paper trained with batches from 512 up to one million and found [Certain, Sections 4.1 and 4.2]:
- Small batches: below 16k, the sigmoid loss is clearly better than the softmax loss.
- Saturation: both losses saturate. SigLIP was best at batch 32k; the softmax baseline needed 98k for its best result and still did not beat it. A batch of 307k hurt both. In the frozen-image setup (SigLiT), performance saturated around 32k, and the model peaked at 256k with only a small gain.
- Many languages: for multilingual SigLIP (100+ languages, 30 billion examples), ImageNet zero-shot was 71.6%, 73.2%, 73.2%, 73.2% and 73.1% for batches 16k, 32k, 64k, 128k and 240k; average cross-lingual retrieval got worse above 32k (34.9% at 32k, 32.7% at 240k).
- Cheap training: because the loss needs less memory, four TPUv4 chips could fit batch 4,096 for a base SigLIP but only 2,048 for the CLIP version. With a frozen ViT-g image tower, SigLiT reached 84.5% ImageNet zero-shot on four TPUv4 chips in two days; SigLIP from scratch reached 73.4% with 32 TPUv4 chips in five days.
Figure 7 shows what changes for one image when the batch grows. We keep one image with its true caption (cosine 0.30) and add captions from a fixed pool as the batch doubles from 2 to 4,096. The scale \(t\) and the bias \(b\) stay fixed, as if we froze a trained model and only changed the batch.
- \(N \leftarrow 2N\): add \(N/2\) new captions; image \(i\) now has \(N - 1\) negatives
- CLIP: \(P_{ii} \leftarrow e^{\ell_{ii}} / \sum_{j} e^{\ell_{ij}}\); term \(\leftarrow -\log P_{ii}\) // one shared denominator
- SigLIP: term \(\leftarrow \log(1 + e^{-\ell_{ii}}) + \sum_{j \ne i} \log(1 + e^{\ell_{ij}})\) // N separate cells
How to read Figure 7
Press Play with \(t = 30\), \(b = -6\). CLIP: the true caption’s probability falls from 0.996 at \(N = 2\) to 0.104 at \(N = 4{,}096\), and its row term grows from 0.004 to 2.263. Nothing about this image changed; only the denominator got more terms. SigLIP: \(\sigma(\ell_{ii}) = 0.953\) at every batch size, because the true pair’s score never sees the other captions. What grows is the number of negative terms: their sum goes from 0.074 to 140.0, roughly doubling with \(N\). That is the base-rate effect of formula (167.5). Switch to \(b = -\ln(N-1)\): the negatives’ sum now grows only from 3.47 to 16.36, but the true pair’s score falls to 0.664. In real training the model also moves \(t\), \(b\) and the embeddings, so this is only the first-order picture.
Two training details from the paper are worth knowing. Large batches made training unstable, with spikes in the gradient norm; lowering Adam’s \(\beta_2\) from 0.999 to 0.95 fixed it (the same reason LLMs use 0.95, see session 4). And when the image tower starts from a pretrained checkpoint, turning off weight decay on the pretrained weights was needed for good results [Certain, Sections 4.5 and 4.7].
The paper also tested the imbalance between positives and negatives by masking negatives. Removing negatives at random hurt; keeping only the easiest ones did not work at all; keeping only the hardest ones almost kept the quality. So the imbalance itself is not a problem, and most of what the model learns from negatives comes from the hard ones [Certain, Section 4.8]. Finally, with corrupted images, corrupted texts or shuffled pairs, models trained with the sigmoid loss stayed better than softmax models as the corruption grew [Certain, Section 4.10]. A plausible reason: a wrong pair only spoils its own term, not a whole softmax row [Speculation].
The released SigLIP models
For the public models, SigLIP was trained for 40 billion examples at batch 32k, with 64 text tokens, then adapted to each resolution with 5 billion more examples at a 100 times smaller learning rate [Certain]. Vision and text representations are pooled with a MAP head (multihead attention pooling): a small attention layer with one learned query that attends over all patch tokens and returns one vector. It replaces the CLS token. The most used size is So400m, a “shape-optimized” ViT (Alabdulmohsin et al., 2023) whose width, depth and MLP size were chosen by a scaling study instead of copying ViT-L or ViT-H. Its config.json on Hugging Face: 27 layers, width 1,152, MLP width 4,304, 16 heads, patch 14 [Certain]. The 27 blocks alone hold 411 million weights (our count from those shapes).
The number of visual tokens is fixed by the resolution \(R\) and the patch size \(p\):
| Model (ImageNet zero-shot, SigLIP paper Table 3) | Patch tokens | ImageNet 0-shot | COCO image → text R@1 |
|---|---|---|---|
| CLIP B/16 (OpenAI) | 196 | 68.3 | 52.4 |
| SigLIP B/16 at 224 | 196 | 76.2 | 64.4 |
| SigLIP B/16 at 512 | 1,024 | 79.2 | 67.6 |
| CLIP L/14 at 336 (OpenAI) | 576 | 76.6 | 57.9 |
| SigLIP L/16 at 384 | 576 | 82.1 | 70.6 |
| SigLIP So400m/14 at 384 | 729 | 83.2 | 70.2 |
These SigLIP models were trained on WebLI, Google’s web image–text dataset, while CLIP used its own 400M pairs, so the table mixes the effect of the loss with the effect of data and training length [Certain]. The loss alone is measured by the batch-size study above, where both losses saw the same data.
SigLIP 2: same towers, more training signals
CLIP-style training has two known weaknesses. The pooled vector is good for “what is in this image” but the per-patch features are weak for “where is it” (localization, segmentation, depth). And the models are mostly English. SigLIP 2 keeps the SigLIP architecture (so you can swap the weights and the tokenizer into existing code) and fixes both with a staged recipe [Certain, SigLIP 2 Section 2]:
- Sigmoid loss + LocCa decoder (whole run). A small transformer decoder reads the un-pooled patch tokens through cross-attention and learns three tasks: write the caption, predict the box for a region description (referring expressions), and write a caption for a given box (grounded captioning). The region–caption pairs are made automatically with an open-vocabulary detector. The decoder has half the layers of the text tower, and it is thrown away after training. The two losses get equal weight.
- Self-distillation and masked prediction (last 20%). At 80% of training, an EMA teacher (an exponential moving average of the student’s weights, as in DINO, see session 165) is added. SILC loss: the student sees 8 small local crops and must match the teacher’s features of the full image. TIPS loss: 50% of the student’s patches are replaced by a mask token and the student must match the teacher’s per-patch features at those places.
- Resolution variants. Fixed-resolution checkpoints branch off at 95% and finish at their target resolution. The NaFlex variant branches off at 90% (next section).
- Small models: distillation through data. B/16 and B/32 get 4 billion more examples with the ACID method: a strong teacher and the student score a super-batch of 64k examples by “learnability”, and the best 32k form the training batch.
- Read the symbols
- \(\mathcal{L}_{\text{LocCa}}\)
- next-token loss of the decoder on captions, referring expressions (box coordinates as text) and grounded captions
- \(\mathcal{L}_{\text{SILC}}\)
- local-to-global self-distillation on pooled features
- \(\mathcal{L}_{\text{TIPS}}\)
- masked prediction on per-patch features
- \(w_{\text{size}}\)
- 0.25, 0.5, 1.0 and 0.5 for the B, L, So400m and g sizes: a balance between global and dense quality
Other settings [Certain]: WebLI with 10 billion images and 12 billion alt-texts in 109 languages, mixed 90% English and 10% other languages, with de-biasing filters; the multilingual Gemma tokenizer (256k vocabulary), text lower-cased, 64 tokens; Adam with learning rate \(10^{-3}\), decoupled weight decay \(10^{-4}\), gradient clipping at 1, batch 32k, cosine schedule with 20k warmup steps, 40 billion examples, up to 2,048 TPUv5e chips with FSDP (session 46). Sizes: ViT-B (86M), L (303M), So400m (400M) and g (1B).
How to read Figure 8
From 0 to 80% only the sigmoid loss and the LocCa decoder train the image tower. At 80% the EMA teacher and the two self-supervised losses switch on, with the weight that depends on the model size (try B versus So400m). At 90% the NaFlex variant leaves the main run; at 95% the fixed-resolution variants leave. The decoder and the teacher are only training tools: the released checkpoint is just the two towers.
- for each step, with progress \(p\) from 0 to 1:
- embed images and captions; \(\mathcal{L} \leftarrow \mathcal{L}_{\text{sigmoid}}\) // formula (167.2)
- run the decoder on the un-pooled patch tokens; \(\mathcal{L} \leftarrow \mathcal{L} + \mathcal{L}_{\text{LocCa}}\) // caption, referring expression, grounded caption
- if \(p \ge 0.8\): build 1 global and 8 local augmented views; teacher \(\leftarrow\) EMA of student
- \(\mathcal{L} \leftarrow \mathcal{L} + w_{\text{size}}(\mathcal{L}_{\text{SILC}} + 0.25\,\mathcal{L}_{\text{TIPS}})\) // formula (167.8); the image–text losses use the original image
- Adam step with clipping at 1, cosine learning rate
- if \(p = 0.90\): fork the NaFlex run; if \(p = 0.95\): fork one run per fixed resolution
- for B/16 and B/32 only: 4B more examples with sigmoid loss, lr \(10^{-5}\), no weight decay, batches picked by ACID from 64k candidates
What did it buy? A few numbers from the SigLIP 2 paper [Certain, Tables 1–3]:
| Task | Model | SigLIP | SigLIP 2 |
|---|---|---|---|
| ImageNet zero-shot (%) | B/16 at 224 | 76.2 | 78.2 |
| ImageNet zero-shot (%) | So400m/14 at 384 | 83.2 | 84.1 |
| XM3600 text → image R@1 (%, 36 languages) | So400m/14 at 384 | 17.8 | 48.4 |
| PASCAL segmentation, frozen features (mIoU) | So400m/14 at 224 | 72.0 | 77.1 |
| ADE20k segmentation, frozen features (mIoU) | So400m/14 at 224 | 37.6 | 41.8 |
| NYUv2 depth, frozen features (RMSE, lower is better) | So400m/14 at 224 | 0.576 | 0.493 |
| Open-vocabulary segmentation ADE-150 with Cat-Seg (mIoU) | L/16 | 37.5 | 38.8 |
The global numbers move by 1–2 points; the multilingual and dense numbers move a lot. That matches the design: the new losses target per-patch features and the data targets languages. The paper also reports that SigLIP 2 beats SigLIP and AIMv2 as the frozen encoder of a PaliGemma-style VLM, and improves referring-expression and open-vocabulary detection results [Certain, Sections 3.2–3.4].
NaFlex: native aspect ratio, several sequence lengths
A fixed-resolution model squashes every image to a square, for example 256 × 256. A wide receipt or a tall phone screenshot gets distorted, and small text becomes unreadable. NaFlex combines two older ideas: NaViT (keep the native aspect ratio, see session 180) and FlexiViT (one model for several sequence lengths). Given a patch size \(p\) and a token budget \(S_{\max}\), it resizes the image by one scale factor \(s\) and rounds each side up to a multiple of \(p\):
- Read the symbols
- \(H, W\)
- the original height and width in pixels
- \(H', W'\)
- the resized sides, multiples of the patch size \(p = 16\)
- \(S_{\max}\)
- the token budget; NaFlex was trained with \(S_{\max} \in \{128, 256, 576, 784, 1024\}\), sampled per mini-batch
Because of the rounding, the aspect ratio changes a little: by at most \((p-1)/W'\) in width and \((p-1)/H'\) in height. Unused token slots are padding, and a mask hides them from attention and from the MAP head. The learned position embedding is a \(16 \times 16\) grid, resized bilinearly to the actual patch grid, for example \(21 \times 12\) [Certain, SigLIP 2 Section 2.4.2]. The Hugging Face processor finds \(s\) with a binary search; Figure 9 runs the same search.
How to read Figure 9
A 1920 × 1080 frame with budget 256 becomes 336 × 192 pixels: a 21 × 12 grid, 252 tokens, 4 padding slots. The aspect ratio goes from 1.778 to 1.750. The square model would use 16 × 16 and stretch the image vertically by 1.78. Try a long strip (2400 × 200): NaFlex gives it a wide, short grid, while the square model squeezes it horizontally by a factor of 12. The paper found NaFlex better on OCR, document and screen retrieval, especially at small budgets, while the square B-sized model was slightly better on natural-image benchmarks [Certain, SigLIP 2 Section 3.1.1].
- input: image \(H \times W\), patch size \(p\), budget \(S_{\max}\)
- binary search the largest scale \(s\) with \(\lceil sH/p\rceil \cdot \lceil sW/p\rceil \le S_{\max}\) // formula (167.9)
- resize to \(H' \times W'\); cut into \(n = H'W'/p^2\) patches, row by row
- resize the \(16 \times 16\) position-embedding grid bilinearly to \(H'/p \times W'/p\)
- pad the patch sequence to \(S_{\max}\); mask \(\leftarrow\) 1 for real patches, 0 for padding
- run the ViT and the MAP head with the mask in every attention layer
The PyTorch version
The code builds the loss of Algorithm 1, checks it against binary cross-entropy and against the gradient formula (167.3), checks that the ring version of Algorithm 2 gives the same number, prints the start-of-training loss for both biases, and trains a tiny SigLIP on toy pairs. It needs only PyTorch and runs in a few seconds on a CPU.
import math, torch import torch.nn.functional as F torch.manual_seed(0) # 1) the sigmoid loss, formula (167.2): every image-text pair is one yes/no question def siglip_loss(img_emb, txt_emb, t_prime, b): x = F.normalize(img_emb, dim=-1) # (n, d) unit vectors y = F.normalize(txt_emb, dim=-1) logits = t_prime.exp() * x @ y.T + b # (n, n) scores n = logits.shape[0] z = 2 * torch.eye(n) - 1 # +1 on the diagonal, -1 elsewhere return -F.logsigmoid(z * logits).sum() / n, logits n, d = 8, 32 img, txt = torch.randn(n, d), torch.randn(n, d) t_prime = torch.tensor(math.log(10.0), requires_grad=True) # t = 10 at start b = torch.tensor(-10.0, requires_grad=True) # b = -10 at start loss, logits = siglip_loss(img, txt, t_prime, b) # 2) the same loss as plain binary cross-entropy with 0/1 labels bce = F.binary_cross_entropy_with_logits(logits, torch.eye(n), reduction="sum") / n print(f"loss {loss.item():.4f} as BCE {bce.item():.4f}") # identical # 3) gradient on each logit, formula (167.3): (sigmoid(l) - label) / n logits.retain_grad(); loss.backward() expected = (torch.sigmoid(logits) - torch.eye(n)) / n print("grad matches:", torch.allclose(logits.grad, expected, atol=1e-7)) print(f"dL/db = {b.grad.item():.4f} (n*sigmoid(b) - 1 when all cosines are 0: {n*torch.sigmoid(b).item()-1:.4f})") # 4) the chunked "ring" version, formula (167.6): D devices, only b x b logits at a time def siglip_chunked(img_emb, txt_emb, t, bias, D): x, y = F.normalize(img_emb, dim=-1), F.normalize(txt_emb, dim=-1) xs, ys = x.chunk(D), y.chunk(D) # device k holds xs[k], ys[k] total = 0.0 for k in range(D): # (runs in parallel on real devices) for step in range(D): # texts move one device per step j = (k + step) % D l = t * xs[k] @ ys[j].T + bias # only (n/D) x (n/D) in memory z = (2 * torch.eye(len(xs[k])) - 1) if j == k else -torch.ones_like(l) total = total - F.logsigmoid(z * l).sum() return total / len(img_emb) with torch.no_grad(): full, _ = siglip_loss(img, txt, t_prime, b) ring = siglip_chunked(img, txt, t_prime.exp(), b, D=4) print(f"full {full.item():.6f} chunked {ring.item():.6f}") # 5) why b starts at -10: loss at the start for a batch of 32,768 with cosines near 0 N = 32768 for b0 in [-10.0, 0.0]: s = torch.tensor(b0) start = F.softplus(-s) + (N - 1) * F.softplus(s) # 1 positive + (N-1) negatives per row print(f"b = {b0:5.1f}: start loss {start.item():10.2f}, best b = -ln(N-1) = {-math.log(N-1):.2f}") # 6) a tiny SigLIP on toy pairs: two linear towers, batch 16 torch.manual_seed(0) K, d_img, d_txt = 64, 20, 12 concepts = torch.randn(K, 8) A, Bm = torch.randn(8, d_img), torch.randn(8, d_txt) def batch(m): c = torch.randint(0, K, (m,)) return concepts[c] @ A + 0.3 * torch.randn(m, d_img), concepts[c] @ Bm + 0.3 * torch.randn(m, d_txt), c f, g = torch.nn.Linear(d_img, 16), torch.nn.Linear(d_txt, 16) tp, bb = torch.nn.Parameter(torch.tensor(math.log(10.0))), torch.nn.Parameter(torch.tensor(-10.0)) opt = torch.optim.AdamW([*f.parameters(), *g.parameters(), tp, bb], lr=1e-2, betas=(0.9, 0.95)) for step in range(1500): xi, yi, _ = batch(16) loss, _ = siglip_loss(f(xi), g(yi), tp, bb) opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): xi, yi, c = batch(256) sims = F.normalize(f(xi), dim=-1) @ F.normalize(g(yi), dim=-1).T acc = (c[sims.argmax(1)] == c).float().mean() # retrieved text has the same concept print(f"final loss {loss.item():.3f}, t = {tp.exp().item():.1f}, b = {bb.item():.2f}, retrieval accuracy: {acc.item():.2f}")
Output on CPU (we ran it twice; the output is identical): the loss is 10.8495 both ways; the gradient check prints True; \(dL/db = -0.9990\) (close to \(8\sigma(-10) - 1 = -0.9996\), because random cosines are not exactly 0); full and chunked losses agree (10.849527 and 10.849526, a float rounding difference); the start loss is 11.49 for \(b = -10\) and 22,713.05 for \(b = 0\), with \(b^\star = -10.40\); after 1,500 steps of batch 16 the loss is 0.244, \(t = 12.1\), \(b = -10.18\), and retrieval accuracy over 256 toy pairs is 1.00. A batch of 16 is far below what CLIP-style training likes, and the toy still learns; real data is much harder, so do not read more into this than “the code works”. In real code, the Hugging Face SiglipModel returns logits_per_image; apply torch.sigmoid to it, not a softmax.
Why open VLMs use a SigLIP-family encoder
A VLM does not use SigLIP’s final pooled vector. It takes the patch tokens from the last (or second-to-last) layer, passes them through a connector, and feeds them to the language model as “visual words” (session 178). What makes SigLIP a good choice for that job?
- Strong features per patch and per image. Image–text training gives features that already line up with words. SigLIP 2 adds the dense and localization signals that VLM tasks (OCR, grounding, counting) need.
- Open weights in many sizes and resolutions. B, L, So400m and g; 224 to 512 pixels; patch 14 or 16; and NaFlex. You choose the token count with formula (167.7).
- Good quality for its size. So400m is about the size of ViT-L but scored higher than much larger CLIP-style models in the SigLIP paper (83.2% vs 82.0% for a 5B EVA-CLIP, Table 3) [Certain].
- Cheaper to train and to continue training. The sigmoid loss works at smaller batches and needs little memory, which matters when a lab wants its own variant.
| VLM | Vision encoder | Input | Visual tokens |
|---|---|---|---|
| PaliGemma | SigLIP So400m/14 [Certain, PaliGemma paper] | 224 (also 448, 896) | 256 at 224 |
| LLaVA-OneVision | SigLIP So400m/14 shape, 26 of the 27 layers (num_hidden_layers: 26 in its config) [Certain] | 384 tiles | 729 per tile |
| Idefics2 | So400m/14 shape (1,152 wide, 27 layers) [Certain, config]; initialized from SigLIP [Likely] | up to 980 | 70 × 70 = 4,900 patches, then a resampler |
| SmolVLM (2.2B) | So400m/14 shape at 384 [Certain, config] | 384 tiles | 729, pixel shuffle by 3 → 81 (image_seq_len: 81) |
| SmolVLM-256M | SigLIP B/16 shape at 512 (12 layers, width 768) [Certain, config] | 512 tiles | 1,024, pixel shuffle by 4 → 64 |
| Gemma 3 | 400M SigLIP-style encoder, So400m/14 shape [Certain, config] | 896 | 64 × 64 = 4,096 patches, pooled to 256 (mm_tokens_per_image: 256) |
Notice how the token budget drives the design: an 896-pixel Gemma 3 image would be 4,096 tokens without pooling, so each VLM adds pooling, pixel shuffle or a resampler to cut the count (session 183). Other families use other encoders: Qwen2-VL and InternVL train their own ViTs (session 182).
Where this is used
| Application | How SigLIP is used |
|---|---|
| Vision encoder of VLMs | Patch tokens from a frozen or fine-tuned SigLIP tower feed the connector and the LLM (table above) |
| Multi-label tagging and filtering | One sigmoid score per tag prompt; keep every tag above a tuned threshold; “no tag” is a valid answer (Figure 3) |
| Multilingual image search | SigLIP 2 embeds queries in 100+ languages into the same space; XM3600 retrieval rose from 17.8 to 48.4 R@1 for So400m/384 |
| Document, screen and receipt retrieval | NaFlex keeps the aspect ratio, so small text survives (Figure 9) |
| Open-vocabulary detection and segmentation | OWL-ViT-style heads and Cat-Seg on SigLIP 2 patch features (session 168) |
| Text-to-video retrieval and video classification | Embed sampled frames, pool over time, score against text prompts (session 208, session 203); motion-heavy actions need a video encoder [Likely] |
| Key-moment detection | Sigmoid score of every frame against “a goal is scored”; the absolute score lets you threshold moments instead of only ranking frames (session 207) |
| Video fingerprinting and Video-RAG | Per-frame image embeddings as a robust fingerprint and as the retrieval index for video question answering (session 209, session 210) |
Common mistakes
- Softmax over SigLIP logits. Ranking is unchanged, but the probabilities lose their meaning, and “none of these” disappears. Use
torch.sigmoid(logits_per_image). - Wrong text preprocessing. SigLIP text towers were trained with captions padded to a fixed length (64 tokens for the released models), and SigLIP 2 lower-cases text. Hugging Face recommends
padding="max_length"for SigLIP [Likely]. Different padding gives silently worse embeddings. - Starting with \(b = 0\). The loss starts in the tens of thousands and the first steps only fix the base rate (Figure 4). Start near \(-\ln N\) (the paper uses −10).
- Dividing by \(N^2\) instead of \(N\). The loss becomes \(N\) times smaller and so do the gradients; with Adam the effect is partly hidden, with SGD the learning rate is effectively wrong.
- Thinking the sigmoid removes false negatives. Duplicates and near-duplicates in a batch are still labeled −1 (the dog/puppy cells in Figure 1). Deduplicate data, or accept the noise.
- Weight decay on a pretrained image tower. When you continue training SigLIP from pretrained weights, the paper needed to switch weight decay off for those weights.
- Mismatched resolution. Feeding 512-pixel images to a 224 checkpoint without resizing the position embeddings, or mixing up patch 14 and 16 checkpoints, changes the token count (formula 167.7) and the quality.
- Using the pooled vector for dense tasks. The MAP-head output is one vector per image. Segmentation, grounding and VLMs need the un-pooled patch tokens.
Interview questions
Write the SigLIP loss and explain each part.basic
What is the bias for, and why is it initialized to −10?core
Why is the sigmoid loss easier to distribute across devices than CLIP’s?core
Does SigLIP remove the need for large batches?core
Compare the gradients of the sigmoid and softmax losses with respect to one negative logit.advanced
When would you prefer sigmoid scores over softmax scores at inference?core
What does SigLIP 2 add to SigLIP, and what does each addition fix?core
Why do many open VLMs use a SigLIP encoder, and which outputs do they use?core
How does NaFlex handle different aspect ratios and sequence lengths in one model?advanced
Is the heavy positive/negative imbalance a problem for the sigmoid loss?advanced
You continue training SigLIP from a pretrained image tower and quality drops. What do you check?basic
Exercises
- A true pair has logit 2 and a negative pair has logit −1. Compute both loss terms of (167.2) and both gradients of (167.3) for \(N = 1\).
- For a batch of 4,096, what is the best starting bias \(b^\star\)? What is \(\sigma(b^\star)\)?
- Your global batch is 65,536 on 512 devices. How much memory does the full logit matrix need in FP32? How much does one block of the ring version need?
- How many patch tokens does SigLIP So400m/14 produce at 224 and at 384 pixels? What NaFlex grid does a 1000 × 200 image get with budget 256 (patch 16)? Use Figure 9 to check.
- In Figure 3 with \(t = 20\), which bias makes a caption with cosine 0.20 sit exactly at 50%?
- Explain in two sentences why gradient accumulation over micro-batches is closer to the real large-batch loss for SigLIP than for CLIP, and what is still different.
Answerscheck
Key takeaways
- SigLIP replaces CLIP’s batch softmax with one sigmoid yes/no question per image–caption pair (167.1, 167.2). The gradient is “probability minus label” per cell (167.3).
- A learned bias, started near \(-\ln N\) (the paper uses −10), matches the base rate of a match and avoids a huge loss at the start (167.5).
- Independent terms make the loss easy to split: a ring of collective permutes keeps only a \(b_d \times b_d\) block in memory (167.6).
- The sigmoid loss is clearly better below batch 16k; both losses saturate around 32k. Use \(\beta_2 = 0.95\) and no weight decay on pretrained towers.
- SigLIP 2 keeps the towers and adds a captioning/grounding decoder, self-distillation and masked prediction late in training, multilingual data and NaFlex (167.8, 167.9). The biggest gains are in dense features and in non-English retrieval.
- Many open VLMs (PaliGemma, LLaVA-OneVision, Idefics2, SmolVLM, Gemma 3) use SigLIP-shaped encoders; the resolution and patch size fix the visual token count (167.7).
Key formulas
References for this session
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Zhai et al. (2022). LiT: Zero-Shot Transfer with Locked-image text Tuning
- Alabdulmohsin, Zhai, Kolesnikov & Beyer (2023). Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
- Chen et al. (2022). PaLI: A Jointly-Scaled Multilingual Language-Image Model
- Wan et al. (2024). LocCa: Visual Pretraining with Location-aware Captioners
- Naeem et al. (2024). SILC: Improving Vision Language Pretraining with Self-Distillation
- Maninis et al. (2025). TIPS: Text-Image Pretraining with Spatial awareness
- Beyer et al. (2023). FlexiViT: One Model for All Patch Sizes
- Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- Udandarao et al. (2025). Active Data Curation Effectively Distills Large-Scale Multimodal Models
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer
- Laurençon et al. (2024). What matters when building vision-language models?
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Gemma Team (2025). Gemma 3 Technical Report
- Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers