LLM & VLM courseBlog
167Track 7: Vision and VLMs · session 5 of 38

SigLIP and SigLIP 2

SigLIP keeps CLIP’s two towers but changes the loss: every image–caption pair in the batch becomes its own yes/no question, scored with a sigmoid. There is no softmax over the batch, so the loss is cheaper to split across devices and works better with small batches. SigLIP 2 adds a captioning decoder, self-distillation and masked prediction, many languages and a native-aspect-ratio variant, and it is the image encoder inside many open VLMs.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 3 (cross-entropy), 164 (the Vision Transformer) and 166 (CLIP)

What you will learn

  • The sigmoid loss: one binary classification per image–caption pair, its gradient, and how it differs from CLIP’s softmax
  • Why SigLIP needs a learned bias, and why −10 is close to the best start −ln(N − 1)
  • The chunked ring implementation: only a b × b block of logits in memory, with collective permutes instead of all-gathers
  • What the batch-size study found: sigmoid wins below 16k, both saturate around 32k
  • Sigmoid scores at inference: multi-label answers and “none of these”
  • SigLIP 2: LocCa decoder, SILC and TIPS losses, multilingual data, ACID distillation and NaFlex resizing
  • Why PaliGemma, LLaVA-OneVision, Idefics2, SmolVLM and Gemma 3 use SigLIP-shaped encoders, and how resolution sets the visual token count

Why this matters

CLIP (session 166) taught us to put images and captions into one shared space with a softmax over the whole batch. SigLIP (Sigmoid Loss for Language–Image Pre-training, Zhai et al., 2023) changes one thing: the loss. Instead of asking “which of these \(N\) captions belongs to this image?”, it asks, for every image–caption pair separately, “do these two belong together, yes or no?”. That small change removes the batch-wide normalization, makes the loss cheaper to compute across many GPUs, and works better when the batch is small.

SigLIP 2 (Tschannen et al., 2025) keeps the same architecture and the same sigmoid loss, and adds three training signals that CLIP-style models lacked: a small captioning decoder that also predicts boxes, self-distillation and masked prediction for better per-patch features, and data curation for the small models. It is trained on many languages and comes in a variant that keeps the image’s own aspect ratio (NaFlex).

Why should an LLM engineer care? Because the image encoder inside many open vision-language models (VLMs) is a SigLIP-family model. PaliGemma uses SigLIP So400m/14 [Certain]; LLaVA-OneVision, Idefics2 and SmolVLM use SigLIP-shaped encoders, and Gemma 3 uses a 400M SigLIP-style encoder at 896 pixels (their config.json files, checked below) [Certain]. When you choose, fine-tune or serve a VLM (session 182, session 177), you are choosing a SigLIP checkpoint, a resolution and a patch size, and that choice fixes the number of visual tokens your language model must read.

From “pick one” to “yes or no”

Start from the CLIP setup. A batch has \(N\) pairs \((x_i, y_i)\): image \(x_i\) and its caption \(y_i\). Each tower produces a vector, and the vectors are scaled to length 1 (L2 normalization):

\[ \mathbf{u}_i = \frac{f(x_i)}{\lVert f(x_i) \rVert}, \qquad \mathbf{v}_j = \frac{g(y_j)}{\lVert g(y_j) \rVert}, \qquad \ell_{ij} = t\, \mathbf{u}_i^\top \mathbf{v}_j + b, \quad t = e^{t'} \tag{167.1} \]
Read the symbols
\(f, g\)
the image tower (a ViT) and the text tower (a transformer), each ending in a pooling head
\(\mathbf{u}_i^\top \mathbf{v}_j\)
the cosine similarity of image \(i\) and caption \(j\), a number between −1 and 1
\(t = e^{t'}\)
the temperature scale; the model learns \(t'\), so \(t\) stays positive. SigLIP starts at \(t = 10\)
\(b\)
a learned bias, one number for the whole model. This is new compared with CLIP. SigLIP starts at \(b = -10\)
\(\ell_{ij}\)
the logit of the pair: a large positive value means “these match”

CLIP turns each row (and each column) of the \(N \times N\) logit matrix into a softmax: “among these \(N\) captions, pick the one for image \(i\)”. The probability of the right caption depends on all the other captions in the batch, because they share the denominator.

SigLIP treats each cell as its own binary classification. The label is \(z_{ij} = +1\) when \(i = j\) (a true pair) and \(z_{ij} = -1\) otherwise. The probability that a pair matches is the sigmoid \(\sigma(\ell) = 1/(1 + e^{-\ell})\), and the loss is the usual logistic (binary cross-entropy) loss, summed over all \(N^2\) cells and divided by \(N\):

\[ \mathcal{L}_{\text{SigLIP}} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{N} \log \sigma\!\big(z_{ij}\, \ell_{ij}\big) = \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{N} \log\!\big(1 + e^{-z_{ij} \ell_{ij}}\big) \tag{167.2} \]
Read the symbols
\(z_{ij}\)
+1 for the \(N\) true pairs on the diagonal, −1 for the \(N^2 - N\) other cells (the negatives)
\(\sigma(z\ell)\)
the probability the model gives to the correct answer for this cell: \(\sigma(\ell)\) for a true pair, \(1 - \sigma(\ell) = \sigma(-\ell)\) for a negative
\(1/N\)
the sum is divided by the batch size, not by \(N^2\), as in the paper

Two things follow at once. First, there is no denominator that couples the cells: each term \(\log\sigma(z_{ij}\ell_{ij})\) depends on one image and one caption only. Second, the loss is symmetric by construction. CLIP must average an image-to-text softmax and a text-to-image softmax; SigLIP has one term per pair, and it is the same in both directions.

Figure 1 shows both losses on the same small batch. Click any cell and move its cosine: watch which loss terms change.

true pair (label +1)negative (label −1)selected cell
Figure 1. One batch of four image–caption pairs. Each cell shows its cosine and its sigmoid match probability \(\sigma(\ell_{ij})\); the shading is the size of that cell’s loss term. Image 1 (a dog) and image 4 (a puppy) are similar, so the off-diagonal cells between them are hard negatives.
How to read Figure 1

Click a cell to select it, then move the third slider. In SigLIP only that one cell’s term changes, so “terms changed” is always 1. In CLIP, the cell sits inside the softmax of its row (image → text) and of its column (text → image), so two of the \(2N = 8\) CLIP terms change, and if the selected cell is a negative, the right answers of its row and column become less likely. Now try the bias slider at \(t = 20\): with \(b = 0\) most negatives get \(\sigma \gt 0.5\), and the SigLIP loss is large; around \(b = -4\) the negatives are near 0 and the true pairs near 1. The dog/puppy cells stay uncertain: they are real near-duplicates that the batch labels as negatives.

Worked example: a 3 × 3 batch by hand

Use the cosine matrix from the CLIP session (dog, cat, car): rows are images, columns captions, \(\begin{pmatrix} 0.30 & 0.10 & 0.05 \\ 0.12 & 0.28 & 0.15 \\ 0.02 & 0.20 & 0.25 \end{pmatrix}\), with \(t = 10\) and \(b = -2\).

Logits \(\ell = 10\cos - 2\): the diagonal is \(1.0, 0.8, 0.5\); the off-diagonal cells are \(-1.0, -1.5, -0.8, -0.5, -1.8, 0.0\).

Diagonal terms \(\log(1 + e^{-\ell})\): \(0.313, 0.371, 0.474\). Negative terms \(\log(1 + e^{\ell})\): \(0.313, 0.201, 0.371, 0.474, 0.153, 0.693\). The worst cell is image 3 with caption 2 (logit 0: the model says “50% match” to a wrong pair).

Sum of all nine terms: \(3.364\); divided by \(N = 3\): \(\mathcal{L}_{\text{SigLIP}} = 1.121\). (CLIP’s loss with scale 10 on the same matrix is 0.368. The two numbers are on different scales and cannot be compared directly: SigLIP sums \(N^2\) terms, CLIP averages \(2N\).)

Figure 2 runs both losses on this worked example, one term at a time. Watch the size of each step: a SigLIP step reads one cell, a CLIP step reads a whole row or column.

true-pair termnegative termCLIP row term (image → text)CLIP column term (text → image)current step
  1. \(\ell_{ij} \leftarrow t \cos_{ij} + b\) for all \(N^2\) cells // formula (167.1)
  2. \(\text{acc} \leftarrow 0\)
  3. for \(i = 1..N\), \(j = 1..N\):
  4. \(z \leftarrow +1\) if \(i = j\), else \(-1\)
  5. \(\text{acc} \leftarrow \text{acc} + \log(1 + e^{-z \ell_{ij}})\) // this cell only
  6. return \(\text{acc} / N\) // formula (167.2)
Figure 2. The two losses computed in your browser on the same cosine matrix. Left: rows are images, columns are captions; each cell shows its cosine and its logit \(\ell\). SigLIP visits the \(N^2\) cells one by one and writes each cell’s term into it. CLIP visits \(N\) rows and \(N\) columns; the active row shows its softmax \(P\), the active column its softmax \(Q\). Right: the running sum as a stacked bar, one block per term. Below: the algorithm (current line highlighted) and the formula of the current step with the real numbers. \(\operatorname{lse}\) is log-sum-exp, \(\log \sum e^{x}\).
How to read Figure 2

Press Play, or Step through by hand. With SigLIP on the 3 × 3 example (\(t = 10\), \(b = -2\)), the first cell gives \(\ell_{11} = 1.00\) and the term \(\log(1 + e^{-1}) = 0.313\). The tallest block in the bar comes from cell (3, 2): \(\ell = 0.00\), term \(\ln 2 = 0.693\). The sum of the nine blocks is 3.364, and the last step divides by \(N = 3\): 1.121, the number of the worked example. Now switch to CLIP. Row 3 needs all three logits (0.20, 2.00, 2.50) at once: \(\operatorname{lse} = 3.035\), term \(0.535\), \(P_{33} = 0.586\). The six terms add to 2.210 and the loss is 0.368. Move the bias slider: the SigLIP bar changes, the CLIP numbers do not. Switch to 4 × 4 to see the dog/puppy hard negatives.

What the gradient does

For one cell, the derivative of \(\log(1 + e^{-z\ell})\) with respect to \(\ell\) is simple. Write the 0/1 label \(y_{ij} = (1 + z_{ij})/2\), so \(y = 1\) on the diagonal and 0 elsewhere:

\[ \frac{\partial \mathcal{L}_{\text{SigLIP}}}{\partial \ell_{ij}} = \frac{1}{N} \big( \sigma(\ell_{ij}) - y_{ij} \big), \qquad \frac{\partial \mathcal{L}_{\text{SigLIP}}}{\partial b} = \frac{1}{N} \sum_{i,j} \big( \sigma(\ell_{ij}) - y_{ij} \big) \tag{167.3} \]

The temperature is learned through \(t'\), with \(t = e^{t'}\), so \(\partial \ell_{ij} / \partial t' = t \cos_{ij}\). Its gradient is the same error, weighted by each cell’s cosine:

\[ \frac{\partial \mathcal{L}_{\text{SigLIP}}}{\partial t'} = \frac{t}{N} \sum_{i,j} \big( \sigma(\ell_{ij}) - y_{ij} \big)\, \mathbf{u}_i^\top \mathbf{v}_j \tag{167.4} \]
Read the symbols
\(\sigma(\ell_{ij}) - y_{ij}\)
the error of one cell: positive when a negative is too likely, negative when a true pair is too unlikely
\(\mathbf{u}_i^\top \mathbf{v}_j\)
the cosine of the cell. A larger \(t\) stretches the logits away from \(b\): it helps when true pairs have larger cosines than negatives

“Predicted probability minus label”: the same form as logistic regression. A true pair is pulled up with force \(1 - \sigma(\ell_{ii})\); a negative is pushed down with force \(\sigma(\ell_{ij})\). Compare CLIP’s gradient \(\tfrac{1}{2N}(P_{ij} + Q_{ij} - 2\delta_{ij})\) (formula 166.6): there \(P_{ij}\) is a softmax over the row, so the push on one negative depends on every other caption in the row. In SigLIP the push depends on that cell alone. An easy negative (\(\sigma \approx 0\)) gets almost no gradient in both losses; a hard negative gets a large one in both. In the worked example, the cell with logit 0 gets the largest push: \((0.5 - 0)/3 = 0.167\).

Probabilities that do not have to sum to 1

The two losses produce different kinds of numbers at test time. With CLIP, zero-shot classification is a softmax over the class prompts (formula 166.9): the probabilities sum to 1, so the model must pick one class even when none fits. With SigLIP, \(\sigma(t\, \mathbf{u}^\top \mathbf{v}_k + b)\) is a separate “yes” probability for each prompt \(k\). Several prompts can be likely at once (a photo with a dog and a ball), and all of them can be unlikely (“none of these”). This is what you want for multi-label tagging, for filtering, and for detection-style scoring where each region–text pair needs its own score.

softmax over the chosen captions (CLIP style)sigmoid per caption (SigLIP style)
Figure 3. One image of a dog on a beach, scored against up to five captions. The cosines are fixed (0.32, 0.27, 0.15, 0.10, 0.05). The softmax uses the same scale \(t\); the sigmoid adds the bias \(b\).
How to read Figure 3

With all five captions, both methods rank the dog captions first. Now untick the two dog captions. The softmax still gives about 0.67 to “a cat on a sofa”, because its three probabilities must sum to 1. The sigmoid scores do not move: the cat caption stays at 0.12, below 0.5, so the model is saying “no match here”. The bias sets the threshold: \(\sigma(\ell) = 0.5\) exactly when \(t\cos = -b\), that is at cosine \(-b/t = 0.25\) with the default sliders.

A practical warning: the sigmoid score is only as calibrated as the training data. A threshold of 0.5 is a good start, but tune it on your own data before you use it for filtering [Likely].

The bias term and the start of training

In a batch of \(N\) pairs there are \(N\) positives and \(N^2 - N\) negatives. With \(N = 16{,}384\) that is 268 million negatives for 16 thousand positives [Certain, Section 4.8 of the paper]. At the start of training the embeddings are random, so all cosines are close to 0 and every logit is close to \(b\). Put \(\ell_{ij} = b\) into (167.2), using \(\operatorname{sp}(a) = \log(1 + e^{a})\) (the softplus):

\[ \mathcal{L}_0(b) = \operatorname{sp}(-b) + (N - 1)\, \operatorname{sp}(b), \qquad \frac{d \mathcal{L}_0}{d b} = N \sigma(b) - 1 = 0 \;\Rightarrow\; b^\star = -\ln(N - 1) \tag{167.5} \]
Read the symbols
\(\mathcal{L}_0(b)\)
the loss at initialization (all cosines 0), per row: one positive term and \(N - 1\) negative terms
\(b^\star\)
the bias that is best before the model has learned anything: it says “a random pair matches with probability \(1/N\)”, the true base rate
Worked example: why \(b = -10\)

With \(N = 32{,}768\) (SigLIP’s usual batch): \(b^\star = -\ln 32{,}767 = -10.40\). The paper’s choice \(b = -10\) is very close to this prior.

At \(b = -10\): \(\sigma(-10) = 4.54 \times 10^{-5}\). The positive term is \(\operatorname{sp}(10) = 10.00\), the negatives add \(32{,}767 \times 4.54\times10^{-5} = 1.49\), so \(\mathcal{L}_0 = 11.49\), and the gradient on \(b\) is only \(+0.49\).

At \(b = 0\): every term is \(\ln 2 = 0.693\), so \(\mathcal{L}_0 = 32{,}768 \times 0.693 = 22{,}713\), and the gradient on \(b\) is \(+16{,}383\). Almost all of it comes from the negatives. The first optimizer steps would spend their effort pushing every pair apart (correcting the base rate) instead of learning which pairs match. That is the “massive over-correction” the paper avoids with \(b = -10\).

positive part \(\operatorname{sp}(-b)\)negative part \((N - 1)\operatorname{sp}(b)\)total \(\mathcal{L}_0\)
Figure 4. The loss at initialization (formula 167.5), on a log scale, as a function of the starting bias. The dashed line marks the best start \(b^\star = -\ln(N-1)\).
How to read Figure 4

The green curve (one positive per row) wants \(b\) large; the orange curve (\(N - 1\) negatives) wants \(b\) very negative. Their sum has its minimum at \(b^\star\), which moves left by \(\ln 2 = 0.69\) every time you double the batch. Note that the temperature does not appear: at initialization the cosines are near 0, so \(t\) multiplies almost nothing. At \(b = 0\) the loss and the gradient on \(b\) are thousands of times larger than at \(b^\star\).

The paper’s ablation (base model, batch 8k, 900M examples) shows that this matters [Certain, their Table 4]. Zero-shot ImageNet accuracy was 63.0% with \(b = -10\), \(t' = \log 10\); 62.0% with no bias at all; 61.7% with \(b = 0\); and only 53.7% with \(b = 0\) together with \(t' = \log 1\) (\(t = 1\)).

Figure 5 trains only these two numbers. The cosines are frozen (the towers do not learn), so we can watch what the loss asks of \(t\) and \(b\) alone. Each Adam step runs the four lines of the algorithm: logits (167.1), loss (167.2), gradients (167.3, 167.4) and the update. Adam uses \(\beta_2 = 0.95\), the paper’s setting.

Batch size \(N\):
true pair termnegative termlossscale \(t\)\(-\ln(N - 1)\)
  1. \(t \leftarrow e^{t'}\); \(\ell_{ij} \leftarrow t \cos_{ij} + b\) // formula (167.1)
  2. \(\mathcal{L} \leftarrow \frac{1}{N} \sum_{i,j} \log(1 + e^{-z_{ij} \ell_{ij}})\) // formula (167.2)
  3. \(g_b \leftarrow \frac1N \sum (\sigma(\ell_{ij}) - y_{ij})\); \(g_{t'} \leftarrow \frac{t}{N} \sum (\sigma(\ell_{ij}) - y_{ij}) \cos_{ij}\) // (167.3), (167.4)
  4. Adam step on \(b\) and \(t'\) with \(\beta_1 = 0.9\), \(\beta_2 = 0.95\), learning rate \(\eta\)
Figure 5. Gradient training of \(t\) and \(b\) only, on a fixed cosine matrix (true pairs near 0.28, negatives near 0.04, plus a few hard negatives up to 0.26). Left: the per-pair loss terms; a dark cell is a large term. Right: the loss, the bias (with the line \(-\ln(N-1)\)) and the scale over 200 Adam steps. Everything is recomputed in your browser when you move a slider.
How to read Figure 5

Press Play with the defaults (\(N = 16\), start \(b = -10\), \(t = 10\)). At step 0 all the dark cells are on the diagonal: every true pair has a logit near \(-7\), so the loss is 7.182 and almost all of it is the positive part. \(\partial \mathcal{L} / \partial b = -0.998\): the bias goes up and \(t\) grows. After 25 steps the loss is 0.770 and \(t = 48.9\); after 200 steps it is 0.405, with \(t = 55.1\) and \(b = -12.0\). Now set the start bias to 0 and press Reset and Play. The start loss is 14.65 and the dark cells are the negatives. The first steps push \(b\) down and even shrink \(t\) (to 2.15 at step 25), because that pushes all logits toward \(b\). After 50 steps the loss is 2.768, against 0.501 with \(b = -10\). Choose \(N = 32\): with \(b = 0\) the start loss doubles to 29.88, and after 50 steps it is still 4.072. A bigger batch has more negatives per row, so a bad start bias costs more steps.

No softmax over the batch: the chunked implementation

Contrastive training uses data parallelism: the global batch \(|B|\) is split over \(D\) devices, each holding \(b_d = |B|/D\) pairs. (We write \(b_d\) to keep it apart from the bias \(b\).) To compute CLIP’s softmax for one image, a device needs the logits against all \(|B|\) captions, because the normalizer is a sum over the whole row. The usual way is to all-gather every embedding onto every device. A naive implementation then builds the full \(|B| \times |B|\) matrix; a careful one builds \(b_d \times |B|\) logits per device.

The sigmoid loss is a plain sum of independent cell terms, so it can be cut into \(D \times D\) blocks and summed in any order:

\[ \mathcal{L}_{\text{SigLIP}} = -\frac{1}{|B|} \sum_{d_i = 1}^{D} \;\sum_{d_j = 1}^{D} \;\sum_{i \in d_i} \;\sum_{j \in d_j} \log \sigma\!\big(z_{ij}\, \ell_{ij}\big) \tag{167.6} \]

The outer sum runs on each device in parallel (sum over \(d_i\)). The middle sum is a ring: at each step every device passes its captions to its neighbour (a collective permute), computes the \(b_d \times b_d\) block of logits against the captions it holds now, adds the block’s loss, and throws the block away. After \(D\) steps every image has met every caption, and one final cross-device sum gives the loss. At no time does a device hold more than \(b_d \times b_d\) logits. The paper reports that \(D\) collective permutes are typically faster than two all-gathers [Certain, Section 3.3].

device 1device 2device 3block in memory now
Figure 6. The chunked sigmoid loss with \(D = 3\) devices and 4 pairs per device (the same mock setup as Figure 1 of the SigLIP paper). Columns are images, rows are captions. The sliders below compute the memory for a real batch: logits in FP32 (4 bytes each).
How to read Figure 6

Step 1: each device holds its own images and captions. Step 2: each device computes its diagonal block, which contains its positives. Steps 3 and 4: captions move one device along the ring, and each device computes a block of pure negatives. The bright block is the only thing in memory; finished blocks are faded. The last step adds the three per-device sums. With the default real numbers, \(|B| = 32{,}768\) on 256 devices, each block is \(128 \times 128\): 65.5 kB, compared with 16.8 MB for one device’s \(128 \times 32{,}768\) slice and 4.29 GB for the full matrix.

Algorithm 1: the SigLIP loss on one device (plain version)
  1. input: image embeddings \(F \in \R^{n \times d}\), text embeddings \(G \in \R^{n \times d}\), parameters \(t', b\)
  2. \(U \leftarrow\) normalize rows of \(F\); \(\;V \leftarrow\) normalize rows of \(G\)
  3. \(L \leftarrow e^{t'}\, U V^\top + b\) // n × n logits, formula (167.1)
  4. \(Z \leftarrow 2I - \mathbf{1}\mathbf{1}^\top\) // \(+1\) on the diagonal, \(-1\) elsewhere
  5. return \(-\tfrac{1}{n} \sum_{i,j} \log\sigma(Z_{ij} L_{ij})\) // formula (167.2); use a stable log-sigmoid
Algorithm 2: the chunked ring version on device k of D
  1. input: local images \(U_k\) and captions \(V_k\) (\(b_d\) each), \(t, b\)
  2. \(\text{acc} \leftarrow 0\); \(\;V \leftarrow V_k\)
  3. for step \(s = 0, \dots, D - 1\):
  4. \(L \leftarrow t\, U_k V^\top + b\) // only \(b_d \times b_d\) in memory
  5. labels: \(Z \leftarrow 2I - \mathbf{1}\mathbf{1}^\top\) if \(s = 0\) (own captions), else \(Z \leftarrow -\mathbf{1}\mathbf{1}^\top\)
  6. \(\text{acc} \leftarrow \text{acc} - \sum \log\sigma(Z \odot L)\)
  7. \(V \leftarrow\) captions received from the neighbour device // collective permute
  8. return \(\tfrac{1}{|B|}\,\text{all-reduce-sum}(\text{acc})\) // formula (167.6)

Does SigLIP still need a huge batch?

CLIP used a batch of 32,768, and the CLIP session showed why softmax training likes big batches: the bound \(I \ge \ln N - \mathcal{L}\) (formula 166.8) cannot certify more than \(\ln N\) nats of shared information. Removing the batch-wide softmax lets us ask the question directly. The SigLIP paper trained with batches from 512 up to one million and found [Certain, Sections 4.1 and 4.2]:

  • Small batches: below 16k, the sigmoid loss is clearly better than the softmax loss.
  • Saturation: both losses saturate. SigLIP was best at batch 32k; the softmax baseline needed 98k for its best result and still did not beat it. A batch of 307k hurt both. In the frozen-image setup (SigLiT), performance saturated around 32k, and the model peaked at 256k with only a small gain.
  • Many languages: for multilingual SigLIP (100+ languages, 30 billion examples), ImageNet zero-shot was 71.6%, 73.2%, 73.2%, 73.2% and 73.1% for batches 16k, 32k, 64k, 128k and 240k; average cross-lingual retrieval got worse above 32k (34.9% at 32k, 32.7% at 240k).
  • Cheap training: because the loss needs less memory, four TPUv4 chips could fit batch 4,096 for a base SigLIP but only 2,048 for the CLIP version. With a frozen ViT-g image tower, SigLiT reached 84.5% ImageNet zero-shot on four TPUv4 chips in two days; SigLIP from scratch reached 73.4% with 32 TPUv4 chips in five days.

Figure 7 shows what changes for one image when the batch grows. We keep one image with its true caption (cosine 0.30) and add captions from a fixed pool as the batch doubles from 2 to 4,096. The scale \(t\) and the bias \(b\) stay fixed, as if we froze a trained model and only changed the batch.

CLIPSigLIP row totalSigLIP negatives part (dashed)SigLIP \(\sigma(\ell_{ii})\)
  1. \(N \leftarrow 2N\): add \(N/2\) new captions; image \(i\) now has \(N - 1\) negatives
  2. CLIP: \(P_{ii} \leftarrow e^{\ell_{ii}} / \sum_{j} e^{\ell_{ij}}\); term \(\leftarrow -\log P_{ii}\) // one shared denominator
  3. SigLIP: term \(\leftarrow \log(1 + e^{-\ell_{ii}}) + \sum_{j \ne i} \log(1 + e^{\ell_{ij}})\) // N separate cells
Figure 7. One row of the logit matrix as the batch grows. Left: the cosines of the negatives (log count; bright bars are the captions that just joined), the true caption in green, and the SigLIP curve \(\sigma(t \cos + b)\). Right: the row’s loss term and the probability of the true caption for both losses, against the batch size.
How to read Figure 7

Press Play with \(t = 30\), \(b = -6\). CLIP: the true caption’s probability falls from 0.996 at \(N = 2\) to 0.104 at \(N = 4{,}096\), and its row term grows from 0.004 to 2.263. Nothing about this image changed; only the denominator got more terms. SigLIP: \(\sigma(\ell_{ii}) = 0.953\) at every batch size, because the true pair’s score never sees the other captions. What grows is the number of negative terms: their sum goes from 0.074 to 140.0, roughly doubling with \(N\). That is the base-rate effect of formula (167.5). Switch to \(b = -\ln(N-1)\): the negatives’ sum now grows only from 3.47 to 16.36, but the true pair’s score falls to 0.664. In real training the model also moves \(t\), \(b\) and the embeddings, so this is only the first-order picture.

Two training details from the paper are worth knowing. Large batches made training unstable, with spikes in the gradient norm; lowering Adam’s \(\beta_2\) from 0.999 to 0.95 fixed it (the same reason LLMs use 0.95, see session 4). And when the image tower starts from a pretrained checkpoint, turning off weight decay on the pretrained weights was needed for good results [Certain, Sections 4.5 and 4.7].

The paper also tested the imbalance between positives and negatives by masking negatives. Removing negatives at random hurt; keeping only the easiest ones did not work at all; keeping only the hardest ones almost kept the quality. So the imbalance itself is not a problem, and most of what the model learns from negatives comes from the hard ones [Certain, Section 4.8]. Finally, with corrupted images, corrupted texts or shuffled pairs, models trained with the sigmoid loss stayed better than softmax models as the corruption grew [Certain, Section 4.10]. A plausible reason: a wrong pair only spoils its own term, not a whole softmax row [Speculation].

The released SigLIP models

For the public models, SigLIP was trained for 40 billion examples at batch 32k, with 64 text tokens, then adapted to each resolution with 5 billion more examples at a 100 times smaller learning rate [Certain]. Vision and text representations are pooled with a MAP head (multihead attention pooling): a small attention layer with one learned query that attends over all patch tokens and returns one vector. It replaces the CLS token. The most used size is So400m, a “shape-optimized” ViT (Alabdulmohsin et al., 2023) whose width, depth and MLP size were chosen by a scaling study instead of copying ViT-L or ViT-H. Its config.json on Hugging Face: 27 layers, width 1,152, MLP width 4,304, 16 heads, patch 14 [Certain]. The 27 blocks alone hold 411 million weights (our count from those shapes).

The number of visual tokens is fixed by the resolution \(R\) and the patch size \(p\):

\[ n_{\text{tokens}} = \left\lfloor \frac{R}{p} \right\rfloor^{2} \qquad \text{e.g. } \lfloor 384/14 \rfloor^2 = 27^2 = 729, \quad (224/14)^2 = 256, \quad (512/16)^2 = 1024 \tag{167.7} \]
Model (ImageNet zero-shot, SigLIP paper Table 3)Patch tokensImageNet 0-shotCOCO image → text R@1
CLIP B/16 (OpenAI)19668.352.4
SigLIP B/16 at 22419676.264.4
SigLIP B/16 at 5121,02479.267.6
CLIP L/14 at 336 (OpenAI)57676.657.9
SigLIP L/16 at 38457682.170.6
SigLIP So400m/14 at 38472983.270.2

These SigLIP models were trained on WebLI, Google’s web image–text dataset, while CLIP used its own 400M pairs, so the table mixes the effect of the loss with the effect of data and training length [Certain]. The loss alone is measured by the batch-size study above, where both losses saw the same data.

SigLIP 2: same towers, more training signals

CLIP-style training has two known weaknesses. The pooled vector is good for “what is in this image” but the per-patch features are weak for “where is it” (localization, segmentation, depth). And the models are mostly English. SigLIP 2 keeps the SigLIP architecture (so you can swap the weights and the tokenizer into existing code) and fixes both with a staged recipe [Certain, SigLIP 2 Section 2]:

  1. Sigmoid loss + LocCa decoder (whole run). A small transformer decoder reads the un-pooled patch tokens through cross-attention and learns three tasks: write the caption, predict the box for a region description (referring expressions), and write a caption for a given box (grounded captioning). The region–caption pairs are made automatically with an open-vocabulary detector. The decoder has half the layers of the text tower, and it is thrown away after training. The two losses get equal weight.
  2. Self-distillation and masked prediction (last 20%). At 80% of training, an EMA teacher (an exponential moving average of the student’s weights, as in DINO, see session 165) is added. SILC loss: the student sees 8 small local crops and must match the teacher’s features of the full image. TIPS loss: 50% of the student’s patches are replaced by a mask token and the student must match the teacher’s per-patch features at those places.
  3. Resolution variants. Fixed-resolution checkpoints branch off at 95% and finish at their target resolution. The NaFlex variant branches off at 90% (next section).
  4. Small models: distillation through data. B/16 and B/32 get 4 billion more examples with the ACID method: a strong teacher and the student score a super-batch of 64k examples by “learnability”, and the best 32k form the training batch.
\[ \mathcal{L}_{\text{SigLIP 2}} = \mathcal{L}_{\text{sigmoid}} + \mathcal{L}_{\text{LocCa}} + \mathbb{1}[\text{progress} \ge 80\%]\; w_{\text{size}} \big( \mathcal{L}_{\text{SILC}} + 0.25\, \mathcal{L}_{\text{TIPS}} \big) \tag{167.8} \]
Read the symbols
\(\mathcal{L}_{\text{LocCa}}\)
next-token loss of the decoder on captions, referring expressions (box coordinates as text) and grounded captions
\(\mathcal{L}_{\text{SILC}}\)
local-to-global self-distillation on pooled features
\(\mathcal{L}_{\text{TIPS}}\)
masked prediction on per-patch features
\(w_{\text{size}}\)
0.25, 0.5, 1.0 and 0.5 for the B, L, So400m and g sizes: a balance between global and dense quality

Other settings [Certain]: WebLI with 10 billion images and 12 billion alt-texts in 109 languages, mixed 90% English and 10% other languages, with de-biasing filters; the multilingual Gemma tokenizer (256k vocabulary), text lower-cased, 64 tokens; Adam with learning rate \(10^{-3}\), decoupled weight decay \(10^{-4}\), gradient clipping at 1, batch 32k, cosine schedule with 20k warmup steps, 40 billion examples, up to 2,048 TPUv5e chips with FSDP (session 46). Sizes: ViT-B (86M), L (303M), So400m (400M) and g (1B).

Figure 8. The SigLIP 2 recipe. Boxes light up when their loss is active at the chosen point of training. Progress above 100% is the extra distillation stage that only the B-sized models get.
How to read Figure 8

From 0 to 80% only the sigmoid loss and the LocCa decoder train the image tower. At 80% the EMA teacher and the two self-supervised losses switch on, with the weight that depends on the model size (try B versus So400m). At 90% the NaFlex variant leaves the main run; at 95% the fixed-resolution variants leave. The decoder and the teacher are only training tools: the released checkpoint is just the two towers.

Algorithm 3: the SigLIP 2 schedule (one model size)
  1. for each step, with progress \(p\) from 0 to 1:
  2. embed images and captions; \(\mathcal{L} \leftarrow \mathcal{L}_{\text{sigmoid}}\) // formula (167.2)
  3. run the decoder on the un-pooled patch tokens; \(\mathcal{L} \leftarrow \mathcal{L} + \mathcal{L}_{\text{LocCa}}\) // caption, referring expression, grounded caption
  4. if \(p \ge 0.8\): build 1 global and 8 local augmented views; teacher \(\leftarrow\) EMA of student
  5. \(\mathcal{L} \leftarrow \mathcal{L} + w_{\text{size}}(\mathcal{L}_{\text{SILC}} + 0.25\,\mathcal{L}_{\text{TIPS}})\) // formula (167.8); the image–text losses use the original image
  6. Adam step with clipping at 1, cosine learning rate
  7. if \(p = 0.90\): fork the NaFlex run; if \(p = 0.95\): fork one run per fixed resolution
  8. for B/16 and B/32 only: 4B more examples with sigmoid loss, lr \(10^{-5}\), no weight decay, batches picked by ACID from 64k candidates

What did it buy? A few numbers from the SigLIP 2 paper [Certain, Tables 1–3]:

TaskModelSigLIPSigLIP 2
ImageNet zero-shot (%)B/16 at 22476.278.2
ImageNet zero-shot (%)So400m/14 at 38483.284.1
XM3600 text → image R@1 (%, 36 languages)So400m/14 at 38417.848.4
PASCAL segmentation, frozen features (mIoU)So400m/14 at 22472.077.1
ADE20k segmentation, frozen features (mIoU)So400m/14 at 22437.641.8
NYUv2 depth, frozen features (RMSE, lower is better)So400m/14 at 2240.5760.493
Open-vocabulary segmentation ADE-150 with Cat-Seg (mIoU)L/1637.538.8

The global numbers move by 1–2 points; the multilingual and dense numbers move a lot. That matches the design: the new losses target per-patch features and the data targets languages. The paper also reports that SigLIP 2 beats SigLIP and AIMv2 as the frozen encoder of a PaliGemma-style VLM, and improves referring-expression and open-vocabulary detection results [Certain, Sections 3.2–3.4].

NaFlex: native aspect ratio, several sequence lengths

A fixed-resolution model squashes every image to a square, for example 256 × 256. A wide receipt or a tall phone screenshot gets distorted, and small text becomes unreadable. NaFlex combines two older ideas: NaViT (keep the native aspect ratio, see session 180) and FlexiViT (one model for several sequence lengths). Given a patch size \(p\) and a token budget \(S_{\max}\), it resizes the image by one scale factor \(s\) and rounds each side up to a multiple of \(p\):

\[ H' = p \left\lceil \frac{s H}{p} \right\rceil, \quad W' = p \left\lceil \frac{s W}{p} \right\rceil, \qquad s = \max \Big\{ s : \frac{H'}{p} \cdot \frac{W'}{p} \le S_{\max} \Big\} \tag{167.9} \]
Read the symbols
\(H, W\)
the original height and width in pixels
\(H', W'\)
the resized sides, multiples of the patch size \(p = 16\)
\(S_{\max}\)
the token budget; NaFlex was trained with \(S_{\max} \in \{128, 256, 576, 784, 1024\}\), sampled per mini-batch

Because of the rounding, the aspect ratio changes a little: by at most \((p-1)/W'\) in width and \((p-1)/H'\) in height. Unused token slots are padding, and a mask hides them from attention and from the MAP head. The learned position embedding is a \(16 \times 16\) grid, resized bilinearly to the actual patch grid, for example \(21 \times 12\) [Certain, SigLIP 2 Section 2.4.2]. The Hugging Face processor finds \(s\) with a binary search; Figure 9 runs the same search.

Token budget \(S_{\max}\):
Figure 9. NaFlex resizing (formula 167.9, patch size 16). Left: the patch grid NaFlex uses, with the aspect ratio kept. Right: a fixed-resolution model with the same budget, which squashes the image into a square grid.
How to read Figure 9

A 1920 × 1080 frame with budget 256 becomes 336 × 192 pixels: a 21 × 12 grid, 252 tokens, 4 padding slots. The aspect ratio goes from 1.778 to 1.750. The square model would use 16 × 16 and stretch the image vertically by 1.78. Try a long strip (2400 × 200): NaFlex gives it a wide, short grid, while the square model squeezes it horizontally by a factor of 12. The paper found NaFlex better on OCR, document and screen retrieval, especially at small budgets, while the square B-sized model was slightly better on natural-image benchmarks [Certain, SigLIP 2 Section 3.1.1].

Algorithm 4: NaFlex preprocessing for one image
  1. input: image \(H \times W\), patch size \(p\), budget \(S_{\max}\)
  2. binary search the largest scale \(s\) with \(\lceil sH/p\rceil \cdot \lceil sW/p\rceil \le S_{\max}\) // formula (167.9)
  3. resize to \(H' \times W'\); cut into \(n = H'W'/p^2\) patches, row by row
  4. resize the \(16 \times 16\) position-embedding grid bilinearly to \(H'/p \times W'/p\)
  5. pad the patch sequence to \(S_{\max}\); mask \(\leftarrow\) 1 for real patches, 0 for padding
  6. run the ViT and the MAP head with the mask in every attention layer

The PyTorch version

The code builds the loss of Algorithm 1, checks it against binary cross-entropy and against the gradient formula (167.3), checks that the ring version of Algorithm 2 gives the same number, prints the start-of-training loss for both biases, and trains a tiny SigLIP on toy pairs. It needs only PyTorch and runs in a few seconds on a CPU.

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)

# 1) the sigmoid loss, formula (167.2): every image-text pair is one yes/no question
def siglip_loss(img_emb, txt_emb, t_prime, b):
    x = F.normalize(img_emb, dim=-1)                 # (n, d) unit vectors
    y = F.normalize(txt_emb, dim=-1)
    logits = t_prime.exp() * x @ y.T + b             # (n, n) scores
    n = logits.shape[0]
    z = 2 * torch.eye(n) - 1                          # +1 on the diagonal, -1 elsewhere
    return -F.logsigmoid(z * logits).sum() / n, logits

n, d = 8, 32
img, txt = torch.randn(n, d), torch.randn(n, d)
t_prime = torch.tensor(math.log(10.0), requires_grad=True)   # t = 10 at start
b = torch.tensor(-10.0, requires_grad=True)                  # b = -10 at start
loss, logits = siglip_loss(img, txt, t_prime, b)

# 2) the same loss as plain binary cross-entropy with 0/1 labels
bce = F.binary_cross_entropy_with_logits(logits, torch.eye(n), reduction="sum") / n
print(f"loss {loss.item():.4f}  as BCE {bce.item():.4f}")   # identical

# 3) gradient on each logit, formula (167.3): (sigmoid(l) - label) / n
logits.retain_grad(); loss.backward()
expected = (torch.sigmoid(logits) - torch.eye(n)) / n
print("grad matches:", torch.allclose(logits.grad, expected, atol=1e-7))
print(f"dL/db = {b.grad.item():.4f}   (n*sigmoid(b) - 1 when all cosines are 0: {n*torch.sigmoid(b).item()-1:.4f})")

# 4) the chunked "ring" version, formula (167.6): D devices, only b x b logits at a time
def siglip_chunked(img_emb, txt_emb, t, bias, D):
    x, y = F.normalize(img_emb, dim=-1), F.normalize(txt_emb, dim=-1)
    xs, ys = x.chunk(D), y.chunk(D)                  # device k holds xs[k], ys[k]
    total = 0.0
    for k in range(D):                               # (runs in parallel on real devices)
        for step in range(D):                        # texts move one device per step
            j = (k + step) % D
            l = t * xs[k] @ ys[j].T + bias           # only (n/D) x (n/D) in memory
            z = (2 * torch.eye(len(xs[k])) - 1) if j == k else -torch.ones_like(l)
            total = total - F.logsigmoid(z * l).sum()
    return total / len(img_emb)

with torch.no_grad():
    full, _ = siglip_loss(img, txt, t_prime, b)
    ring = siglip_chunked(img, txt, t_prime.exp(), b, D=4)
print(f"full {full.item():.6f}  chunked {ring.item():.6f}")

# 5) why b starts at -10: loss at the start for a batch of 32,768 with cosines near 0
N = 32768
for b0 in [-10.0, 0.0]:
    s = torch.tensor(b0)
    start = F.softplus(-s) + (N - 1) * F.softplus(s)       # 1 positive + (N-1) negatives per row
    print(f"b = {b0:5.1f}: start loss {start.item():10.2f}, best b = -ln(N-1) = {-math.log(N-1):.2f}")

# 6) a tiny SigLIP on toy pairs: two linear towers, batch 16
torch.manual_seed(0)
K, d_img, d_txt = 64, 20, 12
concepts = torch.randn(K, 8)
A, Bm = torch.randn(8, d_img), torch.randn(8, d_txt)
def batch(m):
    c = torch.randint(0, K, (m,))
    return concepts[c] @ A + 0.3 * torch.randn(m, d_img), concepts[c] @ Bm + 0.3 * torch.randn(m, d_txt), c
f, g = torch.nn.Linear(d_img, 16), torch.nn.Linear(d_txt, 16)
tp, bb = torch.nn.Parameter(torch.tensor(math.log(10.0))), torch.nn.Parameter(torch.tensor(-10.0))
opt = torch.optim.AdamW([*f.parameters(), *g.parameters(), tp, bb], lr=1e-2, betas=(0.9, 0.95))
for step in range(1500):
    xi, yi, _ = batch(16)
    loss, _ = siglip_loss(f(xi), g(yi), tp, bb)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    xi, yi, c = batch(256)
    sims = F.normalize(f(xi), dim=-1) @ F.normalize(g(yi), dim=-1).T
    acc = (c[sims.argmax(1)] == c).float().mean()          # retrieved text has the same concept
print(f"final loss {loss.item():.3f}, t = {tp.exp().item():.1f}, b = {bb.item():.2f}, retrieval accuracy: {acc.item():.2f}")

Output on CPU (we ran it twice; the output is identical): the loss is 10.8495 both ways; the gradient check prints True; \(dL/db = -0.9990\) (close to \(8\sigma(-10) - 1 = -0.9996\), because random cosines are not exactly 0); full and chunked losses agree (10.849527 and 10.849526, a float rounding difference); the start loss is 11.49 for \(b = -10\) and 22,713.05 for \(b = 0\), with \(b^\star = -10.40\); after 1,500 steps of batch 16 the loss is 0.244, \(t = 12.1\), \(b = -10.18\), and retrieval accuracy over 256 toy pairs is 1.00. A batch of 16 is far below what CLIP-style training likes, and the toy still learns; real data is much harder, so do not read more into this than “the code works”. In real code, the Hugging Face SiglipModel returns logits_per_image; apply torch.sigmoid to it, not a softmax.

Why open VLMs use a SigLIP-family encoder

A VLM does not use SigLIP’s final pooled vector. It takes the patch tokens from the last (or second-to-last) layer, passes them through a connector, and feeds them to the language model as “visual words” (session 178). What makes SigLIP a good choice for that job?

  • Strong features per patch and per image. Image–text training gives features that already line up with words. SigLIP 2 adds the dense and localization signals that VLM tasks (OCR, grounding, counting) need.
  • Open weights in many sizes and resolutions. B, L, So400m and g; 224 to 512 pixels; patch 14 or 16; and NaFlex. You choose the token count with formula (167.7).
  • Good quality for its size. So400m is about the size of ViT-L but scored higher than much larger CLIP-style models in the SigLIP paper (83.2% vs 82.0% for a 5B EVA-CLIP, Table 3) [Certain].
  • Cheaper to train and to continue training. The sigmoid loss works at smaller batches and needs little memory, which matters when a lab wants its own variant.
VLMVision encoderInputVisual tokens
PaliGemmaSigLIP So400m/14 [Certain, PaliGemma paper]224 (also 448, 896)256 at 224
LLaVA-OneVisionSigLIP So400m/14 shape, 26 of the 27 layers (num_hidden_layers: 26 in its config) [Certain]384 tiles729 per tile
Idefics2So400m/14 shape (1,152 wide, 27 layers) [Certain, config]; initialized from SigLIP [Likely]up to 98070 × 70 = 4,900 patches, then a resampler
SmolVLM (2.2B)So400m/14 shape at 384 [Certain, config]384 tiles729, pixel shuffle by 3 → 81 (image_seq_len: 81)
SmolVLM-256MSigLIP B/16 shape at 512 (12 layers, width 768) [Certain, config]512 tiles1,024, pixel shuffle by 4 → 64
Gemma 3400M SigLIP-style encoder, So400m/14 shape [Certain, config]89664 × 64 = 4,096 patches, pooled to 256 (mm_tokens_per_image: 256)

Notice how the token budget drives the design: an 896-pixel Gemma 3 image would be 4,096 tokens without pooling, so each VLM adds pooling, pixel shuffle or a resampler to cut the count (session 183). Other families use other encoders: Qwen2-VL and InternVL train their own ViTs (session 182).

Where this is used

ApplicationHow SigLIP is used
Vision encoder of VLMsPatch tokens from a frozen or fine-tuned SigLIP tower feed the connector and the LLM (table above)
Multi-label tagging and filteringOne sigmoid score per tag prompt; keep every tag above a tuned threshold; “no tag” is a valid answer (Figure 3)
Multilingual image searchSigLIP 2 embeds queries in 100+ languages into the same space; XM3600 retrieval rose from 17.8 to 48.4 R@1 for So400m/384
Document, screen and receipt retrievalNaFlex keeps the aspect ratio, so small text survives (Figure 9)
Open-vocabulary detection and segmentationOWL-ViT-style heads and Cat-Seg on SigLIP 2 patch features (session 168)
Text-to-video retrieval and video classificationEmbed sampled frames, pool over time, score against text prompts (session 208, session 203); motion-heavy actions need a video encoder [Likely]
Key-moment detectionSigmoid score of every frame against “a goal is scored”; the absolute score lets you threshold moments instead of only ranking frames (session 207)
Video fingerprinting and Video-RAGPer-frame image embeddings as a robust fingerprint and as the retrieval index for video question answering (session 209, session 210)

Common mistakes

  • Softmax over SigLIP logits. Ranking is unchanged, but the probabilities lose their meaning, and “none of these” disappears. Use torch.sigmoid(logits_per_image).
  • Wrong text preprocessing. SigLIP text towers were trained with captions padded to a fixed length (64 tokens for the released models), and SigLIP 2 lower-cases text. Hugging Face recommends padding="max_length" for SigLIP [Likely]. Different padding gives silently worse embeddings.
  • Starting with \(b = 0\). The loss starts in the tens of thousands and the first steps only fix the base rate (Figure 4). Start near \(-\ln N\) (the paper uses −10).
  • Dividing by \(N^2\) instead of \(N\). The loss becomes \(N\) times smaller and so do the gradients; with Adam the effect is partly hidden, with SGD the learning rate is effectively wrong.
  • Thinking the sigmoid removes false negatives. Duplicates and near-duplicates in a batch are still labeled −1 (the dog/puppy cells in Figure 1). Deduplicate data, or accept the noise.
  • Weight decay on a pretrained image tower. When you continue training SigLIP from pretrained weights, the paper needed to switch weight decay off for those weights.
  • Mismatched resolution. Feeding 512-pixel images to a 224 checkpoint without resizing the position embeddings, or mixing up patch 14 and 16 checkpoints, changes the token count (formula 167.7) and the quality.
  • Using the pooled vector for dense tasks. The MAP-head output is one vector per image. Segmentation, grounding and VLMs need the un-pooled patch tokens.

Interview questions

Write the SigLIP loss and explain each part.basic
With unit embeddings \(\mathbf{u}_i, \mathbf{v}_j\), logits \(\ell_{ij} = t\,\mathbf{u}_i^\top\mathbf{v}_j + b\) and labels \(z_{ij} = +1\) if \(i = j\), else −1: \(\mathcal{L} = -\frac1N\sum_{i,j}\log\sigma(z_{ij}\ell_{ij})\). Each of the \(N^2\) pairs is a binary classification (“match or not”); \(t = e^{t'}\) is a learned scale, \(b\) a learned bias. There is no normalization across the batch, and one term per pair makes the loss symmetric in images and texts.
What is the bias for, and why is it initialized to −10?core
There are \(N\) positives and \(N^2 - N\) negatives, so the base rate of a match is \(1/N\). At initialization all cosines are near 0 and every logit equals \(b\). The start loss \(\operatorname{sp}(-b) + (N-1)\operatorname{sp}(b)\) is minimized at \(b^\star = -\ln(N-1)\), which is −10.40 for \(N = 32{,}768\). With \(b = 0\) the start loss is 22,713 and the gradient on \(b\) is 16,383, so early steps would only fix the base rate. In the paper’s ablation, \(b = -10\) gave 63.0% ImageNet zero-shot against 61.7% for \(b = 0\) and 53.7% for \(b = 0, t = 1\).
Why is the sigmoid loss easier to distribute across devices than CLIP’s?core
CLIP’s softmax for an image needs the normalizer over all \(|B|\) captions, so implementations all-gather embeddings and hold at least \(b_d \times |B|\) logits per device. The sigmoid loss is a sum of independent pair terms, so each device can pass its captions around a ring (D collective permutes), compute a \(b_d \times b_d\) block, add it and discard it. Peak memory is \(b_d^2\), constant as you add devices. For \(|B| = 32{,}768\) on 256 devices: 65.5 kB per block instead of 16.8 MB per slice, or 4.29 GB for the full matrix in FP32.
Does SigLIP remove the need for large batches?core
Partly. Below 16k the sigmoid loss is clearly better than the softmax, so smaller batches work better. But both losses still improve up to a point and then saturate: SigLIP peaked at 32k, the softmax baseline at 98k (and was still lower), and 307k hurt both. For multilingual training, more than 32k made retrieval worse. The practical message: 32k is enough, and you can go lower with less loss than CLIP.
Compare the gradients of the sigmoid and softmax losses with respect to one negative logit.advanced
Sigmoid: \(\partial\mathcal{L}/\partial\ell_{ij} = \sigma(\ell_{ij})/N\), which depends only on that pair. Softmax (CLIP): \((P_{ij} + Q_{ij})/(2N)\), where \(P_{ij}\) is the row softmax and \(Q_{ij}\) the column softmax, so the push on one negative depends on all logits in its row and column, and the negatives compete for a fixed probability budget. Both give tiny gradients to easy negatives and large ones to hard negatives. Only the softmax creates interactions between cells; with the sigmoid, adding more negatives just adds more independent terms.
When would you prefer sigmoid scores over softmax scores at inference?core
When the answer can be “several” or “none”: multi-label tagging, content filtering, open-vocabulary detection where each region–text pair needs its own score, and key-moment detection where you threshold frames. Softmax probabilities over a class list always sum to 1 and change when you add or remove classes; a SigLIP score \(\sigma(t\cos + b)\) for one pair does not depend on the other candidates. Thresholds still need tuning on your data.
What does SigLIP 2 add to SigLIP, and what does each addition fix?core
(1) A LocCa decoder on the un-pooled patch tokens, trained on captioning, referring expressions (predict a box) and grounded captioning: better localization and OCR. (2) From 80% of training, SILC self-distillation (8 local student views match a global EMA teacher) and TIPS masked prediction (50% of patches masked): better dense features, for example PASCAL segmentation 72.0 → 77.1 mIoU and NYUv2 depth RMSE 0.576 → 0.493 for So400m/224. (3) Multilingual, de-biased WebLI (90% English) with the 256k Gemma tokenizer: XM3600 retrieval 17.8 → 48.4. (4) NaFlex for native aspect ratio. (5) ACID data-curation distillation for the B-sized models. The architecture is unchanged, so checkpoints are drop-in replacements.
Why do many open VLMs use a SigLIP encoder, and which outputs do they use?core
SigLIP towers give features aligned with language, are openly released in many sizes and resolutions, and So400m gives very high quality for its size (83.2% ImageNet zero-shot at 384 pixels). VLMs drop the MAP-head pooled vector and use the patch tokens from the last or second-to-last layer (LLaVA-OneVision keeps 26 of 27 layers), then reduce their number with a projector, pixel shuffle or pooling. PaliGemma, LLaVA-OneVision, Idefics2, SmolVLM and Gemma 3 all use SigLIP-shaped encoders.
How does NaFlex handle different aspect ratios and sequence lengths in one model?advanced
It picks the largest scale whose rounded patch grid fits the token budget (128 to 1024 tokens during training), so the aspect ratio is kept up to the rounding to multiples of 16. The learned 16 × 16 position-embedding grid is resized bilinearly to the actual grid, and padding tokens are masked in attention and in the MAP head. One checkpoint serves all budgets. It helps OCR and document tasks, interpolates well between trained lengths, but does not extrapolate well beyond them.
Is the heavy positive/negative imbalance a problem for the sigmoid loss?advanced
The bias term handles the base rate. Beyond that, the paper masked negatives to change the ratio: random removal hurt, keeping only easy negatives failed, keeping only the hardest negatives almost kept the quality, and training longer on hard negatives to match the total number of pairs was slightly better. So the imbalance is not harmful; the hard negatives carry most of the signal, and better negative mining might help.
You continue training SigLIP from a pretrained image tower and quality drops. What do you check?basic
Weight decay on the pretrained weights: the paper found that decaying them destroys their quality (10-shot ImageNet barely beat training from scratch) and that disabling weight decay for pretrained weights fixed it, together with a 0.1 learning-rate multiplier on the image tower. Also check \(\beta_2\) (0.95 was needed for stability at large batch), the bias initialization, and that the text preprocessing matches the checkpoint.

Exercises

  1. A true pair has logit 2 and a negative pair has logit −1. Compute both loss terms of (167.2) and both gradients of (167.3) for \(N = 1\).
  2. For a batch of 4,096, what is the best starting bias \(b^\star\)? What is \(\sigma(b^\star)\)?
  3. Your global batch is 65,536 on 512 devices. How much memory does the full logit matrix need in FP32? How much does one block of the ring version need?
  4. How many patch tokens does SigLIP So400m/14 produce at 224 and at 384 pixels? What NaFlex grid does a 1000 × 200 image get with budget 256 (patch 16)? Use Figure 9 to check.
  5. In Figure 3 with \(t = 20\), which bias makes a caption with cosine 0.20 sit exactly at 50%?
  6. Explain in two sentences why gradient accumulation over micro-batches is closer to the real large-batch loss for SigLIP than for CLIP, and what is still different.
Answerscheck
1. True pair: \(\log(1 + e^{-2}) = 0.127\), gradient \(\sigma(2) - 1 = 0.881 - 1 = -0.119\). Negative: \(\log(1 + e^{-1}) = 0.313\), gradient \(\sigma(-1) = 0.269\). 2. \(b^\star = -\ln 4095 = -8.32\); \(\sigma(b^\star) = 1/4096 = 2.44\times10^{-4}\). 3. \(65{,}536^2 \times 4\) bytes \(= 17.2\) GB; per device \(b_d = 128\), so a block is \(128^2 \times 4 = 65{,}536\) bytes (65.5 kB). 4. \((224/14)^2 = 256\) and \(\lfloor 384/14\rfloor^2 = 27^2 = 729\). The strip becomes 560 × 112 pixels, a 35 × 7 grid, 245 tokens and 11 padding slots, aspect ratio exactly 5. 5. \(\sigma = 0.5\) when \(20 \times 0.20 + b = 0\), so \(b = -4\). 6. For SigLIP the full-batch loss is a sum over all pairs, so a micro-batch computes an exact subset of its terms (the pairs inside the micro-batch). What is missing are the cross-micro-batch negative pairs, unless you also compute those blocks as in the ring; for CLIP even the terms you do compute are wrong, because each softmax normalizer would need all \(|B|\) captions.

Key takeaways

  • SigLIP replaces CLIP’s batch softmax with one sigmoid yes/no question per image–caption pair (167.1, 167.2). The gradient is “probability minus label” per cell (167.3).
  • A learned bias, started near \(-\ln N\) (the paper uses −10), matches the base rate of a match and avoids a huge loss at the start (167.5).
  • Independent terms make the loss easy to split: a ring of collective permutes keeps only a \(b_d \times b_d\) block in memory (167.6).
  • The sigmoid loss is clearly better below batch 16k; both losses saturate around 32k. Use \(\beta_2 = 0.95\) and no weight decay on pretrained towers.
  • SigLIP 2 keeps the towers and adds a captioning/grounding decoder, self-distillation and masked prediction late in training, multilingual data and NaFlex (167.8, 167.9). The biggest gains are in dense features and in non-English retrieval.
  • Many open VLMs (PaliGemma, LLaVA-OneVision, Idefics2, SmolVLM, Gemma 3) use SigLIP-shaped encoders; the resolution and patch size fix the visual token count (167.7).

Key formulas

\[ \mathcal{L} = -\frac{1}{N}\sum_{i,j}\log\sigma\big(z_{ij}(t\,\mathbf{u}_i^\top\mathbf{v}_j + b)\big), \quad \frac{\partial\mathcal{L}}{\partial\ell_{ij}} = \frac{\sigma(\ell_{ij}) - y_{ij}}{N}, \quad b^\star = -\ln(N-1), \quad n_{\text{tokens}} = \lfloor R/p \rfloor^2 \]

References for this session

  1. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  2. Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
  3. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  4. Zhai et al. (2022). LiT: Zero-Shot Transfer with Locked-image text Tuning
  5. Alabdulmohsin, Zhai, Kolesnikov & Beyer (2023). Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
  6. Chen et al. (2022). PaLI: A Jointly-Scaled Multilingual Language-Image Model
  7. Wan et al. (2024). LocCa: Visual Pretraining with Location-aware Captioners
  8. Naeem et al. (2024). SILC: Improving Vision Language Pretraining with Self-Distillation
  9. Maninis et al. (2025). TIPS: Text-Image Pretraining with Spatial awareness
  10. Beyer et al. (2023). FlexiViT: One Model for All Patch Sizes
  11. Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
  12. Udandarao et al. (2025). Active Data Curation Effectively Distills Large-Scale Multimodal Models
  13. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  14. Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer
  15. Laurençon et al. (2024). What matters when building vision-language models?
  16. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  17. Gemma Team (2025). Gemma 3 Technical Report
  18. Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers