LLM & VLM courseBlog
197Track 7: Vision and VLMs · session 35 of 38

Quantizing VLMs

A 7B vision-language model writes about three times faster with 4-bit weights in its language model, but not because the image is encoded faster. This session finds where the time and memory go in each stage of a VLM request, explains why the vision encoder and connector are usually left in 16-bit, shows why calibration data must contain images, and gives a test plan that checks the model can still see.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on quantization math (103), where quantization helps (104), AWQ (107), SmoothQuant (108) and the anatomy of a VLM (177)

What you will learn

  • The roofline view of a VLM request: why the vision encoder and prefill are compute-bound and decode is bandwidth-bound, with Qwen2-VL-7B numbers on A100, H100 and L4
  • How to compute the memory and decode speed of a quantized VLM part by part, and why quantizing the vision tower saves memory but no decode time
  • The VLM asymmetry: size, run frequency and risk of the vision encoder, the connector, the LLM blocks and the lm_head
  • Why outliers and the calibration distribution decide what a quantizer protects, and why text-only calibration leaves image-token channels unprotected
  • Modality balance in calibration: image tokens outnumber text tokens, but language tokens are more sensitive (MBQ)
  • A precision plan and a test plan: which parts to quantize, which tasks reveal vision damage, and how to read small eval differences

Why this matters

A 7B vision-language model (VLM) in BF16 needs about 16.6 GB just for its weights. That is too much for a 16 GB laptop GPU and leaves little room on a 24 GB card. With 4-bit weights in the language model, the same model fits in about 7 GB, and it writes answers about three times faster. This session explains where that speed comes from, which parts of a VLM you should quantize, and how to check that the model can still see.

Quantization means storing numbers with fewer bits: for example, each weight as a 4-bit integer plus a shared scale, instead of a 16-bit float. You met the math in session 103, the speed model in session 104, and the main methods in session 106, session 107 and session 108. All of those sessions treat a text-only LLM. A VLM adds three new things:

  1. A second network. A vision encoder (a ViT, see session 164) and a connector (session 178) run before the LLM. They run once per image, not once per generated token. So they have a very different cost profile.
  2. A second kind of token. Image tokens enter the LLM next to text tokens. Their activations have different sizes and different outlier channels. A quantizer that only saw text during calibration does not know about them.
  3. A second kind of quality. A quantized VLM can keep its text quality and still lose the small details of an image: the digits on a receipt, a small object, the count of items. You only see this if you test on vision tasks.

Real systems follow this pattern. The official 4-bit AWQ release of Qwen2-VL-7B-Instruct quantizes the LLM to 4 bits with groups of 128 and lists the whole visual module (vision encoder and merger) as “not converted”, so it stays in 16-bit. [Certain] llama.cpp ships the vision part of a VLM as a separate “mmproj” file, which is usually kept in F16 while the LLM file is 4-bit. [Likely] In the previous session, session 196, we followed one request through a VLM. Now we ask what quantization does to each step of that path. In the next session, session 198, we put the quantized model behind a server.

Decode is bandwidth-bound: one request, stage by stage

Every stage of a request does some arithmetic (FLOPs) and reads some bytes from GPU memory. A GPU has a peak compute rate \(P\) (FLOPs per second) and a memory bandwidth \(B_w\) (bytes per second). A stage cannot finish faster than either limit allows. This is the roofline model:

\[ t_{\text{stage}} \;\ge\; \max\!\left( \frac{F}{P},\ \frac{Q}{B_w} \right) \tag{197.1} \]
Read the symbols
\(F\)
FLOPs of the stage. A linear layer with \(N\) weights costs about \(2N\) FLOPs per token (one multiply and one add per weight).
\(Q\)
bytes the stage reads from memory: mostly the weights, plus the KV cache in the LLM
\(P,\ B_w\)
peak compute and memory bandwidth of the GPU, for example 312 TFLOP/s and 2.039 TB/s on an A100 80GB in BF16

Which limit wins? Divide FLOPs by bytes. This ratio is the arithmetic intensity \(I\). For a linear layer with \(N\) weights stored at \(b\) bits that processes \(n\) tokens at once:

\[ I \;=\; \frac{F}{Q} \;\approx\; \frac{2\,N\,n}{N\,b/8} \;=\; \frac{16\,n}{b} \qquad \text{compute-bound if } I \gt \frac{P}{B_w} \tag{197.2} \]

The number \(P/B_w\) is the GPU’s ridge point: about 153 FLOPs per byte on an A100, 295 on an H100 and 403 on an L4. Now look at the stages of a VLM request:

  • Vision encoder. All patches of the image go through the ViT at once. A 448×448 image in Qwen2-VL has \(32 \times 32 = 1{,}024\) patches of 14×14 pixels, so \(n = 1{,}024\) and \(I\) is about 1,000 in BF16. Far above the ridge: compute-bound.
  • Connector. The merger joins each 2×2 block of patches into one token (256 tokens) and maps it to the LLM width. It is small and fast.
  • LLM prefill. The prompt (256 image tokens plus text) goes through the LLM in one pass. With a few hundred tokens, \(I\) is near or above the ridge.
  • LLM decode. Each new token needs one pass through the LLM with \(n = 1\) (at batch size 1). So \(I \approx 16/b = 1\) FLOP per byte in BF16. Far below the ridge: bandwidth-bound. And this stage runs once per output token.

In decode, the time per token is set by the bytes read, so the bound becomes:

\[ t_{\text{token}} \;\approx\; \frac{\tfrac{b}{8}\,N_{\text{lin}} \;+\; 2\,N_{\text{head}} \;+\; \text{KV}(T)}{B_w} \tag{197.3} \]
Read the symbols
\(N_{\text{lin}}\)
weights in the linear layers of the LLM blocks (attention and MLP). These are the ones we quantize.
\(N_{\text{head}}\)
weights of the output layer (lm_head), usually kept in 16-bit, so 2 bytes each
\(\text{KV}(T)\)
bytes of the KV cache for a context of \(T\) tokens; it grows with the number of image tokens
\(b\)
effective bits per quantized weight, including the scales: 4-bit with one 16-bit scale and one 16-bit zero point per group of 128 gives \(4 + 32/128 = 4.25\)

The vision encoder is not in formula (197.3). Its weights are read once per image, during prefill, and never again while the model writes. That one fact explains most of this session. Step through Figure 1.

GPU
compute time \(F/P\)memory time \(Q/B_w\), BF16 weightsmemory time \(Q/B_w\), 4-bit weights (4.25 bits)
Figure 1. Roofline lower bounds (formula 197.1) for each stage of one Qwen2-VL-7B request: a 448×448 image, 50 text tokens, 200 output tokens, batch size 1. Parameter counts come from the model’s config.json. Real kernels reach maybe 50–80% of these peaks, so real times are longer; the ratios are what matter.
How to read Figure 1

For each stage, the longest of the three bars sets the time. In the vision encoder, the orange compute bar is the longest, so shrinking the weights (blue to green) changes nothing. In the decode step, the blue memory bar is about 150 times longer than the compute bar, so 4-bit weights cut the time by about 3×. The last step multiplies the decode step by 200 output tokens: almost all of the request time is decode. Switch to the L4 (a low-cost inference GPU with only 0.30 TB/s) and every decode bar grows by about 7×.

Worked example: Qwen2-VL-7B on an A100

From the config (28 layers, width 3,584, MLP width 18,944, 4 KV heads, vocabulary 152,064) the LLM blocks have \(N_{\text{lin}} = 6.525\) B linear weights and the lm_head has \(152{,}064 \times 3{,}584 = 0.545\) B. The vision encoder (32 layers, width 1,280) has 631 M weights and the merger 44.6 M, so the vision side is 676 M; the whole model is 8.29 B.

Decode, BF16: \(Q = 2 \times 6.525 + 2 \times 0.545 = 14.14\) GB, plus a KV cache of \(306 \times 57{,}344\) bytes = 0.018 GB. At 2.039 TB/s: 6.94 ms per token, at most 144 tokens per second.

Decode, 4-bit LLM: \(Q = 6.525 \times 4.25/8 + 1.090 + 0.018 = 4.57\) GB, so 2.24 ms per token, 3.10× faster.

Vision encoder: \(F = 2 \times 0.6295\text{ B} \times 1{,}024 + \text{attention} = 1.46\) TFLOP, so 4.68 ms of compute; its weights take only 0.62 ms to read. Compute-bound: 4-bit weights would not speed it up.

Whole request: 1,406 ms in BF16 against 466 ms with a 4-bit LLM (3.02×). Decode is 98.7% of the BF16 time.

What quantization buys in a VLM: memory and speed

Memory is a sum over the parts of the model, each at its own precision:

\[ M \;=\; \sum_{c\, \in\, \text{parts}} \frac{b_c}{8}\, N_c \;+\; \underbrace{B \cdot T \cdot \text{kv}_{\text{tok}}}_{\text{KV cache}}, \qquad \text{kv}_{\text{tok}} = 2\, L\, H_{kv}\, d_h \cdot 2 \text{ bytes} \tag{197.4} \]
Read the symbols
\(N_c,\ b_c\)
number of weights in part \(c\) (vision encoder, connector, LLM blocks, embedding, lm_head) and its bits per weight
\(B,\ T\)
batch size (requests at the same time) and tokens per request (image tokens + text + output)
\(L,\ H_{kv},\ d_h\)
LLM layers, KV heads and head size. For Qwen2-VL-7B: \(2 \times 28 \times 4 \times 128 \times 2 = 57{,}344\) bytes per token

The decode speed-up from quantizing only the LLM blocks follows from formula (197.3):

\[ S_{\text{dec}} \;=\; \frac{2\,N_{\text{lin}} + 2\,N_{\text{head}} + \text{KV}}{\tfrac{b}{8}\,N_{\text{lin}} + 2\,N_{\text{head}} + \text{KV}} \tag{197.5} \]

Notice what is missing: no term for the vision encoder. Quantizing it changes \(M\) but never \(S_{\text{dec}}\). Try it in Figure 2.

LLM blocks
Vision encoder + connector
GPU
Figure 2. Memory (formula 197.4) and decode speed (formulas 197.1 and 197.3) of Qwen2-VL-7B. Each request has its images, 50 prompt tokens and 200 output tokens; the KV cache is counted at the end of the answer. The embedding table and lm_head stay in BF16, as in common AWQ and GPTQ checkpoints.
How to read Figure 2

Start with the default (4-bit LLM, BF16 vision): 7.03 GB, and the vision tower is 19.2% of it. Now set the vision side to 4-bit: memory drops by 0.99 GB, but the decode speed does not move at all. Then set the LLM back to BF16: memory jumps to 16.6 GB and the decode time grows 3×. Finally raise the batch size and the number of images: the KV cache grows, and at large batches the compute time starts to matter. Also note the grey part: the embedding and lm_head (2.18 GB) become about a third of a 4-bit model.

Worked example: is the vision tower worth quantizing?

In the 4-bit model above, the BF16 vision tower is 1.35 GB of 7.03 GB. Quantizing it to 4.25 bits saves 0.99 GB (14% of the total) and changes the decode time by 0 ms. Encoding time does not drop either, because the encoder is compute-bound (Figure 1). So the gain is only memory, and only when memory is the real limit: a phone, a 16 GB laptop, or a GPU packed with many requests. The risk is quality on fine visual detail, which we look at next.

The VLM asymmetry: vision encoder and connector

A VLM is not uniform. Its three parts differ in size, in when they run, and in how much damage a quantization error does. The table puts the Qwen2-VL-7B numbers next to each other.

PartWeightsRunsBoundGain from 4-bit weightsRisk
Vision encoder (ViT)631 M (7.6%)once per imagecomputeabout 1 GB of memory; no speederrors reach every image token; fine detail (text, small objects) suffers first [Likely]
Connector (merger / projector)44.6 M (0.5%)once per imagetinyabout 0.07 GB; nothing elsethe only path from vision to language; a shift here moves all image tokens at once [Likely]
LLM blocks6.53 B (78.7%)once per output tokenbandwidth (decode)about 9.6 GB of memory and about 3× decode speedwell studied; 4-bit AWQ or GPTQ usually costs little [Likely]
Embedding + lm_head1.09 B (13.1%)lm_head once per tokenbandwidthpossible, but often kept in 16-bitthe lm_head is sensitive; errors change token probabilities directly [Likely]

Why do many people say the vision side is more sensitive? There are four reasons, and they are partly about risk and partly about measured error.

  1. Small networks have less slack. For LLMs, smaller models lose more quality under the same bit width. [Likely] A 0.6 B encoder has fewer redundant weights than a 7 B LLM.
  2. One error, many tokens. A weight error in the ViT changes the features of every patch. The LLM then reads 256 (or thousands of) slightly wrong tokens. The connector is worse: it is a short MLP with no residual path, and it maps features into the LLM’s embedding space, where the LLM expects a specific range of values.
  3. ViT activations have their own outliers. Large ViTs, supervised and self-supervised, produce a few high-norm tokens in background areas (Darcet et al., “Vision Transformers Need Registers”) and “massive activations” in a few channels (Sun et al.). [Certain] These make activation quantization of the encoder hard, as in LLMs.
  4. The tasks are fine-grained. Reading a 10-pixel digit needs precise features. Captioning a beach photo does not. Damage shows up on documents, charts and counting long before it shows up on general questions. [Likely]

The evidence is mixed, so treat “the vision side is more sensitive” as a starting assumption to test, not a law. The MBQ paper (Li et al., 2024) quantizes the ViT of LLaVA-OneVision with W4A8 SmoothQuant and reports that the encoder is not especially sensitive in its setup; inside the LLM, it finds language tokens about ten times more sensitive than vision tokens, measured by gradients. [Certain] Q-VLM (Wang et al., 2024) also quantizes the visual encoder and reports 2.78× memory compression and 1.44× faster generation on a 13B LLaVA model with no loss on its multimodal benchmarks. [Certain] The practical rule that follows is about cost and benefit: the vision side gives little memory and no decode speed, so the default is to keep it in 16-bit and quantize it only when you need the memory and have measured the damage.

Outliers, and why calibration data must include images

Post-training quantization methods such as GPTQ and AWQ use a small calibration set: a few hundred example inputs run through the model to measure the activations. They then choose the quantized weights that keep the layer’s output close to the original on those inputs. The error of one output, for a weight error \(\Delta W = \hat W - W\), is:

\[ \E_{x \sim \mathcal{D}} \big\lVert \Delta W\, x \big\rVert^2 \;\approx\; \sum_{j} \big\lVert \Delta W_{:,j} \big\rVert^2 \; \E_{\mathcal{D}}\!\big[x_j^2\big] \tag{197.6} \]
Read the symbols
\(\mathcal{D}\)
the distribution of inputs to this layer: what the calibration set says the layer will see
\(\Delta W_{:,j}\)
the error in column \(j\): all weights that multiply input channel \(j\)
\(\E_{\mathcal{D}}[x_j^2]\)
how large input channel \(j\) is on average. Large channels make their column’s error count more. (The approximation drops cross terms between channels.)

So the quantizer protects the columns of channels that are large in the calibration data. In a VLM, the LLM layers see two kinds of tokens with different statistics. If the calibration set is a mixture with a fraction \(p\) of image tokens:

\[ \E_{\mathcal{D}}\big[x_j^2\big] \;=\; (1 - p)\, \E_{\text{text}}\big[x_j^2\big] \;+\; p\, \E_{\text{image}}\big[x_j^2\big] \tag{197.7} \]

With \(p = 0\) (text-only calibration, the default of many LLM tools), a channel that is large only on image tokens looks like an ordinary channel. Its weights get no protection, and every image token pays for it. AWQ makes this concrete. It scales input channel \(j\) up by \(s_j\) before rounding (and the activation down by \(s_j\)), which makes the rounding error of that column relatively smaller. The scales come from the calibration activations:

\[ s_j = \big(\overline{|x_j|}\big)^{\alpha}, \qquad \alpha^{*} = \argmin_{\alpha \in [0, 1]} \sum_{x \in \mathcal{D}} \Big\lVert Q\big(W \operatorname{diag}(\mathbf{s})\big) \operatorname{diag}(\mathbf{s})^{-1} x \;-\; W x \Big\rVert^2 \tag{197.8} \]
Read the symbols
\(\overline{|x_j|}\)
mean absolute value of input channel \(j\) over the calibration tokens
\(\alpha\)
how strongly to follow the activation sizes: 0 means no scaling (plain rounding), 1 means fully proportional
\(Q(\cdot)\)
round-to-nearest quantization per group of weights (here groups of 32 input channels)

Both the scales and the search for \(\alpha\) depend only on \(\mathcal{D}\). Figure 3 is a toy layer with 64 input channels. Text tokens have outlier channels 3 and 17 (standard deviation 8). Image tokens have outlier channels 40 and 52 (standard deviation 12) and a mild channel 3. Move the image share of the calibration set and watch the error on held-out tokens.

Weight bits
text outlier channelimage outlier channeloutlier for bothordinary channel
Figure 3. A toy linear layer (32 outputs, 64 inputs, Gaussian weights, seeded) quantized with AWQ-style scales (formula 197.8). The calibration set has 64 tokens; the test sets have 256 text and 256 image tokens. The error is \(\lVert \hat Y - Y \rVert^2 / \lVert Y \rVert^2\). RTN is plain round-to-nearest with no scaling. This is an illustration of formula (197.7), not a measurement on a real model.
How to read Figure 3

At 0% image tokens, the bars for channels 40 and 52 are short: the calibration set never saw them light up. AWQ protects channels 3 and 17, and the text error falls from 6.06% to 3.06%, but the image error stays near plain rounding (4.92% against 5.53%). At 25% image tokens, the image channels appear, and both errors are low (3.15% and 2.53%). At 100%, the text outliers disappear from view and the text error (6.81%) becomes worse than plain rounding. Turn off the search and move \(\alpha\) to see the trade-off by hand.

Three practical rules follow:

  • Calibrate with real image–text pairs, run through the full model. The LLM layers must see image tokens exactly as they come out of the connector, inside the chat template you serve with. MBQ, for example, uses 128 image–caption pairs from COCO captions (as improved by ShareGPT4V), formatted with the model’s conversation template. [Certain]
  • Match the deployment mix. If you serve documents, put documents in the calibration set; if you serve video, put frames in it. Number of images per sample and image resolution change the token mix too.
  • Do not let image tokens drown the text. One image gives hundreds of tokens, a caption gives twenty. If every token counts the same in formula (197.8), the image tokens dominate, as at the 100% end of Figure 3. MBQ fixes this by weighting the error of each token type by the average gradient of the loss with respect to those tokens; in its experiments, language tokens had about ten times larger gradients than vision tokens. [Certain] MBQ reports up to 4.4% higher accuracy than earlier methods at W3 and up to 11.6% at W4A8 on 7B to 70B VLMs. [Certain]

The modality-balanced objective, in a simple form:

\[ \mathcal{L}_{\text{cal}} \;=\; \lambda_{\text{t}} \sum_{x \in \text{text}} \lVert \Delta W\, x \rVert \;+\; \lambda_{\text{v}} \sum_{x \in \text{image}} \lVert \Delta W\, x \rVert, \qquad \lambda_{\text{m}} = \overline{\big| \partial \mathcal{L}_{\text{SFT}} / \partial y_{\text{m}} \big|} \tag{197.9} \]

Here \(\lambda_{\text{m}}\) is the mean absolute gradient of the training loss with respect to the layer’s outputs on tokens of modality m. MBQ uses an absolute-error (not squared) form, which it reports works better. [Certain]

Algorithm 1: AWQ scale search with VLM calibration data
  1. input: VLM, calibration set of image–text conversations, bits \(b\), group size \(g\), grid of \(\alpha\) values
  2. run the full model (encoder, connector, LLM) on the calibration set; at each LLM linear layer, record its inputs \(X\) and mark each row as text or image
  3. for each LLM linear layer \(W\): // vision encoder and connector skipped (Algorithm 2)
  4.   \(\bar m_j \leftarrow\) mean of \(|X_{:,j}|\) over all recorded tokens
  5.   for each \(\alpha\) in the grid: \(\mathbf{s} \leftarrow \bar{\mathbf{m}}^{\alpha}\); \(\hat W \leftarrow Q_{b,g}(W \operatorname{diag}\mathbf{s})\operatorname{diag}\mathbf{s}^{-1}\)
  6.     error \(\leftarrow \lambda_{\text{t}}\,\lVert (\hat W - W) X_{\text{text}}^\top\rVert + \lambda_{\text{v}}\,\lVert (\hat W - W) X_{\text{image}}^\top \rVert\) // formula (197.9); \(\lambda_{\text{t}} = \lambda_{\text{v}} = 1\) gives plain AWQ
  7.   keep the \(\alpha\) with the lowest error; store \(Q_{b,g}(W\operatorname{diag}\mathbf{s})\) and fold \(\mathbf{s}^{-1}\) into the previous norm or layer
  8. return the quantized LLM

A recipe: which parts to quantize

Algorithm 2: a precision plan for a VLM
  1. input: VLM, target hardware, memory budget, latency target, task suite
  2. LLM linear layers \(\leftarrow\) 4-bit weight-only, group 128 (AWQ or GPTQ) // the decode win, formula (197.5)
  3. embedding, lm_head, norms \(\leftarrow\) BF16 // small and sensitive
  4. vision encoder and connector \(\leftarrow\) BF16
  5. KV cache \(\leftarrow\) BF16, or FP8 if many images or long video fill memory (session 110)
  6. measure memory and quality on the task suite (Algorithm 3)
  7. if memory is still over budget: encoder \(\leftarrow\) 8-bit weights; re-test the vision tasks
  8. if encoder compute dominates (video, many high-resolution images): encoder \(\leftarrow\) W8A8 INT8 or FP8 with smoothing; re-test
  9. connector \(\leftarrow\) quantize last, if ever

The runnable PyTorch below has two parts. Part 1 is the toy of Figure 3 written with tensors (its random numbers differ from the figure, but the story is the same). Part 2 shows the module filter that every real tool needs: quantize the LLM linear layers, skip anything under visual and the lm_head.

import torch
torch.manual_seed(0)

def fake_quant(w, bits=4, group=32):
    """Symmetric per-group round-to-nearest along the input dimension."""
    o, i = w.shape
    g = w.reshape(o, i // group, group)
    qmax = 2 ** (bits - 1) - 1
    d = g.abs().amax(dim=-1, keepdim=True).clamp(min=1e-8) / qmax
    return (torch.round(g / d).clamp(-qmax - 1, qmax) * d).reshape(o, i)

def awq_quant(w, calib, bits=4, group=32, grid=21):
    """AWQ-style: s_j = mean|x_j|^alpha, search alpha on the calibration set (197.8)."""
    m = calib.abs().mean(dim=0).clamp(min=1e-4)
    y = calib @ w.T
    best = (float("inf"), None, None)
    for k in range(grid):
        a = k / (grid - 1)
        s = m ** a
        s = s / (s.max() * s.min()).sqrt()
        wq = fake_quant(w * s, bits, group) / s
        err = ((calib @ wq.T - y) ** 2).sum().item()
        if err < best[0]:
            best = (err, a, wq)
    return best[2], best[1]

def rel_err(w, wq, x):
    y = x @ w.T
    return (((x @ wq.T) - y) ** 2).sum().item() / (y ** 2).sum().item()

# 1) activations: text and image tokens use DIFFERENT outlier channels
d_in, d_out = 64, 32
def tokens(n, outliers):
    sd = torch.ones(d_in)
    for j, v in outliers.items():
        sd[j] = v
    return torch.randn(n, d_in) * sd
TEXT, IMAGE = {3: 8.0, 17: 8.0}, {40: 12.0, 52: 12.0, 3: 2.0}
W = torch.randn(d_out, d_in)
cal_t, cal_i = tokens(64, TEXT), tokens(64, IMAGE)
test_t, test_i = tokens(256, TEXT), tokens(256, IMAGE)

wq = fake_quant(W, bits=3)
print(f"RTN         text {rel_err(W, wq, test_t):.4f}  image {rel_err(W, wq, test_i):.4f}")
for name, calib in [("text only", cal_t),
                    ("75/25 mix", torch.cat([cal_t[:48], cal_i[:16]])),
                    ("image only", cal_i)]:
    wq, a = awq_quant(W, calib, bits=3)
    print(f"{name:11s} alpha {a:.2f}  text {rel_err(W, wq, test_t):.4f}  image {rel_err(W, wq, test_i):.4f}")

# 2) choose WHICH modules to quantize: skip the vision tower, connector and lm_head
class ToyVLM(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.visual = torch.nn.ModuleDict({"block": torch.nn.Linear(64, 64),
                                           "merger": torch.nn.Linear(64, 128)})
        self.layers = torch.nn.ModuleList([torch.nn.Linear(128, 128) for _ in range(4)])
        self.lm_head = torch.nn.Linear(128, 1000, bias=False)

SKIP = ("visual", "lm_head")
model = ToyVLM()
q_params = kept = 0
with torch.no_grad():
    for name, mod in model.named_modules():
        if not isinstance(mod, torch.nn.Linear):
            continue
        if any(name.startswith(s) for s in SKIP):
            kept += mod.weight.numel()
            continue
        mod.weight.copy_(fake_quant(mod.weight, bits=4, group=32))
        q_params += mod.weight.numel()
print("quantized weights:", q_params, " kept in 16-bit:", kept)

# Output (PyTorch 2.8, CPU):
# RTN         text 0.0446  image 0.0465
# text only   alpha 0.35  text 0.0307  image 0.0480   <- image tokens worse than RTN
# 75/25 mix   alpha 0.40  text 0.0309  image 0.0266
# image only  alpha 0.25  text 0.0428  image 0.0212
# quantized weights: 65536  kept in 16-bit: 140288

Real tools have the same switch under different names. The Qwen2-VL AWQ checkpoint lists visual in modules_to_not_convert. [Certain] In Hugging Face BitsAndBytesConfig, the parameter is llm_int8_skip_modules. [Certain] Tools such as llm-compressor take an ignore list of module names or patterns. [Likely] Whatever the tool, print the list of quantized modules and check it before you trust the result.

Test quality on vision tasks, not only text

The usual LLM check after quantization is perplexity on WikiText plus a few text benchmarks. For a VLM this is not enough. Perplexity on text never runs the vision encoder or the connector, and it never puts image tokens through the LLM. A model can pass every text test and still lose the digits on an invoice. Figure 4 shows which tests touch which part.

image vision encoderBF16 (default)or W8A8 for video connectorBF16quantize last LLM blocks4-bit g128, AWQ/GPTQimage+text calibration lm_headBF16 Which tests reach which part document, chart, OCR questions (DocVQA, ChartQA, TextVQA): fine detail hallucination and counting (POPE, counting sets), grounding boxes general VQA and reasoning (MMMU), multi-image, video frames text-only perplexity and text benchmarks all partsall partsall partsLLM only
Figure 4. A default precision plan for a VLM (top) and the tests that exercise each part (bottom). Text-only tests never touch the vision encoder or the connector, and they never put image tokens through the LLM.

Two numbers are useful for each task. The relative drop compares the score of the quantized model with the original. The agreement rate counts how often the two models give the same answer; it catches changes that a score can hide (one error fixed, another added):

\[ \Delta_{\text{rel}} = \frac{\text{acc}_{q} - \text{acc}_{\text{fp}}}{\text{acc}_{\text{fp}}}, \qquad \text{agree} = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1}\big[\, \hat y^{\,q}_i = \hat y^{\,\text{fp}}_i \,\big] \tag{197.10} \]
Read the symbols
\(\text{acc}_q,\ \text{acc}_{\text{fp}}\)
accuracy of the quantized and of the original (full-precision) model on the same questions
\(\hat y^{\,q}_i,\ \hat y^{\,\text{fp}}_i\)
the answers of the two models on question \(i\), with greedy decoding so the comparison is deterministic

Small eval sets are noisy. With \(n\) questions and accuracy near \(a\), the standard error of the score is \(\sqrt{a(1-a)/n}\). For \(a = 0.8\) and \(n = 500\), that is 1.8 points, so a 1-point drop on 500 questions is not evidence of anything. Compare both models on the same questions and look at the questions where they disagree.

Algorithm 3: checking a quantized VLM
  1. input: original model \(f\), quantized model \(f_q\), task suite with at least one task per row of Figure 4
  2. for each task: run \(f\) and \(f_q\) with greedy decoding, the same prompts, image sizes and number of image tokens
  3.   record \(\Delta_{\text{rel}}\) and the agreement rate (formula 197.10)
  4.   save every question where the two answers differ
  5. read 20–50 of the disagreements by hand: are they digits, small objects, counts, or long answers drifting?
  6. if damage is on fine detail: undo vision quantization first, then raise the LLM bits or add matching images to the calibration set
  7. also test the serving settings: largest image resolution, most images per request, long videos // image tokens change the KV cache and the token mix
  8. return a table: task, \(\text{acc}_{\text{fp}}\), \(\text{acc}_q\), \(\Delta_{\text{rel}}\), agreement, memory, tokens per second

See session 192 for the benchmarks themselves and session 193 for why small text and counting break first.

Where this is used

SettingTypical choiceWhy
Serving a 7B VLM on one GPU (chat, visual question answering)4-bit AWQ or GPTQ on the LLM, BF16 vision, e.g. the official Qwen2-VL AWQ checkpointabout 3× faster decode and less than half the memory (Figure 2); the vision side adds no decode cost
Laptop, phone or edge box (llama.cpp, MLX)LLM in 4-bit GGUF or MLX format; vision projector file in F16 [Likely]memory and bandwidth are tight; the projector is small
Document and invoice extractionconservative: 8-bit or 4-bit LLM, BF16 vision, calibration with documentsdigits and small text are the first to break; test on DocVQA-style data
Video question answering, key-moment detection, Video-RAG4-bit LLM; encoder in W8A8 INT8 or FP8 if many frames make encoding slow; FP8 KV cachethousands of frame tokens make the encoder and the KV cache matter
Large-scale video retrieval and fingerprinting with a vision encoderINT8 or FP8 encoder (weights and activations), embeddings checked for recallthe encoder is the whole cost here, and it is compute-bound, so only weight–activation formats help
Fine-tuning a VLM on one GPUQLoRA: 4-bit frozen LLM, adapters in BF16, vision tower in BF16training memory, see session 58 and session 189

Common mistakes

  • Calibrating with text only. Many LLM tools default to a text corpus. The LLM layers then never see image tokens, and their outlier channels are left unprotected (Figure 3).
  • Quantizing the vision tower to “save time”. Weight-only quantization does not speed up a compute-bound encoder. It only saves about 1 GB on a 7B VLM.
  • Quantizing the connector with the LLM by accident. A name filter that matches every Linear catches the projector and the merger too. Print the list of quantized modules.
  • Testing only perplexity. Text perplexity never runs the image path. Test on documents, counting, hallucination and video.
  • Testing at a different resolution than you serve. More image tokens change the token mix and the KV cache. A model that works at 448×448 may degrade at 1,344×1,344.
  • Calling a 1-point drop on 300 questions a regression (or a win). Compute the standard error first, and compare answers question by question.
  • Forgetting the embedding and lm_head. They stay in 16-bit in most checkpoints. In a 4-bit Qwen2-VL-7B they are about 31% of the weights in memory, so the model is bigger than “8.29 B × 4 bits”.
  • Expecting prefill to get 3× faster. With many image tokens prefill is compute-bound; weight-only formats can even make it a little slower, because the kernels must also dequantize. [Likely]

Interview questions

Why does 4-bit weight-only quantization speed up decoding but not image encoding?basic
Decode processes one token per step, so it does about 2 FLOPs per weight it reads: arithmetic intensity near 1, far below the GPU ridge point (about 153 on an A100). Its time is the bytes read divided by bandwidth, and 4-bit weights cut those bytes by about 3.8× (to 4.25 bits). The vision encoder processes all patches at once (1,024 for a 448×448 image in Qwen2-VL), so its intensity is about 1,000: it is compute-bound, and reading fewer bytes does not change its time.
Estimate the decode speed of Qwen2-VL-7B on an A100 in BF16 and with a 4-bit LLM.core
Bytes per token: LLM linear weights 6.525 B and lm_head 0.545 B, so \(2 \times 7.07 = 14.14\) GB in BF16, about 6.94 ms per token at 2.039 TB/s, at most 144 tokens/s. With 4.25-bit LLM weights and a BF16 lm_head: \(3.47 + 1.09 = 4.56\) GB plus a small KV cache, about 2.24 ms, around 446 tokens/s, a 3.1× gain. Real numbers are lower because kernels do not reach peak bandwidth, but the ratio holds roughly.
Would you quantize the vision encoder? Defend your answer.core
By default, no. It is about 8% of a 7B VLM’s weights, it does not appear in the decode cost, and weight-only formats do not speed it up because it is compute-bound. The risk is on fine-grained tasks (small text, counting), because its error reaches every image token. I would quantize it in two cases: memory is the hard limit (edge device), using 8-bit weights first; or encoder compute dominates (video, many images), using W8A8 INT8 or FP8 with smoothing. Either way I re-test document and counting tasks. Some papers (MBQ, Q-VLM) report that the ViT tolerates such quantization well, so it is a measured decision, not a fixed rule.
Why might the connector be the most fragile part?advanced
It is the only path from vision to language. It is a short MLP (or a resampler) with no residual connection, so an error is not diluted by an identity path. It maps vision features into the LLM’s embedding space, where the LLM expects a specific range and distribution; a systematic shift moves all image tokens at once, which looks to the LLM like a different image. And it saves almost nothing: 44.6 M weights in Qwen2-VL-7B, about 0.07 GB at 4 bits. High risk, tiny gain. [Likely]
What calibration data do you use for a VLM, and why?core
Real image–text conversations in the serving chat template, run through the full model, so the LLM layers see image tokens exactly as they come out of the connector. Methods such as AWQ and GPTQ minimize output error weighted by the calibration activations (formula 197.6); channels that are large only on image tokens get no protection if the calibration set has no images. Match the deployment mix (documents, photos, video frames, resolution), use a few hundred samples, and balance modalities so the many image tokens do not drown the text tokens.
Image tokens outnumber text tokens 10 to 1 in your calibration set. Is that a problem?advanced
It can be. If every token counts the same in the reconstruction loss, the quantizer mostly optimizes for image tokens. But the MBQ paper found that the final loss is about ten times more sensitive to errors on language tokens than on vision tokens (measured with gradients). So the effective weighting is backwards. MBQ weights each modality’s error by its mean absolute gradient (formula 197.9) and reports gains of up to 4.4% at W3 and 11.6% at W4A8. A simpler fix is to reweight or subsample tokens per modality.
Your 4-bit VLM has the same WikiText perplexity as the BF16 one, but users say it misreads invoices. What happened and what do you do?core
Perplexity never runs the vision path, so it cannot see damage there. Likely causes: the vision encoder or connector was quantized; the calibration set had no document images, so document-specific channels were unprotected; or the serving resolution differs from the tested one. Steps: compare the two models on a document set (DocVQA-style) with greedy decoding; read the disagreements; undo vision and connector quantization; recalibrate with document images; if still bad, raise the LLM to 8 bits or use a better method (GPTQ with act-order, or AWQ with modality balancing).
Where do the bytes of a 4-bit Qwen2-VL-7B go?basic
About 7.0 GB in total: LLM linear layers 3.47 GB (4.25 bits), embedding plus lm_head 2.18 GB (BF16), vision encoder plus merger 1.35 GB (BF16), and a small KV cache for one short request. So only about half the memory is the 4-bit part; the 16-bit embedding, lm_head and vision tower are the rest.
When does weight–activation quantization (W8A8, FP8) beat weight-only for a VLM?advanced
When the important stages are compute-bound: large-batch serving, long prefill with many image tokens, and the vision encoder on video. W8A8 doubles the tensor-core rate (INT8 or FP8 against BF16), which helps compute-bound work. Weight-only 4-bit only helps when reading weights is the bottleneck (small-batch decode). Activation quantization needs outlier handling, such as SmoothQuant, which moves the difficulty from activations to weights, or rotations; ViTs have high-norm tokens and massive activations, so the encoder needs this care too.
How do you decide whether a 1.5-point drop after quantization is real?core
Compute the standard error: \(\sqrt{a(1-a)/n}\). At 80% accuracy on 500 questions it is 1.8 points, so 1.5 points is within noise. Because both models answer the same questions, a paired test is stronger: count questions only the original got right and questions only the quantized got right, and test whether the two counts differ (McNemar’s test). Also report the agreement rate, and read the disagreements to see whether they cluster on one skill.

Exercises

  1. Using formula (197.2), what is the arithmetic intensity of the Qwen2-VL vision encoder’s linear layers in BF16 for a 448×448 image? Is it compute-bound on an L4?
  2. In Figure 1, switch to the L4. How long is one BF16 decode step, and how long with a 4-bit LLM?
  3. Qwen2-VL-7B serves 32 requests at once, each with 4 images, 50 prompt tokens and 200 output tokens, on an A100 with a 4-bit LLM. Compute the KV cache at the end of the answers. Check with Figure 2.
  4. In Figure 3 (3-bit, \(\alpha\) searched), find the image share of the calibration set where the text and image errors are both closest to their best values. Why does 100% image tokens hurt the text?
  5. You serve 1,000 questions at 70% accuracy. What is the standard error of the accuracy?
  6. A colleague quantizes the vision tower and connector of Qwen2-VL-7B to 4.25 bits. How much memory does that save, and how many ms per decode token?
Answerscheck
1. \(I = 16 \times 1{,}024 / 16 = 1{,}024\) FLOPs per byte (the full count with attention, in Figure 1, is about 1,157). The L4 ridge is \(121/0.30 = 403\), so yes, compute-bound. 2. 47.2 ms in BF16 and 15.2 ms with a 4-bit LLM (14.16 GB and 4.57 GB at 0.30 TB/s). 3. \(T = 4 \times 256 + 250 = 1{,}274\) tokens per request; \(32 \times 1{,}274 \times 57{,}344 = 2.34\) GB. 4. With the grid in the figure, 12.5% or 25% image tokens: both kinds of outlier channel appear in the calibration means, so both get scaled. At 100%, channels 3 and 17 look small in the calibration data (channel 3 only has standard deviation 2 on image tokens), so text outliers lose their protection and the text error goes above plain rounding. 5. \(\sqrt{0.7 \times 0.3/1000} = 0.0145\), about 1.4 points. 6. From 1.35 GB to 0.36 GB, so 0.99 GB saved; 0 ms per decode token, because the vision tower is not read during decode.

Key takeaways

  • Each stage runs at the slower of compute and memory (197.1). The vision encoder and long prefills are compute-bound; decode at small batch is bandwidth-bound (197.2).
  • Decode time is the bytes of LLM weights plus KV divided by bandwidth (197.3). 4-bit LLM weights give about 3× faster decode for Qwen2-VL-7B; the vision encoder is not in this formula at all (197.5).
  • The vision side is small (about 8% of a 7B VLM), runs once per image, and affects every image token. Default: keep the encoder and connector in 16-bit; quantize them only for memory or video compute, and measure.
  • Quantizers protect channels that are large in the calibration data (197.6, 197.7). Calibrate with real image–text conversations, match the serving mix, and balance the modalities (197.8, 197.9).
  • Text perplexity cannot see vision damage. Test documents, counting, hallucination and video, compare answers one by one, and respect the noise of small eval sets (197.10).

Key formulas

\[ t \ge \max\!\left(\frac{F}{P}, \frac{Q}{B_w}\right), \quad I \approx \frac{16\,n}{b}, \quad t_{\text{token}} \approx \frac{\tfrac{b}{8}N_{\text{lin}} + 2N_{\text{head}} + \text{KV}}{B_w} \]
\[ \E\lVert \Delta W x\rVert^2 \approx \sum_j \lVert \Delta W_{:,j}\rVert^2\, \E_{\mathcal{D}}[x_j^2], \quad \E_{\mathcal{D}}[x_j^2] = (1-p)\,\E_{\text{text}}[x_j^2] + p\,\E_{\text{image}}[x_j^2] \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  2. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  3. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  5. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  6. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  7. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  8. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  9. Li, Hu, Ning, Liu, Hong, Jia, Li, Yan, Ran, Dai, Yan, Yang & Wang (2024). MBQ: Modality-Balanced Quantization for Large Vision-Language Models
  10. Wang, Wang, Xu, Tang, Zhou & Lu (2024). Q-VLM: Post-training Quantization for Large Vision-Language Models
  11. Darcet, Oquab, Mairal & Bojanowski (2023). Vision Transformers Need Registers
  12. Sun, Chen, Kolter & Liu (2024). Massive Activations in Large Language Models
  13. Qwen team (2024). Qwen2-VL-7B-Instruct-AWQ model card and config.json