Quantizing VLMs
A 7B vision-language model writes about three times faster with 4-bit weights in its language model, but not because the image is encoded faster. This session finds where the time and memory go in each stage of a VLM request, explains why the vision encoder and connector are usually left in 16-bit, shows why calibration data must contain images, and gives a test plan that checks the model can still see.
What you will learn
- The roofline view of a VLM request: why the vision encoder and prefill are compute-bound and decode is bandwidth-bound, with Qwen2-VL-7B numbers on A100, H100 and L4
- How to compute the memory and decode speed of a quantized VLM part by part, and why quantizing the vision tower saves memory but no decode time
- The VLM asymmetry: size, run frequency and risk of the vision encoder, the connector, the LLM blocks and the lm_head
- Why outliers and the calibration distribution decide what a quantizer protects, and why text-only calibration leaves image-token channels unprotected
- Modality balance in calibration: image tokens outnumber text tokens, but language tokens are more sensitive (MBQ)
- A precision plan and a test plan: which parts to quantize, which tasks reveal vision damage, and how to read small eval differences
Why this matters
A 7B vision-language model (VLM) in BF16 needs about 16.6 GB just for its weights. That is too much for a 16 GB laptop GPU and leaves little room on a 24 GB card. With 4-bit weights in the language model, the same model fits in about 7 GB, and it writes answers about three times faster. This session explains where that speed comes from, which parts of a VLM you should quantize, and how to check that the model can still see.
Quantization means storing numbers with fewer bits: for example, each weight as a 4-bit integer plus a shared scale, instead of a 16-bit float. You met the math in session 103, the speed model in session 104, and the main methods in session 106, session 107 and session 108. All of those sessions treat a text-only LLM. A VLM adds three new things:
- A second network. A vision encoder (a ViT, see session 164) and a connector (session 178) run before the LLM. They run once per image, not once per generated token. So they have a very different cost profile.
- A second kind of token. Image tokens enter the LLM next to text tokens. Their activations have different sizes and different outlier channels. A quantizer that only saw text during calibration does not know about them.
- A second kind of quality. A quantized VLM can keep its text quality and still lose the small details of an image: the digits on a receipt, a small object, the count of items. You only see this if you test on vision tasks.
Real systems follow this pattern. The official 4-bit AWQ release of Qwen2-VL-7B-Instruct quantizes the LLM to 4 bits with groups of 128 and lists the whole visual module (vision encoder and merger) as “not converted”, so it stays in 16-bit. [Certain] llama.cpp ships the vision part of a VLM as a separate “mmproj” file, which is usually kept in F16 while the LLM file is 4-bit. [Likely] In the previous session, session 196, we followed one request through a VLM. Now we ask what quantization does to each step of that path. In the next session, session 198, we put the quantized model behind a server.
Decode is bandwidth-bound: one request, stage by stage
Every stage of a request does some arithmetic (FLOPs) and reads some bytes from GPU memory. A GPU has a peak compute rate \(P\) (FLOPs per second) and a memory bandwidth \(B_w\) (bytes per second). A stage cannot finish faster than either limit allows. This is the roofline model:
- Read the symbols
- \(F\)
- FLOPs of the stage. A linear layer with \(N\) weights costs about \(2N\) FLOPs per token (one multiply and one add per weight).
- \(Q\)
- bytes the stage reads from memory: mostly the weights, plus the KV cache in the LLM
- \(P,\ B_w\)
- peak compute and memory bandwidth of the GPU, for example 312 TFLOP/s and 2.039 TB/s on an A100 80GB in BF16
Which limit wins? Divide FLOPs by bytes. This ratio is the arithmetic intensity \(I\). For a linear layer with \(N\) weights stored at \(b\) bits that processes \(n\) tokens at once:
The number \(P/B_w\) is the GPU’s ridge point: about 153 FLOPs per byte on an A100, 295 on an H100 and 403 on an L4. Now look at the stages of a VLM request:
- Vision encoder. All patches of the image go through the ViT at once. A 448×448 image in Qwen2-VL has \(32 \times 32 = 1{,}024\) patches of 14×14 pixels, so \(n = 1{,}024\) and \(I\) is about 1,000 in BF16. Far above the ridge: compute-bound.
- Connector. The merger joins each 2×2 block of patches into one token (256 tokens) and maps it to the LLM width. It is small and fast.
- LLM prefill. The prompt (256 image tokens plus text) goes through the LLM in one pass. With a few hundred tokens, \(I\) is near or above the ridge.
- LLM decode. Each new token needs one pass through the LLM with \(n = 1\) (at batch size 1). So \(I \approx 16/b = 1\) FLOP per byte in BF16. Far below the ridge: bandwidth-bound. And this stage runs once per output token.
In decode, the time per token is set by the bytes read, so the bound becomes:
- Read the symbols
- \(N_{\text{lin}}\)
- weights in the linear layers of the LLM blocks (attention and MLP). These are the ones we quantize.
- \(N_{\text{head}}\)
- weights of the output layer (lm_head), usually kept in 16-bit, so 2 bytes each
- \(\text{KV}(T)\)
- bytes of the KV cache for a context of \(T\) tokens; it grows with the number of image tokens
- \(b\)
- effective bits per quantized weight, including the scales: 4-bit with one 16-bit scale and one 16-bit zero point per group of 128 gives \(4 + 32/128 = 4.25\)
The vision encoder is not in formula (197.3). Its weights are read once per image, during prefill, and never again while the model writes. That one fact explains most of this session. Step through Figure 1.
How to read Figure 1
For each stage, the longest of the three bars sets the time. In the vision encoder, the orange compute bar is the longest, so shrinking the weights (blue to green) changes nothing. In the decode step, the blue memory bar is about 150 times longer than the compute bar, so 4-bit weights cut the time by about 3×. The last step multiplies the decode step by 200 output tokens: almost all of the request time is decode. Switch to the L4 (a low-cost inference GPU with only 0.30 TB/s) and every decode bar grows by about 7×.
From the config (28 layers, width 3,584, MLP width 18,944, 4 KV heads, vocabulary 152,064) the LLM blocks have \(N_{\text{lin}} = 6.525\) B linear weights and the lm_head has \(152{,}064 \times 3{,}584 = 0.545\) B. The vision encoder (32 layers, width 1,280) has 631 M weights and the merger 44.6 M, so the vision side is 676 M; the whole model is 8.29 B.
Decode, BF16: \(Q = 2 \times 6.525 + 2 \times 0.545 = 14.14\) GB, plus a KV cache of \(306 \times 57{,}344\) bytes = 0.018 GB. At 2.039 TB/s: 6.94 ms per token, at most 144 tokens per second.
Decode, 4-bit LLM: \(Q = 6.525 \times 4.25/8 + 1.090 + 0.018 = 4.57\) GB, so 2.24 ms per token, 3.10× faster.
Vision encoder: \(F = 2 \times 0.6295\text{ B} \times 1{,}024 + \text{attention} = 1.46\) TFLOP, so 4.68 ms of compute; its weights take only 0.62 ms to read. Compute-bound: 4-bit weights would not speed it up.
Whole request: 1,406 ms in BF16 against 466 ms with a 4-bit LLM (3.02×). Decode is 98.7% of the BF16 time.
What quantization buys in a VLM: memory and speed
Memory is a sum over the parts of the model, each at its own precision:
- Read the symbols
- \(N_c,\ b_c\)
- number of weights in part \(c\) (vision encoder, connector, LLM blocks, embedding, lm_head) and its bits per weight
- \(B,\ T\)
- batch size (requests at the same time) and tokens per request (image tokens + text + output)
- \(L,\ H_{kv},\ d_h\)
- LLM layers, KV heads and head size. For Qwen2-VL-7B: \(2 \times 28 \times 4 \times 128 \times 2 = 57{,}344\) bytes per token
The decode speed-up from quantizing only the LLM blocks follows from formula (197.3):
Notice what is missing: no term for the vision encoder. Quantizing it changes \(M\) but never \(S_{\text{dec}}\). Try it in Figure 2.
How to read Figure 2
Start with the default (4-bit LLM, BF16 vision): 7.03 GB, and the vision tower is 19.2% of it. Now set the vision side to 4-bit: memory drops by 0.99 GB, but the decode speed does not move at all. Then set the LLM back to BF16: memory jumps to 16.6 GB and the decode time grows 3×. Finally raise the batch size and the number of images: the KV cache grows, and at large batches the compute time starts to matter. Also note the grey part: the embedding and lm_head (2.18 GB) become about a third of a 4-bit model.
In the 4-bit model above, the BF16 vision tower is 1.35 GB of 7.03 GB. Quantizing it to 4.25 bits saves 0.99 GB (14% of the total) and changes the decode time by 0 ms. Encoding time does not drop either, because the encoder is compute-bound (Figure 1). So the gain is only memory, and only when memory is the real limit: a phone, a 16 GB laptop, or a GPU packed with many requests. The risk is quality on fine visual detail, which we look at next.
The VLM asymmetry: vision encoder and connector
A VLM is not uniform. Its three parts differ in size, in when they run, and in how much damage a quantization error does. The table puts the Qwen2-VL-7B numbers next to each other.
| Part | Weights | Runs | Bound | Gain from 4-bit weights | Risk |
|---|---|---|---|---|---|
| Vision encoder (ViT) | 631 M (7.6%) | once per image | compute | about 1 GB of memory; no speed | errors reach every image token; fine detail (text, small objects) suffers first [Likely] |
| Connector (merger / projector) | 44.6 M (0.5%) | once per image | tiny | about 0.07 GB; nothing else | the only path from vision to language; a shift here moves all image tokens at once [Likely] |
| LLM blocks | 6.53 B (78.7%) | once per output token | bandwidth (decode) | about 9.6 GB of memory and about 3× decode speed | well studied; 4-bit AWQ or GPTQ usually costs little [Likely] |
| Embedding + lm_head | 1.09 B (13.1%) | lm_head once per token | bandwidth | possible, but often kept in 16-bit | the lm_head is sensitive; errors change token probabilities directly [Likely] |
Why do many people say the vision side is more sensitive? There are four reasons, and they are partly about risk and partly about measured error.
- Small networks have less slack. For LLMs, smaller models lose more quality under the same bit width. [Likely] A 0.6 B encoder has fewer redundant weights than a 7 B LLM.
- One error, many tokens. A weight error in the ViT changes the features of every patch. The LLM then reads 256 (or thousands of) slightly wrong tokens. The connector is worse: it is a short MLP with no residual path, and it maps features into the LLM’s embedding space, where the LLM expects a specific range of values.
- ViT activations have their own outliers. Large ViTs, supervised and self-supervised, produce a few high-norm tokens in background areas (Darcet et al., “Vision Transformers Need Registers”) and “massive activations” in a few channels (Sun et al.). [Certain] These make activation quantization of the encoder hard, as in LLMs.
- The tasks are fine-grained. Reading a 10-pixel digit needs precise features. Captioning a beach photo does not. Damage shows up on documents, charts and counting long before it shows up on general questions. [Likely]
The evidence is mixed, so treat “the vision side is more sensitive” as a starting assumption to test, not a law. The MBQ paper (Li et al., 2024) quantizes the ViT of LLaVA-OneVision with W4A8 SmoothQuant and reports that the encoder is not especially sensitive in its setup; inside the LLM, it finds language tokens about ten times more sensitive than vision tokens, measured by gradients. [Certain] Q-VLM (Wang et al., 2024) also quantizes the visual encoder and reports 2.78× memory compression and 1.44× faster generation on a 13B LLaVA model with no loss on its multimodal benchmarks. [Certain] The practical rule that follows is about cost and benefit: the vision side gives little memory and no decode speed, so the default is to keep it in 16-bit and quantize it only when you need the memory and have measured the damage.
Outliers, and why calibration data must include images
Post-training quantization methods such as GPTQ and AWQ use a small calibration set: a few hundred example inputs run through the model to measure the activations. They then choose the quantized weights that keep the layer’s output close to the original on those inputs. The error of one output, for a weight error \(\Delta W = \hat W - W\), is:
- Read the symbols
- \(\mathcal{D}\)
- the distribution of inputs to this layer: what the calibration set says the layer will see
- \(\Delta W_{:,j}\)
- the error in column \(j\): all weights that multiply input channel \(j\)
- \(\E_{\mathcal{D}}[x_j^2]\)
- how large input channel \(j\) is on average. Large channels make their column’s error count more. (The approximation drops cross terms between channels.)
So the quantizer protects the columns of channels that are large in the calibration data. In a VLM, the LLM layers see two kinds of tokens with different statistics. If the calibration set is a mixture with a fraction \(p\) of image tokens:
With \(p = 0\) (text-only calibration, the default of many LLM tools), a channel that is large only on image tokens looks like an ordinary channel. Its weights get no protection, and every image token pays for it. AWQ makes this concrete. It scales input channel \(j\) up by \(s_j\) before rounding (and the activation down by \(s_j\)), which makes the rounding error of that column relatively smaller. The scales come from the calibration activations:
- Read the symbols
- \(\overline{|x_j|}\)
- mean absolute value of input channel \(j\) over the calibration tokens
- \(\alpha\)
- how strongly to follow the activation sizes: 0 means no scaling (plain rounding), 1 means fully proportional
- \(Q(\cdot)\)
- round-to-nearest quantization per group of weights (here groups of 32 input channels)
Both the scales and the search for \(\alpha\) depend only on \(\mathcal{D}\). Figure 3 is a toy layer with 64 input channels. Text tokens have outlier channels 3 and 17 (standard deviation 8). Image tokens have outlier channels 40 and 52 (standard deviation 12) and a mild channel 3. Move the image share of the calibration set and watch the error on held-out tokens.
How to read Figure 3
At 0% image tokens, the bars for channels 40 and 52 are short: the calibration set never saw them light up. AWQ protects channels 3 and 17, and the text error falls from 6.06% to 3.06%, but the image error stays near plain rounding (4.92% against 5.53%). At 25% image tokens, the image channels appear, and both errors are low (3.15% and 2.53%). At 100%, the text outliers disappear from view and the text error (6.81%) becomes worse than plain rounding. Turn off the search and move \(\alpha\) to see the trade-off by hand.
Three practical rules follow:
- Calibrate with real image–text pairs, run through the full model. The LLM layers must see image tokens exactly as they come out of the connector, inside the chat template you serve with. MBQ, for example, uses 128 image–caption pairs from COCO captions (as improved by ShareGPT4V), formatted with the model’s conversation template. [Certain]
- Match the deployment mix. If you serve documents, put documents in the calibration set; if you serve video, put frames in it. Number of images per sample and image resolution change the token mix too.
- Do not let image tokens drown the text. One image gives hundreds of tokens, a caption gives twenty. If every token counts the same in formula (197.8), the image tokens dominate, as at the 100% end of Figure 3. MBQ fixes this by weighting the error of each token type by the average gradient of the loss with respect to those tokens; in its experiments, language tokens had about ten times larger gradients than vision tokens. [Certain] MBQ reports up to 4.4% higher accuracy than earlier methods at W3 and up to 11.6% at W4A8 on 7B to 70B VLMs. [Certain]
The modality-balanced objective, in a simple form:
Here \(\lambda_{\text{m}}\) is the mean absolute gradient of the training loss with respect to the layer’s outputs on tokens of modality m. MBQ uses an absolute-error (not squared) form, which it reports works better. [Certain]
- input: VLM, calibration set of image–text conversations, bits \(b\), group size \(g\), grid of \(\alpha\) values
- run the full model (encoder, connector, LLM) on the calibration set; at each LLM linear layer, record its inputs \(X\) and mark each row as text or image
- for each LLM linear layer \(W\): // vision encoder and connector skipped (Algorithm 2)
- \(\bar m_j \leftarrow\) mean of \(|X_{:,j}|\) over all recorded tokens
- for each \(\alpha\) in the grid: \(\mathbf{s} \leftarrow \bar{\mathbf{m}}^{\alpha}\); \(\hat W \leftarrow Q_{b,g}(W \operatorname{diag}\mathbf{s})\operatorname{diag}\mathbf{s}^{-1}\)
- error \(\leftarrow \lambda_{\text{t}}\,\lVert (\hat W - W) X_{\text{text}}^\top\rVert + \lambda_{\text{v}}\,\lVert (\hat W - W) X_{\text{image}}^\top \rVert\) // formula (197.9); \(\lambda_{\text{t}} = \lambda_{\text{v}} = 1\) gives plain AWQ
- keep the \(\alpha\) with the lowest error; store \(Q_{b,g}(W\operatorname{diag}\mathbf{s})\) and fold \(\mathbf{s}^{-1}\) into the previous norm or layer
- return the quantized LLM
A recipe: which parts to quantize
- input: VLM, target hardware, memory budget, latency target, task suite
- LLM linear layers \(\leftarrow\) 4-bit weight-only, group 128 (AWQ or GPTQ) // the decode win, formula (197.5)
- embedding, lm_head, norms \(\leftarrow\) BF16 // small and sensitive
- vision encoder and connector \(\leftarrow\) BF16
- KV cache \(\leftarrow\) BF16, or FP8 if many images or long video fill memory (session 110)
- measure memory and quality on the task suite (Algorithm 3)
- if memory is still over budget: encoder \(\leftarrow\) 8-bit weights; re-test the vision tasks
- if encoder compute dominates (video, many high-resolution images): encoder \(\leftarrow\) W8A8 INT8 or FP8 with smoothing; re-test
- connector \(\leftarrow\) quantize last, if ever
The runnable PyTorch below has two parts. Part 1 is the toy of Figure 3 written with tensors (its random numbers differ from the figure, but the story is the same). Part 2 shows the module filter that every real tool needs: quantize the LLM linear layers, skip anything under visual and the lm_head.
import torch torch.manual_seed(0) def fake_quant(w, bits=4, group=32): """Symmetric per-group round-to-nearest along the input dimension.""" o, i = w.shape g = w.reshape(o, i // group, group) qmax = 2 ** (bits - 1) - 1 d = g.abs().amax(dim=-1, keepdim=True).clamp(min=1e-8) / qmax return (torch.round(g / d).clamp(-qmax - 1, qmax) * d).reshape(o, i) def awq_quant(w, calib, bits=4, group=32, grid=21): """AWQ-style: s_j = mean|x_j|^alpha, search alpha on the calibration set (197.8).""" m = calib.abs().mean(dim=0).clamp(min=1e-4) y = calib @ w.T best = (float("inf"), None, None) for k in range(grid): a = k / (grid - 1) s = m ** a s = s / (s.max() * s.min()).sqrt() wq = fake_quant(w * s, bits, group) / s err = ((calib @ wq.T - y) ** 2).sum().item() if err < best[0]: best = (err, a, wq) return best[2], best[1] def rel_err(w, wq, x): y = x @ w.T return (((x @ wq.T) - y) ** 2).sum().item() / (y ** 2).sum().item() # 1) activations: text and image tokens use DIFFERENT outlier channels d_in, d_out = 64, 32 def tokens(n, outliers): sd = torch.ones(d_in) for j, v in outliers.items(): sd[j] = v return torch.randn(n, d_in) * sd TEXT, IMAGE = {3: 8.0, 17: 8.0}, {40: 12.0, 52: 12.0, 3: 2.0} W = torch.randn(d_out, d_in) cal_t, cal_i = tokens(64, TEXT), tokens(64, IMAGE) test_t, test_i = tokens(256, TEXT), tokens(256, IMAGE) wq = fake_quant(W, bits=3) print(f"RTN text {rel_err(W, wq, test_t):.4f} image {rel_err(W, wq, test_i):.4f}") for name, calib in [("text only", cal_t), ("75/25 mix", torch.cat([cal_t[:48], cal_i[:16]])), ("image only", cal_i)]: wq, a = awq_quant(W, calib, bits=3) print(f"{name:11s} alpha {a:.2f} text {rel_err(W, wq, test_t):.4f} image {rel_err(W, wq, test_i):.4f}") # 2) choose WHICH modules to quantize: skip the vision tower, connector and lm_head class ToyVLM(torch.nn.Module): def __init__(self): super().__init__() self.visual = torch.nn.ModuleDict({"block": torch.nn.Linear(64, 64), "merger": torch.nn.Linear(64, 128)}) self.layers = torch.nn.ModuleList([torch.nn.Linear(128, 128) for _ in range(4)]) self.lm_head = torch.nn.Linear(128, 1000, bias=False) SKIP = ("visual", "lm_head") model = ToyVLM() q_params = kept = 0 with torch.no_grad(): for name, mod in model.named_modules(): if not isinstance(mod, torch.nn.Linear): continue if any(name.startswith(s) for s in SKIP): kept += mod.weight.numel() continue mod.weight.copy_(fake_quant(mod.weight, bits=4, group=32)) q_params += mod.weight.numel() print("quantized weights:", q_params, " kept in 16-bit:", kept) # Output (PyTorch 2.8, CPU): # RTN text 0.0446 image 0.0465 # text only alpha 0.35 text 0.0307 image 0.0480 <- image tokens worse than RTN # 75/25 mix alpha 0.40 text 0.0309 image 0.0266 # image only alpha 0.25 text 0.0428 image 0.0212 # quantized weights: 65536 kept in 16-bit: 140288
Real tools have the same switch under different names. The Qwen2-VL AWQ checkpoint lists visual in modules_to_not_convert. [Certain] In Hugging Face BitsAndBytesConfig, the parameter is llm_int8_skip_modules. [Certain] Tools such as llm-compressor take an ignore list of module names or patterns. [Likely] Whatever the tool, print the list of quantized modules and check it before you trust the result.
Test quality on vision tasks, not only text
The usual LLM check after quantization is perplexity on WikiText plus a few text benchmarks. For a VLM this is not enough. Perplexity on text never runs the vision encoder or the connector, and it never puts image tokens through the LLM. A model can pass every text test and still lose the digits on an invoice. Figure 4 shows which tests touch which part.
Two numbers are useful for each task. The relative drop compares the score of the quantized model with the original. The agreement rate counts how often the two models give the same answer; it catches changes that a score can hide (one error fixed, another added):
- Read the symbols
- \(\text{acc}_q,\ \text{acc}_{\text{fp}}\)
- accuracy of the quantized and of the original (full-precision) model on the same questions
- \(\hat y^{\,q}_i,\ \hat y^{\,\text{fp}}_i\)
- the answers of the two models on question \(i\), with greedy decoding so the comparison is deterministic
Small eval sets are noisy. With \(n\) questions and accuracy near \(a\), the standard error of the score is \(\sqrt{a(1-a)/n}\). For \(a = 0.8\) and \(n = 500\), that is 1.8 points, so a 1-point drop on 500 questions is not evidence of anything. Compare both models on the same questions and look at the questions where they disagree.
- input: original model \(f\), quantized model \(f_q\), task suite with at least one task per row of Figure 4
- for each task: run \(f\) and \(f_q\) with greedy decoding, the same prompts, image sizes and number of image tokens
- record \(\Delta_{\text{rel}}\) and the agreement rate (formula 197.10)
- save every question where the two answers differ
- read 20–50 of the disagreements by hand: are they digits, small objects, counts, or long answers drifting?
- if damage is on fine detail: undo vision quantization first, then raise the LLM bits or add matching images to the calibration set
- also test the serving settings: largest image resolution, most images per request, long videos // image tokens change the KV cache and the token mix
- return a table: task, \(\text{acc}_{\text{fp}}\), \(\text{acc}_q\), \(\Delta_{\text{rel}}\), agreement, memory, tokens per second
See session 192 for the benchmarks themselves and session 193 for why small text and counting break first.
Where this is used
| Setting | Typical choice | Why |
|---|---|---|
| Serving a 7B VLM on one GPU (chat, visual question answering) | 4-bit AWQ or GPTQ on the LLM, BF16 vision, e.g. the official Qwen2-VL AWQ checkpoint | about 3× faster decode and less than half the memory (Figure 2); the vision side adds no decode cost |
| Laptop, phone or edge box (llama.cpp, MLX) | LLM in 4-bit GGUF or MLX format; vision projector file in F16 [Likely] | memory and bandwidth are tight; the projector is small |
| Document and invoice extraction | conservative: 8-bit or 4-bit LLM, BF16 vision, calibration with documents | digits and small text are the first to break; test on DocVQA-style data |
| Video question answering, key-moment detection, Video-RAG | 4-bit LLM; encoder in W8A8 INT8 or FP8 if many frames make encoding slow; FP8 KV cache | thousands of frame tokens make the encoder and the KV cache matter |
| Large-scale video retrieval and fingerprinting with a vision encoder | INT8 or FP8 encoder (weights and activations), embeddings checked for recall | the encoder is the whole cost here, and it is compute-bound, so only weight–activation formats help |
| Fine-tuning a VLM on one GPU | QLoRA: 4-bit frozen LLM, adapters in BF16, vision tower in BF16 | training memory, see session 58 and session 189 |
Common mistakes
- Calibrating with text only. Many LLM tools default to a text corpus. The LLM layers then never see image tokens, and their outlier channels are left unprotected (Figure 3).
- Quantizing the vision tower to “save time”. Weight-only quantization does not speed up a compute-bound encoder. It only saves about 1 GB on a 7B VLM.
- Quantizing the connector with the LLM by accident. A name filter that matches every
Linearcatches the projector and the merger too. Print the list of quantized modules. - Testing only perplexity. Text perplexity never runs the image path. Test on documents, counting, hallucination and video.
- Testing at a different resolution than you serve. More image tokens change the token mix and the KV cache. A model that works at 448×448 may degrade at 1,344×1,344.
- Calling a 1-point drop on 300 questions a regression (or a win). Compute the standard error first, and compare answers question by question.
- Forgetting the embedding and lm_head. They stay in 16-bit in most checkpoints. In a 4-bit Qwen2-VL-7B they are about 31% of the weights in memory, so the model is bigger than “8.29 B × 4 bits”.
- Expecting prefill to get 3× faster. With many image tokens prefill is compute-bound; weight-only formats can even make it a little slower, because the kernels must also dequantize. [Likely]
Interview questions
Why does 4-bit weight-only quantization speed up decoding but not image encoding?basic
Estimate the decode speed of Qwen2-VL-7B on an A100 in BF16 and with a 4-bit LLM.core
Would you quantize the vision encoder? Defend your answer.core
Why might the connector be the most fragile part?advanced
What calibration data do you use for a VLM, and why?core
Image tokens outnumber text tokens 10 to 1 in your calibration set. Is that a problem?advanced
Your 4-bit VLM has the same WikiText perplexity as the BF16 one, but users say it misreads invoices. What happened and what do you do?core
Where do the bytes of a 4-bit Qwen2-VL-7B go?basic
When does weight–activation quantization (W8A8, FP8) beat weight-only for a VLM?advanced
How do you decide whether a 1.5-point drop after quantization is real?core
Exercises
- Using formula (197.2), what is the arithmetic intensity of the Qwen2-VL vision encoder’s linear layers in BF16 for a 448×448 image? Is it compute-bound on an L4?
- In Figure 1, switch to the L4. How long is one BF16 decode step, and how long with a 4-bit LLM?
- Qwen2-VL-7B serves 32 requests at once, each with 4 images, 50 prompt tokens and 200 output tokens, on an A100 with a 4-bit LLM. Compute the KV cache at the end of the answers. Check with Figure 2.
- In Figure 3 (3-bit, \(\alpha\) searched), find the image share of the calibration set where the text and image errors are both closest to their best values. Why does 100% image tokens hurt the text?
- You serve 1,000 questions at 70% accuracy. What is the standard error of the accuracy?
- A colleague quantizes the vision tower and connector of Qwen2-VL-7B to 4.25 bits. How much memory does that save, and how many ms per decode token?
Answerscheck
Key takeaways
- Each stage runs at the slower of compute and memory (197.1). The vision encoder and long prefills are compute-bound; decode at small batch is bandwidth-bound (197.2).
- Decode time is the bytes of LLM weights plus KV divided by bandwidth (197.3). 4-bit LLM weights give about 3× faster decode for Qwen2-VL-7B; the vision encoder is not in this formula at all (197.5).
- The vision side is small (about 8% of a 7B VLM), runs once per image, and affects every image token. Default: keep the encoder and connector in 16-bit; quantize them only for memory or video compute, and measure.
- Quantizers protect channels that are large in the calibration data (197.6, 197.7). Calibrate with real image–text conversations, match the serving mix, and balance the modalities (197.8, 197.9).
- Text perplexity cannot see vision damage. Test documents, counting, hallucination and video, compare answers one by one, and respect the noise of small eval sets (197.10).
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Li, Hu, Ning, Liu, Hong, Jia, Li, Yan, Ran, Dai, Yan, Yang & Wang (2024). MBQ: Modality-Balanced Quantization for Large Vision-Language Models
- Wang, Wang, Xu, Tang, Zhou & Lu (2024). Q-VLM: Post-training Quantization for Large Vision-Language Models
- Darcet, Oquab, Mairal & Bojanowski (2023). Vision Transformers Need Registers
- Sun, Chen, Kolter & Liu (2024). Massive Activations in Large Language Models
- Qwen team (2024). Qwen2-VL-7B-Instruct-AWQ model card and config.json