LLM & VLM courseBlog
200Track 7: Vision and VLMs · session 38 of 38

serve a VLM and measure

In this lab you serve Qwen2.5-VL-3B-Instruct with vLLM on one NVIDIA L4 and measure how the time to first token grows with the image resolution and with the number of video frames. Then you turn on prefix caching and the multimodal caches, switch the weights to FP8 and INT4, and record speed and quality side by side. A planning model built from the published config tells you what to expect, so you can tell a real effect from a broken measurement.

Time: 4 to 6 hours (mostly hands-on, on a rented GPU)Level: Intermediate · hands-on labBefore this: Sessions 196 to 199 (the VLM inference path, quantizing VLMs, serving VLMs, the L4 latency budget) and the serving-benchmark lab (session 142); Python

What you will learn

  • How to start a VLM server with vLLM with every multimodal flag written down, and how to read its start-up log
  • How to turn an image size or a frame count into image tokens with Qwen2.5-VL’s resize rule, and check the count against the server’s usage field
  • How to split TTFT into CPU preprocessing, vision encoder and LLM prefill, and why the encoder is about half of the GPU work in a 3B VLM
  • How TTFT grows with resolution and with frames, and where max-model-len and max_pixels set hard limits
  • How prefix caching, the processor cache and the encoder cache each remove repeated work, and why the image must come before the question
  • What FP8 and INT4 AWQ change in memory, TTFT, TPOT and throughput, and how to measure their quality cost with a paired test
  • How to write a results sheet that someone else can reproduce

Why this matters

A vision-language model (VLM) server is judged first by how long the user waits before the answer starts: the time to first token (TTFT). For a text model, TTFT is mostly one prefill. For a VLM it is much more: the server must decode and resize the image on the CPU, run a vision encoder on the GPU, and then prefill hundreds or thousands of extra image tokens. Each of these costs grows with the image size and with the number of video frames. In this lab you serve a real small VLM, measure exactly how TTFT grows, and then try the three main tools that make it cheaper: caching, FP8 and INT4.

The lab uses the whole serving part of Track 7. Session 196 followed one request through every stage and counted its cost. Session 197 explained which parts to quantize and why the vision encoder usually stays in 16-bit. Session 198 compared the engines, and Session 199 did the latency math on paper for the same GPU we use here. The method comes from session 142, the text-model version of this lab: write the setup down, predict, measure, change one thing at a time. The next session, session 201, starts the video track, where the frame budget you measure here becomes the central problem.

The lab has four parts, one for each key point of the session:

  1. Serve Qwen2.5-VL-3B-Instruct with vLLM on one NVIDIA L4, and check that the server counts image tokens the way you expect.
  2. Measure TTFT against the image resolution, and against the number of video frames.
  3. Turn on prefix caching and the multimodal caches, and measure what repeated images cost.
  4. Switch to FP8 and to INT4 (AWQ) weights, and record speed and quality for each.
What is measured and what is modeled on this pageThe commands follow the vLLM documentation of November 2026 [Certain for the flags we quote from it]. We did not run vLLM for this page: we had no NVIDIA GPU. The L4 latencies in the figures come from a small planning model (given in full below) that uses the model’s published config, the L4 spec sheet and stated assumptions. It is the “expected result” of the lab, not a measurement. Three things we did measure ourselves, and they are labeled [Certain, our run]: the image-token counts of the Hugging Face processor, its CPU time on one laptop core, and a dry run of the benchmark client against a fake server.

The setup: one L4, one small VLM, every flag written down

A measurement is only useful if someone else can repeat it. So before the first request you write down the GPU, the model, the engine version and every setting that changes what the server does with an image. Do not trust defaults: they change between versions, and for VLMs there are more of them (resize limits, frame sampling, cache sizes).

The GPU. An NVIDIA L4: 24 GB of memory, 300 GB/s of memory bandwidth, 121 TFLOP/s of dense BF16 matrix math and 242 TFLOP/s of dense FP8 [Certain: NVIDIA lists 242 and 485 “with sparsity”; dense is one half]. It supports FP8 tensor-core math (compute capability 8.9) [Certain]. It is a cheap 72 W inference card that most clouds rent.

The model. Qwen2.5-VL-3B-Instruct in BF16. From its config.json [Certain]: a language model with 36 layers, width 2,048, 16 query heads and 2 key-value heads of size 128 (grouped-query attention), a feed-forward width of 11,008, a vocabulary of 151,936 and an output layer that is tied to the input embedding. The vision encoder is a ViT with 32 layers of width 1,280; only layers 7, 15, 23 and 31 use full attention, the others attend inside windows of 112 × 112 pixels. Patches are 14 × 14 pixels, two consecutive video frames form one temporal patch, and a 2 × 2 merger turns four patches into one language-model token. Hugging Face counts 3,754,622,976 parameters; from the shapes we get 668.7 M in the vision tower, 311.2 M in the embedding and 2,774.8 M in the language-model blocks, which adds up exactly [Certain, our count].

\[ C_{\text{KV}} = \left\lfloor \frac{u\,M - W - R}{\text{kv}_{\text{tok}}} \right\rfloor, \qquad \text{kv}_{\text{tok}} = 2\,L\,n_{kv}\,d_h\,s \tag{200.1} \]
Read the symbols
\(C_{\text{KV}}\)
tokens of KV cache that fit on the GPU, summed over all running requests
\(u\)
fraction of GPU memory the engine may use (--gpu-memory-utilization)
\(M, W\)
GPU memory and the bytes of the stored weights (vision tower included)
\(R\)
memory for everything else: activations of the largest step, the vision encoder’s activations for the largest image, CUDA graphs; here an assumed 2.0 GB
\(\text{kv}_{\text{tok}}\)
bytes of cache per token: a key and a value (the 2) for each of \(L\) layers and \(n_{kv}\) heads of size \(d_h\), with \(s\) bytes per number
Worked example: memory of Qwen2.5-VL-3B on an L4

Weights in BF16: \(3{,}754{,}622{,}976 \times 2 = 7.51\) GB. Cache per token: \(2 \cdot 36 \cdot 2 \cdot 128 \cdot 2 = 36{,}864\) bytes (36 KiB). With \(u = 0.9\) and \(R = 2.0\) GB: \(0.9 \cdot 24 - 7.51 - 2.0 = 12.09\) GB for the cache, so \(C_{\text{KV}} = 327{,}982\) tokens.

A 1920 × 1080 image becomes 2,691 image tokens (we show why below); with the chat template the prompt has 2,719 tokens, and with 256 output tokens one request needs 2,975 tokens of cache. So about \(\lfloor 327{,}982 / 2{,}975 \rfloor = 110\) such requests fit at once. One 1080p image costs \(2{,}719 \times 36{,}864 = 100\) MB of cache. Compare a 7B LLaVA-1.5 with full multi-head attention, where one 576-token image costs about 300 MB (session 196): the 2 key-value heads of this model make image tokens cheap to store. They are not cheap to compute, as we will see.

The engine. Install vLLM in a fresh environment, record its version (pip show vllm), and start the server with every setting written out:

# the baseline: BF16, one image or one video per prompt, NO prefix caching, NO processor cache
vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
  --dtype bfloat16 --max-model-len 32768 --gpu-memory-utilization 0.9 \
  --max-num-seqs 64 --max-num-batched-tokens 8192 \
  --limit-mm-per-prompt '{"image": 1, "video": 1}' \
  --disable-prefix-caching --mm-processor-cache-gb 0 \
  --port 8000

What each multimodal flag does, from the engine-argument docs [Certain]. --limit-mm-per-prompt caps the images and videos one prompt may carry; the engine also uses it to size its memory profiling, so a smaller limit leaves more memory for the cache. --mm-processor-cache-gb is the size of a CPU cache that keeps the processed pixels of recent images (default 4 GiB; 0 turns it off). --mm-processor-kwargs passes options to the Hugging Face processor, such as max_pixels (Part 1). --media-io-kwargs '{"video": {"num_frames": 40}}' sets how many frames the server samples when it decodes a video file itself. We turn both caches off for the baseline so that the resolution sweep measures real work; Part 3 turns them on.

When the server is ready, read its start-up log: the KV-cache size in tokens, the maximum concurrency for 32,768-token requests, and whether the model loaded in the precision you asked for [Likely: the exact wording of the log changes between versions]. Then send one request by hand. The response’s usage.prompt_tokens field is your first check: for the 1920 × 1080 test image and our one-sentence question it should be 2,719.

How to measure TTFT for one image

For this lab the question is “how long does one request wait, as a function of what it carries?”, not “how much load can the server take?”. So send one request at a time, with nothing else running: then TTFT has no queueing time in it, and you see the pure cost of the image. (The load question needs open-loop traffic and is the subject of session 142; Figure 4 comes back to batching.)

First predict the token count. Qwen2.5-VL keeps the aspect ratio of the image. Its processor rounds the height and the width to multiples of 28 pixels, then scales the image down if it has more than max_pixels (default 12,845,056) or up if it has fewer than min_pixels (default 3,136) [Certain, from the processor code and config]:

\[ H' = 28 \cdot \operatorname{round}\!\left(\tfrac{H}{28}\right), \quad W' = 28 \cdot \operatorname{round}\!\left(\tfrac{W}{28}\right), \qquad N_{\text{tok}} = \left\lceil \tfrac{T}{2} \right\rceil \cdot \frac{H'}{28} \cdot \frac{W'}{28} \tag{200.2} \]
Read the symbols
\(H, W\)
height and width of the image (or of each video frame) that reaches the processor
\(H', W'\)
after resizing: multiples of 28, because a token covers 2 × 2 patches of 14 pixels
\(T\)
number of frames; an image counts as \(T = 1\) (it is repeated to fill one temporal patch); an odd frame count is padded to the next even one
\(N_{\text{tok}}\)
image or video tokens that the language model sees; the vision encoder works on \(4 N_{\text{tok}}\) patches
Worked example: tokens of a 1920 × 1080 image and of a 32-frame clip

\(1080 / 28 = 38.57\), rounded to 39, so \(H' = 1{,}092\); \(1920 / 28 = 68.57\), rounded to 69, so \(W' = 1{,}932\). That gives \(39 \times 69 = 2{,}691\) image tokens and \(78 \times 138 = 10{,}764\) patches in the encoder. The chat template adds 28 text tokens (system prompt, roles, the two vision markers and our question), so the prompt has 2,719 tokens.

A 640 × 360 clip of 32 frames: each frame becomes \(13 \times 23 = 299\) tokens per temporal patch, and 32 frames make 16 temporal patches, so \(16 \times 299 = 4{,}784\) video tokens, and 4,811 prompt tokens with the video template.

We checked these counts with the Hugging Face processor of the model (transformers 5.23): it returned an image grid of 1 × 78 × 138 and a prompt of 2,719 tokens, and a video grid of 16 × 26 × 46 and a prompt of 4,811 tokens [Certain, our run].

Algorithm 1: measure TTFT for one image size or one frame count
  1. input: width \(W\), height \(H\), frames \(T\) (0 for an image), repeats \(k\)
  2. predict \(N_{\text{tok}}\) with (200.2)
  3. for \(j = 0, 1, \dots, k\):
  4.   make a new test image (or \(T\) new frames) with seed \(j\) // new bytes: no cache can hit
  5.   encode it as a JPEG in a base64 data URL; build the chat message: media first, then the question
  6.   \(t_0 \leftarrow\) now; send the request with streaming on
  7.   TTFT\(_j \leftarrow\) arrival time of the first text chunk \(- t_0\); read usage.prompt_tokens from the last chunk
  8. drop request 0 (warm-up); report the median TTFT and the prompt tokens
  9. check: prompt tokens \(= N_{\text{tok}}\) + template tokens; if not, the server resized differently than you think

The last line is the most useful check in the lab. If the server reports fewer tokens than you predicted, something resized the image: a max_pixels setting, a client library, or a different processor version. If it reports more, your frames were not sampled the way you think. Fix this before you measure any time.

A model to compare against

Before you rent the GPU, predict the result. Our planning model splits TTFT into four parts:

\[ \text{TTFT} = t_{\text{fix}} + t_{\text{prep}} + \frac{F_{\text{enc}}}{\eta_{\text{enc}}\, F_{\text{peak}}} + \sum_{\text{chunks}} \left[ \max\!\left( \frac{F_c}{\eta_c\, F_{\text{peak}}},\ \frac{Q_c}{\eta_m\, \text{BW}} \right) + t_{\text{oh}} \right] \tag{200.3} \]
\[ F_{\text{enc}} = 2 P_{\text{blk}}\, N_p + 4\, d_v\, n_s\, N_s \big( L_{\text{full}}\, N_s + L_{\text{win}}\, w \big), \qquad F_c = 2 P_{\text{LLM}}\, c + 4 L d_a \big( c\,p + \tfrac{c(c+1)}{2} \big) \tag{200.4} \]
Read the symbols
\(t_{\text{fix}}\)
fixed cost per request: HTTP, tokenization, scheduling; assumed 10 ms
\(t_{\text{prep}}\)
CPU time to decode the base64 JPEG and run the image processor (resize, normalize, cut patches); 46 ms per megapixel in our run on one laptop core
\(P_{\text{blk}}, N_p\)
matmul weights of the 32 vision blocks (0.630 B, plus the patch embedding) and the number of patches
\(n_s, N_s\)
temporal slices (1 for an image, \(\lceil T/2 \rceil\) for a video) and patches per slice; full attention runs inside each slice, not across the whole video [Certain, from the model code]
\(d_v, L_{\text{full}}, L_{\text{win}}, w\)
encoder width 1,280; 4 full-attention layers; 28 window layers; 64 patches per window (8 × 8)
\(F_c, c, p\)
FLOPs of one prefill chunk of \(c\) tokens (at most 8,192, the token budget) after \(p\) tokens already in the cache
\(P_{\text{LLM}}, d_a\)
matmul weights of the 36 language-model blocks (2.775 B) and the attention width (16 heads × 128 = 2,048)
\(Q_c\)
bytes the chunk reads: the weights once plus the KV cache so far
\(\eta_{\text{enc}}, \eta_c, \eta_m\)
assumed fractions of peak that real kernels reach: 0.4 for the encoder (narrow layers, short windows), 0.6 for the language model, 0.8 for bandwidth
\(t_{\text{oh}}\)
fixed cost of one engine step; assumed 2 ms

The model ignores two things on purpose: the merger and the patch embedding are small, and attention to the windows at the image border is a bit cheaper than a full window. We checked the per-layer formula with PyTorch’s FLOP counter on one vision block (code below): for 10,752 patches, a window layer costs 0.427 TFLOP and a full-attention layer 1.015 TFLOP, the same as the formula to three digits [Certain, our run].

Worked example: TTFT of one 1920 × 1080 image on an L4, BF16

Encoder. 10,764 patches. Matmuls: \(2 \times 0.6315 \times 10^9 \times 10{,}764 = 13.59\) TFLOP, plus 0.20 for the merger. Attention: \(4 \cdot 1280 \cdot 10{,}764 \cdot (4 \cdot 10{,}764 + 28 \cdot 64) = 2.47\) TFLOP. Total 16.26 TFLOP; at \(0.4 \times 121\) TFLOP/s that is 336 ms.

Prefill. 2,719 tokens: \(2 \times 2.775 \times 10^9 \times 2{,}719 = 15.09\) TFLOP plus 1.09 TFLOP of attention, 16.18 TFLOP in total; at \(0.6 \times 121\) TFLOP/s, 223 ms, plus 2 ms of step overhead: 225 ms. The weights and the cache are only 6.27 GB, 26 ms at 240 GB/s, so the prefill is compute-bound.

CPU. 2.07 megapixels at 46 ms per megapixel: 95 ms. With the 10 ms fixed cost, TTFT ≈ 666 ms.

The surprise: the vision encoder costs as many FLOPs as the whole language-model prefill (16.26 against 16.18 TFLOP), and more time, because its kernels are less efficient. The reason is the ratio from session 196: the encoder sees 4 patches per language-model token, and its 0.63 B weights are almost a quarter of the 2.78 B of the language model, so \(4 \times 0.63 / 2.78 \approx 0.9\). In a 3B VLM, the vision tower is not a small add-on.

Part 1: TTFT against image resolution

Sweep eight 16:9 sizes from 640 × 360 to 3840 × 2160, five new images each, with the baseline server. The client below does one size per call. Plot TTFT against the prompt tokens that the server reports, not against the pixel count: tokens are what the GPU works on. Figure 1 shows what the planning model predicts for BF16, FP8 and INT4.

Weights
fixed costCPU preprocessingvision encoderLLM prefill
Figure 1. TTFT against image size, from the planning model (not measured): Qwen2.5-VL-3B-Instruct on one L4, batch 1, a new image every time (no cache hits), 28 text tokens. The bars split TTFT for the chosen size in all three formats; the lower panel shows TTFT against image tokens for all eight sizes. The checkbox applies the token cap that the model card suggests as a speed setting.
How to read Figure 1

Start at 1920 × 1080 in BF16: 2,691 image tokens, CPU 95 ms, encoder 336 ms, prefill 225 ms, TTFT 666 ms; the encoder is 60% of the GPU time. Move to 640 × 360: 299 tokens and 81 ms. Move to 3840 × 2160: 10,549 tokens and 3.31 s. From the smallest to the largest size the tokens grow 35 times and TTFT 41 times: a bit faster than linear, because the four full-attention encoder layers and the prefill attention grow with the square of the length. Switch to FP8: 435 ms at 1080p, because both the encoder and the prefill run on FP8 tensor cores (we assume the vision tower is quantized too; see Part 4). Switch to INT4: 741 ms, slower than BF16, because AWQ leaves the vision tower in BF16 and its 4-bit matmuls are slower in prefill (an assumption). Now tick the max_pixels box: every image above about one megapixel is shrunk to 1,222 tokens, and 4K costs 633 ms instead of 3.31 s.

What the curves say

  • TTFT follows the token count, almost linearly. Twice the pixels means about twice the tokens, twice the encoder work and twice the prefill. The square terms (full attention) start to show above about 5,000 tokens.
  • The encoder is half of the GPU work at every size. This is special to small VLMs with a large vision tower. For a 72B language model with the same encoder, the encoder would be a few percent.
  • The CPU is a real part of TTFT. In our run the Hugging Face processor took 12 ms for 640 × 360, 98 ms for 1920 × 1080 and 386 ms for 3840 × 2160 on one core [Certain, our run, an Apple M5 Pro laptop; a cloud vCPU may be slower]. If the API server handles requests one at a time, that time also blocks the next request.
  • The cheapest fix is fewer pixels. The max_pixels cap cuts the cost of large images by up to 5 times, and it is one flag: --mm-processor-kwargs '{"max_pixels": 1003520}' [Likely: check that your vLLM version forwards it to this processor]. Whether the answers stay good is a quality question: measure it as in Part 4 on your own images, because fine text in documents needs pixels.

Part 2: TTFT against the number of video frames

Qwen2.5-VL takes a video as a stack of frames. Two consecutive frames are merged into one temporal patch, so a video costs half as many tokens as the same frames sent as separate images [Certain, from temporal_patch_size = 2]. To control the frame count exactly, extract the frames on the client and send them as one video_url of comma-joined base64 JPEGs (data:video/jpeg;base64,...), which the vLLM docs describe for pre-extracted frames [Certain]. If you send a video file instead, the server samples the frames itself, and --media-io-kwargs sets how many.

Sweep 4, 8, 16, 32, 64 and 128 frames at three frame sizes: 640 × 360, 848 × 480 and 1280 × 720. Two limits will appear. The prompt must fit in --max-model-len (32,768 here) with room for the answer; longer prompts are rejected with an error, not truncated. And the tokens per frame depend on the processor version: our transformers 5.23 processor kept 1280 × 720 frames at full size (1,196 tokens per temporal patch) and printed a warning that a per-frame pixel cap, as in the reference qwen-vl-utils code, will become the default in a later version [Certain, our run]. vLLM may also apply its own limits [Likely]. So again: read usage.prompt_tokens.

Frame size
Figure 2. TTFT against video frames, from the planning model (not measured): BF16, batch 1, new frames every time, 27 text tokens, 256 output tokens reserved. The grey dashed part of a curve is rejected by the server, because the prompt plus the answer would be longer than --max-model-len 32768.
How to read Figure 2

At the default, 32 frames of 640 × 360: 16 temporal patches of 299 tokens, 4,784 video tokens, 169 MiB of KV cache, and a TTFT of 1.29 s, of which 0.34 s is CPU work on 32 frames. Switch to 1280 × 720: the same 32 frames now cost 19,136 tokens and 5.78 s. Move the slider and watch the 720p curve turn grey after 54 frames: 56 frames would need \(28 \times 1{,}196 + 27 + 256 = 33{,}771\) tokens. At 848 × 480 the limit is 126 frames, at 640 × 360 it is 216 (beyond the slider). Finally compare 8 frames of 720p with 32 frames of 360p: both are 4,784 tokens and cost about the same (1.31 s and 1.29 s). What counts is frames times pixels, not either one alone.

This is the “budget problem” that the video track starts with in session 201. A 3B model on an L4 needs more than a second of TTFT for 32 small frames, and its context fills up at about 50 frames of 720p. To cover a longer video you must choose: fewer frames (lower fps), smaller frames, or fewer tokens per frame (token merging or pruning, session 183). Each choice trades detail in time against detail in space.

Part 3: prefix caching and the two multimodal caches

Real traffic repeats itself. A user asks three questions about the same photo; many users send the same product image; a client retries a request. A vLLM server has three caches that remove this repeated work, and each one removes a different part of TTFT:

  • Processor cache (CPU, --mm-processor-cache-gb): keeps the processed pixels of recent images, keyed by a hash of the image. A hit skips the resize and normalize step; the server still decodes the upload to compute the hash. In our run, base64 decode, JPEG decode and a SHA-256 of the pixels took 12 ms of the 98 ms that the full processing of a 1080p image took [Certain, our run]. vLLM’s hash function may differ [Likely].
  • Encoder cache (GPU, always on in the V1 engine): keeps the vision encoder’s output embeddings. The source code says it is shared between requests by the image hash, and that old entries with no running request are evicted only when space is needed [Certain, vLLM source]. A hit skips the encoder. When both of the other caches are off, the engine may not compute content hashes at all, so we assume no sharing in that case [Likely].
  • Prefix cache (GPU, --enable-prefix-caching): keeps KV blocks of 16 tokens, keyed by a hash of all tokens up to that block and the hashes of the images inside it (session 134). A hit skips the prefill of those tokens. If every image token is inside the cached prefix, the encoder is not needed either.
\[ h = B \left\lfloor \frac{\min(m,\ n - 1)}{B} \right\rfloor, \qquad n_{\text{new}} = n - h, \qquad \text{encoder needed} \iff h \lt e \tag{200.5} \]
Read the symbols
\(m\)
leading tokens of the new prompt that match a cached sequence (same token ids and same image hashes)
\(B\)
block size, 16 tokens; only full blocks are cached, and the last prompt token is always computed (it produces the first logits)
\(h, n_{\text{new}}\)
tokens served from the cache, and tokens that must still be prefilled
\(e\)
position just after the last image token; if the cached prefix ends before \(e\), the image’s embeddings are needed again (from the encoder cache, or by running the encoder)
Worked example: a second question about the same 1080p image

We tokenized both prompts with the model’s chat template [Certain, our run]. The image tokens sit at positions 15 to 2,705, the end-of-vision marker at 2,706, and the two prompts match for \(m = 2{,}707\) tokens (up to the question). Then \(h = 16 \lfloor 2{,}707 / 16 \rfloor = 2{,}704\): the last two image tokens fall into a block that is not full, so they are not cached. The server prefills \(2{,}720 - 2{,}704 = 16\) tokens, which takes 28 ms (it is now memory-bound: one read of the weights), and it needs the image embeddings for two tokens, which the encoder cache provides. Without that cache, the whole 336 ms encoder would run again for two tokens.

If the question comes before the image, the prompts differ right after the user role marker (\(m = 14\)), so \(h = 0\): all 2,720 tokens are prefilled again. Only the processor cache and the encoder cache help.

Algorithm 2: the caching experiment
  1. start the server with both caches off; send the six requests of Figure 3 in order, one at a time; record TTFT and prompt_tokens
  2. restart with --enable-prefix-caching only; repeat; then with --mm-processor-cache-gb 4 only; then with both
  3. the repeated image must be the same bytes: send the same base64 string, or give it the same uuid field [Certain: the docs describe an optional uuid per media item]
  4. after each run, read the hit counters from the server’s /metrics endpoint (prefix-cache queries and hits, multimodal-cache queries and hits) [Likely: metric names change between versions]
  5. compare each request with the prediction of (200.5); a request that should hit but does not is a bug worth finding: different bytes, a changed template, or an evicted entry
Encoder cache: always on
fixed costCPU preprocessingvision encoderLLM prefill
Figure 3. Six requests in order, from the planning model (not measured): BF16, batch 1, 1920 × 1080 images, token positions from the real chat template. The dashed box behind each bar is the same request with every cache off. Click a row for its details.
How to read Figure 3

With both caches on, requests 2, 3 and 6 cost about 50 ms instead of 666 ms: 12 ms of CPU, 28 ms of prefill, 10 ms fixed. Request 4 brings a new image and pays the full 666 ms. Request 5 has the question before the image, so the prefix cache misses; the processor and encoder caches still save 419 ms, and it costs 247 ms. Turn off the processor cache: the hits cost 134 ms, because the CPU work (95 ms) is now the largest part. Turn off the prefix cache too: every request costs about 666 ms, and the six together take 4.00 s instead of 1.73 s. Click request 2 with prefix caching on and read the token counts: 2,704 cached and 16 prefilled, as in the worked example.

Two lessons for prompt design. Put the shared part first: system prompt, then the image, then the question. And keep the bytes identical: a client that re-encodes the JPEG on every turn, or resizes it slightly differently, produces a new hash and misses every cache.

Part 4: FP8 and INT4, speed and quality

Now change the weights. Each change is one experiment, with the same sweeps as Parts 1 and 2, plus a quality check.

FP8. vLLM can quantize a BF16 checkpoint to FP8 when it loads it, with no calibration data. The current docs spell the option --quantization fp8_per_tensor; older versions use --quantization fp8 [Certain for the current docs; check vllm serve --help]. In this mode “all Linear modules (except for the final lm_head)” get FP8 weights, and the activations get a dynamic per-tensor scale; the docs warn that latency gains are limited in this mode [Certain]. On the L4 the matmuls run on FP8 tensor cores (W8A8). In the current vLLM code the Qwen2.5-VL vision tower receives the same quantization config as the language model, so its linear layers are probably FP8 too [Likely: check the log]. For the best FP8 speed, use a checkpoint that was quantized offline with static scales.

INT4. Serve the official AWQ checkpoint Qwen/Qwen2.5-VL-3B-Instruct-AWQ. Its config says 4 bits, group size 128, with zero points, and "modules_to_not_convert": ["visual"]: the whole vision tower stays in BF16 [Certain]. vLLM reads the method from the config. Expect decode to be fast (fewer bytes per step) and prefill to be no faster, because the 4-bit weights are unpacked and the math still runs in 16 bits (session 197).

\[ \text{TPOT}(b) = \frac{W_{\text{read}} + b \cdot c \cdot \text{kv}_{\text{tok}}}{\eta_m\, \text{BW}} + t_{\text{oh}}, \qquad \text{throughput}(b) = \frac{b}{\text{TPOT}(b)} \tag{200.6} \]
Read the symbols
\(W_{\text{read}}\)
weight bytes one decode step reads: the language-model matmuls in the chosen format, plus the tied 151,936 × 2,048 output layer in BF16 (0.62 GB). The vision tower is not read during decode.
\(b, c\)
requests decoding together, and their average context length
\(\text{TPOT}\)
time per output token for each request; this is a decode-only step, with no prefill mixed in

Decode is memory-bound, so (200.6) uses only bytes. In BF16 a step reads 6.17 GB; in FP8 3.40 GB; in INT4 2.06 GB (4 bits plus a 16-bit scale and a 4-bit zero for every 128 weights: 0.52 bytes per weight). At batch 1 this predicts a TPOT of 28.2, 16.6 and 11.0 ms. Figure 4 adds the batch size.

Workload
Figure 4. The three weight formats, from the planning model (not measured): memory with formula (200.1), TTFT at batch 1 with (200.3), and TPOT and throughput of a decode step with (200.6), for requests with 256 output tokens (average context: prompt + 128). When the KV cache cannot hold the chosen batch, the batch is capped at what fits.
How to read Figure 4

At batch 16 with the image workload: TPOT is 34.7 ms in BF16, 23.2 in FP8 and 17.6 in INT4, so throughput is 461, 691 and 909 output tokens per second. INT4 wins decode, but its TTFT (741 ms) is the worst, and FP8 has the best TTFT (435 ms): the two formats help different metrics. Move to batch 128: BF16 is capped at 110 requests by its KV cache, and the three formats come closer (75.8, 72.1 and 66.6 ms), because the KV cache reads, which quantized weights do not shrink, now dominate each step. Switch to the video workload: each request is longer, so BF16 fits only 64 requests, FP8 83 and INT4 86. The memory gain of quantizing a 3B model is small (7.51 GB against 4.07 and 3.40 GB) next to the 24 GB of the L4.

Quality: measure it on the same questions

A faster wrong answer is not a win. The Qwen team reports, for the 3B model, BF16 against AWQ, evaluated with VLMEvalKit [Certain, from the AWQ model card]:

Qwen2.5-VL-3B-InstructMMMU valDocVQA valMMBench dev (en)MathVista mini
BF1651.793.079.861.4
AWQ (INT4)49.191.878.058.8
Difference−2.6−1.2−1.8−2.6

The card gives no FP8 numbers, and these are their numbers, not yours. Run your own check on the task you care about. The script below sends the first 500 ChartQA test questions to each server at temperature 0 and scores them with relaxed accuracy (a number is right if it is within 5% of the answer) [Certain, ChartQA paper]. Because every server answers the same questions, compare them per question, not only by their totals:

\[ \hat\Delta = \frac{n_{01} - n_{10}}{n}, \qquad \text{SE}(\hat\Delta) = \frac{1}{n}\sqrt{\,n_{01} + n_{10} - \frac{(n_{01} - n_{10})^2}{n}\,} \tag{200.7} \]
Read the symbols
\(n\)
questions, the same for both servers
\(n_{10}, n_{01}\)
questions that the baseline got right and the quantized model wrong, and the other way around (the “flips”)
\(\hat\Delta\)
accuracy of the quantized model minus the baseline; \(\hat\Delta \pm 1.96\,\text{SE}\) is a 95% interval. The exact McNemar test asks whether the flips split 50/50.
Worked example: reading a small difference (made-up counts, to show the arithmetic)

Suppose that on 500 questions the quantized model loses 14 that BF16 got right and wins 6 that BF16 got wrong. Then \(\hat\Delta = (6 - 14)/500 = -1.6\) points, \(\text{SE} = \sqrt{20 - 64/500}\,/\,500 = 0.89\) points, and the 95% interval is \([-3.3, +0.1]\). The exact McNemar p-value for 6 against 14 flips is 0.115. So this run cannot tell a 1.6-point loss from noise; it can only say that the loss is probably not larger than about 3 points. To detect a 1-point difference you need many more questions, or a task with more flips. Note that only the 20 flips carry information; the 480 questions that both got right or both got wrong do not.

The code

Three short programs. The first is the TTFT client. It uses the standard library and Pillow, speaks the OpenAI chat API that vLLM serves, makes new image bytes for every request, sends media before text, streams the answer, and reads prompt_tokens from the usage chunk at the end.

# bench_vlm.py: TTFT of a VLM server against image size and video frames (OpenAI chat API, streaming)
# Needs only the standard library and Pillow. Usage:
#   python3 bench_vlm.py URL MODEL image 1920x1080 [REPEATS]
#   python3 bench_vlm.py URL MODEL video 640x360 32 [REPEATS]
import base64, io, json, math, random, statistics, sys, time, urllib.request
from PIL import Image

def test_image(w, h, seed):
    """A synthetic photo-like JPEG. A new seed gives new bytes, so no cache can hit."""
    rnd = random.Random(seed)
    small = Image.new("RGB", (16, 9))
    small.putdata([tuple(rnd.randrange(256) for _ in range(3)) for _ in range(16 * 9)])
    img = small.resize((w, h), Image.BICUBIC)            # smooth color fields, like a photo
    buf = io.BytesIO(); img.save(buf, "JPEG", quality=90)
    return base64.b64encode(buf.getvalue()).decode()

def one_request(url, model, content, max_tokens=32):
    body = json.dumps({"model": model, "max_tokens": max_tokens, "temperature": 0, "stream": True,
                       "stream_options": {"include_usage": True},
                       "messages": [{"role": "user", "content": content}]}).encode()
    req = urllib.request.Request(url + "/v1/chat/completions", body, {"Content-Type": "application/json"})
    t0 = time.perf_counter(); t_first = None; times = []; usage = None
    with urllib.request.urlopen(req, timeout=600) as resp:
        for line in resp:                                 # server-sent events: b"data: {...}"
            line = line.strip()
            if not line.startswith(b"data: ") or line == b"data: [DONE]":
                continue
            ev = json.loads(line[6:])
            if ev.get("usage"):
                usage = ev["usage"]
            if ev.get("choices") and ev["choices"][0]["delta"].get("content"):
                times.append(time.perf_counter())
                t_first = t_first or times[-1]
    tpot = (times[-1] - times[0]) / (len(times) - 1) if len(times) > 1 else float("nan")
    return t_first - t0, tpot, usage["prompt_tokens"]

def run(url, model, kind, size, frames=0, repeats=5):
    w, h = map(int, size.split("x")); rows = []
    for k in range(repeats + 1):                          # request 0 is a warm-up and is dropped
        if kind == "image":
            media = {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64," + test_image(w, h, k)}}
        else:                                             # pre-extracted frames, comma-joined base64 JPEGs
            fr = ",".join(test_image(w, h, 1000 * k + j) for j in range(frames))
            media = {"type": "video_url", "video_url": {"url": "data:video/jpeg;base64," + fr}}
        content = [media, {"type": "text", "text": "Describe this in one sentence."}]
        rows.append(one_request(url, model, content))
    rows = rows[1:]
    ttft = statistics.median(r[0] for r in rows); tpot = statistics.median(r[1] for r in rows)
    print(json.dumps({"kind": kind, "size": size, "frames": frames, "prompt_tokens": rows[0][2],
                      "ttft_p50_ms": round(1e3 * ttft, 1), "ttft_max_ms": round(1e3 * max(r[0] for r in rows), 1),
                      "tpot_p50_ms": round(1e3 * tpot, 2)}))

if __name__ == "__main__":
    a = sys.argv[1:]
    if a[2] == "image":
        run(a[0], a[1], "image", a[3], 0, int(a[4]) if len(a) > 4 else 5)
    else:
        run(a[0], a[1], "video", a[3], int(a[4]), int(a[5]) if len(a) > 5 else 5)

A shell loop runs the sweeps: for s in 640x360 960x540 1280x720 1600x900 1920x1080 2560x1440 3200x1800 3840x2160; do python3 bench_vlm.py http://localhost:8000 Qwen/Qwen2.5-VL-3B-Instruct image $s; done, and the same with video 640x360 $t for each frame count. We dry-ran the client without a GPU, against a fake server that counts tokens with the Qwen resize rule and waits 20 ms plus 0.05 ms per token. It printed prompt_tokens of 327, 2,719 and 10,577 for 640 × 360, 1920 × 1080 and 3840 × 2160, and 4,811 for 32 frames of 640 × 360: the same counts as the real Hugging Face processor [Certain, our run]. Its TTFTs were about 10 ms above the fake server’s sleep, which is the time the fake server spent parsing the JSON and the JPEG; a real server pays that too, and it belongs in TTFT.

For a load test with images, vLLM’s own client has a random-mm dataset: for example vllm bench serve --backend openai-chat --endpoint /v1/chat/completions --dataset-name random-mm --random-mm-bucket-config '{(1080, 1920, 1): 1.0}' --random-mm-base-items-per-request 1 --request-rate 2. The bucket keys are (height, width, frames); the docs note that only image sampling is supported for now [Certain].

The second program is the quality check of Part 4. We tested its scoring and its paired comparison on made-up correctness lists (they print the numbers of the worked example above), but we did not run it against a server.

# quality_vlm.py: the same 500 ChartQA questions against two servers, then a paired comparison
# Usage: python3 quality_vlm.py URL MODEL OUT.json      (once per server: BF16, FP8, AWQ)
#        python3 quality_vlm.py compare A.json B.json
import base64, io, json, math, sys, urllib.request

def relaxed_match(pred, gold):                     # ChartQA: numbers within 5%, else exact (case-insensitive)
    p, g = pred.strip().rstrip(".%").replace(",", ""), gold.strip().rstrip("%").replace(",", "")
    try:
        return abs(float(p) - float(g)) <= 0.05 * abs(float(g))
    except ValueError:
        return p.lower() == g.lower()

def ask(url, model, png, question):
    msg = [{"type": "image_url", "image_url": {"url": "data:image/png;base64," + base64.b64encode(png).decode()}},
           {"type": "text", "text": question + "\nAnswer with a single word or number."}]
    body = json.dumps({"model": model, "messages": [{"role": "user", "content": msg}],
                       "max_tokens": 16, "temperature": 0}).encode()
    req = urllib.request.Request(url + "/v1/chat/completions", body, {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        return json.loads(r.read())["choices"][0]["message"]["content"]

def evaluate(url, model, out, n=500):
    from datasets import load_dataset                # pip install datasets
    ds = load_dataset("HuggingFaceM4/ChartQA", split="test").select(range(n))
    rows = []
    for ex in ds:
        buf = io.BytesIO(); ex["image"].save(buf, "PNG")
        pred = ask(url, model, buf.getvalue(), ex["query"])
        rows.append({"q": ex["query"], "gold": ex["label"][0], "pred": pred, "ok": relaxed_match(pred, ex["label"][0])})
    json.dump(rows, open(out, "w"), indent=0)
    print(f"{out}: relaxed accuracy {100 * sum(r['ok'] for r in rows) / n:.1f}% on {n} questions")

def compare(a_file, b_file):                         # paired: the same questions in the same order
    a, b = json.load(open(a_file)), json.load(open(b_file)); n = len(a)
    lost = sum(x["ok"] and not y["ok"] for x, y in zip(a, b))    # right in A, wrong in B
    won = sum(y["ok"] and not x["ok"] for x, y in zip(a, b))     # wrong in A, right in B
    d = (won - lost) / n; se = math.sqrt(lost + won - (lost - won) ** 2 / n) / n
    k = min(lost, won); p = min(1.0, 2 * sum(math.comb(lost + won, i) for i in range(k + 1)) / 2 ** (lost + won))
    print(f"B - A = {100 * d:+.1f} points, 95% CI [{100 * (d - 1.96 * se):+.1f}, {100 * (d + 1.96 * se):+.1f}], "
          f"flips {lost} lost / {won} won, exact McNemar p = {p:.3f}")

if __name__ == "__main__":
    compare(*sys.argv[2:4]) if sys.argv[1] == "compare" else evaluate(*sys.argv[1:4])

The third program is the planning model behind Figures 1 to 4. It is deterministic and has no dependencies. When you have measurements, fit it: change \(\eta_m\) until TPOT at batch 1 matches, \(\eta_c\) until the TTFT of a tiny image matches (it is mostly prefill), and \(\eta_{\text{enc}}\) until the large images match. If the fitted model then predicts the video sweep too, you understand your server.

# vlm_sim.py: the planning model behind Figures 1 to 4 (Qwen2.5-VL-3B-Instruct on one NVIDIA L4)
import math
DV, IV, DEPTH, L_FULL, L_WIN, WIN = 1280, 3420, 32, 4, 28, 64           # vision_config
P_BLK = (DV * 3 * DV + DV * DV + 3 * DV * IV) * DEPTH                    # 0.630 B: qkv, proj, gated MLP
P_PATCH, P_MERGE = 3 * 2 * 14 * 14 * DV, 5120 * 5120 + 5120 * 2048       # patch embedding, 2x2 merger
D, L, H, KVH, DH, V, FF = 2048, 36, 16, 2, 128, 151936, 11008            # text config
P_LIN = L * (2 * D * D + 2 * D * KVH * DH + 3 * D * FF)                  # 2.775 B LLM matmul weights
P_EMB, P_ALL, P_VIS = V * D, 3_754_622_976, 668_684_288                  # tied embedding; totals from HF
KV_TOK = 2 * L * KVH * DH * 2                                            # 36,864 bytes per token (BF16)
BW, F16, F8, MEM = 300e9, 121e12, 242e12, 24e9                           # L4 spec sheet (dense)
ETA_M, ETA_ENC, T_OH, T_FIX, BUDGET = 0.8, 0.4, 0.002, 0.010, 8192       # assumptions
UTIL, RESERVE, PREP_S_PER_MP = 0.9, 2.0e9, 0.046                         # assumptions; CPU rate from our run
FMT = {"bf16": (2.0, F16, 0.6, False), "fp8": (1.0, F8, 0.5, True),      # (bytes/weight, peak, eta, vision in FP8)
       "int4": (0.5 + 2.5 / 128, F16, 0.45, False)}                      # AWQ: 4 bits + group-128 scale and zero

def smart_resize(h, w, f=28, mn=3136, mx=12845056):                     # Qwen2-VL image processor rule
    hb, wb = round(h / f) * f, round(w / f) * f
    if hb * wb > mx:
        b = math.sqrt(h * w / mx); hb, wb = max(f, math.floor(h / b / f) * f), max(f, math.floor(w / b / f) * f)
    elif hb * wb < mn:
        b = math.sqrt(mn / (h * w)); hb, wb = math.ceil(h * b / f) * f, math.ceil(w * b / f) * f
    return hb, wb

def encoder_time(h, w, slices, fmt):                                     # slices = 1 (image) or ceil(T/2) (video)
    hb, wb = smart_resize(h, w); per = (hb // 14) * (wb // 14)           # patches per temporal slice
    lin = 2 * (P_BLK + P_PATCH) * per * slices + 2 * P_MERGE * per * slices // 4
    att = slices * 4 * DV * per * (L_FULL * per + L_WIN * WIN)           # full attention inside each slice
    return lin / (ETA_ENC * (F8 if FMT[fmt][3] else F16)) + att / (ETA_ENC * F16), slices * per // 4

def w_read(fmt): return P_LIN * FMT[fmt][0] + 2 * P_EMB                  # bytes per decode step (tied head in BF16)

def prefill_time(n, fmt, past=0):                                        # chunks of BUDGET tokens
    s, peak, eta, _ = FMT[fmt]; t = done = 0
    while done < n:
        c = min(BUDGET, n - done); p = past + done
        flops = 2 * P_LIN * c + 4 * L * H * DH * (c * p + c * (c + 1) / 2) + 2 * P_EMB
        t += max(flops / (eta * peak), (w_read(fmt) + (p + c) * KV_TOK) / (ETA_M * BW)) + T_OH; done += c
    return t

def ttft(w, h, frames=0, fmt="bf16"):                                    # frames = 0 means one image
    slices = math.ceil(frames / 2) if frames else 1
    t_enc, n_vis = encoder_time(h, w, slices, fmt)
    n = n_vis + (27 if frames else 28)                                   # text tokens of the chat template
    t_prep = PREP_S_PER_MP * w * h * max(frames, 1) / 1e6
    return n, T_FIX + t_prep + t_enc + prefill_time(n, fmt)

def w_stored(fmt):
    vis_lin = P_BLK + P_PATCH + P_MERGE; vis_b = 1 if FMT[fmt][3] else 2
    return P_LIN * FMT[fmt][0] + vis_lin * vis_b + (P_ALL - P_LIN - vis_lin) * 2

def kv_capacity(fmt): return int((UTIL * MEM - w_stored(fmt) - RESERVE) // KV_TOK)
def tpot(batch, ctx, fmt): return (w_read(fmt) + batch * ctx * KV_TOK) / (ETA_M * BW) + T_OH

if __name__ == "__main__":
    for w, h in [(640, 360), (1280, 720), (1920, 1080), (3840, 2160)]:
        print(f"image {w}x{h}: " + ", ".join(f"{f} {1e3 * ttft(w, h, 0, f)[1]:.0f} ms" for f in FMT),
              f"({ttft(w, h)[0]} prompt tokens)")
    for T in [8, 32, 64]:
        n, t = ttft(640, 360, T); print(f"video 640x360, {T} frames: {n} prompt tokens, TTFT {1e3 * t:.0f} ms")
    for f in FMT:
        print(f"{f}: weights {w_stored(f) / 1e9:.2f} GB, KV capacity {kv_capacity(f):,} tokens, "
              f"TPOT batch 1 {1e3 * tpot(1, 2847, f):.1f} ms, batch 16 {1e3 * tpot(16, 2847, f):.1f} ms")

Its output:

image 640x360: bf16 81 ms, fp8 54 ms, int4 89 ms (327 prompt tokens)
image 1280x720: bf16 288 ms, fp8 186 ms, int4 320 ms (1224 prompt tokens)
image 1920x1080: bf16 666 ms, fp8 435 ms, int4 741 ms (2719 prompt tokens)
image 3840x2160: bf16 3310 ms, fp8 2337 ms, int4 3655 ms (10577 prompt tokens)
video 640x360, 8 frames: 1223 prompt tokens, TTFT 323 ms
video 640x360, 32 frames: 4811 prompt tokens, TTFT 1286 ms
video 640x360, 64 frames: 9595 prompt tokens, TTFT 2652 ms
bf16: weights 7.51 GB, KV capacity 327,982 tokens, TPOT batch 1 28.2 ms, batch 16 34.7 ms
fp8: weights 4.07 GB, KV capacity 421,371 tokens, TPOT batch 1 16.6 ms, batch 16 23.2 ms
int4: weights 3.40 GB, KV capacity 439,408 tokens, TPOT batch 1 11.0 ms, batch 16 17.6 ms

Last, the PyTorch check of the encoder formula (200.4). It builds one Qwen2.5-VL vision block with random weights and counts its FLOPs with torch.utils.flop_counter, once with 64-patch windows and once with full attention. It runs on a CPU in a few seconds.

import torch, torch.nn as nn, torch.nn.functional as F
from torch.utils.flop_counter import FlopCounterMode
torch.manual_seed(0)
DV, IV, HEADS = 1280, 3420, 16                       # Qwen2.5-VL vision_config
class Block(nn.Module):                              # one vision block: attention + gated SiLU MLP
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = nn.RMSNorm(DV), nn.RMSNorm(DV)
        s.qkv, s.proj = nn.Linear(DV, 3 * DV), nn.Linear(DV, DV)
        s.gate, s.up, s.down = nn.Linear(DV, IV), nn.Linear(DV, IV), nn.Linear(IV, DV)
    def forward(s, x, win):                          # x: (N, DV); win: patches per attention segment
        N = x.shape[0]
        q, k, v = s.qkv(s.n1(x)).view(N // win, win, 3, HEADS, DV // HEADS).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v)  # attention inside each segment only
        x = x + s.proj(a.permute(0, 2, 1, 3).reshape(N, DV))
        h = s.n2(x)
        return x + s.down(F.silu(s.gate(h)) * s.up(h))
blk = Block().eval()
P_LIN = sum(m.weight.numel() for m in blk.modules() if isinstance(m, nn.Linear))
for N, win, name in [(10752, 64, "window layer"), (10752, 10752, "full-attention layer")]:
    x = torch.randn(N, DV)
    with torch.no_grad(), FlopCounterMode(display=False) as fc:
        blk(x, win)
    model = 2 * P_LIN * N + 4 * DV * N * win          # our formula: matmuls + attention
    print(f"{name:21s} N={N} measured {fc.get_total_flops()/1e12:.3f} TFLOP, formula {model/1e12:.3f} TFLOP")

Our output (PyTorch 2.15): window layer N=10752 measured 0.427 TFLOP, formula 0.427 TFLOP and full-attention layer N=10752 measured 1.015 TFLOP, formula 1.015 TFLOP [Certain, our run]. We used 10,752 patches (168 windows of 64) instead of the 10,764 of a 1080p image so that the windows divide evenly.

The results sheet

Fill one row per configuration. The “model” columns are the predictions of this page; the empty columns are yours. A row without the vLLM version and the full command line is not a result.

ConfigurationTTFT 1080p, modelTTFT 32 × 360p, modelTPOT batch 1, modelTTFT 1080p, yoursTTFT 32 × 360p, yoursTPOT batch 1, yoursChartQA-500, yours
BF16, no caches666 ms1.29 s28.2 ms
BF16, repeated image, all caches50 msn/a28.2 mssame as BF16
BF16, max_pixels 1,003,520347 ms1.29 s28.2 ms
FP8 (online, per tensor)435 ms0.87 s16.6 ms
INT4 AWQ741 ms1.42 s11.0 ms

The planning model is a sketch with the right shape, good to maybe 30 to 50% before you fit it [Speculation]. The interesting part of the lab is the gap: if FP8 does not lower TTFT on your server, check whether the vision tower was quantized; if INT4 prefill is faster than predicted, your engine uses better 4-bit kernels than we assumed; if a cached request is not fast, the cache did not hit.

Where this is used

WhereWhat you measureWhat it decides
Chat with photos (assistants, support apps)TTFT against resolution; prefix-cache hits on follow-up turnsThe max_pixels cap, client-side resizing, keeping the image before the question
Document and chart understandingQuality against max_pixels on your documentsHow many pixels fine text needs; whether FP8 or INT4 hurts reading accuracy
Video question answering and Video-RAGTTFT and tokens against frames and frame sizeFrames per clip, fps, frame resolution, max-model-len
Video retrieval and key-moment detection with VLM captionsThroughput at large batch on short clipsCost to caption an archive; INT4 for decode-heavy captioning
Action recognition and video classification with a VLMAccuracy against frames (8, 16, 32)The smallest frame budget that keeps accuracy
E-commerce and catalog tagging (same image, many questions)Processor-, encoder- and prefix-cache hit ratesPrompt layout, stable image bytes or uuids, cache sizes
Edge and small-GPU deploymentMemory and TTFT of FP8 and INT4Which format fits the device and the latency target

Common mistakes

  • Measuring the same image again and again with caches on. The second request hits the processor, encoder or prefix cache and looks five to ten times faster. Use new bytes for every timing request, or turn the caches off for the sweeps.
  • Not checking the token count. A client library, a proxy or a processor version may resize the image. Compare usage.prompt_tokens with formula (200.2) before you trust any time.
  • Plotting TTFT against pixels or file size. The GPU works on tokens. Two images with the same pixel count but different aspect ratios can round to different token counts; a JPEG’s file size says nothing about its tokens.
  • Sending frames as separate images. Each image is its own temporal patch, so a frame costs twice as many tokens as inside a video. Send a video when the model supports it.
  • Forgetting the CPU. Preprocessing a 4K image took 386 ms on one core in our run. Measure the API server’s CPU time, and give it enough cores or workers before blaming the GPU.
  • Expecting INT4 to fix TTFT. AWQ keeps the vision tower in BF16 and does not speed up compute-bound prefill. It helps decode. Choose the format for the metric you need.
  • Judging quality from one total. A 1-point change on 500 questions is usually noise. Compare per question (paired), count the flips, and report an interval.
  • Putting the question before the image. Every follow-up question then misses the prefix cache for all image tokens.

Interview questions

Why does TTFT of a VLM grow with the image resolution, and roughly how fast?basic
Models with native resolution, such as Qwen2.5-VL, turn every 28 × 28 pixels into one language-model token (14-pixel patches, merged 2 × 2). So the token count, the encoder work and the prefill work all grow about linearly with the pixel count, and the attention terms grow with its square. In our planning model for Qwen2.5-VL-3B on an L4, TTFT goes from 81 ms at 640 × 360 (299 tokens) to 666 ms at 1920 × 1080 (2,691 tokens) and 3.31 s at 4K (10,549 tokens): tokens grow 35 times, TTFT 41 times. Models that resize every image to a fixed size (LLaVA-1.5 at 336 × 336) have a flat GPU cost and only the CPU cost grows.
In a 3B VLM, why can the vision encoder cost as much as the language-model prefill?core
Matmul FLOPs are about \(2 \times \text{parameters} \times \text{tokens}\) for each part. The encoder processes 4 patches for every language-model token (the 2 × 2 merger), and Qwen2.5-VL’s encoder has 0.63 B matmul weights against 2.78 B in the 3B language model. So encoder over prefill \(\approx 4 \times 0.63 / 2.78 \approx 0.9\); with its attention terms, the encoder is 16.26 TFLOP for a 1080p image against 16.18 TFLOP for the prefill. The encoder also tends to run at lower efficiency (narrow layers, windowed attention), so it can take more time. For a 72B language model with the same encoder the ratio would be a few percent; the share of the encoder is a property of small VLMs.
How do you make sure a TTFT-versus-resolution measurement is correct?core
Send one request at a time so there is no queueing; drop a warm-up request; use new image bytes for every request (or turn off the processor and prefix caches) so no cache hits; send the media before the text; stream and time the first text chunk; and read usage.prompt_tokens to confirm that the server saw the token count you predicted from the resize rule. Report the median and the spread of several requests per size, and plot against tokens. Record the vLLM version, the full command line and the processor settings (max_pixels), because they change the token count.
A colleague says “just send 64 frames of 720p to the model”. What happens on Qwen2.5-VL-3B with max-model-len 32,768?core
Two frames make one temporal patch, and a 1280 × 720 frame is 1,196 tokens per patch (if the processor does not shrink it), so 64 frames are \(32 \times 1{,}196 = 38{,}272\) tokens. That is more than 32,768, so the server rejects the request. The most that fits with 256 output tokens is 54 frames. Even 32 frames would cost 19,136 tokens and about 5.8 s of TTFT on an L4 in our model. Options: fewer frames (lower fps), smaller frames (848 × 480 costs 510 tokens per patch, 640 × 360 costs 299), a per-frame pixel cap, token pruning, or a longer max-model-len if memory allows.
What are the processor cache, the encoder cache and the prefix cache in vLLM, and what does each skip?core
The processor cache is on the CPU: it keeps the processed pixels of recent images by hash and skips the resize and normalize step (decoding the upload and hashing still happen). The encoder cache is on the GPU: it keeps the vision encoder’s output embeddings, shared between requests by image hash, and skips the encoder. The prefix cache keeps KV blocks of 16 tokens keyed by the tokens and the image hashes in them, and skips the prefill of matching blocks; if all image tokens are in the cached prefix, the encoder is not needed at all. In our model for a repeated 1080p image they cut TTFT from 666 ms to about 50 ms together; the processor cache alone gives 247 ms, the prefix cache alone 134 ms.
Why should the image come before the question in the prompt?basic
The prefix cache matches tokens from the start of the prompt. If the layout is system prompt, image, question, then a second question about the same image matches everything up to the question, so only a few tokens are prefilled. If the layout is system prompt, question, image, the prompts differ right after the user marker, so all image tokens are prefilled again (only the encoder cache can still help). In our example the first layout costs 50 ms and the second 247 ms for the same follow-up.
FP8 lowers TTFT but INT4 AWQ does not. Why, and which one would you pick?advanced
TTFT is dominated by the encoder and the prefill, which are compute-bound. FP8 W8A8 runs the matmuls on FP8 tensor cores at twice the BF16 rate on Ada and Hopper GPUs, and vLLM’s online FP8 also quantizes the vision tower’s linear layers, so both parts get faster (666 to 435 ms at 1080p in our model). AWQ stores 4-bit weights but computes in 16 bits after unpacking, and the official checkpoint keeps the whole vision tower in BF16, so prefill does not get faster and can get slower. AWQ reads the fewest bytes per decode step, so it has the lowest TPOT (11.0 against 16.6 and 28.2 ms at batch 1). Pick FP8 for short answers and interactive TTFT, INT4 for long answers or a small memory budget, and decide only after a paired quality check.
Quantizing the 3B VLM saves 3 to 4 GB. Why does that change the batch size so little on an L4?advanced
The weights are only 7.51 GB of the 24 GB in BF16. The KV cache gets what is left after the weights and the reserve: 12.1 GB in BF16, 15.5 GB in FP8, 16.2 GB in INT4. With 36 KiB per token (grouped-query attention with 2 KV heads), that is 328 k, 421 k and 439 k tokens, or 110, 141 and 147 requests of one 1080p image and 256 output tokens. For a 7B or 13B model on the same GPU the weights are most of the memory, and quantization changes the batch much more. At large batch the KV reads, which weight quantization does not shrink, also come to dominate the decode step, so TPOT of the three formats converges.
How would you check that INT4 did not hurt quality, and how many questions do you need?advanced
Run the same questions through both servers at temperature 0, on a task like your product (charts, documents, photos), and compare per question. Count the flips: \(n_{10}\) right in BF16 and wrong in INT4, \(n_{01}\) the other way. The difference is \((n_{01} - n_{10})/n\), with standard error \(\sqrt{n_{01} + n_{10} - (n_{01}-n_{10})^2/n}/n\), and the exact McNemar test checks whether the flips split evenly. With 500 questions and 20 flips the interval is about \(\pm 1.7\) points, so a 1-point loss is invisible; you need more questions, or a task with more disagreement. The Qwen card reports drops of 1.2 to 2.6 points for the 3B AWQ model on four benchmarks, which is the size of effect you are trying to see.
Your measured TTFT for 1080p images is 1.5 s, but the model predicts 0.67 s. Where do you look?core
First the token count in usage.prompt_tokens: if it is much larger than 2,719, the image was not resized the way you think. Then the CPU: time the API server process; a slow vCPU or a single-threaded processor can add hundreds of milliseconds, and a queue of preprocessing requests adds more. Then the GPU split: profile the encoder and the prefill separately (or compare a text-only prompt of the same length, which has no encoder); a slow encoder often means the wrong attention backend or no CUDA graphs for it. Then the server: is something else running, is the request waiting for a large batch, did the first requests pay for warm-up? Fit the planning model’s efficiencies to small and large images to see which part is off.

Exercises

  1. Use (200.2) to compute the image tokens and the encoder patches of a 1280 × 960 photo.
  2. A clip has 48 frames of 848 × 480. How many video tokens does Qwen2.5-VL produce, how much KV cache does the prompt take (27 text tokens), and does it fit in max-model-len 32,768 with 256 output tokens?
  3. Show that 54 is the largest even number of 1280 × 720 frames that fits in 32,768 tokens with 27 text tokens and 256 output tokens.
  4. Compute the lowest possible TPOT at batch 1 for the INT4 model (2.064 GB read per step), first at the spec bandwidth, then with \(\eta_m = 0.8\) and 2 ms of overhead.
  5. In the worked example of Part 3, what would be cached and what prefilled if the block size were 32 instead of 16? Is the encoder needed?
  6. On 500 questions, FP8 loses 9 questions that BF16 got right and wins 3. Compute the difference, its 95% interval with (200.7), and the exact McNemar p-value.
  7. Run the lab: the resolution sweep, the frame sweep, the six caching requests and at least one quantized model on a real GPU. Fit the planning model and report where it was wrong.
Answerscheck
1. \(960/28 = 34.29 \to 34\), \(H' = 952\); \(1280/28 = 45.71 \to 46\), \(W' = 1{,}288\). Tokens \(34 \times 46 = 1{,}564\); patches \(68 \times 92 = 6{,}256\). (The planning model gives a TTFT of 379 ms in BF16.) 2. Each 848 × 480 frame becomes 840 × 476, so \(17 \times 30 = 510\) tokens per temporal patch; 48 frames are 24 patches, so 12,240 video tokens and a prompt of 12,267 tokens. KV cache \(12{,}267 \times 36{,}864 = 452\) MB (431 MiB). It fits easily: \(12{,}267 + 256 \lt 32{,}768\). 3. Each pair of frames costs 1,196 tokens. \((32{,}768 - 256 - 27)/1{,}196 = 27.2\), so at most 27 pairs, 54 frames (\(32{,}292 + 27 + 256 = 32{,}575\) tokens); 56 frames would need 33,771. 4. \(2.064 / 300 = 6.9\) ms at the spec bandwidth; \(2.064 / 240 + 2 = 10.6\) ms with the assumptions (11.0 ms with the KV cache of a 1080p conversation). 5. \(h = 32 \lfloor 2{,}707/32 \rfloor = 2{,}688\), so 32 tokens are prefilled and the last 18 image tokens (positions 2,688 to 2,705) are not cached; the image embeddings are needed again, from the encoder cache if the entry is still there, else the encoder runs (336 ms). Bigger blocks lose more of the prefix. 6. \(\hat\Delta = (3 - 9)/500 = -1.2\) points; \(\text{SE} = \sqrt{12 - 36/500}/500 = 0.69\) points; interval \([-2.6, +0.2]\); McNemar \(p = 2 \sum_{i=0}^{3} \binom{12}{i} / 2^{12} = 2 \cdot 299/4096 = 0.146\). Not significant. 7. Open: compare your numbers with the results sheet.

Key takeaways

  • Write down every multimodal flag (--limit-mm-per-prompt, --mm-processor-kwargs, cache sizes, --max-model-len) and the vLLM version; check usage.prompt_tokens against the resize rule before trusting any time.
  • Qwen2.5-VL spends one token per 28 × 28 pixels and one temporal patch per two frames; TTFT grows a bit faster than linearly with tokens. Plot against tokens.
  • TTFT = CPU preprocessing + vision encoder + prefill. In a 3B VLM the encoder is about half of the GPU work, and the CPU can be hundreds of milliseconds for large images.
  • Video hits hard limits: 32 frames of 720p are 19,136 tokens; the context fills at about 54 frames. Frames times pixels is the budget.
  • The processor, encoder and prefix caches remove repeated work; with all three, a repeated 1080p image costs about 50 ms instead of 666 ms in our model. Keep the image before the question and its bytes identical.
  • FP8 speeds up the compute-bound TTFT; INT4 AWQ speeds up the memory-bound decode and leaves the vision tower in BF16. Measure quality per question with a paired test.

Key formulas

\[ N_{\text{tok}} = \left\lceil \tfrac{T}{2} \right\rceil \frac{H'}{28} \frac{W'}{28}, \qquad C_{\text{KV}} = \frac{uM - W - R}{2 L n_{kv} d_h s}, \qquad h = B \left\lfloor \frac{\min(m, n-1)}{B} \right\rfloor \]
\[ \text{TTFT} = t_{\text{fix}} + t_{\text{prep}} + \frac{F_{\text{enc}}}{\eta_{\text{enc}} F_{\text{peak}}} + \sum_{\text{chunks}} \max\!\left(\frac{F_c}{\eta_c F_{\text{peak}}}, \frac{Q_c}{\eta_m \text{BW}}\right) \]
\[ \text{TPOT}(b) = \frac{W_{\text{read}} + b\,c\,\text{kv}_{\text{tok}}}{\eta_m\,\text{BW}} + t_{\text{oh}}, \qquad \text{SE}(\hat\Delta) = \frac{\sqrt{n_{01} + n_{10} - (n_{01} - n_{10})^2/n}}{n} \]

References for this session

  1. vLLM project (docs). vLLM documentation
  2. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  3. Bai et al. (2025). Qwen2.5-VL Technical Report
  4. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  5. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  6. Micikevicius et al. (2022). FP8 Formats for Deep Learning
  7. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  8. Qwen Team (2025). Qwen2.5-VL-3B-Instruct model card, config.json and preprocessor_config.json (Hugging Face)
  9. Qwen Team (2025). Qwen2.5-VL-3B-Instruct-AWQ model card and config.json (Hugging Face)
  10. vLLM project (2026). vLLM documentation: Multimodal Inputs
  11. vLLM project (2026). vLLM documentation: Engine Arguments
  12. vLLM project (2026). vllm bench serve: command-line reference (random-mm dataset)
  13. vLLM project (2026). vLLM documentation: FP8 W8A8 quantization
  14. vLLM project (2026). vllm/v1/core/encoder_cache_manager.py (source code)
  15. Hugging Face (2026). transformers: Qwen2-VL image and video processors (smart_resize, video_processing_qwen2_vl.py)
  16. NVIDIA (2023). NVIDIA L4 Tensor Core GPU (product page and specifications)
  17. Masry, Long, Tan, Joty & Hoque (2022). ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning