serve a VLM and measure
In this lab you serve Qwen2.5-VL-3B-Instruct with vLLM on one NVIDIA L4 and measure how the time to first token grows with the image resolution and with the number of video frames. Then you turn on prefix caching and the multimodal caches, switch the weights to FP8 and INT4, and record speed and quality side by side. A planning model built from the published config tells you what to expect, so you can tell a real effect from a broken measurement.
What you will learn
- How to start a VLM server with vLLM with every multimodal flag written down, and how to read its start-up log
- How to turn an image size or a frame count into image tokens with Qwen2.5-VL’s resize rule, and check the count against the server’s usage field
- How to split TTFT into CPU preprocessing, vision encoder and LLM prefill, and why the encoder is about half of the GPU work in a 3B VLM
- How TTFT grows with resolution and with frames, and where max-model-len and max_pixels set hard limits
- How prefix caching, the processor cache and the encoder cache each remove repeated work, and why the image must come before the question
- What FP8 and INT4 AWQ change in memory, TTFT, TPOT and throughput, and how to measure their quality cost with a paired test
- How to write a results sheet that someone else can reproduce
Why this matters
A vision-language model (VLM) server is judged first by how long the user waits before the answer starts: the time to first token (TTFT). For a text model, TTFT is mostly one prefill. For a VLM it is much more: the server must decode and resize the image on the CPU, run a vision encoder on the GPU, and then prefill hundreds or thousands of extra image tokens. Each of these costs grows with the image size and with the number of video frames. In this lab you serve a real small VLM, measure exactly how TTFT grows, and then try the three main tools that make it cheaper: caching, FP8 and INT4.
The lab uses the whole serving part of Track 7. Session 196 followed one request through every stage and counted its cost. Session 197 explained which parts to quantize and why the vision encoder usually stays in 16-bit. Session 198 compared the engines, and Session 199 did the latency math on paper for the same GPU we use here. The method comes from session 142, the text-model version of this lab: write the setup down, predict, measure, change one thing at a time. The next session, session 201, starts the video track, where the frame budget you measure here becomes the central problem.
The lab has four parts, one for each key point of the session:
- Serve Qwen2.5-VL-3B-Instruct with vLLM on one NVIDIA L4, and check that the server counts image tokens the way you expect.
- Measure TTFT against the image resolution, and against the number of video frames.
- Turn on prefix caching and the multimodal caches, and measure what repeated images cost.
- Switch to FP8 and to INT4 (AWQ) weights, and record speed and quality for each.
The setup: one L4, one small VLM, every flag written down
A measurement is only useful if someone else can repeat it. So before the first request you write down the GPU, the model, the engine version and every setting that changes what the server does with an image. Do not trust defaults: they change between versions, and for VLMs there are more of them (resize limits, frame sampling, cache sizes).
The GPU. An NVIDIA L4: 24 GB of memory, 300 GB/s of memory bandwidth, 121 TFLOP/s of dense BF16 matrix math and 242 TFLOP/s of dense FP8 [Certain: NVIDIA lists 242 and 485 “with sparsity”; dense is one half]. It supports FP8 tensor-core math (compute capability 8.9) [Certain]. It is a cheap 72 W inference card that most clouds rent.
The model. Qwen2.5-VL-3B-Instruct in BF16. From its config.json [Certain]: a language model with 36 layers, width 2,048, 16 query heads and 2 key-value heads of size 128 (grouped-query attention), a feed-forward width of 11,008, a vocabulary of 151,936 and an output layer that is tied to the input embedding. The vision encoder is a ViT with 32 layers of width 1,280; only layers 7, 15, 23 and 31 use full attention, the others attend inside windows of 112 × 112 pixels. Patches are 14 × 14 pixels, two consecutive video frames form one temporal patch, and a 2 × 2 merger turns four patches into one language-model token. Hugging Face counts 3,754,622,976 parameters; from the shapes we get 668.7 M in the vision tower, 311.2 M in the embedding and 2,774.8 M in the language-model blocks, which adds up exactly [Certain, our count].
- Read the symbols
- \(C_{\text{KV}}\)
- tokens of KV cache that fit on the GPU, summed over all running requests
- \(u\)
- fraction of GPU memory the engine may use (
--gpu-memory-utilization) - \(M, W\)
- GPU memory and the bytes of the stored weights (vision tower included)
- \(R\)
- memory for everything else: activations of the largest step, the vision encoder’s activations for the largest image, CUDA graphs; here an assumed 2.0 GB
- \(\text{kv}_{\text{tok}}\)
- bytes of cache per token: a key and a value (the 2) for each of \(L\) layers and \(n_{kv}\) heads of size \(d_h\), with \(s\) bytes per number
Weights in BF16: \(3{,}754{,}622{,}976 \times 2 = 7.51\) GB. Cache per token: \(2 \cdot 36 \cdot 2 \cdot 128 \cdot 2 = 36{,}864\) bytes (36 KiB). With \(u = 0.9\) and \(R = 2.0\) GB: \(0.9 \cdot 24 - 7.51 - 2.0 = 12.09\) GB for the cache, so \(C_{\text{KV}} = 327{,}982\) tokens.
A 1920 × 1080 image becomes 2,691 image tokens (we show why below); with the chat template the prompt has 2,719 tokens, and with 256 output tokens one request needs 2,975 tokens of cache. So about \(\lfloor 327{,}982 / 2{,}975 \rfloor = 110\) such requests fit at once. One 1080p image costs \(2{,}719 \times 36{,}864 = 100\) MB of cache. Compare a 7B LLaVA-1.5 with full multi-head attention, where one 576-token image costs about 300 MB (session 196): the 2 key-value heads of this model make image tokens cheap to store. They are not cheap to compute, as we will see.
The engine. Install vLLM in a fresh environment, record its version (pip show vllm), and start the server with every setting written out:
# the baseline: BF16, one image or one video per prompt, NO prefix caching, NO processor cache
vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--dtype bfloat16 --max-model-len 32768 --gpu-memory-utilization 0.9 \
--max-num-seqs 64 --max-num-batched-tokens 8192 \
--limit-mm-per-prompt '{"image": 1, "video": 1}' \
--disable-prefix-caching --mm-processor-cache-gb 0 \
--port 8000
What each multimodal flag does, from the engine-argument docs [Certain]. --limit-mm-per-prompt caps the images and videos one prompt may carry; the engine also uses it to size its memory profiling, so a smaller limit leaves more memory for the cache. --mm-processor-cache-gb is the size of a CPU cache that keeps the processed pixels of recent images (default 4 GiB; 0 turns it off). --mm-processor-kwargs passes options to the Hugging Face processor, such as max_pixels (Part 1). --media-io-kwargs '{"video": {"num_frames": 40}}' sets how many frames the server samples when it decodes a video file itself. We turn both caches off for the baseline so that the resolution sweep measures real work; Part 3 turns them on.
When the server is ready, read its start-up log: the KV-cache size in tokens, the maximum concurrency for 32,768-token requests, and whether the model loaded in the precision you asked for [Likely: the exact wording of the log changes between versions]. Then send one request by hand. The response’s usage.prompt_tokens field is your first check: for the 1920 × 1080 test image and our one-sentence question it should be 2,719.
How to measure TTFT for one image
For this lab the question is “how long does one request wait, as a function of what it carries?”, not “how much load can the server take?”. So send one request at a time, with nothing else running: then TTFT has no queueing time in it, and you see the pure cost of the image. (The load question needs open-loop traffic and is the subject of session 142; Figure 4 comes back to batching.)
First predict the token count. Qwen2.5-VL keeps the aspect ratio of the image. Its processor rounds the height and the width to multiples of 28 pixels, then scales the image down if it has more than max_pixels (default 12,845,056) or up if it has fewer than min_pixels (default 3,136) [Certain, from the processor code and config]:
- Read the symbols
- \(H, W\)
- height and width of the image (or of each video frame) that reaches the processor
- \(H', W'\)
- after resizing: multiples of 28, because a token covers 2 × 2 patches of 14 pixels
- \(T\)
- number of frames; an image counts as \(T = 1\) (it is repeated to fill one temporal patch); an odd frame count is padded to the next even one
- \(N_{\text{tok}}\)
- image or video tokens that the language model sees; the vision encoder works on \(4 N_{\text{tok}}\) patches
\(1080 / 28 = 38.57\), rounded to 39, so \(H' = 1{,}092\); \(1920 / 28 = 68.57\), rounded to 69, so \(W' = 1{,}932\). That gives \(39 \times 69 = 2{,}691\) image tokens and \(78 \times 138 = 10{,}764\) patches in the encoder. The chat template adds 28 text tokens (system prompt, roles, the two vision markers and our question), so the prompt has 2,719 tokens.
A 640 × 360 clip of 32 frames: each frame becomes \(13 \times 23 = 299\) tokens per temporal patch, and 32 frames make 16 temporal patches, so \(16 \times 299 = 4{,}784\) video tokens, and 4,811 prompt tokens with the video template.
We checked these counts with the Hugging Face processor of the model (transformers 5.23): it returned an image grid of 1 × 78 × 138 and a prompt of 2,719 tokens, and a video grid of 16 × 26 × 46 and a prompt of 4,811 tokens [Certain, our run].
- input: width \(W\), height \(H\), frames \(T\) (0 for an image), repeats \(k\)
- predict \(N_{\text{tok}}\) with (200.2)
- for \(j = 0, 1, \dots, k\):
- make a new test image (or \(T\) new frames) with seed \(j\) // new bytes: no cache can hit
- encode it as a JPEG in a base64 data URL; build the chat message: media first, then the question
- \(t_0 \leftarrow\) now; send the request with streaming on
- TTFT\(_j \leftarrow\) arrival time of the first text chunk \(- t_0\); read
usage.prompt_tokensfrom the last chunk - drop request 0 (warm-up); report the median TTFT and the prompt tokens
- check: prompt tokens \(= N_{\text{tok}}\) + template tokens; if not, the server resized differently than you think
The last line is the most useful check in the lab. If the server reports fewer tokens than you predicted, something resized the image: a max_pixels setting, a client library, or a different processor version. If it reports more, your frames were not sampled the way you think. Fix this before you measure any time.
A model to compare against
Before you rent the GPU, predict the result. Our planning model splits TTFT into four parts:
- Read the symbols
- \(t_{\text{fix}}\)
- fixed cost per request: HTTP, tokenization, scheduling; assumed 10 ms
- \(t_{\text{prep}}\)
- CPU time to decode the base64 JPEG and run the image processor (resize, normalize, cut patches); 46 ms per megapixel in our run on one laptop core
- \(P_{\text{blk}}, N_p\)
- matmul weights of the 32 vision blocks (0.630 B, plus the patch embedding) and the number of patches
- \(n_s, N_s\)
- temporal slices (1 for an image, \(\lceil T/2 \rceil\) for a video) and patches per slice; full attention runs inside each slice, not across the whole video [Certain, from the model code]
- \(d_v, L_{\text{full}}, L_{\text{win}}, w\)
- encoder width 1,280; 4 full-attention layers; 28 window layers; 64 patches per window (8 × 8)
- \(F_c, c, p\)
- FLOPs of one prefill chunk of \(c\) tokens (at most 8,192, the token budget) after \(p\) tokens already in the cache
- \(P_{\text{LLM}}, d_a\)
- matmul weights of the 36 language-model blocks (2.775 B) and the attention width (16 heads × 128 = 2,048)
- \(Q_c\)
- bytes the chunk reads: the weights once plus the KV cache so far
- \(\eta_{\text{enc}}, \eta_c, \eta_m\)
- assumed fractions of peak that real kernels reach: 0.4 for the encoder (narrow layers, short windows), 0.6 for the language model, 0.8 for bandwidth
- \(t_{\text{oh}}\)
- fixed cost of one engine step; assumed 2 ms
The model ignores two things on purpose: the merger and the patch embedding are small, and attention to the windows at the image border is a bit cheaper than a full window. We checked the per-layer formula with PyTorch’s FLOP counter on one vision block (code below): for 10,752 patches, a window layer costs 0.427 TFLOP and a full-attention layer 1.015 TFLOP, the same as the formula to three digits [Certain, our run].
Encoder. 10,764 patches. Matmuls: \(2 \times 0.6315 \times 10^9 \times 10{,}764 = 13.59\) TFLOP, plus 0.20 for the merger. Attention: \(4 \cdot 1280 \cdot 10{,}764 \cdot (4 \cdot 10{,}764 + 28 \cdot 64) = 2.47\) TFLOP. Total 16.26 TFLOP; at \(0.4 \times 121\) TFLOP/s that is 336 ms.
Prefill. 2,719 tokens: \(2 \times 2.775 \times 10^9 \times 2{,}719 = 15.09\) TFLOP plus 1.09 TFLOP of attention, 16.18 TFLOP in total; at \(0.6 \times 121\) TFLOP/s, 223 ms, plus 2 ms of step overhead: 225 ms. The weights and the cache are only 6.27 GB, 26 ms at 240 GB/s, so the prefill is compute-bound.
CPU. 2.07 megapixels at 46 ms per megapixel: 95 ms. With the 10 ms fixed cost, TTFT ≈ 666 ms.
The surprise: the vision encoder costs as many FLOPs as the whole language-model prefill (16.26 against 16.18 TFLOP), and more time, because its kernels are less efficient. The reason is the ratio from session 196: the encoder sees 4 patches per language-model token, and its 0.63 B weights are almost a quarter of the 2.78 B of the language model, so \(4 \times 0.63 / 2.78 \approx 0.9\). In a 3B VLM, the vision tower is not a small add-on.
Part 1: TTFT against image resolution
Sweep eight 16:9 sizes from 640 × 360 to 3840 × 2160, five new images each, with the baseline server. The client below does one size per call. Plot TTFT against the prompt tokens that the server reports, not against the pixel count: tokens are what the GPU works on. Figure 1 shows what the planning model predicts for BF16, FP8 and INT4.
How to read Figure 1
Start at 1920 × 1080 in BF16: 2,691 image tokens, CPU 95 ms, encoder 336 ms, prefill 225 ms, TTFT 666 ms; the encoder is 60% of the GPU time. Move to 640 × 360: 299 tokens and 81 ms. Move to 3840 × 2160: 10,549 tokens and 3.31 s. From the smallest to the largest size the tokens grow 35 times and TTFT 41 times: a bit faster than linear, because the four full-attention encoder layers and the prefill attention grow with the square of the length. Switch to FP8: 435 ms at 1080p, because both the encoder and the prefill run on FP8 tensor cores (we assume the vision tower is quantized too; see Part 4). Switch to INT4: 741 ms, slower than BF16, because AWQ leaves the vision tower in BF16 and its 4-bit matmuls are slower in prefill (an assumption). Now tick the max_pixels box: every image above about one megapixel is shrunk to 1,222 tokens, and 4K costs 633 ms instead of 3.31 s.
What the curves say
- TTFT follows the token count, almost linearly. Twice the pixels means about twice the tokens, twice the encoder work and twice the prefill. The square terms (full attention) start to show above about 5,000 tokens.
- The encoder is half of the GPU work at every size. This is special to small VLMs with a large vision tower. For a 72B language model with the same encoder, the encoder would be a few percent.
- The CPU is a real part of TTFT. In our run the Hugging Face processor took 12 ms for 640 × 360, 98 ms for 1920 × 1080 and 386 ms for 3840 × 2160 on one core [Certain, our run, an Apple M5 Pro laptop; a cloud vCPU may be slower]. If the API server handles requests one at a time, that time also blocks the next request.
- The cheapest fix is fewer pixels. The
max_pixelscap cuts the cost of large images by up to 5 times, and it is one flag:--mm-processor-kwargs '{"max_pixels": 1003520}'[Likely: check that your vLLM version forwards it to this processor]. Whether the answers stay good is a quality question: measure it as in Part 4 on your own images, because fine text in documents needs pixels.
Part 2: TTFT against the number of video frames
Qwen2.5-VL takes a video as a stack of frames. Two consecutive frames are merged into one temporal patch, so a video costs half as many tokens as the same frames sent as separate images [Certain, from temporal_patch_size = 2]. To control the frame count exactly, extract the frames on the client and send them as one video_url of comma-joined base64 JPEGs (data:video/jpeg;base64,...), which the vLLM docs describe for pre-extracted frames [Certain]. If you send a video file instead, the server samples the frames itself, and --media-io-kwargs sets how many.
Sweep 4, 8, 16, 32, 64 and 128 frames at three frame sizes: 640 × 360, 848 × 480 and 1280 × 720. Two limits will appear. The prompt must fit in --max-model-len (32,768 here) with room for the answer; longer prompts are rejected with an error, not truncated. And the tokens per frame depend on the processor version: our transformers 5.23 processor kept 1280 × 720 frames at full size (1,196 tokens per temporal patch) and printed a warning that a per-frame pixel cap, as in the reference qwen-vl-utils code, will become the default in a later version [Certain, our run]. vLLM may also apply its own limits [Likely]. So again: read usage.prompt_tokens.
--max-model-len 32768.How to read Figure 2
At the default, 32 frames of 640 × 360: 16 temporal patches of 299 tokens, 4,784 video tokens, 169 MiB of KV cache, and a TTFT of 1.29 s, of which 0.34 s is CPU work on 32 frames. Switch to 1280 × 720: the same 32 frames now cost 19,136 tokens and 5.78 s. Move the slider and watch the 720p curve turn grey after 54 frames: 56 frames would need \(28 \times 1{,}196 + 27 + 256 = 33{,}771\) tokens. At 848 × 480 the limit is 126 frames, at 640 × 360 it is 216 (beyond the slider). Finally compare 8 frames of 720p with 32 frames of 360p: both are 4,784 tokens and cost about the same (1.31 s and 1.29 s). What counts is frames times pixels, not either one alone.
This is the “budget problem” that the video track starts with in session 201. A 3B model on an L4 needs more than a second of TTFT for 32 small frames, and its context fills up at about 50 frames of 720p. To cover a longer video you must choose: fewer frames (lower fps), smaller frames, or fewer tokens per frame (token merging or pruning, session 183). Each choice trades detail in time against detail in space.
Part 3: prefix caching and the two multimodal caches
Real traffic repeats itself. A user asks three questions about the same photo; many users send the same product image; a client retries a request. A vLLM server has three caches that remove this repeated work, and each one removes a different part of TTFT:
- Processor cache (CPU,
--mm-processor-cache-gb): keeps the processed pixels of recent images, keyed by a hash of the image. A hit skips the resize and normalize step; the server still decodes the upload to compute the hash. In our run, base64 decode, JPEG decode and a SHA-256 of the pixels took 12 ms of the 98 ms that the full processing of a 1080p image took [Certain, our run]. vLLM’s hash function may differ [Likely]. - Encoder cache (GPU, always on in the V1 engine): keeps the vision encoder’s output embeddings. The source code says it is shared between requests by the image hash, and that old entries with no running request are evicted only when space is needed [Certain, vLLM source]. A hit skips the encoder. When both of the other caches are off, the engine may not compute content hashes at all, so we assume no sharing in that case [Likely].
- Prefix cache (GPU,
--enable-prefix-caching): keeps KV blocks of 16 tokens, keyed by a hash of all tokens up to that block and the hashes of the images inside it (session 134). A hit skips the prefill of those tokens. If every image token is inside the cached prefix, the encoder is not needed either.
- Read the symbols
- \(m\)
- leading tokens of the new prompt that match a cached sequence (same token ids and same image hashes)
- \(B\)
- block size, 16 tokens; only full blocks are cached, and the last prompt token is always computed (it produces the first logits)
- \(h, n_{\text{new}}\)
- tokens served from the cache, and tokens that must still be prefilled
- \(e\)
- position just after the last image token; if the cached prefix ends before \(e\), the image’s embeddings are needed again (from the encoder cache, or by running the encoder)
We tokenized both prompts with the model’s chat template [Certain, our run]. The image tokens sit at positions 15 to 2,705, the end-of-vision marker at 2,706, and the two prompts match for \(m = 2{,}707\) tokens (up to the question). Then \(h = 16 \lfloor 2{,}707 / 16 \rfloor = 2{,}704\): the last two image tokens fall into a block that is not full, so they are not cached. The server prefills \(2{,}720 - 2{,}704 = 16\) tokens, which takes 28 ms (it is now memory-bound: one read of the weights), and it needs the image embeddings for two tokens, which the encoder cache provides. Without that cache, the whole 336 ms encoder would run again for two tokens.
If the question comes before the image, the prompts differ right after the user role marker (\(m = 14\)), so \(h = 0\): all 2,720 tokens are prefilled again. Only the processor cache and the encoder cache help.
- start the server with both caches off; send the six requests of Figure 3 in order, one at a time; record TTFT and
prompt_tokens - restart with
--enable-prefix-cachingonly; repeat; then with--mm-processor-cache-gb 4only; then with both - the repeated image must be the same bytes: send the same base64 string, or give it the same
uuidfield [Certain: the docs describe an optional uuid per media item] - after each run, read the hit counters from the server’s
/metricsendpoint (prefix-cache queries and hits, multimodal-cache queries and hits) [Likely: metric names change between versions] - compare each request with the prediction of (200.5); a request that should hit but does not is a bug worth finding: different bytes, a changed template, or an evicted entry
How to read Figure 3
With both caches on, requests 2, 3 and 6 cost about 50 ms instead of 666 ms: 12 ms of CPU, 28 ms of prefill, 10 ms fixed. Request 4 brings a new image and pays the full 666 ms. Request 5 has the question before the image, so the prefix cache misses; the processor and encoder caches still save 419 ms, and it costs 247 ms. Turn off the processor cache: the hits cost 134 ms, because the CPU work (95 ms) is now the largest part. Turn off the prefix cache too: every request costs about 666 ms, and the six together take 4.00 s instead of 1.73 s. Click request 2 with prefix caching on and read the token counts: 2,704 cached and 16 prefilled, as in the worked example.
Two lessons for prompt design. Put the shared part first: system prompt, then the image, then the question. And keep the bytes identical: a client that re-encodes the JPEG on every turn, or resizes it slightly differently, produces a new hash and misses every cache.
Part 4: FP8 and INT4, speed and quality
Now change the weights. Each change is one experiment, with the same sweeps as Parts 1 and 2, plus a quality check.
FP8. vLLM can quantize a BF16 checkpoint to FP8 when it loads it, with no calibration data. The current docs spell the option --quantization fp8_per_tensor; older versions use --quantization fp8 [Certain for the current docs; check vllm serve --help]. In this mode “all Linear modules (except for the final lm_head)” get FP8 weights, and the activations get a dynamic per-tensor scale; the docs warn that latency gains are limited in this mode [Certain]. On the L4 the matmuls run on FP8 tensor cores (W8A8). In the current vLLM code the Qwen2.5-VL vision tower receives the same quantization config as the language model, so its linear layers are probably FP8 too [Likely: check the log]. For the best FP8 speed, use a checkpoint that was quantized offline with static scales.
INT4. Serve the official AWQ checkpoint Qwen/Qwen2.5-VL-3B-Instruct-AWQ. Its config says 4 bits, group size 128, with zero points, and "modules_to_not_convert": ["visual"]: the whole vision tower stays in BF16 [Certain]. vLLM reads the method from the config. Expect decode to be fast (fewer bytes per step) and prefill to be no faster, because the 4-bit weights are unpacked and the math still runs in 16 bits (session 197).
- Read the symbols
- \(W_{\text{read}}\)
- weight bytes one decode step reads: the language-model matmuls in the chosen format, plus the tied 151,936 × 2,048 output layer in BF16 (0.62 GB). The vision tower is not read during decode.
- \(b, c\)
- requests decoding together, and their average context length
- \(\text{TPOT}\)
- time per output token for each request; this is a decode-only step, with no prefill mixed in
Decode is memory-bound, so (200.6) uses only bytes. In BF16 a step reads 6.17 GB; in FP8 3.40 GB; in INT4 2.06 GB (4 bits plus a 16-bit scale and a 4-bit zero for every 128 weights: 0.52 bytes per weight). At batch 1 this predicts a TPOT of 28.2, 16.6 and 11.0 ms. Figure 4 adds the batch size.
How to read Figure 4
At batch 16 with the image workload: TPOT is 34.7 ms in BF16, 23.2 in FP8 and 17.6 in INT4, so throughput is 461, 691 and 909 output tokens per second. INT4 wins decode, but its TTFT (741 ms) is the worst, and FP8 has the best TTFT (435 ms): the two formats help different metrics. Move to batch 128: BF16 is capped at 110 requests by its KV cache, and the three formats come closer (75.8, 72.1 and 66.6 ms), because the KV cache reads, which quantized weights do not shrink, now dominate each step. Switch to the video workload: each request is longer, so BF16 fits only 64 requests, FP8 83 and INT4 86. The memory gain of quantizing a 3B model is small (7.51 GB against 4.07 and 3.40 GB) next to the 24 GB of the L4.
Quality: measure it on the same questions
A faster wrong answer is not a win. The Qwen team reports, for the 3B model, BF16 against AWQ, evaluated with VLMEvalKit [Certain, from the AWQ model card]:
| Qwen2.5-VL-3B-Instruct | MMMU val | DocVQA val | MMBench dev (en) | MathVista mini |
|---|---|---|---|---|
| BF16 | 51.7 | 93.0 | 79.8 | 61.4 |
| AWQ (INT4) | 49.1 | 91.8 | 78.0 | 58.8 |
| Difference | −2.6 | −1.2 | −1.8 | −2.6 |
The card gives no FP8 numbers, and these are their numbers, not yours. Run your own check on the task you care about. The script below sends the first 500 ChartQA test questions to each server at temperature 0 and scores them with relaxed accuracy (a number is right if it is within 5% of the answer) [Certain, ChartQA paper]. Because every server answers the same questions, compare them per question, not only by their totals:
- Read the symbols
- \(n\)
- questions, the same for both servers
- \(n_{10}, n_{01}\)
- questions that the baseline got right and the quantized model wrong, and the other way around (the “flips”)
- \(\hat\Delta\)
- accuracy of the quantized model minus the baseline; \(\hat\Delta \pm 1.96\,\text{SE}\) is a 95% interval. The exact McNemar test asks whether the flips split 50/50.
Suppose that on 500 questions the quantized model loses 14 that BF16 got right and wins 6 that BF16 got wrong. Then \(\hat\Delta = (6 - 14)/500 = -1.6\) points, \(\text{SE} = \sqrt{20 - 64/500}\,/\,500 = 0.89\) points, and the 95% interval is \([-3.3, +0.1]\). The exact McNemar p-value for 6 against 14 flips is 0.115. So this run cannot tell a 1.6-point loss from noise; it can only say that the loss is probably not larger than about 3 points. To detect a 1-point difference you need many more questions, or a task with more flips. Note that only the 20 flips carry information; the 480 questions that both got right or both got wrong do not.
The code
Three short programs. The first is the TTFT client. It uses the standard library and Pillow, speaks the OpenAI chat API that vLLM serves, makes new image bytes for every request, sends media before text, streams the answer, and reads prompt_tokens from the usage chunk at the end.
# bench_vlm.py: TTFT of a VLM server against image size and video frames (OpenAI chat API, streaming) # Needs only the standard library and Pillow. Usage: # python3 bench_vlm.py URL MODEL image 1920x1080 [REPEATS] # python3 bench_vlm.py URL MODEL video 640x360 32 [REPEATS] import base64, io, json, math, random, statistics, sys, time, urllib.request from PIL import Image def test_image(w, h, seed): """A synthetic photo-like JPEG. A new seed gives new bytes, so no cache can hit.""" rnd = random.Random(seed) small = Image.new("RGB", (16, 9)) small.putdata([tuple(rnd.randrange(256) for _ in range(3)) for _ in range(16 * 9)]) img = small.resize((w, h), Image.BICUBIC) # smooth color fields, like a photo buf = io.BytesIO(); img.save(buf, "JPEG", quality=90) return base64.b64encode(buf.getvalue()).decode() def one_request(url, model, content, max_tokens=32): body = json.dumps({"model": model, "max_tokens": max_tokens, "temperature": 0, "stream": True, "stream_options": {"include_usage": True}, "messages": [{"role": "user", "content": content}]}).encode() req = urllib.request.Request(url + "/v1/chat/completions", body, {"Content-Type": "application/json"}) t0 = time.perf_counter(); t_first = None; times = []; usage = None with urllib.request.urlopen(req, timeout=600) as resp: for line in resp: # server-sent events: b"data: {...}" line = line.strip() if not line.startswith(b"data: ") or line == b"data: [DONE]": continue ev = json.loads(line[6:]) if ev.get("usage"): usage = ev["usage"] if ev.get("choices") and ev["choices"][0]["delta"].get("content"): times.append(time.perf_counter()) t_first = t_first or times[-1] tpot = (times[-1] - times[0]) / (len(times) - 1) if len(times) > 1 else float("nan") return t_first - t0, tpot, usage["prompt_tokens"] def run(url, model, kind, size, frames=0, repeats=5): w, h = map(int, size.split("x")); rows = [] for k in range(repeats + 1): # request 0 is a warm-up and is dropped if kind == "image": media = {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64," + test_image(w, h, k)}} else: # pre-extracted frames, comma-joined base64 JPEGs fr = ",".join(test_image(w, h, 1000 * k + j) for j in range(frames)) media = {"type": "video_url", "video_url": {"url": "data:video/jpeg;base64," + fr}} content = [media, {"type": "text", "text": "Describe this in one sentence."}] rows.append(one_request(url, model, content)) rows = rows[1:] ttft = statistics.median(r[0] for r in rows); tpot = statistics.median(r[1] for r in rows) print(json.dumps({"kind": kind, "size": size, "frames": frames, "prompt_tokens": rows[0][2], "ttft_p50_ms": round(1e3 * ttft, 1), "ttft_max_ms": round(1e3 * max(r[0] for r in rows), 1), "tpot_p50_ms": round(1e3 * tpot, 2)})) if __name__ == "__main__": a = sys.argv[1:] if a[2] == "image": run(a[0], a[1], "image", a[3], 0, int(a[4]) if len(a) > 4 else 5) else: run(a[0], a[1], "video", a[3], int(a[4]), int(a[5]) if len(a) > 5 else 5)
A shell loop runs the sweeps: for s in 640x360 960x540 1280x720 1600x900 1920x1080 2560x1440 3200x1800 3840x2160; do python3 bench_vlm.py http://localhost:8000 Qwen/Qwen2.5-VL-3B-Instruct image $s; done, and the same with video 640x360 $t for each frame count. We dry-ran the client without a GPU, against a fake server that counts tokens with the Qwen resize rule and waits 20 ms plus 0.05 ms per token. It printed prompt_tokens of 327, 2,719 and 10,577 for 640 × 360, 1920 × 1080 and 3840 × 2160, and 4,811 for 32 frames of 640 × 360: the same counts as the real Hugging Face processor [Certain, our run]. Its TTFTs were about 10 ms above the fake server’s sleep, which is the time the fake server spent parsing the JSON and the JPEG; a real server pays that too, and it belongs in TTFT.
For a load test with images, vLLM’s own client has a random-mm dataset: for example vllm bench serve --backend openai-chat --endpoint /v1/chat/completions --dataset-name random-mm --random-mm-bucket-config '{(1080, 1920, 1): 1.0}' --random-mm-base-items-per-request 1 --request-rate 2. The bucket keys are (height, width, frames); the docs note that only image sampling is supported for now [Certain].
The second program is the quality check of Part 4. We tested its scoring and its paired comparison on made-up correctness lists (they print the numbers of the worked example above), but we did not run it against a server.
# quality_vlm.py: the same 500 ChartQA questions against two servers, then a paired comparison # Usage: python3 quality_vlm.py URL MODEL OUT.json (once per server: BF16, FP8, AWQ) # python3 quality_vlm.py compare A.json B.json import base64, io, json, math, sys, urllib.request def relaxed_match(pred, gold): # ChartQA: numbers within 5%, else exact (case-insensitive) p, g = pred.strip().rstrip(".%").replace(",", ""), gold.strip().rstrip("%").replace(",", "") try: return abs(float(p) - float(g)) <= 0.05 * abs(float(g)) except ValueError: return p.lower() == g.lower() def ask(url, model, png, question): msg = [{"type": "image_url", "image_url": {"url": "data:image/png;base64," + base64.b64encode(png).decode()}}, {"type": "text", "text": question + "\nAnswer with a single word or number."}] body = json.dumps({"model": model, "messages": [{"role": "user", "content": msg}], "max_tokens": 16, "temperature": 0}).encode() req = urllib.request.Request(url + "/v1/chat/completions", body, {"Content-Type": "application/json"}) with urllib.request.urlopen(req, timeout=300) as r: return json.loads(r.read())["choices"][0]["message"]["content"] def evaluate(url, model, out, n=500): from datasets import load_dataset # pip install datasets ds = load_dataset("HuggingFaceM4/ChartQA", split="test").select(range(n)) rows = [] for ex in ds: buf = io.BytesIO(); ex["image"].save(buf, "PNG") pred = ask(url, model, buf.getvalue(), ex["query"]) rows.append({"q": ex["query"], "gold": ex["label"][0], "pred": pred, "ok": relaxed_match(pred, ex["label"][0])}) json.dump(rows, open(out, "w"), indent=0) print(f"{out}: relaxed accuracy {100 * sum(r['ok'] for r in rows) / n:.1f}% on {n} questions") def compare(a_file, b_file): # paired: the same questions in the same order a, b = json.load(open(a_file)), json.load(open(b_file)); n = len(a) lost = sum(x["ok"] and not y["ok"] for x, y in zip(a, b)) # right in A, wrong in B won = sum(y["ok"] and not x["ok"] for x, y in zip(a, b)) # wrong in A, right in B d = (won - lost) / n; se = math.sqrt(lost + won - (lost - won) ** 2 / n) / n k = min(lost, won); p = min(1.0, 2 * sum(math.comb(lost + won, i) for i in range(k + 1)) / 2 ** (lost + won)) print(f"B - A = {100 * d:+.1f} points, 95% CI [{100 * (d - 1.96 * se):+.1f}, {100 * (d + 1.96 * se):+.1f}], " f"flips {lost} lost / {won} won, exact McNemar p = {p:.3f}") if __name__ == "__main__": compare(*sys.argv[2:4]) if sys.argv[1] == "compare" else evaluate(*sys.argv[1:4])
The third program is the planning model behind Figures 1 to 4. It is deterministic and has no dependencies. When you have measurements, fit it: change \(\eta_m\) until TPOT at batch 1 matches, \(\eta_c\) until the TTFT of a tiny image matches (it is mostly prefill), and \(\eta_{\text{enc}}\) until the large images match. If the fitted model then predicts the video sweep too, you understand your server.
# vlm_sim.py: the planning model behind Figures 1 to 4 (Qwen2.5-VL-3B-Instruct on one NVIDIA L4) import math DV, IV, DEPTH, L_FULL, L_WIN, WIN = 1280, 3420, 32, 4, 28, 64 # vision_config P_BLK = (DV * 3 * DV + DV * DV + 3 * DV * IV) * DEPTH # 0.630 B: qkv, proj, gated MLP P_PATCH, P_MERGE = 3 * 2 * 14 * 14 * DV, 5120 * 5120 + 5120 * 2048 # patch embedding, 2x2 merger D, L, H, KVH, DH, V, FF = 2048, 36, 16, 2, 128, 151936, 11008 # text config P_LIN = L * (2 * D * D + 2 * D * KVH * DH + 3 * D * FF) # 2.775 B LLM matmul weights P_EMB, P_ALL, P_VIS = V * D, 3_754_622_976, 668_684_288 # tied embedding; totals from HF KV_TOK = 2 * L * KVH * DH * 2 # 36,864 bytes per token (BF16) BW, F16, F8, MEM = 300e9, 121e12, 242e12, 24e9 # L4 spec sheet (dense) ETA_M, ETA_ENC, T_OH, T_FIX, BUDGET = 0.8, 0.4, 0.002, 0.010, 8192 # assumptions UTIL, RESERVE, PREP_S_PER_MP = 0.9, 2.0e9, 0.046 # assumptions; CPU rate from our run FMT = {"bf16": (2.0, F16, 0.6, False), "fp8": (1.0, F8, 0.5, True), # (bytes/weight, peak, eta, vision in FP8) "int4": (0.5 + 2.5 / 128, F16, 0.45, False)} # AWQ: 4 bits + group-128 scale and zero def smart_resize(h, w, f=28, mn=3136, mx=12845056): # Qwen2-VL image processor rule hb, wb = round(h / f) * f, round(w / f) * f if hb * wb > mx: b = math.sqrt(h * w / mx); hb, wb = max(f, math.floor(h / b / f) * f), max(f, math.floor(w / b / f) * f) elif hb * wb < mn: b = math.sqrt(mn / (h * w)); hb, wb = math.ceil(h * b / f) * f, math.ceil(w * b / f) * f return hb, wb def encoder_time(h, w, slices, fmt): # slices = 1 (image) or ceil(T/2) (video) hb, wb = smart_resize(h, w); per = (hb // 14) * (wb // 14) # patches per temporal slice lin = 2 * (P_BLK + P_PATCH) * per * slices + 2 * P_MERGE * per * slices // 4 att = slices * 4 * DV * per * (L_FULL * per + L_WIN * WIN) # full attention inside each slice return lin / (ETA_ENC * (F8 if FMT[fmt][3] else F16)) + att / (ETA_ENC * F16), slices * per // 4 def w_read(fmt): return P_LIN * FMT[fmt][0] + 2 * P_EMB # bytes per decode step (tied head in BF16) def prefill_time(n, fmt, past=0): # chunks of BUDGET tokens s, peak, eta, _ = FMT[fmt]; t = done = 0 while done < n: c = min(BUDGET, n - done); p = past + done flops = 2 * P_LIN * c + 4 * L * H * DH * (c * p + c * (c + 1) / 2) + 2 * P_EMB t += max(flops / (eta * peak), (w_read(fmt) + (p + c) * KV_TOK) / (ETA_M * BW)) + T_OH; done += c return t def ttft(w, h, frames=0, fmt="bf16"): # frames = 0 means one image slices = math.ceil(frames / 2) if frames else 1 t_enc, n_vis = encoder_time(h, w, slices, fmt) n = n_vis + (27 if frames else 28) # text tokens of the chat template t_prep = PREP_S_PER_MP * w * h * max(frames, 1) / 1e6 return n, T_FIX + t_prep + t_enc + prefill_time(n, fmt) def w_stored(fmt): vis_lin = P_BLK + P_PATCH + P_MERGE; vis_b = 1 if FMT[fmt][3] else 2 return P_LIN * FMT[fmt][0] + vis_lin * vis_b + (P_ALL - P_LIN - vis_lin) * 2 def kv_capacity(fmt): return int((UTIL * MEM - w_stored(fmt) - RESERVE) // KV_TOK) def tpot(batch, ctx, fmt): return (w_read(fmt) + batch * ctx * KV_TOK) / (ETA_M * BW) + T_OH if __name__ == "__main__": for w, h in [(640, 360), (1280, 720), (1920, 1080), (3840, 2160)]: print(f"image {w}x{h}: " + ", ".join(f"{f} {1e3 * ttft(w, h, 0, f)[1]:.0f} ms" for f in FMT), f"({ttft(w, h)[0]} prompt tokens)") for T in [8, 32, 64]: n, t = ttft(640, 360, T); print(f"video 640x360, {T} frames: {n} prompt tokens, TTFT {1e3 * t:.0f} ms") for f in FMT: print(f"{f}: weights {w_stored(f) / 1e9:.2f} GB, KV capacity {kv_capacity(f):,} tokens, " f"TPOT batch 1 {1e3 * tpot(1, 2847, f):.1f} ms, batch 16 {1e3 * tpot(16, 2847, f):.1f} ms")
Its output:
image 640x360: bf16 81 ms, fp8 54 ms, int4 89 ms (327 prompt tokens) image 1280x720: bf16 288 ms, fp8 186 ms, int4 320 ms (1224 prompt tokens) image 1920x1080: bf16 666 ms, fp8 435 ms, int4 741 ms (2719 prompt tokens) image 3840x2160: bf16 3310 ms, fp8 2337 ms, int4 3655 ms (10577 prompt tokens) video 640x360, 8 frames: 1223 prompt tokens, TTFT 323 ms video 640x360, 32 frames: 4811 prompt tokens, TTFT 1286 ms video 640x360, 64 frames: 9595 prompt tokens, TTFT 2652 ms bf16: weights 7.51 GB, KV capacity 327,982 tokens, TPOT batch 1 28.2 ms, batch 16 34.7 ms fp8: weights 4.07 GB, KV capacity 421,371 tokens, TPOT batch 1 16.6 ms, batch 16 23.2 ms int4: weights 3.40 GB, KV capacity 439,408 tokens, TPOT batch 1 11.0 ms, batch 16 17.6 ms
Last, the PyTorch check of the encoder formula (200.4). It builds one Qwen2.5-VL vision block with random weights and counts its FLOPs with torch.utils.flop_counter, once with 64-patch windows and once with full attention. It runs on a CPU in a few seconds.
import torch, torch.nn as nn, torch.nn.functional as F from torch.utils.flop_counter import FlopCounterMode torch.manual_seed(0) DV, IV, HEADS = 1280, 3420, 16 # Qwen2.5-VL vision_config class Block(nn.Module): # one vision block: attention + gated SiLU MLP def __init__(s): super().__init__() s.n1, s.n2 = nn.RMSNorm(DV), nn.RMSNorm(DV) s.qkv, s.proj = nn.Linear(DV, 3 * DV), nn.Linear(DV, DV) s.gate, s.up, s.down = nn.Linear(DV, IV), nn.Linear(DV, IV), nn.Linear(IV, DV) def forward(s, x, win): # x: (N, DV); win: patches per attention segment N = x.shape[0] q, k, v = s.qkv(s.n1(x)).view(N // win, win, 3, HEADS, DV // HEADS).permute(2, 0, 3, 1, 4) a = F.scaled_dot_product_attention(q, k, v) # attention inside each segment only x = x + s.proj(a.permute(0, 2, 1, 3).reshape(N, DV)) h = s.n2(x) return x + s.down(F.silu(s.gate(h)) * s.up(h)) blk = Block().eval() P_LIN = sum(m.weight.numel() for m in blk.modules() if isinstance(m, nn.Linear)) for N, win, name in [(10752, 64, "window layer"), (10752, 10752, "full-attention layer")]: x = torch.randn(N, DV) with torch.no_grad(), FlopCounterMode(display=False) as fc: blk(x, win) model = 2 * P_LIN * N + 4 * DV * N * win # our formula: matmuls + attention print(f"{name:21s} N={N} measured {fc.get_total_flops()/1e12:.3f} TFLOP, formula {model/1e12:.3f} TFLOP")
Our output (PyTorch 2.15): window layer N=10752 measured 0.427 TFLOP, formula 0.427 TFLOP and full-attention layer N=10752 measured 1.015 TFLOP, formula 1.015 TFLOP [Certain, our run]. We used 10,752 patches (168 windows of 64) instead of the 10,764 of a 1080p image so that the windows divide evenly.
The results sheet
Fill one row per configuration. The “model” columns are the predictions of this page; the empty columns are yours. A row without the vLLM version and the full command line is not a result.
| Configuration | TTFT 1080p, model | TTFT 32 × 360p, model | TPOT batch 1, model | TTFT 1080p, yours | TTFT 32 × 360p, yours | TPOT batch 1, yours | ChartQA-500, yours |
|---|---|---|---|---|---|---|---|
| BF16, no caches | 666 ms | 1.29 s | 28.2 ms | ||||
| BF16, repeated image, all caches | 50 ms | n/a | 28.2 ms | same as BF16 | |||
BF16, max_pixels 1,003,520 | 347 ms | 1.29 s | 28.2 ms | ||||
| FP8 (online, per tensor) | 435 ms | 0.87 s | 16.6 ms | ||||
| INT4 AWQ | 741 ms | 1.42 s | 11.0 ms |
The planning model is a sketch with the right shape, good to maybe 30 to 50% before you fit it [Speculation]. The interesting part of the lab is the gap: if FP8 does not lower TTFT on your server, check whether the vision tower was quantized; if INT4 prefill is faster than predicted, your engine uses better 4-bit kernels than we assumed; if a cached request is not fast, the cache did not hit.
Where this is used
| Where | What you measure | What it decides |
|---|---|---|
| Chat with photos (assistants, support apps) | TTFT against resolution; prefix-cache hits on follow-up turns | The max_pixels cap, client-side resizing, keeping the image before the question |
| Document and chart understanding | Quality against max_pixels on your documents | How many pixels fine text needs; whether FP8 or INT4 hurts reading accuracy |
| Video question answering and Video-RAG | TTFT and tokens against frames and frame size | Frames per clip, fps, frame resolution, max-model-len |
| Video retrieval and key-moment detection with VLM captions | Throughput at large batch on short clips | Cost to caption an archive; INT4 for decode-heavy captioning |
| Action recognition and video classification with a VLM | Accuracy against frames (8, 16, 32) | The smallest frame budget that keeps accuracy |
| E-commerce and catalog tagging (same image, many questions) | Processor-, encoder- and prefix-cache hit rates | Prompt layout, stable image bytes or uuids, cache sizes |
| Edge and small-GPU deployment | Memory and TTFT of FP8 and INT4 | Which format fits the device and the latency target |
Common mistakes
- Measuring the same image again and again with caches on. The second request hits the processor, encoder or prefix cache and looks five to ten times faster. Use new bytes for every timing request, or turn the caches off for the sweeps.
- Not checking the token count. A client library, a proxy or a processor version may resize the image. Compare
usage.prompt_tokenswith formula (200.2) before you trust any time. - Plotting TTFT against pixels or file size. The GPU works on tokens. Two images with the same pixel count but different aspect ratios can round to different token counts; a JPEG’s file size says nothing about its tokens.
- Sending frames as separate images. Each image is its own temporal patch, so a frame costs twice as many tokens as inside a video. Send a video when the model supports it.
- Forgetting the CPU. Preprocessing a 4K image took 386 ms on one core in our run. Measure the API server’s CPU time, and give it enough cores or workers before blaming the GPU.
- Expecting INT4 to fix TTFT. AWQ keeps the vision tower in BF16 and does not speed up compute-bound prefill. It helps decode. Choose the format for the metric you need.
- Judging quality from one total. A 1-point change on 500 questions is usually noise. Compare per question (paired), count the flips, and report an interval.
- Putting the question before the image. Every follow-up question then misses the prefix cache for all image tokens.
Interview questions
Why does TTFT of a VLM grow with the image resolution, and roughly how fast?basic
In a 3B VLM, why can the vision encoder cost as much as the language-model prefill?core
How do you make sure a TTFT-versus-resolution measurement is correct?core
usage.prompt_tokens to confirm that the server saw the token count you predicted from the resize rule. Report the median and the spread of several requests per size, and plot against tokens. Record the vLLM version, the full command line and the processor settings (max_pixels), because they change the token count.A colleague says “just send 64 frames of 720p to the model”. What happens on Qwen2.5-VL-3B with max-model-len 32,768?core
What are the processor cache, the encoder cache and the prefix cache in vLLM, and what does each skip?core
Why should the image come before the question in the prompt?basic
FP8 lowers TTFT but INT4 AWQ does not. Why, and which one would you pick?advanced
Quantizing the 3B VLM saves 3 to 4 GB. Why does that change the batch size so little on an L4?advanced
How would you check that INT4 did not hurt quality, and how many questions do you need?advanced
Your measured TTFT for 1080p images is 1.5 s, but the model predicts 0.67 s. Where do you look?core
usage.prompt_tokens: if it is much larger than 2,719, the image was not resized the way you think. Then the CPU: time the API server process; a slow vCPU or a single-threaded processor can add hundreds of milliseconds, and a queue of preprocessing requests adds more. Then the GPU split: profile the encoder and the prefill separately (or compare a text-only prompt of the same length, which has no encoder); a slow encoder often means the wrong attention backend or no CUDA graphs for it. Then the server: is something else running, is the request waiting for a large batch, did the first requests pay for warm-up? Fit the planning model’s efficiencies to small and large images to see which part is off.Exercises
- Use (200.2) to compute the image tokens and the encoder patches of a 1280 × 960 photo.
- A clip has 48 frames of 848 × 480. How many video tokens does Qwen2.5-VL produce, how much KV cache does the prompt take (27 text tokens), and does it fit in max-model-len 32,768 with 256 output tokens?
- Show that 54 is the largest even number of 1280 × 720 frames that fits in 32,768 tokens with 27 text tokens and 256 output tokens.
- Compute the lowest possible TPOT at batch 1 for the INT4 model (2.064 GB read per step), first at the spec bandwidth, then with \(\eta_m = 0.8\) and 2 ms of overhead.
- In the worked example of Part 3, what would be cached and what prefilled if the block size were 32 instead of 16? Is the encoder needed?
- On 500 questions, FP8 loses 9 questions that BF16 got right and wins 3. Compute the difference, its 95% interval with (200.7), and the exact McNemar p-value.
- Run the lab: the resolution sweep, the frame sweep, the six caching requests and at least one quantized model on a real GPU. Fit the planning model and report where it was wrong.
Answerscheck
Key takeaways
- Write down every multimodal flag (
--limit-mm-per-prompt,--mm-processor-kwargs, cache sizes,--max-model-len) and the vLLM version; checkusage.prompt_tokensagainst the resize rule before trusting any time. - Qwen2.5-VL spends one token per 28 × 28 pixels and one temporal patch per two frames; TTFT grows a bit faster than linearly with tokens. Plot against tokens.
- TTFT = CPU preprocessing + vision encoder + prefill. In a 3B VLM the encoder is about half of the GPU work, and the CPU can be hundreds of milliseconds for large images.
- Video hits hard limits: 32 frames of 720p are 19,136 tokens; the context fills at about 54 frames. Frames times pixels is the budget.
- The processor, encoder and prefix caches remove repeated work; with all three, a repeated 1080p image costs about 50 ms instead of 666 ms in our model. Keep the image before the question and its bytes identical.
- FP8 speeds up the compute-bound TTFT; INT4 AWQ speeds up the memory-bound decode and leaves the vision tower in BF16. Measure quality per question with a paired test.
Key formulas
References for this session
- vLLM project (docs). vLLM documentation
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Qwen Team (2025). Qwen2.5-VL-3B-Instruct model card, config.json and preprocessor_config.json (Hugging Face)
- Qwen Team (2025). Qwen2.5-VL-3B-Instruct-AWQ model card and config.json (Hugging Face)
- vLLM project (2026). vLLM documentation: Multimodal Inputs
- vLLM project (2026). vLLM documentation: Engine Arguments
- vLLM project (2026). vllm bench serve: command-line reference (random-mm dataset)
- vLLM project (2026). vLLM documentation: FP8 W8A8 quantization
- vLLM project (2026). vllm/v1/core/encoder_cache_manager.py (source code)
- Hugging Face (2026). transformers: Qwen2-VL image and video processors (smart_resize, video_processing_qwen2_vl.py)
- NVIDIA (2023). NVIDIA L4 Tensor Core GPU (product page and specifications)
- Masry, Long, Tan, Joty & Hoque (2022). ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning