The VLM inference path
A VLM request does more work than a text request before the first word comes out: the image is decoded and resized on the CPU, run through a vision encoder, and turned into hundreds or thousands of extra prompt tokens that the language model must prefill and keep in its KV cache. This session follows one request through every stage, counts the cost of each with real model configs, and shows how encoder caching and prefix caching remove repeated work. It ends with a breakdown of where the time goes.
What you will learn
- The ten stages of a VLM request, from uploaded bytes to streamed text, and which ones run on the CPU and which on the GPU
- Why image decode and resize can cost more than the GPU work for a small VLM, and how to move or shrink it (GPU decode, JPEG draft mode, client-side resize, overlap)
- How to count vision-encoder FLOPs, why the encoder is about 5% of prefill in LLaVA-1.5 but about 30% in Qwen2.5-VL, and what window attention saves
- Why image tokens make prefill long and compute-bound, and how to estimate time to first token from FLOPs
- How much KV cache an image takes (288 MiB for LLaVA-1.5-7B, 147 MiB for a 1080p image in Qwen2.5-VL-7B) and what it does to decode
- Encoder caching and multimodal prefix caching: hashing image bytes into block hashes, why the image should come before the question, and what each cache saves
- A stage-by-stage time breakdown, and why TTFT and end-to-end latency have different bottlenecks
Why this matters
Send a text question to a language model and the server does two things: prefill the prompt, then decode the answer token by token. Send the same question with a photo and the server does much more before the first word appears. It decodes the JPEG, resizes it, runs a vision encoder, maps the result into the language model’s space, and then prefills a prompt that is now hundreds or thousands of tokens longer.
Each of these extra steps has a cost, and the costs land in different places. Some run on the CPU, some on the GPU. Some add to the time to first token (TTFT, the delay before the first output token), some fill GPU memory, and some slow every decode step. A team that profiles only the language model will miss half of the picture. For example, for a 7B LLaVA-style model, the GPU part of the prefill takes tens of milliseconds, while decoding and resizing a 12-megapixel phone photo on one CPU core can take longer than that. For a model with a bigger encoder and native resolution, such as Qwen2.5-VL, the vision encoder alone is about 30% of the prefill FLOPs.
In session 177 we built the model: encoder, connector, language model. In session 183 we counted visual tokens and learned to remove some. This session looks at the serving path of a request through that model, the way an inference engineer sees it. We follow one request stage by stage, count the cost of each stage using real model configs (LLaVA-1.5-7B and Qwen2.5-VL-7B), add the two caches that remove repeated work, and finish with a breakdown of where the time goes. The next session, session 197, then makes the remaining work cheaper.
The path of one request
Here is the full path, from the bytes the user uploads to the text that comes back. We give each stage a name, because a profile is only useful if you can say which stage a number belongs to.
- Receive. The image arrives as a file upload, a URL to fetch, or a base64 string inside JSON. Base64 is a way to write binary data as text; it makes the data \(4/3\) times larger.
- Decode. The JPEG or PNG is decompressed into a grid of pixels: \(H \times W \times 3\) bytes (one byte per color channel).
- Resize and normalize. The pixels are resized to what the encoder expects (a fixed 336 × 336 for LLaVA-1.5, a multiple of 28 pixels for Qwen2.5-VL), converted to floats, and normalized with a per-channel mean and standard deviation.
- Patchify. The image is cut into square patches (14 × 14 pixels for both models), and each patch is flattened into a vector.
- Vision encoder. A vision transformer (ViT) turns the patch vectors into feature vectors.
- Connector. A small network maps the features into the language model’s embedding space. It may also merge neighbors to reduce the count (Qwen2.5-VL merges 2 × 2 patches into one token).
- Merge into the prompt. The prompt has placeholder tokens where the image goes. Their embeddings are replaced by the image embeddings.
- Prefill. The language model processes the whole prompt (text and image tokens) in one forward pass and writes keys and values for every token into the KV cache.
- Decode. The model produces one token per step, each step reading all weights and the whole KV cache.
- Detokenize and stream. Token ids are turned back into text and sent to the user.
Stages 1 to 3 usually run on the CPU, in the server’s front-end process. Stages 4 to 9 run on the GPU (some processors, such as Qwen2.5-VL’s Hugging Face processor, already cut the patches on the CPU). Stage 10 is on the CPU again. Step through Figure 1 to see the tensor at every stage, for both models and three image sizes.
How to read Figure 1
Pick a model and an image size, then press Next. Watch three things. First, the pixel tensor after decode: a 4000 × 3000 photo is 36 MB of raw pixels, which the CPU must produce and then shrink. Second, the token count at the connector: LLaVA-1.5 always gives 576 tokens because it resizes every image to 336 × 336, while Qwen2.5-VL’s count grows with the image (2,691 tokens for 1920 × 1080). Third, the KV cache written at prefill: LLaVA-1.5 uses 0.5 MiB per token, Qwen2.5-VL only 56 KiB, because Qwen uses grouped-query attention with 4 key-value heads.
The general algorithm, as a serving engine runs it:
- input: request with text and image bytes \(r_1, \dots, r_k\)
- for each image \(r_i\) // CPU, front-end process
- \(\text{key}_i \leftarrow \text{hash}(r_i)\) // used by both caches
- \(x_i \leftarrow \text{normalize}(\text{resize}(\text{decode}(r_i)))\) // skip if the processed pixels are cached
- tokens \(\leftarrow\) chat template with \(n_i\) placeholder tokens per image
- \(h \leftarrow\) longest cached prefix of tokens (with image keys) // prefix cache, Algorithm 2
- for each image whose tokens are not all inside the first \(h\) // GPU
- \(E_i \leftarrow\) encoder cache[\(\text{key}_i\)] if present, else \(\text{connector}(\text{ViT}(\text{patchify}(x_i)))\)
- embeddings \(\leftarrow\) text embeddings with placeholders replaced by \(E_i\)
- KV, logits \(\leftarrow\) LLM prefill on positions \(h, \dots, n-1\) using the cached KV for \(0, \dots, h-1\)
- repeat decode one token, append its KV, stream it until end token or length limit
Stage 1 to 3: preprocessing on the CPU or the GPU
The first cost is easy to forget because it is not on the GPU. Decoding a JPEG is real work: the CPU must undo the entropy coding, invert the discrete cosine transform for every 8 × 8 block, and convert colors. The output size is simple to compute:
- Read the symbols
- \(H, W\)
- height and width of the decoded image in pixels
- \(B_{\text{pix}}\)
- bytes of raw pixels after decode (8-bit RGB)
- \(B_{\text{file}}, B_{\text{base64}}\)
- bytes of the compressed file, and of the same file written as base64 text
A 12-megapixel phone photo (4000 × 3000) becomes \(4000 \times 3000 \times 3 = 36\) MB of pixels. LLaVA-1.5 then throws almost all of it away: \(336 \times 336 \times 3\) is only 0.34 MB. The resize itself is also work, because a good resize filter (bilinear or bicubic with antialiasing) reads every input pixel. In our own run on one laptop CPU core with Pillow, decoding plus resizing a synthetic 12-megapixel JPEG took about 0.1 to 0.25 seconds, depending on the image content. That is one machine and one image; measure your own with the snippet below. The point is the order of magnitude: it is in the same range as, or larger than, the whole GPU prefill of a 7B LLaVA model (about 18 ms in our estimate below).
Why does this matter in a server? Because the front-end process is often a single Python process. If it decodes and resizes images one at a time, the GPU waits. Four fixes are used in practice:
- Resize on the client. If the model will see 336 × 336 anyway, there is no reason to upload 12 MP. A phone app can resize before sending. This also cuts network time.
- Decode at reduced size. JPEG can be decoded directly at \(1/2\), \(1/4\) or \(1/8\) of the size, by skipping high-frequency DCT coefficients. In Pillow this is
Image.draft(). It is cheaper than a full decode followed by a resize. - Decode on the GPU. NVIDIA’s nvJPEG library decodes on the GPU;
torchvision.io.decode_jpeg(..., device="cuda")and NVIDIA DALI use it. Then resize on the GPU too. The CPU only moves the compressed bytes. The cost is some GPU time and memory, taken from the model. - Overlap. Run preprocessing in a pool of worker processes, so the CPU prepares the next requests while the GPU runs the current batch. Most serving engines do this in a separate front-end process and also cache the processed result (vLLM keeps a processor cache for this [Likely]).
The snippet below measures decode and resize on your machine. Run it on your real images (photos, scans, screenshots), because the cost depends on the content and on the JPEG quality.
import io, time from PIL import Image src = Image.merge("RGB", [Image.effect_mandelbrot((4000, 3000), (-2, -1.2, 1, 1.2), q) for q in (40, 80, 160)]) # a 12-megapixel test image buf = io.BytesIO(); src.save(buf, "JPEG", quality=90); raw = buf.getvalue() def once(): t0 = time.perf_counter(); im = Image.open(io.BytesIO(raw)); im.load() # decode t1 = time.perf_counter(); im.resize((336, 336), Image.BICUBIC) # resize t2 = time.perf_counter(); return t1 - t0, t2 - t1 once() # warm up d, r = min(once() for _ in range(5)) print(f"file {len(raw)/1e6:.1f} MB, decode {1e3*d:.0f} ms, resize {1e3*r:.0f} ms, " f"{12/(d + r):.0f} megapixels per second on one core") im = Image.open(io.BytesIO(raw)); im.draft("RGB", (1000, 750)) # JPEG DCT scaling: decode at 1/4 size t0 = time.perf_counter(); im.load(); print("draft decode", im.size, f"{1e3*(time.perf_counter()-t0):.0f} ms")
Stage 4 to 6: the vision encoder and connector
The vision encoder is a ViT. Its cost follows the same rule as any transformer: about 2 FLOPs per parameter per token for the matrix multiplications, plus an attention term that grows with the square of the sequence length. The difference from the language model is what a “token” is: here it is a patch.
How many patches, how many tokens
LLaVA-1.5 uses CLIP ViT-L/14 at 336 pixels: \(336/14 = 24\), so \(24 \times 24 = 576\) patches, plus one class token. Every image gives 576 language-model tokens, whatever its original size. Qwen2.5-VL keeps the original aspect ratio. Its processor rounds the height and width to multiples of 28 (and shrinks very large images to at most 12,845,056 pixels), cuts 14 × 14 patches, and the connector merges each 2 × 2 group of patches into one token [Certain, from the model’s processor config]:
- Read the symbols
- \(H', W'\)
- the resized height and width, each a multiple of 28 (for example 1080 × 1920 becomes 1092 × 1932)
- \(N_{\text{patch}}\)
- sequence length inside the vision encoder
- \(N_{\text{tok}}\)
- number of image tokens the language model sees
For a 1920 × 1080 image: \(N_{\text{patch}} = 78 \times 138 = 10{,}764\) and \(N_{\text{tok}} = 2{,}691\). Note the factor 4: the Qwen encoder works on a sequence 4 times longer than the one the language model sees.
Encoder FLOPs
For a ViT with \(L_v\) layers, width \(d_v\) and \(P_{\text{enc}}\) parameters, the forward cost is
- Read the symbols
- \(2 P_{\text{enc}} N_{\text{patch}}\)
- the matrix multiplications: 2 FLOPs (a multiply and an add) per weight per patch
- \(4 d_v N \cdot N\)
- attention: \(QK^\top\) and the weighted sum of values, each \(2 d_v N^2\) FLOPs per layer when every patch sees every patch
- \(L_{\text{full}}, L_{\text{win}}\)
- number of layers with full attention, and with window attention
- \(w\)
- patches per window; Qwen2.5-VL uses 112 × 112 pixel windows, so \(w = 8 \times 8 = 64\)
LLaVA-1.5’s encoder has 24 full-attention layers, width 1024, and about 0.30 B parameters in its layers; for 577 patches that is about 0.38 TFLOPs. Qwen2.5-VL’s encoder has 32 layers, width 1280 and about 0.68 B parameters; only 4 of its layers (numbers 7, 15, 23 and 31) use full attention, the other 28 use windows [Certain, from config.json]. Figure 3 shows what that choice saves.
A quick rule tells you how big the encoder is relative to the language model. Ignoring attention, the ratio of encoder FLOPs to LLM prefill FLOPs for the image tokens is
- Read the symbols
- \(s\)
- encoder patches per language-model token: 1 for LLaVA-1.5, 4 for Qwen2.5-VL (2 × 2 merge)
- \(P_{\text{enc}}, P_{\text{LLM}}\)
- parameters of the encoder and of the language model (without the embedding table)
LLaVA-1.5-7B: \(\rho = 1 \times 0.302 / 6.48 \approx 0.047\). The encoder is a rounding error next to the 7B language model. Qwen2.5-VL-7B: \(\rho = 4 \times 0.676 / 6.53 \approx 0.41\). The encoder is now a big part of the cost, and it grows faster than linearly with image size because of the full-attention layers.
How to read Figure 3
At small sizes all three curves grow roughly linearly with the number of pixels, and the window encoder is about 25 to 30% of the total. As the image grows, the full-attention curve bends upward fast: at 4000 × 3000 it would cost more than the language model itself. Window attention keeps most layers linear; only the 4 full layers still grow with the square. The readout converts FLOPs to milliseconds assuming an H100 at 50% of its dense BF16 peak, which is an assumption, not a measurement.
Resize: \(1080 \to 1092\) and \(1920 \to 1932\) (nearest multiples of 28). Patches: \(78 \times 138 = 10{,}764\). Tokens: \(10{,}764 / 4 = 2{,}691\).
Encoder (formula 196.3, with window attention): about 16.3 TFLOPs, of which attention is about 15%. With full attention in all 32 layers it would be about 32.8 TFLOPs, and attention would be 58% of it.
Language-model prefill of 2,691 + 50 tokens (formula 196.5 below): about 38.8 TFLOPs. So the encoder is 30% of the GPU work before the first token.
Time, if we assume an H100 SXM (989 TFLOP/s dense BF16 [Certain, NVIDIA spec]) running at 50% of peak: \((16.3 + 38.8) / 494.5 \approx 0.111\) s, about 111 ms. Of that, 33 ms is the encoder.
Stage 8: the long prefill of image tokens
Now the image tokens enter the language model. For the language model they are just prompt tokens: each one goes through every layer, and each one gets keys and values in the KV cache. The prefill cost for a prompt of \(n\) tokens is the familiar formula from session 78:
- Read the symbols
- \(P_{\text{LLM}}\)
- parameters of the language model used per token (all layers, without the embedding table)
- \(L, d\)
- number of layers and model width of the language model
- \(N_{\text{img}}, N_{\text{text}}\)
- image tokens (sum over all images) and text tokens in the prompt
We count the full \(n \times n\) attention; a causal kernel skips about half of it, so this is a slight overestimate. The key fact: with an image, \(n\) is dominated by \(N_{\text{img}}\). A 50-token question to LLaVA-1.5-7B costs about 0.65 TFLOPs of prefill. Add one image and the prompt is 626 tokens and about 8.3 TFLOPs, 12.8 times more. With Qwen2.5-VL and a high-resolution photo, the image can be 15,301 tokens (a 4000 × 3000 photo after resizing to 4004 × 2996).
Prefill with many tokens is compute-bound: the GPU does many FLOPs per byte of weights it reads, so its speed is set by its arithmetic throughput, not its memory bandwidth (see session 82). So a FLOP count gives a usable estimate of TTFT:
- Read the symbols
- \(t_{\text{prep}}\)
- CPU time for receive, decode and resize (stages 1 to 3), if it is not overlapped
- \(u\)
- the fraction of peak the kernels really reach (model FLOPs utilization); 0.4 to 0.6 is a common range for large prefills [Likely], but you must measure it
- \(\Pi_{\text{peak}}\)
- the GPU’s peak FLOP rate, for example 989 TFLOP/s dense BF16 on an H100 SXM
- \(t_{\text{queue}}\)
- time waiting behind other requests in the scheduler
Encoder plus projector: \(0.381 + 0.024 = 0.405\) TFLOPs. Prefill of 626 tokens with \(P_{\text{LLM}} = 6.48\) B, \(L = 32\), \(d = 4096\): \(F_{\text{pre}} \approx 8.31\) TFLOPs. Total GPU work: 8.72 TFLOPs.
At 50% of an H100’s peak: \(8.72 / 494.5 \approx 0.0176\) s, about 18 ms. The encoder’s share is under 1 ms. Without the image, the 50-token prefill would take about 1.3 ms.
Compare with the CPU: if decode and resize run at 100 megapixels per second on one core (in the range of our laptop run), a 12 MP photo costs 120 ms of preprocessing, almost 7 times the GPU work. For this model, the image upload and the CPU are the first places to look when TTFT is high.
Long image prefills also hurt other users. A 15,000-token prefill keeps the GPU busy for a long time, and the decode steps of other requests in the same batch wait for it. Chunked prefill (from session 135) splits a long prompt into pieces and mixes them with decode steps. For a VLM there is one extra rule: the encoder runs on a whole image at once, so the engine must run the encoder for an image before the first chunk that contains its tokens, and keep the output until the last such chunk has been processed. vLLM schedules this with an encoder compute budget and keeps the outputs in an encoder cache on the GPU [Likely]. Some systems go further and put the encoder on separate GPUs, next to separate prefill and decode GPUs (EPD disaggregation; session 137).
Image tokens in the KV cache
Every prompt token, image or text, leaves keys and values in every layer. The bytes per token are
- Read the symbols
- \(2\)
- one key and one value vector
- \(n_{\text{kv}}, d_h\)
- number of key-value heads and the size of each head
- \(b\)
- bytes per number: 2 for FP16 or BF16
- \(M_{\text{img}}\)
- KV-cache memory taken by one image
LLaVA-1.5-7B (32 layers, 32 KV heads of size 128): \(b_{\text{KV}} = 2 \cdot 32 \cdot 32 \cdot 128 \cdot 2 = 524{,}288\) bytes, exactly 0.5 MiB per token, so 288 MiB per image. Qwen2.5-VL-7B (28 layers, 4 KV heads of size 128, grouped-query attention): 57,344 bytes, 56 KiB per token. A 1920 × 1080 image (2,691 tokens) takes 147 MiB, about half of LLaVA’s image, although it has 4.7 times more tokens. Eight such images in one prompt take 1.15 GiB. This is the memory that limits how many requests fit in a batch.
The KV cache also slows decode. Each decode step reads all the weights once and the KV cache of every request in the batch. Decode is memory-bound, so its time is bytes divided by bandwidth:
- Read the symbols
- \(M_W\)
- bytes of the language-model weights read per step (13.2 GB for LLaVA-1.5-7B in BF16)
- \(B, n_j\)
- batch size, and the current context length of request \(j\) (its image and text tokens so far)
- \(\eta \cdot \text{BW}\)
- achieved memory bandwidth: a fraction \(\eta\) of the peak (3.35 TB/s on an H100 SXM [Certain, NVIDIA spec])
At batch size 1, one LLaVA image adds 313 MiB (\(626 \times 0.5\) MiB) to the 13.2 GB of weights per step: about 2.4% slower decode. At batch size 32, the KV caches add 10.5 GB per step, almost as much as the weights. So image tokens hurt decode mainly at large batch sizes, and much less for GQA models like Qwen2.5-VL. The visual-token reduction methods from session 183 help here too, and so does KV-cache quantization.
Caching: do not do the same work twice
Real traffic repeats images. A user asks three questions about the same chart in a chat: the client sends the full history, image included, every turn. A shop has one product photo that thousands of users ask about. A document pipeline asks ten extraction questions about each page. Without caching, every one of these requests decodes the image, runs the encoder, and prefills all the image tokens again. Two caches remove this.
Encoder cache
The encoder cache maps a hash of the image to the encoder-plus-connector output \(E\) (the image embeddings, \(N_{\text{tok}} \times d\) numbers). The key must be the image content, for example SHA-256 of the file bytes or of the decoded pixels, not the URL or the file name: two users can upload the same file under different names, and one URL can change its content. A hit skips decode, resize and the encoder. It does not skip the language-model prefill.
Prefix cache with images
The prefix cache (session 134) keeps the KV blocks of earlier prompts. A new request that starts with the same tokens reuses them and prefills only the rest. The KV cache is stored in blocks of \(B\) tokens (16 by default in vLLM), and each full block gets a hash that depends on everything before it:
- Read the symbols
- \(h_j\)
- the hash of block \(j\); because it includes \(h_{j-1}\), equal hashes mean the whole prefix up to that block is equal
- \(t_i\)
- the token ids in the block
- \(\text{extra}_j\)
- extra keys: here, the content hash of every image whose placeholder tokens appear in the block
The extra key is essential for VLMs. The placeholder token ids are the same for every image (LLaVA uses token 32000 repeated 576 times). Without the image hash, two different photos would produce the same block hashes, and the second user would get answers about the first user’s photo. vLLM puts the image hash into the hash of every block that holds that image’s tokens [Certain, vLLM design docs]. SGLang’s radix tree uses image hashes in the same role [Likely].
The number of tokens a request must prefill is then
- Read the symbols
- \(m\)
- the number of leading tokens that match a cached prompt (same ids and same images)
- \(h\)
- the cached prefix length: only full blocks count, and at least the last token is always computed (it produces the first output logits)
- input: token ids \(t_0, \dots, t_{n-1}\), image keys with their token positions, block size \(B\)
- \(h_{-1} \leftarrow\) empty; \(\; h \leftarrow 0\)
- for \(j = 0, 1, \dots\) while block \(j\) is full
- extra \(\leftarrow\) keys of images with any token in block \(j\)
- \(h_j \leftarrow \text{hash}(h_{j-1}, t_{jB..jB+B-1}, \text{extra})\) // formula (196.9)
- if \(h_j\) is in the block table then \(h \leftarrow (j+1)B\) else break
- \(h \leftarrow \min(h, B\lfloor (n-1)/B \rfloor)\) // always compute the last token
- for each image with a token at position \(\ge h\)
- \(E \leftarrow\) encoder cache[key] if present, else run the encoder and store it
- prefill positions \(h..n-1\); store the hashes of the new full blocks
Order matters. The prefix cache only helps if the shared part comes first. Put the image before the question (system prompt, image, question) and a second question about the same image hits the cache for the system prompt and all image tokens. Put the question first (system prompt, question, image) and the prefix differs right after the system prompt: the image tokens must be prefilled again, and only the encoder cache can help. Figure 4 lets you try both.
How to read Figure 4
Each row is one request’s prompt, drawn to scale. Green parts come from the prefix cache, orange parts are prefilled. The dark band marks the 576 image tokens. Step through the five requests with both caches on, then turn the prefix cache off: every request prefills everything. Turn only the encoder cache on: the encoder runs twice instead of five times, but the saving is small for LLaVA, because its encoder is only about 5% of the work. Then switch to “Question before image” and look at request 4: the image tokens are recomputed. Note also that only 32 of the 40 system-prompt tokens are cached in request 5: only full blocks of 16 count.
Prompt lengths: turn 1 is \(40 + 576 + 30 = 646\) tokens, turn 2 adds the first answer and a new question (736), turn 3 adds another 90 (826). Without caches the engine prefills \(646 + 736 + 826 = 2{,}208\) tokens and runs the encoder 3 times.
With the prefix cache, turn 2 finds the previous prompt and answer (706 tokens) in cache; full blocks give \(h = 16 \lfloor 706/16 \rfloor = 704\), so it prefills only 32 tokens. Turn 3: \(h = 16 \lfloor 796/16 \rfloor = 784\), so 42 tokens. Total: \(646 + 32 + 42 = 720\) tokens, and the encoder runs once (later turns have all image tokens inside the cached prefix).
In FLOPs, the three turns cost 30.7 TFLOPs without caches and 10.0 TFLOPs with the prefix cache. The encoder cache alone saves only \(2 \times 0.405 = 0.81\) TFLOPs here. For Qwen2.5-VL with a high-resolution image, where the encoder is 30% of the work, the encoder cache matters much more.
Where the time goes: a breakdown
We now have a formula for every stage. End-to-end latency for one request is
- Read the symbols
- \(t_{\text{enc}}, t_{\text{pre}}\)
- encoder and prefill time: FLOPs divided by \(u \cdot \Pi_{\text{peak}}\)
- \(N_{\text{out}}\)
- number of output tokens; the first one comes out of the prefill, the other \(N_{\text{out}} - 1\) need a decode step each
- \(t_{\text{step}}(\cdot)\)
- decode step time at the current context length, formula (196.8)
Figure 5 puts it all together for batch size 1 on an H100. The answer to “where does the time go?” depends on what you measure. For TTFT, the image dominates: preprocessing for small encoders and big photos, encoder plus prefill for native-resolution models. For end-to-end latency with a long answer, decode dominates, because each of hundreds of steps reads 13 to 14 GB of weights. A short answer (a yes/no question, a label, a bounding box) makes the image stages matter much more.
How to read Figure 5
The top bar is TTFT, the bottom bar is the whole request, both on their own time scale. Start with LLaVA-1.5 and a 4000 × 3000 photo: preprocessing is most of TTFT, the encoder is a thin sliver. Switch to Qwen2.5-VL with the same photo: now the encoder and prefill take about 0.9 s and preprocessing is a smaller share. Set output tokens to 1: the end-to-end bar becomes almost all image work. Set it to 1000: decode takes over. Move the CPU rate to see how much a GPU decoder or client-side resizing could save.
How to get these numbers for your own system: put timers (or profiler ranges, see session 123) around each stage, synchronize the GPU before reading a timer, and report TTFT and time per output token separately, with their distributions, not only the mean (session 81). Compare the measured numbers with the estimates above. A large gap tells you where to look: a slow stage that the model says should be fast is usually a bug or a missing overlap.
Code: a tiny VLM server with both caches
The code below builds a toy VLM (a small ViT encoder with an MLP connector, and a one-layer language model with a KV cache) and serves a three-turn chat. It implements the encoder cache keyed by a SHA-256 of the uploaded bytes, and the block-hash prefix cache of formula (196.9) with the image hash as an extra key. It counts encoder runs and prefilled tokens, checks that the cached path gives the same logits as the uncached path, and checks that a different image with the same placeholder tokens does not hit the image blocks. It runs on a CPU in a few seconds.
import hashlib, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) D, P, S, BLK = 64, 8, 32, 4 # model width, patch size, encoder input side, KV block size IMG = 1000 # placeholder token id for one image token def decode(raw, H=480, W=640): # stands in for JPEG decode: file bytes -> (3, H, W) uint8 return torch.frombuffer(bytearray(raw), dtype=torch.uint8).view(3, H, W) def preprocess(raw): x = decode(raw).float().unsqueeze(0) / 255.0 x = F.interpolate(x, size=(S, S), mode="bilinear", antialias=True, align_corners=False) mean = torch.tensor([0.481, 0.458, 0.408]).view(1, 3, 1, 1) std = torch.tensor([0.269, 0.261, 0.276]).view(1, 3, 1, 1) return (x - mean) / std # (1, 3, S, S) class Encoder(nn.Module): # tiny ViT + MLP connector def __init__(s): super().__init__() s.patch = nn.Linear(3 * P * P, D) s.pos = nn.Parameter(torch.randn((S // P) ** 2, D) * 0.02) s.blocks = nn.TransformerEncoder(nn.TransformerEncoderLayer(D, 4, 4 * D, batch_first=True, dropout=0.0), 2) s.proj = nn.Sequential(nn.Linear(D, D), nn.GELU(), nn.Linear(D, D)) def forward(s, x): p = F.unfold(x, P, stride=P).transpose(1, 2) # (1, 16 patches, 192) return s.proj(s.blocks(s.patch(p) + s.pos))[0] # (16, D): one LLM token per patch class TinyLM(nn.Module): # one causal attention layer + MLP, with a KV cache def __init__(s, V=1001): super().__init__() s.emb = nn.Embedding(V, D); s.qkv = nn.Linear(D, 3 * D); s.o = nn.Linear(D, D) s.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D)); s.head = nn.Linear(D, V) def forward(s, h, past=None): # h: (n, D) new embeddings; past: (K, V) of earlier positions q, k, v = s.qkv(h).chunk(3, -1) if past is not None: k, v = torch.cat([past[0], k]), torch.cat([past[1], v]) n, T = q.shape[0], k.shape[0] mask = torch.ones(n, T, dtype=torch.bool).tril(T - n) # causal, offset by the cache a = F.scaled_dot_product_attention(q[None], k[None], v[None], attn_mask=mask[None])[0] h = h + s.o(a); h = h + s.mlp(h) return s.head(h), (k, v) enc, lm = Encoder().eval(), TinyLM().eval() stats = {"encoder_runs": 0, "prefill_tokens": 0} enc_cache, prefix_cache = {}, {} # image hash -> embeddings ; block-chain hash -> (K, V) up to that block def img_key(raw): return hashlib.sha256(raw).hexdigest() # hash the uploaded bytes def embed(tokens, images, use_enc_cache): out, it, i = [], iter(images), 0 while i < len(tokens): if tokens[i] == IMG: # a run of 16 placeholders = one image raw = next(it); key = img_key(raw) if not (use_enc_cache and key in enc_cache): stats["encoder_runs"] += 1; enc_cache[key] = enc(preprocess(raw)) out.extend(enc_cache[key]); i += (S // P) ** 2 else: out.append(lm.emb.weight[tokens[i]]); i += 1 return torch.stack(out) def block_hashes(tokens, image_keys): # chain hash per full block; image hash is an "extra key", formula (196.9) hs, parent = [], b"" for b in range(len(tokens) // BLK): blk = tokens[b * BLK:(b + 1) * BLK] extra = "|".join(image_keys) if IMG in tokens[:(b + 1) * BLK] else "" parent = hashlib.sha256(parent + str(blk).encode() + extra.encode()).digest(); hs.append(parent) return hs @torch.no_grad() def prefill(tokens, images, use_enc_cache=True, use_prefix_cache=True): keys = [img_key(r) for r in images] h_all = embed(tokens, images, use_enc_cache) start, past = 0, None if use_prefix_cache: # longest cached prefix of full blocks for b, hb in enumerate(block_hashes(tokens, keys)): if hb in prefix_cache and (b + 1) * BLK < len(tokens): start, past = (b + 1) * BLK, prefix_cache[hb] logits, kv = lm(h_all[start:], past) stats["prefill_tokens"] += len(tokens) - start for b, hb in enumerate(block_hashes(tokens, keys)): # store every full block prefix_cache[hb] = (kv[0][:(b + 1) * BLK], kv[1][:(b + 1) * BLK]) return logits[-1] g = torch.Generator().manual_seed(1) photo = bytes(torch.randint(0, 256, (3 * 480 * 640,), generator=g).tolist()) # "uploaded files" other = bytes(torch.randint(0, 256, (3 * 480 * 640,), generator=g).tolist()) sys_ = [1, 2, 3, 4, 5, 6, 7, 8] # system prompt (8 tokens) img = [IMG] * 16 turns = [[11, 12, 13, 14], [21, 22, 23, 24, 25, 26, 27, 28], [31, 32, 33, 34]] for enc_c, pre_c in [(False, False), (True, False), (True, True)]: enc_cache.clear(); prefix_cache.clear(); stats.update(encoder_runs=0, prefill_tokens=0); hist = sys_ + img for q in turns: hist = hist + q; out = prefill(hist, [photo], enc_c, pre_c) hist = hist + [int(out.argmax())] * 4 # pretend the model answered with 4 tokens print(f"encoder cache={enc_c!s:5} prefix cache={pre_c!s:5} -> {stats}") # correctness: cached and uncached prefill give the same next-token logits enc_cache.clear(); prefix_cache.clear() a = prefill(sys_ + img + turns[0], [photo]); b = prefill(sys_ + img + turns[1], [photo]) c = prefill(sys_ + img + turns[1], [photo], False, False) print("max |cached - uncached| =", f"{(b - c).abs().max().item():.1e}") # a different image with identical placeholder tokens must NOT hit the image blocks stats["prefill_tokens"] = 0; prefill(sys_ + img + turns[1], [other]); print("other image prefilled:", stats["prefill_tokens"])
Output (PyTorch 2.8 on a CPU):
encoder cache=False prefix cache=False -> {'encoder_runs': 3, 'prefill_tokens': 116}
encoder cache=True prefix cache=False -> {'encoder_runs': 1, 'prefill_tokens': 116}
encoder cache=True prefix cache=True -> {'encoder_runs': 1, 'prefill_tokens': 48}
max |cached - uncached| = 2.4e-07
other image prefilled: 24
Read it like Figure 4. The prompts are 28, 40 and 48 tokens long (116 in total). The encoder cache cuts encoder runs from 3 to 1 but prefills the same 116 tokens. The prefix cache prefills only 28, then 12, then 8 tokens. The logits match to \(2.4 \times 10^{-7}\), so caching changes the cost, not the answer. And the other image reuses only the two system-prompt blocks (8 tokens) and prefills its 24 remaining tokens: the image hash in the block hash keeps the users apart.
Where this is used
| Application | What dominates | What helps most |
|---|---|---|
| Multi-turn chat about a photo or chart | Re-prefilling the image every turn | Prefix cache (image before the question); stable chat template |
| Document AI: many questions per page (invoices, forms, DocVQA) | High-resolution pages give thousands of tokens; same page asked many times | Prefix cache with the page first; batch the questions as one prompt if possible |
| Product or catalog images seen by many users | Same image, different questions | Encoder cache keyed by content hash, shared across requests; prefix cache |
| Video QA, action recognition, key-moment detection | Many frames: encoder and prefill grow with the frame count; video decode on the CPU | GPU video decoding, frame sampling, token reduction; cache per-frame embeddings when the same clip is asked about again |
| Video retrieval, Video-RAG, fingerprinting | Encoding a large library once | Run the encoder offline in big batches; store embeddings; at query time only retrieve and prefill the chosen clips |
| Screen and GUI agents (a screenshot every step) | A new large image every step; history grows | Prefix cache for the history; drop or downscale old screenshots |
| On-device VLMs (phones, laptops) | Encoder and preprocessing on a weak CPU or NPU; memory | Small encoders, fixed low resolution, quantization (session 197) |
Common mistakes
- Profiling only the language model. TTFT includes upload, decode, resize, encoder and queue time. Put a timer around every stage and synchronize the GPU before reading it.
- Uploading full-size photos to a fixed-resolution model. A 12 MP photo for a 336-pixel encoder wastes network and CPU time. Resize on the client, or decode at reduced size.
- Hashing the URL or file name instead of the content. The same image under two names misses the cache, and a changed image under the same URL hits a stale entry. Hash the bytes.
- Prefix cache without image hashes. Placeholder tokens are identical for all images. Without the image hash in the block hash, one user can get answers about another user’s image: a privacy bug, not only a quality bug.
- Putting the question before the image when many questions share one image. The prefix diverges before the image, and the image tokens are prefilled again for every question.
- Mismatched preprocessing. A different resize filter, mean and standard deviation, or channel order than in training lowers accuracy with no error. Compare the server’s pixel tensors with the reference processor.
- Forgetting that image KV limits the batch. At 0.5 MiB per token, one LLaVA image is 288 MiB. Size the KV budget with the image tokens, not only the text tokens.
- Ignoring the encoder for native-resolution models. For Qwen2.5-VL the encoder is about 30% of prefill FLOPs, and a full-attention encoder grows with the square of the patch count. Cap the maximum pixels for your task.
Interview questions
Walk through what happens when a VLM server gets a request with one image.basic
How many tokens and how much KV cache does one image take in LLaVA-1.5-7B?core
Our VLM has a high TTFT. Where do you look, in order?core
Why is the vision encoder a small share of compute in LLaVA-1.5 but a large one in Qwen2.5-VL?core
How does prefix caching work when the prompt contains images?core
What is the difference between an encoder cache and a prefix cache? When does each help?core
Is VLM prefill compute-bound or memory-bound? And decode?basic
Estimate TTFT for LLaVA-1.5-7B with one image and a 50-token question on an H100.advanced
How does chunked prefill interact with the vision encoder?advanced
A user sends a 4000 × 3000 photo to Qwen2.5-VL-7B. What happens to cost, and what would you change?advanced
Why can prefix caching with images be a security issue?advanced
Exercises
- Qwen2.5-VL gets a 1280 × 720 image. What size does the processor resize it to, and how many LLM tokens result?
- A prompt has eight 1920 × 1080 images for Qwen2.5-VL-7B. How many image tokens, and how much KV cache in BF16?
- Without caches, the three-turn chat of Figure 4 prefills 2,208 tokens. How many with the prefix cache, and why is it not just 646?
- A client sends a 3 MB JPEG as base64 inside JSON. How many bytes go over the network for the image?
- For LLaVA-1.5-7B at batch size 32, with each request holding 626 tokens, how many bytes does one decode step read for KV, and how does that compare with the weights?
- Using formula (196.4), how large would the encoder of a model with \(s = 4\) and a 7B language model have to be for the encoder to cost as much as the image-token prefill?
Answerscheck
Key takeaways
- A VLM request has ten stages; decode and resize run on the CPU (196.1) and can cost more than the GPU prefill for small encoders and big photos.
- Patches and tokens follow the processor rule (196.2); encoder FLOPs follow (196.3), and the ratio \(s P_{\text{enc}} / P_{\text{LLM}}\) (196.4) says if the encoder matters: about 5% for LLaVA-1.5, about 30–40% for Qwen2.5-VL.
- Image tokens make the prefill long and compute-bound (196.5, 196.6), and they fill the KV cache (196.7): 288 MiB per LLaVA image, 147 MiB for a 1080p Qwen2.5-VL image. They also make each decode step read more (196.8).
- The encoder cache skips the encoder for a repeated image; the prefix cache skips the prefill, with the image hash inside every block hash (196.9, 196.10). Put the image before the question.
- TTFT is dominated by image work; end-to-end latency with long answers by decode (196.11). Measure every stage separately.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
- Singh et al. (2025). Efficiently Serving Large Multimodal Models Using EPD Disaggregation
- vLLM team (2025). vLLM design docs: Automatic Prefix Caching (multimodal block hashes)
- PyTorch team (2024). torchvision.io.decode_jpeg (CPU and CUDA/nvJPEG decoding)
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models