LLM & VLM courseBlog
196Track 7: Vision and VLMs · session 34 of 38

The VLM inference path

A VLM request does more work than a text request before the first word comes out: the image is decoded and resized on the CPU, run through a vision encoder, and turned into hundreds or thousands of extra prompt tokens that the language model must prefill and keep in its KV cache. This session follows one request through every stage, counts the cost of each with real model configs, and shows how encoder caching and prefix caching remove repeated work. It ends with a breakdown of where the time goes.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on the generation loop (78), the KV cache (80), memory management and prefix caching (134), anatomy of a VLM (177) and visual token efficiency (183)

What you will learn

  • The ten stages of a VLM request, from uploaded bytes to streamed text, and which ones run on the CPU and which on the GPU
  • Why image decode and resize can cost more than the GPU work for a small VLM, and how to move or shrink it (GPU decode, JPEG draft mode, client-side resize, overlap)
  • How to count vision-encoder FLOPs, why the encoder is about 5% of prefill in LLaVA-1.5 but about 30% in Qwen2.5-VL, and what window attention saves
  • Why image tokens make prefill long and compute-bound, and how to estimate time to first token from FLOPs
  • How much KV cache an image takes (288 MiB for LLaVA-1.5-7B, 147 MiB for a 1080p image in Qwen2.5-VL-7B) and what it does to decode
  • Encoder caching and multimodal prefix caching: hashing image bytes into block hashes, why the image should come before the question, and what each cache saves
  • A stage-by-stage time breakdown, and why TTFT and end-to-end latency have different bottlenecks

Why this matters

Send a text question to a language model and the server does two things: prefill the prompt, then decode the answer token by token. Send the same question with a photo and the server does much more before the first word appears. It decodes the JPEG, resizes it, runs a vision encoder, maps the result into the language model’s space, and then prefills a prompt that is now hundreds or thousands of tokens longer.

Each of these extra steps has a cost, and the costs land in different places. Some run on the CPU, some on the GPU. Some add to the time to first token (TTFT, the delay before the first output token), some fill GPU memory, and some slow every decode step. A team that profiles only the language model will miss half of the picture. For example, for a 7B LLaVA-style model, the GPU part of the prefill takes tens of milliseconds, while decoding and resizing a 12-megapixel phone photo on one CPU core can take longer than that. For a model with a bigger encoder and native resolution, such as Qwen2.5-VL, the vision encoder alone is about 30% of the prefill FLOPs.

In session 177 we built the model: encoder, connector, language model. In session 183 we counted visual tokens and learned to remove some. This session looks at the serving path of a request through that model, the way an inference engineer sees it. We follow one request stage by stage, count the cost of each stage using real model configs (LLaVA-1.5-7B and Qwen2.5-VL-7B), add the two caches that remove repeated work, and finish with a breakdown of where the time goes. The next session, session 197, then makes the remaining work cheaper.

The path of one request

Here is the full path, from the bytes the user uploads to the text that comes back. We give each stage a name, because a profile is only useful if you can say which stage a number belongs to.

  1. Receive. The image arrives as a file upload, a URL to fetch, or a base64 string inside JSON. Base64 is a way to write binary data as text; it makes the data \(4/3\) times larger.
  2. Decode. The JPEG or PNG is decompressed into a grid of pixels: \(H \times W \times 3\) bytes (one byte per color channel).
  3. Resize and normalize. The pixels are resized to what the encoder expects (a fixed 336 × 336 for LLaVA-1.5, a multiple of 28 pixels for Qwen2.5-VL), converted to floats, and normalized with a per-channel mean and standard deviation.
  4. Patchify. The image is cut into square patches (14 × 14 pixels for both models), and each patch is flattened into a vector.
  5. Vision encoder. A vision transformer (ViT) turns the patch vectors into feature vectors.
  6. Connector. A small network maps the features into the language model’s embedding space. It may also merge neighbors to reduce the count (Qwen2.5-VL merges 2 × 2 patches into one token).
  7. Merge into the prompt. The prompt has placeholder tokens where the image goes. Their embeddings are replaced by the image embeddings.
  8. Prefill. The language model processes the whole prompt (text and image tokens) in one forward pass and writes keys and values for every token into the KV cache.
  9. Decode. The model produces one token per step, each step reading all weights and the whole KV cache.
  10. Detokenize and stream. Token ids are turned back into text and sent to the user.

Stages 1 to 3 usually run on the CPU, in the server’s front-end process. Stages 4 to 9 run on the GPU (some processors, such as Qwen2.5-VL’s Hugging Face processor, already cut the patches on the CPU). Stage 10 is on the CPU again. Step through Figure 1 to see the tensor at every stage, for both models and three image sizes.

runs on the CPUruns on the GPU
Figure 1. One request through a VLM, stage by stage. The readout gives the tensor shape, its size in memory, and the work done at each stage, computed from the models’ published configs (a prompt with 50 text tokens).
How to read Figure 1

Pick a model and an image size, then press Next. Watch three things. First, the pixel tensor after decode: a 4000 × 3000 photo is 36 MB of raw pixels, which the CPU must produce and then shrink. Second, the token count at the connector: LLaVA-1.5 always gives 576 tokens because it resizes every image to 336 × 336, while Qwen2.5-VL’s count grows with the image (2,691 tokens for 1920 × 1080). Third, the KV cache written at prefill: LLaVA-1.5 uses 0.5 MiB per token, Qwen2.5-VL only 56 KiB, because Qwen uses grouped-query attention with 4 key-value heads.

The general algorithm, as a serving engine runs it:

Algorithm 1: serve one VLM request
  1. input: request with text and image bytes \(r_1, \dots, r_k\)
  2. for each image \(r_i\) // CPU, front-end process
  3.   \(\text{key}_i \leftarrow \text{hash}(r_i)\) // used by both caches
  4.   \(x_i \leftarrow \text{normalize}(\text{resize}(\text{decode}(r_i)))\) // skip if the processed pixels are cached
  5. tokens \(\leftarrow\) chat template with \(n_i\) placeholder tokens per image
  6. \(h \leftarrow\) longest cached prefix of tokens (with image keys) // prefix cache, Algorithm 2
  7. for each image whose tokens are not all inside the first \(h\) // GPU
  8.   \(E_i \leftarrow\) encoder cache[\(\text{key}_i\)] if present, else \(\text{connector}(\text{ViT}(\text{patchify}(x_i)))\)
  9. embeddings \(\leftarrow\) text embeddings with placeholders replaced by \(E_i\)
  10. KV, logits \(\leftarrow\) LLM prefill on positions \(h, \dots, n-1\) using the cached KV for \(0, \dots, h-1\)
  11. repeat decode one token, append its KV, stream it until end token or length limit

Stage 1 to 3: preprocessing on the CPU or the GPU

The first cost is easy to forget because it is not on the GPU. Decoding a JPEG is real work: the CPU must undo the entropy coding, invert the discrete cosine transform for every 8 × 8 block, and convert colors. The output size is simple to compute:

\[ B_{\text{pix}} = H \cdot W \cdot 3 \ \text{bytes}, \qquad B_{\text{base64}} = \tfrac{4}{3}\, B_{\text{file}} \tag{196.1} \]
Read the symbols
\(H, W\)
height and width of the decoded image in pixels
\(B_{\text{pix}}\)
bytes of raw pixels after decode (8-bit RGB)
\(B_{\text{file}}, B_{\text{base64}}\)
bytes of the compressed file, and of the same file written as base64 text

A 12-megapixel phone photo (4000 × 3000) becomes \(4000 \times 3000 \times 3 = 36\) MB of pixels. LLaVA-1.5 then throws almost all of it away: \(336 \times 336 \times 3\) is only 0.34 MB. The resize itself is also work, because a good resize filter (bilinear or bicubic with antialiasing) reads every input pixel. In our own run on one laptop CPU core with Pillow, decoding plus resizing a synthetic 12-megapixel JPEG took about 0.1 to 0.25 seconds, depending on the image content. That is one machine and one image; measure your own with the snippet below. The point is the order of magnitude: it is in the same range as, or larger than, the whole GPU prefill of a 7B LLaVA model (about 18 ms in our estimate below).

Why does this matter in a server? Because the front-end process is often a single Python process. If it decodes and resizes images one at a time, the GPU waits. Four fixes are used in practice:

  • Resize on the client. If the model will see 336 × 336 anyway, there is no reason to upload 12 MP. A phone app can resize before sending. This also cuts network time.
  • Decode at reduced size. JPEG can be decoded directly at \(1/2\), \(1/4\) or \(1/8\) of the size, by skipping high-frequency DCT coefficients. In Pillow this is Image.draft(). It is cheaper than a full decode followed by a resize.
  • Decode on the GPU. NVIDIA’s nvJPEG library decodes on the GPU; torchvision.io.decode_jpeg(..., device="cuda") and NVIDIA DALI use it. Then resize on the GPU too. The CPU only moves the compressed bytes. The cost is some GPU time and memory, taken from the model.
  • Overlap. Run preprocessing in a pool of worker processes, so the CPU prepares the next requests while the GPU runs the current batch. Most serving engines do this in a separate front-end process and also cache the processed result (vLLM keeps a processor cache for this [Likely]).
Serial: one process does everything CPUGPU prep A run A prep B run B prep C run C GPU idle 56% Overlapped: worker pool prepares ahead CPUGPU prep A prep B prep C run A run B run C done at 400 instead of 540
Figure 2. A schematic timeline (not measured). With one process, the GPU waits while each image is prepared: here preparation takes 90 units and the GPU work 70, so the GPU is busy only 210 of 480 units after the start. With a worker pool, preparation of the next image overlaps the GPU work of the current one. With more CPU workers, the CPU line can be split across cores until the GPU is the bottleneck.

The snippet below measures decode and resize on your machine. Run it on your real images (photos, scans, screenshots), because the cost depends on the content and on the JPEG quality.

import io, time
from PIL import Image
src = Image.merge("RGB", [Image.effect_mandelbrot((4000, 3000), (-2, -1.2, 1, 1.2), q)
                         for q in (40, 80, 160)])                  # a 12-megapixel test image
buf = io.BytesIO(); src.save(buf, "JPEG", quality=90); raw = buf.getvalue()
def once():
    t0 = time.perf_counter(); im = Image.open(io.BytesIO(raw)); im.load()     # decode
    t1 = time.perf_counter(); im.resize((336, 336), Image.BICUBIC)            # resize
    t2 = time.perf_counter(); return t1 - t0, t2 - t1
once()                                                               # warm up
d, r = min(once() for _ in range(5))
print(f"file {len(raw)/1e6:.1f} MB, decode {1e3*d:.0f} ms, resize {1e3*r:.0f} ms, "
      f"{12/(d + r):.0f} megapixels per second on one core")
im = Image.open(io.BytesIO(raw)); im.draft("RGB", (1000, 750))       # JPEG DCT scaling: decode at 1/4 size
t0 = time.perf_counter(); im.load(); print("draft decode", im.size, f"{1e3*(time.perf_counter()-t0):.0f} ms")

Stage 4 to 6: the vision encoder and connector

The vision encoder is a ViT. Its cost follows the same rule as any transformer: about 2 FLOPs per parameter per token for the matrix multiplications, plus an attention term that grows with the square of the sequence length. The difference from the language model is what a “token” is: here it is a patch.

How many patches, how many tokens

LLaVA-1.5 uses CLIP ViT-L/14 at 336 pixels: \(336/14 = 24\), so \(24 \times 24 = 576\) patches, plus one class token. Every image gives 576 language-model tokens, whatever its original size. Qwen2.5-VL keeps the original aspect ratio. Its processor rounds the height and width to multiples of 28 (and shrinks very large images to at most 12,845,056 pixels), cuts 14 × 14 patches, and the connector merges each 2 × 2 group of patches into one token [Certain, from the model’s processor config]:

\[ N_{\text{patch}} = \frac{H'}{14} \cdot \frac{W'}{14}, \qquad N_{\text{tok}} = \frac{N_{\text{patch}}}{4} = \frac{H' \cdot W'}{28^2} \tag{196.2} \]
Read the symbols
\(H', W'\)
the resized height and width, each a multiple of 28 (for example 1080 × 1920 becomes 1092 × 1932)
\(N_{\text{patch}}\)
sequence length inside the vision encoder
\(N_{\text{tok}}\)
number of image tokens the language model sees

For a 1920 × 1080 image: \(N_{\text{patch}} = 78 \times 138 = 10{,}764\) and \(N_{\text{tok}} = 2{,}691\). Note the factor 4: the Qwen encoder works on a sequence 4 times longer than the one the language model sees.

Encoder FLOPs

For a ViT with \(L_v\) layers, width \(d_v\) and \(P_{\text{enc}}\) parameters, the forward cost is

\[ F_{\text{enc}} \approx 2\, P_{\text{enc}}\, N_{\text{patch}} \;+\; 4\, d_v\, N_{\text{patch}} \Big( L_{\text{full}}\, N_{\text{patch}} + L_{\text{win}}\, w \Big) \tag{196.3} \]
Read the symbols
\(2 P_{\text{enc}} N_{\text{patch}}\)
the matrix multiplications: 2 FLOPs (a multiply and an add) per weight per patch
\(4 d_v N \cdot N\)
attention: \(QK^\top\) and the weighted sum of values, each \(2 d_v N^2\) FLOPs per layer when every patch sees every patch
\(L_{\text{full}}, L_{\text{win}}\)
number of layers with full attention, and with window attention
\(w\)
patches per window; Qwen2.5-VL uses 112 × 112 pixel windows, so \(w = 8 \times 8 = 64\)

LLaVA-1.5’s encoder has 24 full-attention layers, width 1024, and about 0.30 B parameters in its layers; for 577 patches that is about 0.38 TFLOPs. Qwen2.5-VL’s encoder has 32 layers, width 1280 and about 0.68 B parameters; only 4 of its layers (numbers 7, 15, 23 and 31) use full attention, the other 28 use windows [Certain, from config.json]. Figure 3 shows what that choice saves.

A quick rule tells you how big the encoder is relative to the language model. Ignoring attention, the ratio of encoder FLOPs to LLM prefill FLOPs for the image tokens is

\[ \rho \approx \frac{s \cdot P_{\text{enc}}}{P_{\text{LLM}}} \tag{196.4} \]
Read the symbols
\(s\)
encoder patches per language-model token: 1 for LLaVA-1.5, 4 for Qwen2.5-VL (2 × 2 merge)
\(P_{\text{enc}}, P_{\text{LLM}}\)
parameters of the encoder and of the language model (without the embedding table)

LLaVA-1.5-7B: \(\rho = 1 \times 0.302 / 6.48 \approx 0.047\). The encoder is a rounding error next to the 7B language model. Qwen2.5-VL-7B: \(\rho = 4 \times 0.676 / 6.53 \approx 0.41\). The encoder is now a big part of the cost, and it grows faster than linearly with image size because of the full-attention layers.

encoder, window attention (Qwen2.5-VL)same encoder, full attention in all 32 layersLLM prefill (image tokens + 50 text tokens)
Figure 3. Forward FLOPs of the Qwen2.5-VL-7B vision encoder (formula 196.3) and of the language-model prefill (formula 196.5), computed from the published config, as the image grows. The image is resized with the model’s own rule (multiples of 28, at most 12.8 MP).
How to read Figure 3

At small sizes all three curves grow roughly linearly with the number of pixels, and the window encoder is about 25 to 30% of the total. As the image grows, the full-attention curve bends upward fast: at 4000 × 3000 it would cost more than the language model itself. Window attention keeps most layers linear; only the 4 full layers still grow with the square. The readout converts FLOPs to milliseconds assuming an H100 at 50% of its dense BF16 peak, which is an assumption, not a measurement.

Worked example: one 1920 × 1080 image in Qwen2.5-VL-7B

Resize: \(1080 \to 1092\) and \(1920 \to 1932\) (nearest multiples of 28). Patches: \(78 \times 138 = 10{,}764\). Tokens: \(10{,}764 / 4 = 2{,}691\).

Encoder (formula 196.3, with window attention): about 16.3 TFLOPs, of which attention is about 15%. With full attention in all 32 layers it would be about 32.8 TFLOPs, and attention would be 58% of it.

Language-model prefill of 2,691 + 50 tokens (formula 196.5 below): about 38.8 TFLOPs. So the encoder is 30% of the GPU work before the first token.

Time, if we assume an H100 SXM (989 TFLOP/s dense BF16 [Certain, NVIDIA spec]) running at 50% of peak: \((16.3 + 38.8) / 494.5 \approx 0.111\) s, about 111 ms. Of that, 33 ms is the encoder.

Stage 8: the long prefill of image tokens

Now the image tokens enter the language model. For the language model they are just prompt tokens: each one goes through every layer, and each one gets keys and values in the KV cache. The prefill cost for a prompt of \(n\) tokens is the familiar formula from session 78:

\[ F_{\text{pre}}(n) \approx 2\, P_{\text{LLM}}\, n + 4\, L\, d\, n^2, \qquad n = N_{\text{img}} + N_{\text{text}} \tag{196.5} \]
Read the symbols
\(P_{\text{LLM}}\)
parameters of the language model used per token (all layers, without the embedding table)
\(L, d\)
number of layers and model width of the language model
\(N_{\text{img}}, N_{\text{text}}\)
image tokens (sum over all images) and text tokens in the prompt

We count the full \(n \times n\) attention; a causal kernel skips about half of it, so this is a slight overestimate. The key fact: with an image, \(n\) is dominated by \(N_{\text{img}}\). A 50-token question to LLaVA-1.5-7B costs about 0.65 TFLOPs of prefill. Add one image and the prompt is 626 tokens and about 8.3 TFLOPs, 12.8 times more. With Qwen2.5-VL and a high-resolution photo, the image can be 15,301 tokens (a 4000 × 3000 photo after resizing to 4004 × 2996).

Prefill with many tokens is compute-bound: the GPU does many FLOPs per byte of weights it reads, so its speed is set by its arithmetic throughput, not its memory bandwidth (see session 82). So a FLOP count gives a usable estimate of TTFT:

\[ \text{TTFT} \approx t_{\text{prep}} + \frac{F_{\text{enc}} + F_{\text{pre}}(n)}{u \cdot \Pi_{\text{peak}}} + t_{\text{queue}} \tag{196.6} \]
Read the symbols
\(t_{\text{prep}}\)
CPU time for receive, decode and resize (stages 1 to 3), if it is not overlapped
\(u\)
the fraction of peak the kernels really reach (model FLOPs utilization); 0.4 to 0.6 is a common range for large prefills [Likely], but you must measure it
\(\Pi_{\text{peak}}\)
the GPU’s peak FLOP rate, for example 989 TFLOP/s dense BF16 on an H100 SXM
\(t_{\text{queue}}\)
time waiting behind other requests in the scheduler
Worked example: LLaVA-1.5-7B, one image, 50 text tokens

Encoder plus projector: \(0.381 + 0.024 = 0.405\) TFLOPs. Prefill of 626 tokens with \(P_{\text{LLM}} = 6.48\) B, \(L = 32\), \(d = 4096\): \(F_{\text{pre}} \approx 8.31\) TFLOPs. Total GPU work: 8.72 TFLOPs.

At 50% of an H100’s peak: \(8.72 / 494.5 \approx 0.0176\) s, about 18 ms. The encoder’s share is under 1 ms. Without the image, the 50-token prefill would take about 1.3 ms.

Compare with the CPU: if decode and resize run at 100 megapixels per second on one core (in the range of our laptop run), a 12 MP photo costs 120 ms of preprocessing, almost 7 times the GPU work. For this model, the image upload and the CPU are the first places to look when TTFT is high.

Long image prefills also hurt other users. A 15,000-token prefill keeps the GPU busy for a long time, and the decode steps of other requests in the same batch wait for it. Chunked prefill (from session 135) splits a long prompt into pieces and mixes them with decode steps. For a VLM there is one extra rule: the encoder runs on a whole image at once, so the engine must run the encoder for an image before the first chunk that contains its tokens, and keep the output until the last such chunk has been processed. vLLM schedules this with an encoder compute budget and keeps the outputs in an encoder cache on the GPU [Likely]. Some systems go further and put the encoder on separate GPUs, next to separate prefill and decode GPUs (EPD disaggregation; session 137).

Image tokens in the KV cache

Every prompt token, image or text, leaves keys and values in every layer. The bytes per token are

\[ b_{\text{KV}} = 2 \cdot L \cdot n_{\text{kv}} \cdot d_h \cdot b, \qquad M_{\text{img}} = N_{\text{tok}} \cdot b_{\text{KV}} \tag{196.7} \]
Read the symbols
\(2\)
one key and one value vector
\(n_{\text{kv}}, d_h\)
number of key-value heads and the size of each head
\(b\)
bytes per number: 2 for FP16 or BF16
\(M_{\text{img}}\)
KV-cache memory taken by one image

LLaVA-1.5-7B (32 layers, 32 KV heads of size 128): \(b_{\text{KV}} = 2 \cdot 32 \cdot 32 \cdot 128 \cdot 2 = 524{,}288\) bytes, exactly 0.5 MiB per token, so 288 MiB per image. Qwen2.5-VL-7B (28 layers, 4 KV heads of size 128, grouped-query attention): 57,344 bytes, 56 KiB per token. A 1920 × 1080 image (2,691 tokens) takes 147 MiB, about half of LLaVA’s image, although it has 4.7 times more tokens. Eight such images in one prompt take 1.15 GiB. This is the memory that limits how many requests fit in a batch.

The KV cache also slows decode. Each decode step reads all the weights once and the KV cache of every request in the batch. Decode is memory-bound, so its time is bytes divided by bandwidth:

\[ t_{\text{step}} \approx \frac{M_W + b_{\text{KV}} \sum_{j=1}^{B} n_j}{\eta \cdot \text{BW}} \tag{196.8} \]
Read the symbols
\(M_W\)
bytes of the language-model weights read per step (13.2 GB for LLaVA-1.5-7B in BF16)
\(B, n_j\)
batch size, and the current context length of request \(j\) (its image and text tokens so far)
\(\eta \cdot \text{BW}\)
achieved memory bandwidth: a fraction \(\eta\) of the peak (3.35 TB/s on an H100 SXM [Certain, NVIDIA spec])

At batch size 1, one LLaVA image adds 313 MiB (\(626 \times 0.5\) MiB) to the 13.2 GB of weights per step: about 2.4% slower decode. At batch size 32, the KV caches add 10.5 GB per step, almost as much as the weights. So image tokens hurt decode mainly at large batch sizes, and much less for GQA models like Qwen2.5-VL. The visual-token reduction methods from session 183 help here too, and so does KV-cache quantization.

Caching: do not do the same work twice

Real traffic repeats images. A user asks three questions about the same chart in a chat: the client sends the full history, image included, every turn. A shop has one product photo that thousands of users ask about. A document pipeline asks ten extraction questions about each page. Without caching, every one of these requests decodes the image, runs the encoder, and prefills all the image tokens again. Two caches remove this.

Encoder cache

The encoder cache maps a hash of the image to the encoder-plus-connector output \(E\) (the image embeddings, \(N_{\text{tok}} \times d\) numbers). The key must be the image content, for example SHA-256 of the file bytes or of the decoded pixels, not the URL or the file name: two users can upload the same file under different names, and one URL can change its content. A hit skips decode, resize and the encoder. It does not skip the language-model prefill.

Prefix cache with images

The prefix cache (session 134) keeps the KV blocks of earlier prompts. A new request that starts with the same tokens reuses them and prefills only the rest. The KV cache is stored in blocks of \(B\) tokens (16 by default in vLLM), and each full block gets a hash that depends on everything before it:

\[ h_j = \text{hash}\big(h_{j-1},\ t_{jB}, \dots, t_{jB+B-1},\ \text{extra}_j\big) \tag{196.9} \]
Read the symbols
\(h_j\)
the hash of block \(j\); because it includes \(h_{j-1}\), equal hashes mean the whole prefix up to that block is equal
\(t_i\)
the token ids in the block
\(\text{extra}_j\)
extra keys: here, the content hash of every image whose placeholder tokens appear in the block

The extra key is essential for VLMs. The placeholder token ids are the same for every image (LLaVA uses token 32000 repeated 576 times). Without the image hash, two different photos would produce the same block hashes, and the second user would get answers about the first user’s photo. vLLM puts the image hash into the hash of every block that holds that image’s tokens [Certain, vLLM design docs]. SGLang’s radix tree uses image hashes in the same role [Likely].

The number of tokens a request must prefill is then

\[ n_{\text{new}} = n - h, \qquad h = B \left\lfloor \frac{\min(m,\ n - 1)}{B} \right\rfloor \tag{196.10} \]
Read the symbols
\(m\)
the number of leading tokens that match a cached prompt (same ids and same images)
\(h\)
the cached prefix length: only full blocks count, and at least the last token is always computed (it produces the first output logits)
Algorithm 2: prefix lookup with image hashes, plus the encoder cache
  1. input: token ids \(t_0, \dots, t_{n-1}\), image keys with their token positions, block size \(B\)
  2. \(h_{-1} \leftarrow\) empty; \(\; h \leftarrow 0\)
  3. for \(j = 0, 1, \dots\) while block \(j\) is full
  4.   extra \(\leftarrow\) keys of images with any token in block \(j\)
  5.   \(h_j \leftarrow \text{hash}(h_{j-1}, t_{jB..jB+B-1}, \text{extra})\) // formula (196.9)
  6.   if \(h_j\) is in the block table then \(h \leftarrow (j+1)B\) else break
  7. \(h \leftarrow \min(h, B\lfloor (n-1)/B \rfloor)\) // always compute the last token
  8. for each image with a token at position \(\ge h\)
  9.   \(E \leftarrow\) encoder cache[key] if present, else run the encoder and store it
  10. prefill positions \(h..n-1\); store the hashes of the new full blocks

Order matters. The prefix cache only helps if the shared part comes first. Put the image before the question (system prompt, image, question) and a second question about the same image hits the cache for the system prompt and all image tokens. Put the question first (system prompt, question, image) and the prefix differs right after the system prompt: the image tokens must be prefilled again, and only the encoder cache can help. Figure 4 lets you try both.

from prefix cacheprefilled againimage tokens (dark band)
Figure 4. Five requests to LLaVA-1.5-7B: a three-turn chat about one photo, then a new user with the same photo and a new question, then a new user with a different photo. System prompt 40 tokens, image 576, each question 30, each answer 60, block size 16. FLOPs from formulas (196.3) and (196.5).
How to read Figure 4

Each row is one request’s prompt, drawn to scale. Green parts come from the prefix cache, orange parts are prefilled. The dark band marks the 576 image tokens. Step through the five requests with both caches on, then turn the prefix cache off: every request prefills everything. Turn only the encoder cache on: the encoder runs twice instead of five times, but the saving is small for LLaVA, because its encoder is only about 5% of the work. Then switch to “Question before image” and look at request 4: the image tokens are recomputed. Note also that only 32 of the 40 system-prompt tokens are cached in request 5: only full blocks of 16 count.

Worked example: the three-turn chat of Figure 4

Prompt lengths: turn 1 is \(40 + 576 + 30 = 646\) tokens, turn 2 adds the first answer and a new question (736), turn 3 adds another 90 (826). Without caches the engine prefills \(646 + 736 + 826 = 2{,}208\) tokens and runs the encoder 3 times.

With the prefix cache, turn 2 finds the previous prompt and answer (706 tokens) in cache; full blocks give \(h = 16 \lfloor 706/16 \rfloor = 704\), so it prefills only 32 tokens. Turn 3: \(h = 16 \lfloor 796/16 \rfloor = 784\), so 42 tokens. Total: \(646 + 32 + 42 = 720\) tokens, and the encoder runs once (later turns have all image tokens inside the cached prefix).

In FLOPs, the three turns cost 30.7 TFLOPs without caches and 10.0 TFLOPs with the prefix cache. The encoder cache alone saves only \(2 \times 0.405 = 0.81\) TFLOPs here. For Qwen2.5-VL with a high-resolution image, where the encoder is 30% of the work, the encoder cache matters much more.

Where the time goes: a breakdown

We now have a formula for every stage. End-to-end latency for one request is

\[ T_{\text{e2e}} = \underbrace{t_{\text{prep}} + t_{\text{enc}} + t_{\text{pre}}}_{\text{TTFT (no queue)}} + \sum_{k=1}^{N_{\text{out}}-1} t_{\text{step}}(n + k) \tag{196.11} \]
Read the symbols
\(t_{\text{enc}}, t_{\text{pre}}\)
encoder and prefill time: FLOPs divided by \(u \cdot \Pi_{\text{peak}}\)
\(N_{\text{out}}\)
number of output tokens; the first one comes out of the prefill, the other \(N_{\text{out}} - 1\) need a decode step each
\(t_{\text{step}}(\cdot)\)
decode step time at the current context length, formula (196.8)

Figure 5 puts it all together for batch size 1 on an H100. The answer to “where does the time go?” depends on what you measure. For TTFT, the image dominates: preprocessing for small encoders and big photos, encoder plus prefill for native-resolution models. For end-to-end latency with a long answer, decode dominates, because each of hundreds of steps reads 13 to 14 GB of weights. A short answer (a yes/no question, a label, a bounding box) makes the image stages matter much more.

CPU preprocessingvision encoder + connectorLLM prefilldecode
Figure 5. An estimated time breakdown for one request at batch size 1 on an H100 SXM (989 TFLOP/s, 3.35 TB/s), using formulas (196.3), (196.5), (196.8) and (196.11) with 50 text tokens. The efficiencies and the CPU rate are assumptions you set; the model shapes come from the published configs. This is a planning model, not a benchmark.
How to read Figure 5

The top bar is TTFT, the bottom bar is the whole request, both on their own time scale. Start with LLaVA-1.5 and a 4000 × 3000 photo: preprocessing is most of TTFT, the encoder is a thin sliver. Switch to Qwen2.5-VL with the same photo: now the encoder and prefill take about 0.9 s and preprocessing is a smaller share. Set output tokens to 1: the end-to-end bar becomes almost all image work. Set it to 1000: decode takes over. Move the CPU rate to see how much a GPU decoder or client-side resizing could save.

How to get these numbers for your own system: put timers (or profiler ranges, see session 123) around each stage, synchronize the GPU before reading a timer, and report TTFT and time per output token separately, with their distributions, not only the mean (session 81). Compare the measured numbers with the estimates above. A large gap tells you where to look: a slow stage that the model says should be fast is usually a bug or a missing overlap.

Code: a tiny VLM server with both caches

The code below builds a toy VLM (a small ViT encoder with an MLP connector, and a one-layer language model with a KV cache) and serves a three-turn chat. It implements the encoder cache keyed by a SHA-256 of the uploaded bytes, and the block-hash prefix cache of formula (196.9) with the image hash as an extra key. It counts encoder runs and prefilled tokens, checks that the cached path gives the same logits as the uncached path, and checks that a different image with the same placeholder tokens does not hit the image blocks. It runs on a CPU in a few seconds.

import hashlib, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
D, P, S, BLK = 64, 8, 32, 4          # model width, patch size, encoder input side, KV block size
IMG = 1000                            # placeholder token id for one image token

def decode(raw, H=480, W=640):        # stands in for JPEG decode: file bytes -> (3, H, W) uint8
    return torch.frombuffer(bytearray(raw), dtype=torch.uint8).view(3, H, W)

def preprocess(raw):
    x = decode(raw).float().unsqueeze(0) / 255.0
    x = F.interpolate(x, size=(S, S), mode="bilinear", antialias=True, align_corners=False)
    mean = torch.tensor([0.481, 0.458, 0.408]).view(1, 3, 1, 1)
    std = torch.tensor([0.269, 0.261, 0.276]).view(1, 3, 1, 1)
    return (x - mean) / std           # (1, 3, S, S)

class Encoder(nn.Module):             # tiny ViT + MLP connector
    def __init__(s):
        super().__init__()
        s.patch = nn.Linear(3 * P * P, D)
        s.pos = nn.Parameter(torch.randn((S // P) ** 2, D) * 0.02)
        s.blocks = nn.TransformerEncoder(nn.TransformerEncoderLayer(D, 4, 4 * D, batch_first=True, dropout=0.0), 2)
        s.proj = nn.Sequential(nn.Linear(D, D), nn.GELU(), nn.Linear(D, D))
    def forward(s, x):
        p = F.unfold(x, P, stride=P).transpose(1, 2)    # (1, 16 patches, 192)
        return s.proj(s.blocks(s.patch(p) + s.pos))[0]   # (16, D): one LLM token per patch

class TinyLM(nn.Module):              # one causal attention layer + MLP, with a KV cache
    def __init__(s, V=1001):
        super().__init__()
        s.emb = nn.Embedding(V, D); s.qkv = nn.Linear(D, 3 * D); s.o = nn.Linear(D, D)
        s.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D)); s.head = nn.Linear(D, V)
    def forward(s, h, past=None):     # h: (n, D) new embeddings; past: (K, V) of earlier positions
        q, k, v = s.qkv(h).chunk(3, -1)
        if past is not None: k, v = torch.cat([past[0], k]), torch.cat([past[1], v])
        n, T = q.shape[0], k.shape[0]
        mask = torch.ones(n, T, dtype=torch.bool).tril(T - n)            # causal, offset by the cache
        a = F.scaled_dot_product_attention(q[None], k[None], v[None], attn_mask=mask[None])[0]
        h = h + s.o(a); h = h + s.mlp(h)
        return s.head(h), (k, v)

enc, lm = Encoder().eval(), TinyLM().eval()
stats = {"encoder_runs": 0, "prefill_tokens": 0}
enc_cache, prefix_cache = {}, {}      # image hash -> embeddings ; block-chain hash -> (K, V) up to that block

def img_key(raw): return hashlib.sha256(raw).hexdigest()    # hash the uploaded bytes

def embed(tokens, images, use_enc_cache):
    out, it, i = [], iter(images), 0
    while i < len(tokens):
        if tokens[i] == IMG:                                 # a run of 16 placeholders = one image
            raw = next(it); key = img_key(raw)
            if not (use_enc_cache and key in enc_cache):
                stats["encoder_runs"] += 1; enc_cache[key] = enc(preprocess(raw))
            out.extend(enc_cache[key]); i += (S // P) ** 2
        else: out.append(lm.emb.weight[tokens[i]]); i += 1
    return torch.stack(out)

def block_hashes(tokens, image_keys):  # chain hash per full block; image hash is an "extra key", formula (196.9)
    hs, parent = [], b""
    for b in range(len(tokens) // BLK):
        blk = tokens[b * BLK:(b + 1) * BLK]
        extra = "|".join(image_keys) if IMG in tokens[:(b + 1) * BLK] else ""
        parent = hashlib.sha256(parent + str(blk).encode() + extra.encode()).digest(); hs.append(parent)
    return hs

@torch.no_grad()
def prefill(tokens, images, use_enc_cache=True, use_prefix_cache=True):
    keys = [img_key(r) for r in images]
    h_all = embed(tokens, images, use_enc_cache)
    start, past = 0, None
    if use_prefix_cache:                                     # longest cached prefix of full blocks
        for b, hb in enumerate(block_hashes(tokens, keys)):
            if hb in prefix_cache and (b + 1) * BLK < len(tokens): start, past = (b + 1) * BLK, prefix_cache[hb]
    logits, kv = lm(h_all[start:], past)
    stats["prefill_tokens"] += len(tokens) - start
    for b, hb in enumerate(block_hashes(tokens, keys)):      # store every full block
        prefix_cache[hb] = (kv[0][:(b + 1) * BLK], kv[1][:(b + 1) * BLK])
    return logits[-1]

g = torch.Generator().manual_seed(1)
photo = bytes(torch.randint(0, 256, (3 * 480 * 640,), generator=g).tolist())   # "uploaded files"
other = bytes(torch.randint(0, 256, (3 * 480 * 640,), generator=g).tolist())
sys_ = [1, 2, 3, 4, 5, 6, 7, 8]                              # system prompt (8 tokens)
img = [IMG] * 16
turns = [[11, 12, 13, 14], [21, 22, 23, 24, 25, 26, 27, 28], [31, 32, 33, 34]]

for enc_c, pre_c in [(False, False), (True, False), (True, True)]:
    enc_cache.clear(); prefix_cache.clear(); stats.update(encoder_runs=0, prefill_tokens=0); hist = sys_ + img
    for q in turns:
        hist = hist + q; out = prefill(hist, [photo], enc_c, pre_c)
        hist = hist + [int(out.argmax())] * 4              # pretend the model answered with 4 tokens
    print(f"encoder cache={enc_c!s:5} prefix cache={pre_c!s:5} -> {stats}")

# correctness: cached and uncached prefill give the same next-token logits
enc_cache.clear(); prefix_cache.clear()
a = prefill(sys_ + img + turns[0], [photo]); b = prefill(sys_ + img + turns[1], [photo])
c = prefill(sys_ + img + turns[1], [photo], False, False)
print("max |cached - uncached| =", f"{(b - c).abs().max().item():.1e}")
# a different image with identical placeholder tokens must NOT hit the image blocks
stats["prefill_tokens"] = 0; prefill(sys_ + img + turns[1], [other]); print("other image prefilled:", stats["prefill_tokens"])

Output (PyTorch 2.8 on a CPU):

encoder cache=False prefix cache=False -> {'encoder_runs': 3, 'prefill_tokens': 116}
encoder cache=True  prefix cache=False -> {'encoder_runs': 1, 'prefill_tokens': 116}
encoder cache=True  prefix cache=True  -> {'encoder_runs': 1, 'prefill_tokens': 48}
max |cached - uncached| = 2.4e-07
other image prefilled: 24

Read it like Figure 4. The prompts are 28, 40 and 48 tokens long (116 in total). The encoder cache cuts encoder runs from 3 to 1 but prefills the same 116 tokens. The prefix cache prefills only 28, then 12, then 8 tokens. The logits match to \(2.4 \times 10^{-7}\), so caching changes the cost, not the answer. And the other image reuses only the two system-prompt blocks (8 tokens) and prefills its 24 remaining tokens: the image hash in the block hash keeps the users apart.

Where this is used

ApplicationWhat dominatesWhat helps most
Multi-turn chat about a photo or chartRe-prefilling the image every turnPrefix cache (image before the question); stable chat template
Document AI: many questions per page (invoices, forms, DocVQA)High-resolution pages give thousands of tokens; same page asked many timesPrefix cache with the page first; batch the questions as one prompt if possible
Product or catalog images seen by many usersSame image, different questionsEncoder cache keyed by content hash, shared across requests; prefix cache
Video QA, action recognition, key-moment detectionMany frames: encoder and prefill grow with the frame count; video decode on the CPUGPU video decoding, frame sampling, token reduction; cache per-frame embeddings when the same clip is asked about again
Video retrieval, Video-RAG, fingerprintingEncoding a large library onceRun the encoder offline in big batches; store embeddings; at query time only retrieve and prefill the chosen clips
Screen and GUI agents (a screenshot every step)A new large image every step; history growsPrefix cache for the history; drop or downscale old screenshots
On-device VLMs (phones, laptops)Encoder and preprocessing on a weak CPU or NPU; memorySmall encoders, fixed low resolution, quantization (session 197)

Common mistakes

  • Profiling only the language model. TTFT includes upload, decode, resize, encoder and queue time. Put a timer around every stage and synchronize the GPU before reading it.
  • Uploading full-size photos to a fixed-resolution model. A 12 MP photo for a 336-pixel encoder wastes network and CPU time. Resize on the client, or decode at reduced size.
  • Hashing the URL or file name instead of the content. The same image under two names misses the cache, and a changed image under the same URL hits a stale entry. Hash the bytes.
  • Prefix cache without image hashes. Placeholder tokens are identical for all images. Without the image hash in the block hash, one user can get answers about another user’s image: a privacy bug, not only a quality bug.
  • Putting the question before the image when many questions share one image. The prefix diverges before the image, and the image tokens are prefilled again for every question.
  • Mismatched preprocessing. A different resize filter, mean and standard deviation, or channel order than in training lowers accuracy with no error. Compare the server’s pixel tensors with the reference processor.
  • Forgetting that image KV limits the batch. At 0.5 MiB per token, one LLaVA image is 288 MiB. Size the KV budget with the image tokens, not only the text tokens.
  • Ignoring the encoder for native-resolution models. For Qwen2.5-VL the encoder is about 30% of prefill FLOPs, and a full-attention encoder grows with the square of the patch count. Cap the maximum pixels for your task.

Interview questions

Walk through what happens when a VLM server gets a request with one image.basic
Receive the bytes (often base64, \(4/3\) larger), decode the JPEG to pixels, resize and normalize to the encoder’s input, cut into patches, run the vision encoder, run the connector (projector, maybe a \(2 \times 2\) merge), replace the placeholder token embeddings in the prompt with the image embeddings, prefill the whole prompt to fill the KV cache, then decode token by token and stream the detokenized text. Steps 1–3 and the last one are usually on the CPU; the rest on the GPU.
How many tokens and how much KV cache does one image take in LLaVA-1.5-7B?core
The image is resized to \(336 \times 336\) and cut into \(14 \times 14\) patches: \(24 \times 24 = 576\) tokens. KV per token is \(2 \times 32\ \text{layers} \times 32\ \text{KV heads} \times 128 \times 2\ \text{bytes} = 0.5\ \text{MiB}\), so \(576 \times 0.5\ \text{MiB} = 288\ \text{MiB}\) per image in FP16. A GQA model like Qwen2.5-VL-7B (4 KV heads, 28 layers) needs only 56 KiB per token, so a 1080p image with 2,691 tokens takes about 147 MiB.
Our VLM has a high TTFT. Where do you look, in order?core
First measure each stage separately. Then: (1) queueing in the scheduler; (2) network and base64 size; (3) CPU decode and resize, especially for big photos and a single-threaded front end; (4) the encoder, which matters for large or native-resolution encoders; (5) prefill length: count image tokens, which can be thousands; (6) cache hit rates. Fixes map to stages: client-side resizing or GPU decode, overlap with workers, capping max pixels or token reduction, chunked prefill, prefix and encoder caches.
Why is the vision encoder a small share of compute in LLaVA-1.5 but a large one in Qwen2.5-VL?core
Ignoring attention, the encoder/LLM FLOP ratio is \(\rho \approx s \cdot P_{\text{enc}} / P_{\text{LLM}}\) (196.4), where \(s\) is patches per LLM token. LLaVA-1.5: \(s = 1\), \(P_{\text{enc}} \approx 0.30\) B, \(P_{\text{LLM}} \approx 6.5\) B, \(\rho \approx 0.05\). Qwen2.5-VL: the \(2 \times 2\) merge gives \(s = 4\), and \(P_{\text{enc}} \approx 0.68\) B, so \(\rho \approx 0.41\). The encoder also works on a sequence \(4\times\) longer than the LLM sees, so its attention cost grows faster with resolution; that is why Qwen2.5-VL uses window attention in 28 of 32 layers.
How does prefix caching work when the prompt contains images?core
The KV cache is split into blocks; each full block gets a hash of the parent block’s hash, its token ids, and extra keys. For images, the extra key is the content hash of every image whose placeholder tokens are in the block. This is needed because placeholder ids are identical for all images. A new request walks its block hashes and reuses the longest run that is in the table, then prefills the rest. Put shared content (system prompt, image) first to get hits.
What is the difference between an encoder cache and a prefix cache? When does each help?core
The encoder cache stores the image embeddings (encoder + connector output) by image hash; a hit skips decode, resize and the encoder, but the LLM must still prefill the image tokens. The prefix cache stores KV blocks; a hit skips the LLM prefill for the matching prefix, and if the whole image is inside the hit, the encoder is not needed either. The encoder cache helps when the same image appears in a different context (after a different question, at a different position); the prefix cache helps when the same prefix repeats (multi-turn chat, shared system prompt and image). The encoder cache also lets chunked prefill reuse one encoder run across chunks.
Is VLM prefill compute-bound or memory-bound? And decode?basic
Prefill of hundreds or thousands of image tokens is compute-bound: each weight read is used for many tokens, so \(t \approx \text{FLOPs} / (u \cdot \Pi_{\text{peak}})\). Decode is memory-bound: each step reads all weights and the KV cache for one new token per request, so \(t \approx \text{bytes} / \text{bandwidth}\). Image tokens make prefill longer and make every decode step read more KV, which matters most at large batch sizes and for models without GQA.
Estimate TTFT for LLaVA-1.5-7B with one image and a 50-token question on an H100.advanced
Prompt: \(576 + 50 = 626\) tokens. Prefill (196.5) \(\approx 2 \cdot 6.48 \times 10^{9} \cdot 626 + 4 \cdot 32 \cdot 4096 \cdot 626^2 \approx 8.31\) TFLOPs; encoder \(+\) projector \(\approx 0.41\) TFLOPs. At 50% of 989 TFLOP/s: \(\approx 18\) ms of GPU time. Add CPU preprocessing (can be 100+ ms for a 12 MP photo on one core), network and queueing. So for this model, the GPU part is often not the main part of TTFT. State the efficiency assumption and verify it by measuring.
How does chunked prefill interact with the vision encoder?advanced
Chunked prefill splits a long prompt into token chunks to interleave with decode steps. The encoder, though, processes a whole image at once (its attention spans all patches). So the scheduler must run the encoder for an image before the first chunk containing its tokens and keep the output until the last such chunk is done, which is what a GPU encoder cache does. It also needs a budget so that one huge image or many frames do not stall a step. Disaggregated designs (EPD) go further and run encoders on separate GPUs.
A user sends a 4000 × 3000 photo to Qwen2.5-VL-7B. What happens to cost, and what would you change?advanced
The processor keeps it under 12.8 MP: about \(4004 \times 2996\), 61,204 patches and \(61{,}204 / 4 = 15{,}301\) LLM tokens. Encoder \(\approx 156\) TFLOPs with window attention (full attention everywhere would be \(\approx 692\)), prefill \(\approx 295\) TFLOPs, so roughly 0.9 s of H100 time at 50% efficiency, plus 837 MiB of KV cache. For most tasks set max_pixels much lower (for example 1–2 MP), or let the model zoom in on crops only when needed. Keep high resolution only for tasks that need small text.
Why can prefix caching with images be a security issue?advanced
If the block hash covers only token ids, two different images give identical hashes, and one user’s cached KV (computed from their image) is reused for another user’s request: the model answers about the wrong image and leaks information. The image content hash must be part of the hash. Also, cache timing can reveal whether someone else sent the same image or prompt; multi-tenant systems use per-tenant cache salts for that.

Exercises

  1. Qwen2.5-VL gets a 1280 × 720 image. What size does the processor resize it to, and how many LLM tokens result?
  2. A prompt has eight 1920 × 1080 images for Qwen2.5-VL-7B. How many image tokens, and how much KV cache in BF16?
  3. Without caches, the three-turn chat of Figure 4 prefills 2,208 tokens. How many with the prefix cache, and why is it not just 646?
  4. A client sends a 3 MB JPEG as base64 inside JSON. How many bytes go over the network for the image?
  5. For LLaVA-1.5-7B at batch size 32, with each request holding 626 tokens, how many bytes does one decode step read for KV, and how does that compare with the weights?
  6. Using formula (196.4), how large would the encoder of a model with \(s = 4\) and a 7B language model have to be for the encoder to cost as much as the image-token prefill?
Answerscheck
1. Round each side to a multiple of 28: \(720/28 = 25.7 \to 26 \times 28 = 728\), and \(1280/28 = 45.7 \to 46 \times 28 = 1288\). Tokens: \(26 \times 46 = 1{,}196\). 2. \(8 \times 2{,}691 = 21{,}528\) tokens; \(\times\, 57{,}344\) bytes \(= 1.15\) GiB. 3. \(646 + 32 + 42 = 720\). Turns 2 and 3 reuse the previous prompt and answer, but only full blocks of 16: 704 of 706 and 784 of 796 tokens, so a few tokens before the new question are recomputed. 4. \(3\ \text{MB} \times 4/3 = 4\ \text{MB}\). 5. \(32 \times 626 \times 0.5\ \text{MiB} = 10.5\ \text{GB}\) (10,502,537,216 bytes), about 80% of the 13.2 GB of weights read in the same step. Per step, the bytes grow from 13.5 GB at batch size 1 to 23.7 GB, so each step is about 1.75 times slower (but it serves 32 requests). 6. \(\rho = 1\) means \(4 \cdot P_{\text{enc}} = P_{\text{LLM}}\), so \(P_{\text{enc}} \approx 6.5/4 \approx 1.6\) B parameters (ignoring attention).

Key takeaways

  • A VLM request has ten stages; decode and resize run on the CPU (196.1) and can cost more than the GPU prefill for small encoders and big photos.
  • Patches and tokens follow the processor rule (196.2); encoder FLOPs follow (196.3), and the ratio \(s P_{\text{enc}} / P_{\text{LLM}}\) (196.4) says if the encoder matters: about 5% for LLaVA-1.5, about 30–40% for Qwen2.5-VL.
  • Image tokens make the prefill long and compute-bound (196.5, 196.6), and they fill the KV cache (196.7): 288 MiB per LLaVA image, 147 MiB for a 1080p Qwen2.5-VL image. They also make each decode step read more (196.8).
  • The encoder cache skips the encoder for a repeated image; the prefix cache skips the prefill, with the image hash inside every block hash (196.9, 196.10). Put the image before the question.
  • TTFT is dominated by image work; end-to-end latency with long answers by decode (196.11). Measure every stage separately.

Key formulas

\[ N_{\text{tok}} = \frac{H' W'}{28^2}, \qquad \rho \approx \frac{s\, P_{\text{enc}}}{P_{\text{LLM}}}, \qquad \text{TTFT} \approx t_{\text{prep}} + \frac{F_{\text{enc}} + F_{\text{pre}}}{u\, \Pi_{\text{peak}}} \]
\[ b_{\text{KV}} = 2 L\, n_{\text{kv}}\, d_h\, b, \qquad h_j = \text{hash}(h_{j-1}, \text{tokens}_j, \text{image keys}_j), \qquad n_{\text{new}} = n - h \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  2. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
  5. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  6. Bai et al. (2025). Qwen2.5-VL Technical Report
  7. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  8. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  9. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  10. Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
  11. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  12. Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
  13. Singh et al. (2025). Efficiently Serving Large Multimodal Models Using EPD Disaggregation
  14. vLLM team (2025). vLLM design docs: Automatic Prefix Caching (multimodal block hashes)
  15. PyTorch team (2024). torchvision.io.decode_jpeg (CPU and CUDA/nvJPEG decoding)
  16. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models