LLM & VLM courseBlog
198Track 7: Vision and VLMs · session 36 of 38

Serving VLMs: vLLM, TensorRT-LLM and the edge

Serving a VLM to many users means planning for the largest request you accept, keeping every user's next token fast when a big image arrives, and, on phones and laptops, fitting a vision part and a language part into a few gigabytes. This session explains how vLLM and TensorRT-LLM handle images, why the vision encoder is sometimes moved to its own GPU, how llama.cpp, MLX and ONNX run VLMs on devices, and how LoRA adapters let one base model serve many tasks.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on the serving stack (133), memory management (134), batching (135), disaggregated serving (137), LoRA (57), the VLM inference path (196) and quantizing VLMs (197)

What you will learn

  • Where a serving engine handles images: the CPU front end with a processor cache, the scheduler with an encoder budget, the encoder cache and the image-aware prefix cache, in vLLM and TensorRT-LLM
  • Why limits on images, frames and pixels per prompt decide the KV pool and the number of concurrent requests, with Qwen2.5-VL-7B numbers on an H100 and an L4
  • Why chunked prefill cannot split the vision encoder, how a large image stretches every user's inter-token latency, and how E→PD disaggregation and data-parallel encoders fix it
  • How llama.cpp (model + mmproj), MLX, ONNX with transformers.js and ONNX Runtime store and run a VLM on a device
  • How to estimate device memory, decode speed and time to first token, and why the encoder dominates TTFT for small VLMs
  • How LoRA adapters are served on devices and servers, merged or unmerged, and what each choice costs

Why this matters

A vision-language model (VLM) that works in a notebook is not yet a product. A product must take many requests at once, each with its own images or video, keep every user’s answer fast, never run out of GPU memory, and sometimes run on a phone or a laptop with no GPU server at all. This is the job of a serving engine: the program that loads the model once and turns a stream of requests into batched GPU work.

For text-only models, the engine’s main problems are the KV cache and batching, which we studied in session 133, session 134 and session 135. Images add three new problems. First, one request can be tiny or huge: a 640 × 480 photo gives Qwen2.5-VL-7B 391 tokens, a 4000 × 3000 photo gives it 15,301. The engine must put limits on this, or one user can fill the GPU. Second, the vision encoder is a separate model that runs on a whole image at once, and it does not fit neatly into the token-by-token schedule of prefill and decode. Third, on edge devices, the vision part is a large fraction of the model, and the tools (ONNX, transformers.js, llama.cpp, MLX) each store and run it differently.

In session 196 we followed one request through all its stages, and in session 197 we made each stage cheaper. This session looks at many requests at once. We use two real data-center engines, vLLM and TensorRT-LLM, and four edge runtimes. All model sizes come from the published Hugging Face configs and weight files of Qwen2.5-VL (3B and 7B) and SmolVLM (256M and 2.2B). The next session, session 199, uses these tools to plan a real deployment.

Inside a multimodal serving engine

A serving engine for a VLM does the same loop as for a text model: accept requests, pick which tokens to compute this step, run one forward pass for the whole batch, stream new tokens, repeat. Each run of this loop is one step (also called an iteration). What changes is the work around the loop. We describe vLLM, because its design is documented in detail, and then say what is the same and what is different in TensorRT-LLM.

vLLM: four places where images are handled

  1. The front end (API server). It receives the OpenAI-style chat request, fetches image URLs or decodes base64, applies the chat template, and runs the model’s Hugging Face processor (resize, normalize, patchify, and inserting the right number of placeholder tokens). This runs on the CPU, in a separate process from the GPU work [Likely]. A processor cache keeps the processed result of recent images, keyed by a hash of the content, so a repeated image is not processed again. Its size is mm_processor_cache_gb, 4 GiB by default [Certain, vLLM docs].
  2. The scheduler. Each step it chooses tokens up to a token budget, max_num_batched_tokens. With chunked prefill (on by default [Certain, vLLM docs]), a long prompt is split across steps. For images there is a second budget: the encoder compute budget, measured in encoder output tokens. vLLM sets it to max_num_batched_tokens, or to the size of the largest single image if that is bigger [Certain, vLLM scheduler docs]. The last part is important: an image is never split. Its encoder runs on all of it in one step.
  3. The encoder cache on the GPU. The encoder output of an image is kept until the last prefill chunk that needs it has run. Its size follows the same rule as the encoder budget [Certain, vLLM scheduler docs].
  4. The prefix cache. As in the previous session, the hash of each KV block includes the content hash of any image whose tokens are in that block, so two different images with the same placeholder ids never share KV blocks.

The scheduling rule for one request with images, in a simplified form:

Algorithm 1: schedule one prefill chunk of a request with images (simplified from vLLM’s design)
  1. input: request with computed tokens \(c\), prompt length \(n\), images \(i\) with token ranges \([s_i, s_i + \tau_i)\); remaining token budget \(b_{\text{tok}}\) and encoder budget \(b_{\text{enc}}\)
  2. \(k \leftarrow \min(n - c,\ b_{\text{tok}})\) // tentative chunk: positions \(c, \dots, c+k-1\)
  3. for each image \(i\) whose range overlaps \([c, c+k)\)
  4.   if image \(i\) is in the encoder cache then continue // already encoded in an earlier step
  5.   if \(\tau_i \le b_{\text{enc}}\) then schedule the encoder for image \(i\); \(b_{\text{enc}} \leftarrow b_{\text{enc}} - \tau_i\)
  6.   else \(k \leftarrow s_i - c\); break // stop the chunk just before this image; it waits for a later step
  7. if \(k = 0\) then skip this request in this step
  8. \(b_{\text{tok}} \leftarrow b_{\text{tok}} - k\); run the scheduled encoders first, then the LLM on the chunk
  9. free an image’s encoder-cache entry when \(c + k \ge s_i + \tau_i\) // its last token has been prefilled

TensorRT-LLM: the same ideas, NVIDIA’s implementation

TensorRT-LLM is NVIDIA’s inference library. Its current multimodal path runs models on its PyTorch backend and serves them with trtllm-serve, which exposes the same OpenAI-compatible chat API (for example trtllm-serve Qwen/Qwen2-VL-7B-Instruct --backend pytorch) [Certain, TensorRT-LLM docs]. It hashes the raw image data and uses the hashes for KV-cache reuse, so it has the same image-aware prefix cache as vLLM, and it overlaps CPU preprocessing with GPU encoding [Certain, TensorRT-LLM docs]. Older releases took a different path: the vision encoder was compiled into its own TensorRT engine next to the LLM engine, and a runner passed the image features from one to the other [Likely]. Even in that old design you can see the main idea of this session: the encoder and the language model are two separate programs with a clean interface, a tensor of image embeddings.

For splitting the encoder onto other machines, NVIDIA uses Dynamo, a layer above the engine. Dynamo lists an “E/PD” mode for TensorRT-LLM: a separate encode worker sends embeddings to a combined prefill-and-decode worker [Certain, Dynamo docs].

ConcernvLLMTensorRT-LLM
Servervllm serve <model>, OpenAI chat APItrtllm-serve <model> --backend pytorch, OpenAI chat API
Items per prompt--limit-mm-per-prompt '{"image": 4, "video": 1}'set by the model’s input processor and request [Likely]
Image size--mm-processor-kwargs '{"max_pixels": ...}'processor arguments of the model [Likely]
Prefix cache with imagesimage hash in each block hashimage hashes plus token-chunk information
Encoder on other GPUsdisaggregated encoder: E→PD or E→P→D, embeddings sent by an EC connectorthrough NVIDIA Dynamo (E/PD)
Encoder parallelismmm_encoder_tp_mode="data": each GPU runs a full encoder copy on part of the imagesmodel dependent [Likely]
Many fine-tuned adapters--enable-lora, --max-lora-rank, per-request adapter nameLoRA support in the PyTorch backend [Likely]

Limits per request: images, frames and pixels

Why does vLLM have a flag called limit_mm_per_prompt? Because a serving engine must be ready for the worst request it accepts. At start-up, vLLM runs a profiling forward pass with dummy inputs of the largest allowed size, measures the peak memory, and gives what remains (up to the fraction gpu_memory_utilization of the GPU) to the KV cache. More items per prompt and bigger items mean a bigger worst case. The docs say it directly: allowing fewer multimodal items per prompt reduces the memory footprint, and setting a modality to 0 (for example {"video": 0}) turns it off [Certain, vLLM docs]. The limit can also carry size hints (width and height for images, number of frames for video); these hints change only the profiling, not how real inputs are processed [Certain, vLLM docs].

The token count of the worst request is a sum over modalities:

\[ T_{\max} = \sum_{m \in \{\text{image},\, \text{video}\}} c_m\, \tau_m \;+\; T_{\text{text}} \tag{198.1} \]
Read the symbols
\(c_m\)
the limit: the most items of modality \(m\) one prompt may contain
\(\tau_m\)
the most LLM tokens one item can produce, set by the processor’s size limits
\(T_{\text{text}}\)
room kept for the text of the prompt and the answer

For Qwen2.5-VL, \(\tau\) follows from the pixel limit. Each LLM token covers 28 × 28 pixels (a 2 × 2 merge of 14-pixel patches), and a video merges 2 frames in time (temporal patch size 2 [Certain, config.json]):

\[ \tau_{\text{img}} \approx \left\lfloor \frac{\text{max\_pixels}}{28^2} \right\rfloor, \qquad \tau_{\text{vid}} = \left\lceil \frac{F}{2} \right\rceil \cdot \frac{H' W'}{28^2} \tag{198.2} \]
Read the symbols
max_pixels
the processor’s upper limit on \(H' \times W'\) for one image; 12,845,056 in Qwen2.5-VL’s processor config, often lowered with mm_processor_kwargs
\(F\)
number of video frames given to the model after sampling
\(H', W'\)
the resized frame size, each a multiple of 28

Then the memory left for the KV cache, and how many worst-case requests fit in it at the same time, are

\[ M_{\text{KV}} = u\, M_{\text{GPU}} - M_W - M_{\text{act}}, \qquad N_{\text{conc}} = \left\lfloor \frac{M_{\text{KV}}}{b_{\text{KV}}\, T_{\max}} \right\rfloor \tag{198.3} \]
Read the symbols
\(u\)
gpu_memory_utilization, the share of GPU memory the engine may use (0.9 by default in vLLM [Certain])
\(M_W\)
weight memory: 16.58 GB for Qwen2.5-VL-7B in BF16 (8.29 B parameters, from its safetensors index)
\(M_{\text{act}}\)
peak activation memory of the profiling pass, plus CUDA graphs; we assume 2 GB, the real value is measured at start-up
\(b_{\text{KV}}\)
KV bytes per token: \(2 \cdot 28 \cdot 4 \cdot 128 \cdot 2 = 57{,}344\) for Qwen2.5-VL-7B (formula 196.7)

Paged attention allocates blocks only for tokens that really exist, so the engine can run more than \(N_{\text{conc}}\) small requests. But \(N_{\text{conc}}\) is the number of worst-case requests that can be in flight together, and a single request with \(b_{\text{KV}} T_{\max} \gt M_{\text{KV}}\) can never run. Figure 1 shows how much the limits matter.

weights (16.58 GB)activations and graphs (2 GB, assumed)KV cache: one box per worst-case requestnot used by the engine
Figure 1. A planning model of GPU memory for Qwen2.5-VL-7B in BF16, using formulas (198.1) to (198.3) with 512 text tokens per request. Video frames are 448 × 448 (256 tokens per pair of frames). The activation reserve is an assumption; vLLM measures the real one at start-up.
How to read Figure 1

The bar is the whole GPU, split as in the legend: weights, the activation reserve, and the KV pool, which is cut into boxes, one per worst-case request. The dashed line marks gpu_memory_utilization; the rest is not used. Start at the default: H100, 4 images at 12.8 MP. Each worst-case request needs 66,048 tokens (3.79 GB of KV), so only 14 fit. Now pick 1.0 MP: each request needs 5,632 tokens and 165 fit. Switch to the L4: the weights take most of the card, the pool is about 3 GB, and the default limits do not fit even one request. Turn on a video and move the frame slider to see how fast frames add up.

Worked example: what the defaults cost on an H100 and an L4

Qwen2.5-VL-7B, 4 images per prompt, default max_pixels = 12,845,056. Formula (198.2): \(\tau_{\text{img}} = \lfloor 12{,}845{,}056 / 784 \rfloor = 16{,}384\) tokens per image. Formula (198.1): \(T_{\max} = 4 \times 16{,}384 + 512 = 66{,}048\) tokens, and its KV cache is \(66{,}048 \times 57{,}344 = 3.79\) GB.

H100 80 GB at \(u = 0.9\): \(M_{\text{KV}} = 72 - 16.58 - 2 = 53.42\) GB, room for 931,495 tokens, so \(N_{\text{conc}} = 14\) worst-case requests. Lower max_pixels to \(1280 \times 28 \times 28 = 1{,}003{,}520\) (1,280 tokens per image): \(T_{\max} = 5{,}632\), and \(N_{\text{conc}} = 165\). The limit changed the worst-case concurrency by a factor of 11.8.

L4 24 GB: \(M_{\text{KV}} = 21.6 - 16.58 - 2 = 3.02\) GB, room for 52,589 tokens. The default worst case (66,048 tokens) does not fit at all. With 4 images at 1.0 MP it fits 9 times; with 1 image at 1.0 MP, 29 times. On a small GPU the limits are not optional. A 4-bit language model (session 197) is the other lever.

Video makes this sharper. Each frame adds tokens, and the frame count is set by the sampling rate and the clip length. 64 frames at 448 × 448 give \(32 \times 256 = 8{,}192\) tokens, already more than six 1 MP images. vLLM’s docs warn that for the Qwen2-VL family a larger max_model_len can increase the profiled video size and so reduce the KV cache [Certain, vLLM docs]. In practice, teams fix the frame count and frame size in the server config, so the worst case is known.

A configuration that applies these limits, for vLLM:

# Qwen2.5-VL-7B on one GPU: at most 4 images and 1 video per prompt,
# images capped at about 1 MP (1280 tokens each), context capped at 16k tokens
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --limit-mm-per-prompt '{"image": 4, "video": 1}' \
  --mm-processor-kwargs '{"max_pixels": 1003520}' \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --mm-processor-cache-gb 4

# a request with more images than the limit is rejected with an error, not silently cut

There are two other things a limit protects. One is fairness: without a limit, one user with 20 photos makes everyone else wait through a long prefill. The other is the context length: a request whose image tokens exceed max_model_len fails. Always check the error path: a request over the limit should get a clear error message, and the client should resize or drop images before it retries.

Separating the encoder from prefill and decode

Every step of a continuous-batching engine does decode for all running requests plus some prefill. The time of the step is the inter-token latency (ITL) that every running user sees: their next token cannot come out before the step ends. Chunked prefill exists to keep this step short. But there is a piece it cannot cut: the vision encoder runs on a whole image in one step. A step that contains the encoder of a large image is long for everybody.

We can model a step as bound by memory or by compute, whichever is slower (the roofline from session 82), plus the encoder if it is scheduled:

\[ t_{\text{step}} \approx \max\!\left( \frac{M_W + b_{\text{KV}} \sum_j n_j}{\eta\, \text{BW}},\ \frac{F_{\text{LLM}}(\text{batch})}{u\, \Pi} \right) + \frac{F_{\text{enc}}}{u\, \Pi} \tag{198.4} \]
Read the symbols
\(M_W\)
LLM weight bytes read per step: 14.14 GB for Qwen2.5-VL-7B in BF16 (layers and output head; the vision tower is not read in decode)
\(\sum_j n_j\)
the context lengths of all requests in the batch, whose KV must be read
\(F_{\text{LLM}}(\text{batch})\)
LLM FLOPs for this step: one token for each decoding request plus the prefill chunk (formula 196.5)
\(F_{\text{enc}}\)
encoder FLOPs for the images whose encoder runs in this step (formula 196.3)
\(\eta, u, \Pi, \text{BW}\)
bandwidth efficiency, compute efficiency, peak FLOP rate and peak bandwidth of the GPU

The fix is to run the encoder somewhere else. In an E→PD layout, an encoder worker (its own GPU or GPUs) runs the vision tower and sends the image embeddings to the prefill-and-decode worker. In E→P→D, prefill and decode are also split, as in session 137. vLLM implements both: the encoder instance writes into an encoder cache that the PD instance reads through an EC connector (an example connector, or one based on the Mooncake transfer engine) [Certain, vLLM docs]. The cost is a transfer of the embeddings:

\[ t_{\text{xfer}} = \frac{N_{\text{tok}} \cdot d \cdot b}{\text{BW}_{\text{link}}} \tag{198.5} \]
Read the symbols
\(N_{\text{tok}}\)
image tokens after the connector (2,691 for a 1920 × 1080 image in Qwen2.5-VL)
\(d, b\)
LLM width (3,584) and bytes per number (2 for BF16)
\(\text{BW}_{\text{link}}\)
bandwidth of the link between the two GPUs; we assume 25 GB/s, a PCIe-class or fast network link

A 1920 × 1080 image is \(2{,}691 \times 3{,}584 \times 2 = 19.3\) MB of embeddings, 0.77 ms at 25 GB/s. A 4000 × 3000 photo is 109.7 MB, 4.4 ms. This is small next to the encoder time it removes from the shared step. (For comparison, sending its KV cache from a prefill GPU to a decode GPU would be \(15{,}301 \times 57{,}344 = 877\) MB, 8 times more. Embeddings are cheap to move.)

decode onlydecode + prefill chunkencoder inside the stepencoder on the other GPU
Figure 2. A step-by-step simulation (formula 198.4) of Qwen2.5-VL-7B on an H100 (989 TFLOP/s, 3.35 TB/s, \(u = 0.5\), \(\eta = 0.8\)). Sixteen users are decoding with 3,000 tokens of context each; at step 4 a new request arrives with one image and 50 text tokens. The token budget is max_num_batched_tokens. This is a model, not a measurement.
How to read Figure 2

The top row is the shared GPU. Each box is one step, and its width is its duration, so the width is the inter-token latency the 16 users feel. Start with the 4000 × 3000 photo and budget 512 on the same GPU: one box is 330 ms wide, because the whole encoder (156 TFLOPs) runs inside it. Switch to E→PD: the encoder moves to the second row (another GPU), the shared steps stay below 21 ms, and the new request’s first token comes at almost the same time. Now raise the budget to 8192: the encoder is still gone, but each prefill chunk is now big, so ITL rises again. The budget sets the ITL of prefill; the encoder placement sets the ITL of images.

Worked example: one large photo arrives while 16 users decode

Plain decode step: \((14.14 + 16 \times 3{,}000 \times 57{,}344 / 10^9) / (0.8 \times 3.35) = 16.89 / 2.68 \approx 6.3\) ms. That is what each user sees between tokens when nothing else happens.

The photo (4000 × 3000) becomes 15,301 tokens; its encoder is 155.9 TFLOPs, which is \(155.9 / 494.5 = 315\) ms at 50% of peak. Budget 512, same GPU: the step that runs the encoder also runs the first 496-token chunk, so it takes about 330 ms. Sixteen users see a pause of a third of a second in the middle of their answer. The prefill then needs 31 chunks, and TTFT for the new request is about 868 ms.

E→PD: the PD GPU keeps decoding at 6.3 ms per step while the encoder GPU works, receives 109.7 MB of embeddings (4.4 ms), and then runs the same 31 chunks. The longest step is about 21 ms. TTFT is about 875 ms, almost unchanged: disaggregation does not make one image faster, it stops one image from slowing everyone else. Under real load it also lets the encoder of request A run while the LLM does prefill for request B, which raises throughput.

(a) One engine: E + P + D on the same GPUs encoder · prefill · decode vLLM / trtllm-serve default (b) E→PD: encoder workers feed one LLM pool encoder embeddings prefill · decode vLLM disaggregated encoder; Dynamo E/PD (c) E→P→D: three pools encoder prefill KV cache decode very large fleets (d) Edge: one process, one device mmproj / vision_encoder LLM (4-bit) LoRA llama.cpp, MLX, ONNX / transformers.js
Figure 3. Four deployment shapes. Moving right in (b) and (c), each pool can be sized and scaled for its own work: encoders are compute-bound and short, prefill is compute-bound, decode is bandwidth-bound. On the edge (d), everything shares one memory, and the vision part is a separate file or graph.

When is a separate encoder worth it? It costs extra GPUs and a network hop, so the answer depends on the traffic.

  • Worth it when images are large or many (documents, high-resolution photos, video), when the encoder is a big share of the work (Qwen2.5-VL’s encoder is about 30% of prefill FLOPs for a 1080p image, see session 196), and when the ITL target is strict.
  • Not worth it for small fixed-size encoders (LLaVA-1.5’s encoder is about 5% of prefill), mostly text traffic, or one-GPU deployments.
  • A cheaper middle step when the LLM is split over several GPUs with tensor parallelism: run the encoder data-parallel instead. With mm_encoder_tp_mode="data", vLLM gives each GPU a full copy of the (small) encoder and splits the images of a batch across the GPUs, while the language model stays tensor-parallel [Certain, vLLM docs]. A small encoder split with tensor parallelism spends much of its time on communication, so this is often faster; the cost is a copy of the encoder weights on each GPU.

The edge: four ways to run a VLM on a device

“The edge” means running the model on the user’s device: a phone, a laptop, a car computer, a camera box. There is no batch of other users, so the goal changes from throughput to fitting in memory and answering fast for one user, often on battery. There is also privacy: the photo never leaves the device. The same model is split in a similar way by every edge runtime: a vision part and a language part, in separate files or graphs.

RuntimeWhere it runsHow the VLM is storedNotes
llama.cpp (libmtmd)CPU, Apple Metal, CUDA, Vulkan; laptops, phones, JetsonLLM in one GGUF file (-m), vision encoder and projector in a second GGUF file (--mmproj)llama-server serves the OpenAI chat API, llama-mtmd-cli is for testing; convert_hf_to_gguf.py --mmproj writes the projector file; supports Gemma 3, Qwen2/2.5-VL, SmolVLM, InternVL, Pixtral and others [Certain, llama.cpp docs]
MLX (mlx-vlm)Apple silicon, unified memoryMLX safetensors, quantized with mlx_vlm.convert --quantize --q-bits 4Python; generate, an OpenAI-style server, and LoRA/QLoRA fine-tuning on the Mac [Certain, mlx-vlm README]
ONNX + transformers.jsThe browser (WebGPU or WebAssembly), Node.jsSeveral ONNX graphs: vision_encoder, embed_tokens, decoder_model_mergedA precision per graph: for example vision_encoder: "fp16" and decoder_model_merged: "q4" [Certain, transformers.js docs]
ONNX Runtime (GenAI)Windows, Android, iOS; CPU, GPU, NPUThe same split into ONNX graphsUsed for Phi vision models; has an adapter API for LoRA [Likely]

Notice the split in every row. It matches what we learned in session 197: the vision part is more sensitive, so the runtimes let you keep it in 16-bit (an F16 mmproj file, an fp16 vision_encoder) while the language model goes to 4 bits. The transformers.js docs give exactly this example for Florence-2. But on a small device this has a price. Count it.

Memory and speed on a device

The memory needed is the two files plus the KV cache:

\[ M_{\text{dev}} \approx \frac{q}{8} P_{\text{LLM}}^{\text{all}} + \frac{q_v}{8} P_{\text{vis}} + C\, b_{\text{KV}} \tag{198.6} \]
Read the symbols
\(q, q_v\)
bits per weight of the LLM file and of the vision file; GGUF Q8_0 is 8.5 bits (32 int8 weights plus one 16-bit scale), Q4_0 is 4.5 bits, F16 is 16
\(P_{\text{LLM}}^{\text{all}}\)
all LLM parameters, including the embedding table and an untied output head
\(P_{\text{vis}}\)
vision encoder plus connector parameters: 0.669 B for Qwen2.5-VL-3B, 0.094 B for SmolVLM-256M
\(C\)
context length the app allows (image tokens + text)

Decode on a device is bandwidth-bound, like decode on a GPU at batch size 1. Each token reads the LLM layers and the output head once (the embedding table only gives one row, and the vision file is not read at all):

\[ \text{tokens/s} \le \frac{\eta\, \text{BW}}{\frac{q}{8}\left(P_{\text{LLM}} + P_{\text{head}}\right) + n\, b_{\text{KV}}} \tag{198.7} \]
Read the symbols
\(P_{\text{LLM}}, P_{\text{head}}\)
parameters of the transformer layers, and of the output head (for tied embeddings, the embedding table is read as the head)
\(n\)
current context length
\(\eta\, \text{BW}\)
the achieved share of the device’s memory bandwidth

Time to first token is compute-bound. For small VLMs the encoder is often the bigger part, because each LLM token stands for \(s\) encoder patches (\(s = 4\) for Qwen2.5-VL’s 2 × 2 merge, 16 for SmolVLM-256M’s pixel shuffle with scale 4, 9 for SmolVLM-2.2B’s scale 3 [Certain, config.json]):

\[ \text{TTFT}_{\text{dev}} \approx \frac{2 P_{\text{vis}}\, s\, N_{\text{img}} + 2 P_{\text{LLM}} \left(N_{\text{img}} + N_{\text{txt}}\right)}{C_{\text{eff}}} \tag{198.8} \]
Read the symbols
\(N_{\text{img}}, N_{\text{txt}}\)
image tokens and text tokens of the prompt
\(s\)
encoder patches per LLM image token
\(C_{\text{eff}}\)
the FLOP rate the device really reaches on these kernels; you must measure it (we leave it as a slider)

The ratio of the two terms is \(s P_{\text{vis}} / P_{\text{LLM}}\) (formula 196.4). For SmolVLM-256M it is \(16 \times 93.5 / 106.2 \approx 14\): the encoder costs 14 times more than the LLM for each image token. For Qwen2.5-VL-3B it is \(4 \times 668.7 / 2{,}774.8 \approx 0.96\), and for Qwen2.5-VL-7B about 0.41. The smaller the language model, the more the vision side dominates TTFT.

LLM filevision file (mmproj)KV cache at Cusable memory line
Figure 4. An edge planner using formulas (198.6) to (198.8) with 50 text tokens, F16 KV cache and bandwidth efficiency \(\eta = 0.7\) (assumed). Parameter counts come from the models’ safetensors files. Bandwidths: M2 100 GB/s, Orin Nano Super 102 GB/s, M4 Pro 273 GB/s [Certain, vendor specs]; the phone class (8 GB, 60 GB/s) is a typical value, not one product [Likely]. Runtime buffers are not counted.
How to read Figure 4

The bar shows the memory the model needs; the line marked “usable” is what the app may use. Start with Qwen2.5-VL-3B at Q4_0 on the M2: the LLM file is 1.74 GB and the F16 vision file 1.34 GB, so the vision side is 41% of the memory although it has 18% of the parameters. Switch the mmproj to Q8_0 and it drops to 0.71 GB. Now pick SmolVLM-256M and look at the TTFT split: almost all of it is the encoder. Pick Qwen2.5-VL-7B on the phone: at F16 it does not fit at all; at Q4_0 it fits only if the app may use about 0.75 of the RAM.

Worked example: Qwen2.5-VL-3B on a MacBook Air M2

Parameters (from the safetensors file): 2,774.8 M in the layers, 311.2 M in the tied embedding table, 668.7 M in the vision tower and merger. LLM file at Q4_0: \((2{,}774.8 + 311.2) \times 10^6 \times 4.5 / 8 = 1.74\) GB. Vision file at F16: \(668.7 \times 10^6 \times 2 = 1.34\) GB. KV at \(C = 4{,}096\): \(4{,}096 \times 2 \cdot 36 \cdot 2 \cdot 128 \cdot 2 = 0.15\) GB. Total 3.22 GB (formula 198.6), well inside 60% of 16 GB.

Decode (formula 198.7) with 1,050 tokens in context: \(0.7 \times 100 / (1.736 + 0.039) \approx 39\) tokens per second at most. On a Mac mini M4 Pro (273 GB/s) the bound is about 108 tokens per second. The vision file does not appear in this formula: quantizing it saves memory but not decode time, the same lesson as on a GPU.

TTFT (formula 198.8) for 1,000 image tokens: encoder \(2 \times 668.7\text{M} \times 4 \times 1{,}000 = 5.35\) TFLOPs, LLM \(2 \times 2{,}774.8\text{M} \times 1{,}050 = 5.83\) TFLOPs. At an assumed 2 TFLOP/s, about 5.6 s. On a device, reducing image tokens is the main TTFT lever.

The steps to put a VLM on a device are almost the same in every runtime:

Algorithm 2: ship a VLM to an edge device (llama.cpp names; MLX and ONNX are similar)
  1. input: Hugging Face checkpoint, target device, task test set with images
  2. convert the LLM: convert_hf_to_gguf.py → model-f16.gguf
  3. convert the vision part: convert_hf_to_gguf.py --mmproj → mmproj-f16.gguf // keep 16-bit first
  4. quantize the LLM: llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
  5. fix the image budget: --image-max-tokens (dynamic-resolution models) and context -c // sets TTFT and KV
  6. measure on the device: memory peak, TTFT, decode tokens/s, battery and heat for a sustained run
  7. measure quality on the task set at the chosen quantization and image budget // vision tasks, not only text
  8. if memory is too high then try a Q8_0 vision file, a smaller context, or a smaller model; repeat from step 6
  9. ship base files once; ship task changes as small LoRA files (next section)
# llama.cpp on a laptop: OpenAI-compatible server with a vision projector and a LoRA adapter
llama-server -m Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf \
  --mmproj mmproj-Qwen2.5-VL-3B-Instruct-f16.gguf \
  --lora receipts-lora.gguf \
  -c 8192 --image-max-tokens 1024
# add --no-mmproj-offload to keep the vision part on the CPU and save GPU memory

# MLX on a Mac
mlx_vlm.convert --hf-path Qwen/Qwen2.5-VL-3B-Instruct --mlx-path qwen3b-4bit --quantize --q-bits 4
mlx_vlm.generate --model qwen3b-4bit --image receipt.jpg --prompt "Total amount?" --max-tokens 50

PEFT adapters on edge devices

Often you need many variants of one VLM: one tuned for receipts, one for medical forms, one for a customer’s product catalog. Fine-tuning the whole model for each gives many full copies. LoRA (session 57, session 189) gives one shared base and a small file per task. The adapted layer computes

\[ \mathbf{h} = W \mathbf{x} + \frac{\alpha}{r} B A \mathbf{x}, \qquad P_{\text{LoRA}} = r \sum_{\ell} \sum_{j \in \text{targets}} \left( d_{\text{in}}^{(j)} + d_{\text{out}}^{(j)} \right) \tag{198.9} \]
Read the symbols
\(W\)
the frozen base weight, \(d_{\text{out}} \times d_{\text{in}}\); on a device it is usually 4-bit
\(A, B\)
the adapter: \(A\) is \(r \times d_{\text{in}}\), \(B\) is \(d_{\text{out}} \times r\)
\(r, \alpha\)
rank and scaling of the adapter
\(\ell, j\)
the layers, and the adapted matrices in each layer (for example q, k, v, o, and the three MLP matrices)

For Qwen2.5-VL-3B (36 layers, width 2,048, KV width 256, MLP width 11,008), one layer’s attention matrices give \(4{,}096 + 2{,}304 + 2{,}304 + 4{,}096 = 12{,}800\) and the MLP adds \(3 \times 13{,}056 = 39{,}168\). At \(r = 16\) on all seven matrices: \(16 \times 51{,}968 \times 36 = 29.9\) M parameters, 59.9 MB in FP16. The 4-bit base is 1.74 GB. Ten task adapters are 0.6 GB; ten merged models would be 17.4 GB.

Merged or unmerged?

There are two ways to run an adapter. Merged: compute \(W' = W + \frac{\alpha}{r}BA\) once, quantize it, and ship a full model. No runtime cost, but one full file per task, and you cannot switch tasks without loading another model. Unmerged: keep the base and apply \(B(A\mathbf{x})\) on the side at runtime. Since decode is bandwidth-bound, the cost is roughly the extra bytes read per token:

\[ \frac{\Delta t_{\text{token}}}{t_{\text{token}}} \approx \frac{b_a\, P_{\text{LoRA}}}{\frac{q}{8}\left(P_{\text{LLM}} + P_{\text{head}}\right)} \tag{198.10} \]
Read the symbols
\(b_a\)
bytes per adapter weight (2 for FP16)
denominator
bytes of the base read per decode token (formula 198.7 without the KV term)

For the \(r = 16\) adapter on the Q4_0 3B base: \(59.9 / 1{,}736 = 3.4\%\) more bytes per token. The extra FLOPs are only about 1.1%, but on a device the bytes and the extra small kernels are what you pay. llama.cpp supports both ways: --lora and --lora-scaled FILE:SCALE apply adapters at runtime (comma-separated for several), --lora-init-without-apply loads them switched off, and a request can set its own adapter list and scales [Certain, llama.cpp server docs]; adapters are converted with convert_lora_to_gguf.py, and llama-export-lora merges them [Likely]. MLX trains and loads adapters in the same tool. In the data center, vLLM serves many adapters in one batch: --enable-lora, --max-lora-rank set to the largest rank you serve (a much larger value wastes memory), and the adapter chosen per request by name [Certain, vLLM docs]. The kernels for this, which apply a different \(A, B\) to each row of the batch, come from Punica and S-LoRA.

4-bit base (1.74 GB)LoRA adapters (FP16)merged full copies
Figure 5. LoRA adapters for the language model of Qwen2.5-VL-3B with a Q4_0 base (formulas 198.9 and 198.10, dimensions from config.json). The vision file is the same in both options and is left out.
How to read Figure 5

The top bar is \(K\) merged models, the bottom bar is one base plus \(K\) adapters, on the same scale. At \(r = 16\) on all matrices and \(K = 10\), merged needs 17.4 GB and unmerged 2.3 GB. Raise the rank to 128: each adapter is now 479 MB and runtime overhead is about 28% of the bytes per token, so for a single fixed task, merging starts to look better. Switch to attention only: the adapter shrinks by a factor of about 4.

Two cautions for VLM adapters. First, an adapter is tied to its exact base. A LoRA trained on a QLoRA NF4 base and then used on a Q4_K_M GGUF base sees slightly different base weights than it was trained with; usually this is small, but test the task again after conversion [Likely]. Second, many VLM fine-tunes also train the connector or put LoRA in the vision tower. Not every runtime can apply those: vLLM’s support for LoRA on the vision tower and connector is still experimental [Certain, vLLM docs]. If the runtime cannot apply part of an adapter, merge that part into the vision file instead and keep only the LLM part as a runtime adapter.

Code: LoRA on a 4-bit base, merged and unmerged, and many adapters in one batch

The code below quantizes a weight matrix with 4-bit groups of 32 (like Q4_0), applies one adapter in the two ways, and measures how far each result is from the exact fine-tuned layer. Then it serves a batch where each request uses a different adapter, by gathering each row’s \(A\) and \(B\), and checks that this gives the same result as a loop. It runs on a CPU in under a second.

import torch
torch.manual_seed(0)
d_in, d_out, r, G = 512, 512, 16, 32          # layer size, LoRA rank, quantization group size

def q4(W):                                     # symmetric 4-bit, one fp16 scale per group of 32 weights (like Q4_0)
    Wg = W.reshape(-1, G)
    s = Wg.abs().amax(1, keepdim=True) / 7
    return (torch.clamp(torch.round(Wg / s), -8, 7) * s).reshape(W.shape)

W = torch.randn(d_out, d_in) / d_in ** 0.5     # base weight
A = torch.randn(r, d_in) / d_in ** 0.5         # one trained adapter: delta = (alpha / r) B A
B = torch.randn(d_out, r) * 0.02
scale = 2.0                                    # alpha / r with alpha = 32
x = torch.randn(64, d_in)                      # 64 tokens

ref = x @ (W + scale * B @ A).T                # what the fine-tuned model should compute
Wq = q4(W)                                     # the 4-bit base we ship
unmerged = x @ Wq.T + scale * (x @ A.T) @ B.T  # base stays 4-bit, adapter runs on the side
merged = x @ q4(W + scale * B @ A).T           # merge first, then quantize again
base_only = x @ Wq.T
rel = lambda y: ((y - ref).norm() / ref.norm()).item()
print(f"adapter off      rel. error {rel(base_only):.4f}")
print(f"unmerged adapter rel. error {rel(unmerged):.4f}")
print(f"merge + requant  rel. error {rel(merged):.4f}")
d_ref = x @ (scale * B @ A).T                  # the adapter's own contribution
kept = lambda y: ((y - base_only) * d_ref).sum().item() / (d_ref * d_ref).sum().item()
print(f"share of adapter effect kept: unmerged {kept(unmerged):.3f}, merged {kept(merged):.3f}")

# many adapters, one batch: each request picks its adapter by id (the idea behind multi-LoRA serving)
K = 4
As = torch.randn(K, r, d_in) / d_in ** 0.5
Bs = torch.randn(K, d_out, r) * 0.02
ids = torch.tensor([0, 2, 2, 1, 3, 0, 1, 3])   # adapter id per request
xb = torch.randn(len(ids), d_in)
y_batched = xb @ Wq.T + scale * torch.einsum("bor,bri,bi->bo", Bs[ids], As[ids], xb)
y_loop = torch.stack([xb[i] @ Wq.T + scale * Bs[k] @ (As[k] @ xb[i]) for i, k in enumerate(ids.tolist())])
print("batched == loop:", torch.allclose(y_batched, y_loop, atol=1e-5))
print(f"adapter size {2 * r * (d_in + d_out) / 1e3:.1f} kB in fp16 vs base {d_in * d_out * 4.5 / 8 / 1e3:.1f} kB in 4.5-bit")

Output (PyTorch 2.8 on a CPU):

adapter off      rel. error 0.1858
unmerged adapter rel. error 0.0958
merge + requant  rel. error 0.0973
share of adapter effect kept: unmerged 1.000, merged 1.004
batched == loop: True
adapter size 32.8 kB in fp16 vs base 147.5 kB in 4.5-bit

Read it this way. Without the adapter, the output is 18.6% away from the fine-tuned layer. Both ways of applying it bring the error down to about 9.6 to 9.7%, and what remains is the 4-bit rounding of the base, not the adapter. Both keep the whole adapter effect (the share is 1.000 and 1.004). So in this test, merging into a quantized base does not lose accuracy; the real reasons to choose are operational: storage, switching between tasks, and the small runtime cost of formula (198.10). The batched gather gives exactly the loop’s result, which is how one engine can serve many adapters in one batch.

Where this is used

ApplicationTypical setupWhat decides the design
Chat assistant with photo uploadvLLM or TensorRT-LLM, one engine per GPU group, prefix cache onImage limits and max_pixels set concurrency (formula 198.3); ITL under mixed traffic
Document AI (invoices, forms, contracts)High max_pixels, few pages per prompt; separate encoder pool for high volumeLarge images per request; encoder time per page
Video QA, action recognition, key-moment detectionFixed frame count and frame size in the server config; video limit 1 per promptFrames × tokens per frame (formula 198.2); encoder load per clip, where E→PD helps most
Video retrieval, Video-RAG, fingerprintingOffline encoder jobs in large batches; embeddings stored; online engine only for the final answerEncoder throughput, not latency; the encoder fleet scales on its own
Phone and laptop assistants (screen reading, photo questions)llama.cpp, MLX, or ONNX Runtime with a 4-bit LLM and 16-bit or 8-bit vision partMemory (formula 198.6) and TTFT on the device (198.8); battery
In-browser demos and private toolstransformers.js on WebGPU with a small VLM (for example SmolVLM)Download size; per-graph precision
Robots, drones and cameras (Jetson-class)llama.cpp or TensorRT on the device; fixed low resolution; few framesShared memory of 8 to 64 GB; real-time TTFT
Many customer-specific variantsOne base plus a LoRA per customer: vLLM multi-LoRA on servers, --lora on devicesAdapter size and rank (formula 198.9); per-request adapter selection

Common mistakes

  • Leaving the image limits at their defaults. On a 24 GB GPU, the defaults for Qwen2.5-VL-7B (several images at up to 12.8 MP) do not even fit one worst-case request. Set limit_mm_per_prompt, max_pixels and the frame count for your task, and turn off modalities you do not use.
  • Lowering max_pixels without testing accuracy. The limit changes what the model sees. Test OCR and small-object tasks at the limit you deploy.
  • Tuning only the token budget for ITL. Chunked prefill cannot split the encoder. A large image still puts its whole encoder into one step; watch the p99 ITL when large images arrive, not the average.
  • Disaggregating too early. A separate encoder pool costs GPUs and a network hop. For small fixed encoders or text-heavy traffic it adds cost with no gain. Measure the encoder share first.
  • Counting only the 4-bit LLM on the edge. An F16 vision file can be as large as the 4-bit LLM (1.34 GB vs 1.74 GB for Qwen2.5-VL-3B). Include it, and the KV cache at the real context length, in the memory budget.
  • Trusting edge preprocessing. Reimplemented resize and tiling can differ from the reference processor. Compare tokens or embeddings on sample images.
  • Using an adapter with a different base. A LoRA trained on one base (or one quantization of it) is not guaranteed to work on another. Retest after conversion to GGUF or MLX.
  • Setting max-lora-rank far above the ranks you serve. The engine reserves memory for the largest rank. Set it to the largest rank you actually use.

Interview questions

Why does vLLM ask for a limit on images per prompt?basic
At start-up the engine profiles peak memory with a dummy request of the largest allowed size, and gives the rest of the GPU to the KV cache. More and bigger items per prompt make the worst case bigger, so the profiling peak and the worst-case KV need grow and fewer requests fit. The limit also protects fairness and the context length, and setting a modality to 0 turns it off. For Qwen2.5-VL-7B, 4 images at the default 12.8 MP is 66,048 tokens (3.79 GB of KV) per worst-case request; capping images at 1 MP gives 5,632 tokens.
How many tokens can one image and one video produce in Qwen2.5-VL?core
One LLM token covers 28 × 28 pixels (14-pixel patches, 2 × 2 merge), so an image gives about \(\text{max\_pixels}/784\) tokens at most: 16,384 at the default 12,845,056 pixels, 1,280 at 1,003,520. A video also merges 2 frames in time (temporal patch size 2), so \(F\) frames of \(H' \times W'\) give \(\lceil F/2 \rceil \cdot H'W'/784\) tokens: 64 frames at 448 × 448 give \(32 \times 256 = 8{,}192\) tokens.
Chunked prefill keeps ITL low. Why can a large image still cause a long pause for other users?core
Chunked prefill splits the LLM part of a long prompt into token chunks, but the vision encoder runs on a whole image at once: its attention spans all patches. vLLM’s encoder budget is max_num_batched_tokens or, if larger, the size of the largest single item, so one huge image is encoded in one step. Every request in that step waits. For a 4000 × 3000 photo in Qwen2.5-VL-7B, the encoder is about 156 TFLOPs, around 315 ms on an H100 at 50% of peak, so a step can grow from 6 ms to over 300 ms.
What is E→PD disaggregation, and what does it cost?core
The vision encoder runs in its own worker on separate GPUs; it sends image embeddings to the prefill-and-decode worker, which reads them from an encoder cache. Benefits: encoder bursts no longer stretch the shared decode steps, encoders scale independently, text-only requests skip them, and a shared cache allows reuse across workers. Costs: extra GPUs, a network transfer of \(N_{\text{tok}} \cdot d \cdot 2\) bytes (19.3 MB for a 1080p image in Qwen2.5-VL-7B, under 1 ms at 25 GB/s), and more moving parts. It does not make a single image faster; it protects the ITL of everyone else and raises throughput under load.
When would you use mm_encoder_tp_mode="data" instead of a separate encoder pool?advanced
When the LLM already spans several GPUs with tensor parallelism. A small encoder (0.67 B parameters for Qwen2.5-VL) split by tensor parallelism spends much time on all-reduce communication. In data mode each GPU holds a full copy of the encoder and processes a share of the images of the batch, while the language model stays tensor-parallel. It costs a copy of the encoder weights per GPU but no extra machines or network hop. A separate pool is for when encoder load is large and bursty enough to need its own scaling.
What are TensorRT-LLM’s and vLLM’s similarities for VLM serving?basic
Both run an OpenAI-compatible server (vllm serve, trtllm-serve with the PyTorch backend), both apply the chat template and the model’s input processor, both reuse KV blocks across requests using hashes of the image data, and both overlap CPU preprocessing with GPU work. Older TensorRT-LLM releases compiled the vision encoder as a separate TensorRT engine next to the LLM engine. For disaggregating the encoder, vLLM has its own E→PD support; with TensorRT-LLM, NVIDIA Dynamo provides E/PD.
How do you run a VLM with llama.cpp, and why are there two files?basic
The language model is one GGUF file (-m) and the vision encoder plus projector is a second GGUF file (--mmproj), produced by convert_hf_to_gguf.py with --mmproj. llama-server then serves the OpenAI chat API with images; llama-mtmd-cli is for testing. Two files let you quantize the parts differently (for example a Q4_K_M LLM and an F16 or Q8_0 projector), share one projector across LLM quantizations, and keep the projector on the CPU with --no-mmproj-offload when GPU memory is tight.
Estimate the memory and decode speed of Qwen2.5-VL-3B on a laptop with 100 GB/s.core
LLM at Q4_0: \((2{,}774.8\,\text{M} + 311.2\,\text{M tied embedding}) \times 4.5/8\) bytes \(\approx 1.74\) GB. Vision at F16: \(668.7\,\text{M} \times 2 \approx 1.34\) GB. KV at 4,096 tokens: 36 layers, 2 KV heads, head size 128, K and V, 2 bytes, so \(36 \times 2 \times 128 \times 2 \times 2 = 36{,}864\) bytes per token, 0.15 GB. Total about 3.2 GB. Decode reads about 1.74 GB per token plus KV, so at 70% of 100 GB/s, at most about 39 tokens/s. The vision file is not read in decode.
For a small VLM on a phone, why is the encoder often the bottleneck for time to first token?advanced
Encoder FLOPs per image token are about \(2 P_{\text{vis}}\, s\), LLM FLOPs about \(2 P_{\text{LLM}}\), so the ratio is \(s P_{\text{vis}} / P_{\text{LLM}}\). SmolVLM-256M has a 93.5 M vision part, a 106.2 M LLM, and \(s = 16\) (pixel shuffle with scale 4), so the encoder costs about 14 times more than the LLM per image token. For Qwen2.5-VL-7B the ratio is only about 0.41. On a device with limited compute, image tokens (and the patches behind them) set the TTFT, so the image budget is the main lever.
You must serve 50 customer-specific fine-tunes of one VLM. Merge or not?core
On a server, keep one base and serve LoRA adapters per request (vLLM --enable-lora with --max-lora-rank set to the largest rank; batched kernels in the style of Punica and S-LoRA apply a different adapter per row). Fifty merged 7B models would need fifty copies of the weights. On a device with one fixed task, merging gives zero runtime cost. With \(r = 16\) on all matrices of Qwen2.5-VL-3B, an adapter is 59.9 MB against a 1.74 GB 4-bit base, and costs about 3.4% more bytes per decode token when applied at runtime.
Does merging a LoRA into a 4-bit base lose the fine-tune?advanced
Not by itself. Quantizing \(W + \frac{\alpha}{r}BA\) rounds the merged matrix; the rounding error is about the same size as the base’s own rounding error, and on average the adapter’s effect survives (in our test the share of the effect kept was 1.004 merged versus 1.000 unmerged, with errors of 9.7% vs 9.6%). The real risks are elsewhere: an adapter used with a different quantization of the base than it was trained with, LoRA parts in the vision tower or connector that the runtime cannot apply, and losing the ability to switch tasks.

Exercises

  1. Qwen2.5-VL-7B on an H100 80 GB at \(u = 0.9\), 2 images per prompt at max_pixels 1,003,520, no video, 512 text tokens. How many worst-case requests fit? (Use the 2 GB reserve.)
  2. A video limit of 1 with 128 frames at 448 × 448 is added to exercise 1. How many tokens is the worst case now?
  3. How large are the embeddings of a 640 × 480 image (391 tokens) for Qwen2.5-VL-7B, and how long does sending them take at 25 GB/s?
  4. Compute the size of the Qwen2.5-VL-7B files at Q4_0 (LLM) and Q8_0 (vision). Parameters: 6,525.6 M layers, 545.0 M embedding, 545.0 M output head, 676.6 M vision and merger.
  5. With formula (198.7), what is the decode bound of the model in exercise 4 on a device with 273 GB/s at \(\eta = 0.7\), ignoring KV?
  6. A LoRA with \(r = 8\) on the attention matrices only of Qwen2.5-VL-3B: how many parameters and how many MB in FP16?
Answerscheck
1. \(T_{\max} = 2 \times 1{,}280 + 512 = 3{,}072\) tokens, \(3{,}072 \times 57{,}344 = 0.176\) GB per request; \(53.42 / 0.176\), so 303 requests. 2. \(64 \times 256 = 16{,}384\) video tokens, so \(T_{\max} = 19{,}456\) tokens (1.12 GB of KV); 47 worst-case requests fit. 3. \(391 \times 3{,}584 \times 2 = 2.80\) MB; 0.11 ms. 4. LLM: \((6{,}525.6 + 545.0 + 545.0) \times 10^6 \times 4.5/8 = 4.28\) GB. Vision: \(676.6 \times 10^6 \times 8.5/8 = 0.72\) GB. 5. Bytes per token: \((6{,}525.6 + 545.0) \times 10^6 \times 4.5/8 = 3.98\) GB; \(0.7 \times 273 / 3.98 \approx 48\) tokens/s. 6. \(8 \times 12{,}800 \times 36 = 3{,}686{,}400\) parameters, 7.37 MB.

Key takeaways

  • A serving engine must plan for the worst request it accepts. Image count, video frames and max_pixels set that worst case (198.1, 198.2) and so the KV pool and the number of concurrent requests (198.3). On small GPUs the defaults may not fit at all.
  • vLLM and TensorRT-LLM handle images in the same places: a CPU front end with a processor cache, a scheduler with an encoder budget, an encoder cache, and a prefix cache with image hashes.
  • Chunked prefill cannot split the vision encoder. A large image stretches one shared step for every user (198.4). A separate encoder pool (E→PD) removes this at the price of extra GPUs and a cheap embedding transfer (198.5); data-parallel encoders are a cheaper middle step.
  • On the edge, every runtime splits the VLM into a vision part and a language part. Count both and the KV cache (198.6). Decode speed depends only on the LLM bytes (198.7); TTFT is often dominated by the encoder for small models (198.8).
  • LoRA gives one base and small task files (198.9). Unmerged adapters cost a few percent per token on a device (198.10) and allow switching; merging costs no runtime and, in our test, no accuracy, but needs a full copy per task.

Key formulas

\[ T_{\max} = \sum_m c_m \tau_m + T_{\text{text}}, \qquad N_{\text{conc}} = \left\lfloor \frac{u M_{\text{GPU}} - M_W - M_{\text{act}}}{b_{\text{KV}} T_{\max}} \right\rfloor \]
\[ t_{\text{step}} \approx \max\!\left(\frac{\text{bytes}}{\eta\,\text{BW}}, \frac{F_{\text{LLM}}}{u\,\Pi}\right) + \frac{F_{\text{enc}}}{u\,\Pi}, \qquad t_{\text{xfer}} = \frac{N_{\text{tok}}\, d\, b}{\text{BW}_{\text{link}}} \]
\[ \text{tokens/s} \le \frac{\eta\,\text{BW}}{\tfrac{q}{8}(P_{\text{LLM}} + P_{\text{head}})}, \qquad \frac{\text{TTFT}_{\text{enc}}}{\text{TTFT}_{\text{LLM}}} \approx \frac{s\,P_{\text{vis}}}{P_{\text{LLM}}}, \qquad P_{\text{LoRA}} = r \sum (d_{\text{in}} + d_{\text{out}}) \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  2. vLLM project (docs). vLLM documentation
  3. NVIDIA (code). TensorRT-LLM
  4. ggml-org (code). llama.cpp
  5. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  6. vLLM project (docs). vLLM documentation
  7. NVIDIA (code). TensorRT-LLM
  8. ggml-org (code). llama.cpp
  9. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  10. Bai et al. (2025). Qwen2.5-VL Technical Report
  11. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  12. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  13. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  14. Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
  15. Singh et al. (2025). Efficiently Serving Large Multimodal Models Using EPD Disaggregation
  16. vLLM team (2025). vLLM docs: Conserving memory (limit_mm_per_prompt, mm_processor_kwargs) and Optimization (mm_processor_cache_gb, mm_encoder_tp_mode)
  17. vLLM team (2025). vLLM docs: Disaggregated Encoder
  18. vLLM team (2025). vLLM docs: LoRA Adapters
  19. NVIDIA (2025). TensorRT-LLM docs: Multimodal support
  20. NVIDIA (2025). NVIDIA Dynamo docs: Encoder Disaggregation
  21. ggml-org (2025). llama.cpp docs: Multimodal (libmtmd, mmproj) and server options
  22. Blaizzy et al. (2025). MLX-VLM: vision language models on Apple silicon
  23. Hugging Face (2025). Transformers.js docs: Using quantized models (dtypes), per-module dtypes
  24. Chen et al. (2023). Punica: Multi-Tenant LoRA Serving
  25. Sheng et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters