Serving VLMs: vLLM, TensorRT-LLM and the edge
Serving a VLM to many users means planning for the largest request you accept, keeping every user's next token fast when a big image arrives, and, on phones and laptops, fitting a vision part and a language part into a few gigabytes. This session explains how vLLM and TensorRT-LLM handle images, why the vision encoder is sometimes moved to its own GPU, how llama.cpp, MLX and ONNX run VLMs on devices, and how LoRA adapters let one base model serve many tasks.
What you will learn
- Where a serving engine handles images: the CPU front end with a processor cache, the scheduler with an encoder budget, the encoder cache and the image-aware prefix cache, in vLLM and TensorRT-LLM
- Why limits on images, frames and pixels per prompt decide the KV pool and the number of concurrent requests, with Qwen2.5-VL-7B numbers on an H100 and an L4
- Why chunked prefill cannot split the vision encoder, how a large image stretches every user's inter-token latency, and how E→PD disaggregation and data-parallel encoders fix it
- How llama.cpp (model + mmproj), MLX, ONNX with transformers.js and ONNX Runtime store and run a VLM on a device
- How to estimate device memory, decode speed and time to first token, and why the encoder dominates TTFT for small VLMs
- How LoRA adapters are served on devices and servers, merged or unmerged, and what each choice costs
Why this matters
A vision-language model (VLM) that works in a notebook is not yet a product. A product must take many requests at once, each with its own images or video, keep every user’s answer fast, never run out of GPU memory, and sometimes run on a phone or a laptop with no GPU server at all. This is the job of a serving engine: the program that loads the model once and turns a stream of requests into batched GPU work.
For text-only models, the engine’s main problems are the KV cache and batching, which we studied in session 133, session 134 and session 135. Images add three new problems. First, one request can be tiny or huge: a 640 × 480 photo gives Qwen2.5-VL-7B 391 tokens, a 4000 × 3000 photo gives it 15,301. The engine must put limits on this, or one user can fill the GPU. Second, the vision encoder is a separate model that runs on a whole image at once, and it does not fit neatly into the token-by-token schedule of prefill and decode. Third, on edge devices, the vision part is a large fraction of the model, and the tools (ONNX, transformers.js, llama.cpp, MLX) each store and run it differently.
In session 196 we followed one request through all its stages, and in session 197 we made each stage cheaper. This session looks at many requests at once. We use two real data-center engines, vLLM and TensorRT-LLM, and four edge runtimes. All model sizes come from the published Hugging Face configs and weight files of Qwen2.5-VL (3B and 7B) and SmolVLM (256M and 2.2B). The next session, session 199, uses these tools to plan a real deployment.
Inside a multimodal serving engine
A serving engine for a VLM does the same loop as for a text model: accept requests, pick which tokens to compute this step, run one forward pass for the whole batch, stream new tokens, repeat. Each run of this loop is one step (also called an iteration). What changes is the work around the loop. We describe vLLM, because its design is documented in detail, and then say what is the same and what is different in TensorRT-LLM.
vLLM: four places where images are handled
- The front end (API server). It receives the OpenAI-style chat request, fetches image URLs or decodes base64, applies the chat template, and runs the model’s Hugging Face processor (resize, normalize, patchify, and inserting the right number of placeholder tokens). This runs on the CPU, in a separate process from the GPU work [Likely]. A processor cache keeps the processed result of recent images, keyed by a hash of the content, so a repeated image is not processed again. Its size is
mm_processor_cache_gb, 4 GiB by default [Certain, vLLM docs]. - The scheduler. Each step it chooses tokens up to a token budget,
max_num_batched_tokens. With chunked prefill (on by default [Certain, vLLM docs]), a long prompt is split across steps. For images there is a second budget: the encoder compute budget, measured in encoder output tokens. vLLM sets it tomax_num_batched_tokens, or to the size of the largest single image if that is bigger [Certain, vLLM scheduler docs]. The last part is important: an image is never split. Its encoder runs on all of it in one step. - The encoder cache on the GPU. The encoder output of an image is kept until the last prefill chunk that needs it has run. Its size follows the same rule as the encoder budget [Certain, vLLM scheduler docs].
- The prefix cache. As in the previous session, the hash of each KV block includes the content hash of any image whose tokens are in that block, so two different images with the same placeholder ids never share KV blocks.
The scheduling rule for one request with images, in a simplified form:
- input: request with computed tokens \(c\), prompt length \(n\), images \(i\) with token ranges \([s_i, s_i + \tau_i)\); remaining token budget \(b_{\text{tok}}\) and encoder budget \(b_{\text{enc}}\)
- \(k \leftarrow \min(n - c,\ b_{\text{tok}})\) // tentative chunk: positions \(c, \dots, c+k-1\)
- for each image \(i\) whose range overlaps \([c, c+k)\)
- if image \(i\) is in the encoder cache then continue // already encoded in an earlier step
- if \(\tau_i \le b_{\text{enc}}\) then schedule the encoder for image \(i\); \(b_{\text{enc}} \leftarrow b_{\text{enc}} - \tau_i\)
- else \(k \leftarrow s_i - c\); break // stop the chunk just before this image; it waits for a later step
- if \(k = 0\) then skip this request in this step
- \(b_{\text{tok}} \leftarrow b_{\text{tok}} - k\); run the scheduled encoders first, then the LLM on the chunk
- free an image’s encoder-cache entry when \(c + k \ge s_i + \tau_i\) // its last token has been prefilled
TensorRT-LLM: the same ideas, NVIDIA’s implementation
TensorRT-LLM is NVIDIA’s inference library. Its current multimodal path runs models on its PyTorch backend and serves them with trtllm-serve, which exposes the same OpenAI-compatible chat API (for example trtllm-serve Qwen/Qwen2-VL-7B-Instruct --backend pytorch) [Certain, TensorRT-LLM docs]. It hashes the raw image data and uses the hashes for KV-cache reuse, so it has the same image-aware prefix cache as vLLM, and it overlaps CPU preprocessing with GPU encoding [Certain, TensorRT-LLM docs]. Older releases took a different path: the vision encoder was compiled into its own TensorRT engine next to the LLM engine, and a runner passed the image features from one to the other [Likely]. Even in that old design you can see the main idea of this session: the encoder and the language model are two separate programs with a clean interface, a tensor of image embeddings.
For splitting the encoder onto other machines, NVIDIA uses Dynamo, a layer above the engine. Dynamo lists an “E/PD” mode for TensorRT-LLM: a separate encode worker sends embeddings to a combined prefill-and-decode worker [Certain, Dynamo docs].
| Concern | vLLM | TensorRT-LLM |
|---|---|---|
| Server | vllm serve <model>, OpenAI chat API | trtllm-serve <model> --backend pytorch, OpenAI chat API |
| Items per prompt | --limit-mm-per-prompt '{"image": 4, "video": 1}' | set by the model’s input processor and request [Likely] |
| Image size | --mm-processor-kwargs '{"max_pixels": ...}' | processor arguments of the model [Likely] |
| Prefix cache with images | image hash in each block hash | image hashes plus token-chunk information |
| Encoder on other GPUs | disaggregated encoder: E→PD or E→P→D, embeddings sent by an EC connector | through NVIDIA Dynamo (E/PD) |
| Encoder parallelism | mm_encoder_tp_mode="data": each GPU runs a full encoder copy on part of the images | model dependent [Likely] |
| Many fine-tuned adapters | --enable-lora, --max-lora-rank, per-request adapter name | LoRA support in the PyTorch backend [Likely] |
Limits per request: images, frames and pixels
Why does vLLM have a flag called limit_mm_per_prompt? Because a serving engine must be ready for the worst request it accepts. At start-up, vLLM runs a profiling forward pass with dummy inputs of the largest allowed size, measures the peak memory, and gives what remains (up to the fraction gpu_memory_utilization of the GPU) to the KV cache. More items per prompt and bigger items mean a bigger worst case. The docs say it directly: allowing fewer multimodal items per prompt reduces the memory footprint, and setting a modality to 0 (for example {"video": 0}) turns it off [Certain, vLLM docs]. The limit can also carry size hints (width and height for images, number of frames for video); these hints change only the profiling, not how real inputs are processed [Certain, vLLM docs].
The token count of the worst request is a sum over modalities:
- Read the symbols
- \(c_m\)
- the limit: the most items of modality \(m\) one prompt may contain
- \(\tau_m\)
- the most LLM tokens one item can produce, set by the processor’s size limits
- \(T_{\text{text}}\)
- room kept for the text of the prompt and the answer
For Qwen2.5-VL, \(\tau\) follows from the pixel limit. Each LLM token covers 28 × 28 pixels (a 2 × 2 merge of 14-pixel patches), and a video merges 2 frames in time (temporal patch size 2 [Certain, config.json]):
- Read the symbols
- max_pixels
- the processor’s upper limit on \(H' \times W'\) for one image; 12,845,056 in Qwen2.5-VL’s processor config, often lowered with
mm_processor_kwargs - \(F\)
- number of video frames given to the model after sampling
- \(H', W'\)
- the resized frame size, each a multiple of 28
Then the memory left for the KV cache, and how many worst-case requests fit in it at the same time, are
- Read the symbols
- \(u\)
gpu_memory_utilization, the share of GPU memory the engine may use (0.9 by default in vLLM [Certain])- \(M_W\)
- weight memory: 16.58 GB for Qwen2.5-VL-7B in BF16 (8.29 B parameters, from its safetensors index)
- \(M_{\text{act}}\)
- peak activation memory of the profiling pass, plus CUDA graphs; we assume 2 GB, the real value is measured at start-up
- \(b_{\text{KV}}\)
- KV bytes per token: \(2 \cdot 28 \cdot 4 \cdot 128 \cdot 2 = 57{,}344\) for Qwen2.5-VL-7B (formula 196.7)
Paged attention allocates blocks only for tokens that really exist, so the engine can run more than \(N_{\text{conc}}\) small requests. But \(N_{\text{conc}}\) is the number of worst-case requests that can be in flight together, and a single request with \(b_{\text{KV}} T_{\max} \gt M_{\text{KV}}\) can never run. Figure 1 shows how much the limits matter.
How to read Figure 1
The bar is the whole GPU, split as in the legend: weights, the activation reserve, and the KV pool, which is cut into boxes, one per worst-case request. The dashed line marks gpu_memory_utilization; the rest is not used. Start at the default: H100, 4 images at 12.8 MP. Each worst-case request needs 66,048 tokens (3.79 GB of KV), so only 14 fit. Now pick 1.0 MP: each request needs 5,632 tokens and 165 fit. Switch to the L4: the weights take most of the card, the pool is about 3 GB, and the default limits do not fit even one request. Turn on a video and move the frame slider to see how fast frames add up.
Qwen2.5-VL-7B, 4 images per prompt, default max_pixels = 12,845,056. Formula (198.2): \(\tau_{\text{img}} = \lfloor 12{,}845{,}056 / 784 \rfloor = 16{,}384\) tokens per image. Formula (198.1): \(T_{\max} = 4 \times 16{,}384 + 512 = 66{,}048\) tokens, and its KV cache is \(66{,}048 \times 57{,}344 = 3.79\) GB.
H100 80 GB at \(u = 0.9\): \(M_{\text{KV}} = 72 - 16.58 - 2 = 53.42\) GB, room for 931,495 tokens, so \(N_{\text{conc}} = 14\) worst-case requests. Lower max_pixels to \(1280 \times 28 \times 28 = 1{,}003{,}520\) (1,280 tokens per image): \(T_{\max} = 5{,}632\), and \(N_{\text{conc}} = 165\). The limit changed the worst-case concurrency by a factor of 11.8.
L4 24 GB: \(M_{\text{KV}} = 21.6 - 16.58 - 2 = 3.02\) GB, room for 52,589 tokens. The default worst case (66,048 tokens) does not fit at all. With 4 images at 1.0 MP it fits 9 times; with 1 image at 1.0 MP, 29 times. On a small GPU the limits are not optional. A 4-bit language model (session 197) is the other lever.
Video makes this sharper. Each frame adds tokens, and the frame count is set by the sampling rate and the clip length. 64 frames at 448 × 448 give \(32 \times 256 = 8{,}192\) tokens, already more than six 1 MP images. vLLM’s docs warn that for the Qwen2-VL family a larger max_model_len can increase the profiled video size and so reduce the KV cache [Certain, vLLM docs]. In practice, teams fix the frame count and frame size in the server config, so the worst case is known.
A configuration that applies these limits, for vLLM:
# Qwen2.5-VL-7B on one GPU: at most 4 images and 1 video per prompt, # images capped at about 1 MP (1280 tokens each), context capped at 16k tokens vllm serve Qwen/Qwen2.5-VL-7B-Instruct \ --limit-mm-per-prompt '{"image": 4, "video": 1}' \ --mm-processor-kwargs '{"max_pixels": 1003520}' \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --mm-processor-cache-gb 4 # a request with more images than the limit is rejected with an error, not silently cut
There are two other things a limit protects. One is fairness: without a limit, one user with 20 photos makes everyone else wait through a long prefill. The other is the context length: a request whose image tokens exceed max_model_len fails. Always check the error path: a request over the limit should get a clear error message, and the client should resize or drop images before it retries.
Separating the encoder from prefill and decode
Every step of a continuous-batching engine does decode for all running requests plus some prefill. The time of the step is the inter-token latency (ITL) that every running user sees: their next token cannot come out before the step ends. Chunked prefill exists to keep this step short. But there is a piece it cannot cut: the vision encoder runs on a whole image in one step. A step that contains the encoder of a large image is long for everybody.
We can model a step as bound by memory or by compute, whichever is slower (the roofline from session 82), plus the encoder if it is scheduled:
- Read the symbols
- \(M_W\)
- LLM weight bytes read per step: 14.14 GB for Qwen2.5-VL-7B in BF16 (layers and output head; the vision tower is not read in decode)
- \(\sum_j n_j\)
- the context lengths of all requests in the batch, whose KV must be read
- \(F_{\text{LLM}}(\text{batch})\)
- LLM FLOPs for this step: one token for each decoding request plus the prefill chunk (formula 196.5)
- \(F_{\text{enc}}\)
- encoder FLOPs for the images whose encoder runs in this step (formula 196.3)
- \(\eta, u, \Pi, \text{BW}\)
- bandwidth efficiency, compute efficiency, peak FLOP rate and peak bandwidth of the GPU
The fix is to run the encoder somewhere else. In an E→PD layout, an encoder worker (its own GPU or GPUs) runs the vision tower and sends the image embeddings to the prefill-and-decode worker. In E→P→D, prefill and decode are also split, as in session 137. vLLM implements both: the encoder instance writes into an encoder cache that the PD instance reads through an EC connector (an example connector, or one based on the Mooncake transfer engine) [Certain, vLLM docs]. The cost is a transfer of the embeddings:
- Read the symbols
- \(N_{\text{tok}}\)
- image tokens after the connector (2,691 for a 1920 × 1080 image in Qwen2.5-VL)
- \(d, b\)
- LLM width (3,584) and bytes per number (2 for BF16)
- \(\text{BW}_{\text{link}}\)
- bandwidth of the link between the two GPUs; we assume 25 GB/s, a PCIe-class or fast network link
A 1920 × 1080 image is \(2{,}691 \times 3{,}584 \times 2 = 19.3\) MB of embeddings, 0.77 ms at 25 GB/s. A 4000 × 3000 photo is 109.7 MB, 4.4 ms. This is small next to the encoder time it removes from the shared step. (For comparison, sending its KV cache from a prefill GPU to a decode GPU would be \(15{,}301 \times 57{,}344 = 877\) MB, 8 times more. Embeddings are cheap to move.)
max_num_batched_tokens. This is a model, not a measurement.How to read Figure 2
The top row is the shared GPU. Each box is one step, and its width is its duration, so the width is the inter-token latency the 16 users feel. Start with the 4000 × 3000 photo and budget 512 on the same GPU: one box is 330 ms wide, because the whole encoder (156 TFLOPs) runs inside it. Switch to E→PD: the encoder moves to the second row (another GPU), the shared steps stay below 21 ms, and the new request’s first token comes at almost the same time. Now raise the budget to 8192: the encoder is still gone, but each prefill chunk is now big, so ITL rises again. The budget sets the ITL of prefill; the encoder placement sets the ITL of images.
Plain decode step: \((14.14 + 16 \times 3{,}000 \times 57{,}344 / 10^9) / (0.8 \times 3.35) = 16.89 / 2.68 \approx 6.3\) ms. That is what each user sees between tokens when nothing else happens.
The photo (4000 × 3000) becomes 15,301 tokens; its encoder is 155.9 TFLOPs, which is \(155.9 / 494.5 = 315\) ms at 50% of peak. Budget 512, same GPU: the step that runs the encoder also runs the first 496-token chunk, so it takes about 330 ms. Sixteen users see a pause of a third of a second in the middle of their answer. The prefill then needs 31 chunks, and TTFT for the new request is about 868 ms.
E→PD: the PD GPU keeps decoding at 6.3 ms per step while the encoder GPU works, receives 109.7 MB of embeddings (4.4 ms), and then runs the same 31 chunks. The longest step is about 21 ms. TTFT is about 875 ms, almost unchanged: disaggregation does not make one image faster, it stops one image from slowing everyone else. Under real load it also lets the encoder of request A run while the LLM does prefill for request B, which raises throughput.
When is a separate encoder worth it? It costs extra GPUs and a network hop, so the answer depends on the traffic.
- Worth it when images are large or many (documents, high-resolution photos, video), when the encoder is a big share of the work (Qwen2.5-VL’s encoder is about 30% of prefill FLOPs for a 1080p image, see session 196), and when the ITL target is strict.
- Not worth it for small fixed-size encoders (LLaVA-1.5’s encoder is about 5% of prefill), mostly text traffic, or one-GPU deployments.
- A cheaper middle step when the LLM is split over several GPUs with tensor parallelism: run the encoder data-parallel instead. With
mm_encoder_tp_mode="data", vLLM gives each GPU a full copy of the (small) encoder and splits the images of a batch across the GPUs, while the language model stays tensor-parallel [Certain, vLLM docs]. A small encoder split with tensor parallelism spends much of its time on communication, so this is often faster; the cost is a copy of the encoder weights on each GPU.
The edge: four ways to run a VLM on a device
“The edge” means running the model on the user’s device: a phone, a laptop, a car computer, a camera box. There is no batch of other users, so the goal changes from throughput to fitting in memory and answering fast for one user, often on battery. There is also privacy: the photo never leaves the device. The same model is split in a similar way by every edge runtime: a vision part and a language part, in separate files or graphs.
| Runtime | Where it runs | How the VLM is stored | Notes |
|---|---|---|---|
| llama.cpp (libmtmd) | CPU, Apple Metal, CUDA, Vulkan; laptops, phones, Jetson | LLM in one GGUF file (-m), vision encoder and projector in a second GGUF file (--mmproj) | llama-server serves the OpenAI chat API, llama-mtmd-cli is for testing; convert_hf_to_gguf.py --mmproj writes the projector file; supports Gemma 3, Qwen2/2.5-VL, SmolVLM, InternVL, Pixtral and others [Certain, llama.cpp docs] |
| MLX (mlx-vlm) | Apple silicon, unified memory | MLX safetensors, quantized with mlx_vlm.convert --quantize --q-bits 4 | Python; generate, an OpenAI-style server, and LoRA/QLoRA fine-tuning on the Mac [Certain, mlx-vlm README] |
| ONNX + transformers.js | The browser (WebGPU or WebAssembly), Node.js | Several ONNX graphs: vision_encoder, embed_tokens, decoder_model_merged | A precision per graph: for example vision_encoder: "fp16" and decoder_model_merged: "q4" [Certain, transformers.js docs] |
| ONNX Runtime (GenAI) | Windows, Android, iOS; CPU, GPU, NPU | The same split into ONNX graphs | Used for Phi vision models; has an adapter API for LoRA [Likely] |
Notice the split in every row. It matches what we learned in session 197: the vision part is more sensitive, so the runtimes let you keep it in 16-bit (an F16 mmproj file, an fp16 vision_encoder) while the language model goes to 4 bits. The transformers.js docs give exactly this example for Florence-2. But on a small device this has a price. Count it.
Memory and speed on a device
The memory needed is the two files plus the KV cache:
- Read the symbols
- \(q, q_v\)
- bits per weight of the LLM file and of the vision file; GGUF Q8_0 is 8.5 bits (32 int8 weights plus one 16-bit scale), Q4_0 is 4.5 bits, F16 is 16
- \(P_{\text{LLM}}^{\text{all}}\)
- all LLM parameters, including the embedding table and an untied output head
- \(P_{\text{vis}}\)
- vision encoder plus connector parameters: 0.669 B for Qwen2.5-VL-3B, 0.094 B for SmolVLM-256M
- \(C\)
- context length the app allows (image tokens + text)
Decode on a device is bandwidth-bound, like decode on a GPU at batch size 1. Each token reads the LLM layers and the output head once (the embedding table only gives one row, and the vision file is not read at all):
- Read the symbols
- \(P_{\text{LLM}}, P_{\text{head}}\)
- parameters of the transformer layers, and of the output head (for tied embeddings, the embedding table is read as the head)
- \(n\)
- current context length
- \(\eta\, \text{BW}\)
- the achieved share of the device’s memory bandwidth
Time to first token is compute-bound. For small VLMs the encoder is often the bigger part, because each LLM token stands for \(s\) encoder patches (\(s = 4\) for Qwen2.5-VL’s 2 × 2 merge, 16 for SmolVLM-256M’s pixel shuffle with scale 4, 9 for SmolVLM-2.2B’s scale 3 [Certain, config.json]):
- Read the symbols
- \(N_{\text{img}}, N_{\text{txt}}\)
- image tokens and text tokens of the prompt
- \(s\)
- encoder patches per LLM image token
- \(C_{\text{eff}}\)
- the FLOP rate the device really reaches on these kernels; you must measure it (we leave it as a slider)
The ratio of the two terms is \(s P_{\text{vis}} / P_{\text{LLM}}\) (formula 196.4). For SmolVLM-256M it is \(16 \times 93.5 / 106.2 \approx 14\): the encoder costs 14 times more than the LLM for each image token. For Qwen2.5-VL-3B it is \(4 \times 668.7 / 2{,}774.8 \approx 0.96\), and for Qwen2.5-VL-7B about 0.41. The smaller the language model, the more the vision side dominates TTFT.
How to read Figure 4
The bar shows the memory the model needs; the line marked “usable” is what the app may use. Start with Qwen2.5-VL-3B at Q4_0 on the M2: the LLM file is 1.74 GB and the F16 vision file 1.34 GB, so the vision side is 41% of the memory although it has 18% of the parameters. Switch the mmproj to Q8_0 and it drops to 0.71 GB. Now pick SmolVLM-256M and look at the TTFT split: almost all of it is the encoder. Pick Qwen2.5-VL-7B on the phone: at F16 it does not fit at all; at Q4_0 it fits only if the app may use about 0.75 of the RAM.
Parameters (from the safetensors file): 2,774.8 M in the layers, 311.2 M in the tied embedding table, 668.7 M in the vision tower and merger. LLM file at Q4_0: \((2{,}774.8 + 311.2) \times 10^6 \times 4.5 / 8 = 1.74\) GB. Vision file at F16: \(668.7 \times 10^6 \times 2 = 1.34\) GB. KV at \(C = 4{,}096\): \(4{,}096 \times 2 \cdot 36 \cdot 2 \cdot 128 \cdot 2 = 0.15\) GB. Total 3.22 GB (formula 198.6), well inside 60% of 16 GB.
Decode (formula 198.7) with 1,050 tokens in context: \(0.7 \times 100 / (1.736 + 0.039) \approx 39\) tokens per second at most. On a Mac mini M4 Pro (273 GB/s) the bound is about 108 tokens per second. The vision file does not appear in this formula: quantizing it saves memory but not decode time, the same lesson as on a GPU.
TTFT (formula 198.8) for 1,000 image tokens: encoder \(2 \times 668.7\text{M} \times 4 \times 1{,}000 = 5.35\) TFLOPs, LLM \(2 \times 2{,}774.8\text{M} \times 1{,}050 = 5.83\) TFLOPs. At an assumed 2 TFLOP/s, about 5.6 s. On a device, reducing image tokens is the main TTFT lever.
The steps to put a VLM on a device are almost the same in every runtime:
- input: Hugging Face checkpoint, target device, task test set with images
- convert the LLM:
convert_hf_to_gguf.py→model-f16.gguf - convert the vision part:
convert_hf_to_gguf.py --mmproj→mmproj-f16.gguf// keep 16-bit first - quantize the LLM:
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M - fix the image budget:
--image-max-tokens(dynamic-resolution models) and context-c// sets TTFT and KV - measure on the device: memory peak, TTFT, decode tokens/s, battery and heat for a sustained run
- measure quality on the task set at the chosen quantization and image budget // vision tasks, not only text
- if memory is too high then try a Q8_0 vision file, a smaller context, or a smaller model; repeat from step 6
- ship base files once; ship task changes as small LoRA files (next section)
# llama.cpp on a laptop: OpenAI-compatible server with a vision projector and a LoRA adapter llama-server -m Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf \ --mmproj mmproj-Qwen2.5-VL-3B-Instruct-f16.gguf \ --lora receipts-lora.gguf \ -c 8192 --image-max-tokens 1024 # add --no-mmproj-offload to keep the vision part on the CPU and save GPU memory # MLX on a Mac mlx_vlm.convert --hf-path Qwen/Qwen2.5-VL-3B-Instruct --mlx-path qwen3b-4bit --quantize --q-bits 4 mlx_vlm.generate --model qwen3b-4bit --image receipt.jpg --prompt "Total amount?" --max-tokens 50
PEFT adapters on edge devices
Often you need many variants of one VLM: one tuned for receipts, one for medical forms, one for a customer’s product catalog. Fine-tuning the whole model for each gives many full copies. LoRA (session 57, session 189) gives one shared base and a small file per task. The adapted layer computes
- Read the symbols
- \(W\)
- the frozen base weight, \(d_{\text{out}} \times d_{\text{in}}\); on a device it is usually 4-bit
- \(A, B\)
- the adapter: \(A\) is \(r \times d_{\text{in}}\), \(B\) is \(d_{\text{out}} \times r\)
- \(r, \alpha\)
- rank and scaling of the adapter
- \(\ell, j\)
- the layers, and the adapted matrices in each layer (for example q, k, v, o, and the three MLP matrices)
For Qwen2.5-VL-3B (36 layers, width 2,048, KV width 256, MLP width 11,008), one layer’s attention matrices give \(4{,}096 + 2{,}304 + 2{,}304 + 4{,}096 = 12{,}800\) and the MLP adds \(3 \times 13{,}056 = 39{,}168\). At \(r = 16\) on all seven matrices: \(16 \times 51{,}968 \times 36 = 29.9\) M parameters, 59.9 MB in FP16. The 4-bit base is 1.74 GB. Ten task adapters are 0.6 GB; ten merged models would be 17.4 GB.
Merged or unmerged?
There are two ways to run an adapter. Merged: compute \(W' = W + \frac{\alpha}{r}BA\) once, quantize it, and ship a full model. No runtime cost, but one full file per task, and you cannot switch tasks without loading another model. Unmerged: keep the base and apply \(B(A\mathbf{x})\) on the side at runtime. Since decode is bandwidth-bound, the cost is roughly the extra bytes read per token:
- Read the symbols
- \(b_a\)
- bytes per adapter weight (2 for FP16)
- denominator
- bytes of the base read per decode token (formula 198.7 without the KV term)
For the \(r = 16\) adapter on the Q4_0 3B base: \(59.9 / 1{,}736 = 3.4\%\) more bytes per token. The extra FLOPs are only about 1.1%, but on a device the bytes and the extra small kernels are what you pay. llama.cpp supports both ways: --lora and --lora-scaled FILE:SCALE apply adapters at runtime (comma-separated for several), --lora-init-without-apply loads them switched off, and a request can set its own adapter list and scales [Certain, llama.cpp server docs]; adapters are converted with convert_lora_to_gguf.py, and llama-export-lora merges them [Likely]. MLX trains and loads adapters in the same tool. In the data center, vLLM serves many adapters in one batch: --enable-lora, --max-lora-rank set to the largest rank you serve (a much larger value wastes memory), and the adapter chosen per request by name [Certain, vLLM docs]. The kernels for this, which apply a different \(A, B\) to each row of the batch, come from Punica and S-LoRA.
How to read Figure 5
The top bar is \(K\) merged models, the bottom bar is one base plus \(K\) adapters, on the same scale. At \(r = 16\) on all matrices and \(K = 10\), merged needs 17.4 GB and unmerged 2.3 GB. Raise the rank to 128: each adapter is now 479 MB and runtime overhead is about 28% of the bytes per token, so for a single fixed task, merging starts to look better. Switch to attention only: the adapter shrinks by a factor of about 4.
Two cautions for VLM adapters. First, an adapter is tied to its exact base. A LoRA trained on a QLoRA NF4 base and then used on a Q4_K_M GGUF base sees slightly different base weights than it was trained with; usually this is small, but test the task again after conversion [Likely]. Second, many VLM fine-tunes also train the connector or put LoRA in the vision tower. Not every runtime can apply those: vLLM’s support for LoRA on the vision tower and connector is still experimental [Certain, vLLM docs]. If the runtime cannot apply part of an adapter, merge that part into the vision file instead and keep only the LLM part as a runtime adapter.
Code: LoRA on a 4-bit base, merged and unmerged, and many adapters in one batch
The code below quantizes a weight matrix with 4-bit groups of 32 (like Q4_0), applies one adapter in the two ways, and measures how far each result is from the exact fine-tuned layer. Then it serves a batch where each request uses a different adapter, by gathering each row’s \(A\) and \(B\), and checks that this gives the same result as a loop. It runs on a CPU in under a second.
import torch torch.manual_seed(0) d_in, d_out, r, G = 512, 512, 16, 32 # layer size, LoRA rank, quantization group size def q4(W): # symmetric 4-bit, one fp16 scale per group of 32 weights (like Q4_0) Wg = W.reshape(-1, G) s = Wg.abs().amax(1, keepdim=True) / 7 return (torch.clamp(torch.round(Wg / s), -8, 7) * s).reshape(W.shape) W = torch.randn(d_out, d_in) / d_in ** 0.5 # base weight A = torch.randn(r, d_in) / d_in ** 0.5 # one trained adapter: delta = (alpha / r) B A B = torch.randn(d_out, r) * 0.02 scale = 2.0 # alpha / r with alpha = 32 x = torch.randn(64, d_in) # 64 tokens ref = x @ (W + scale * B @ A).T # what the fine-tuned model should compute Wq = q4(W) # the 4-bit base we ship unmerged = x @ Wq.T + scale * (x @ A.T) @ B.T # base stays 4-bit, adapter runs on the side merged = x @ q4(W + scale * B @ A).T # merge first, then quantize again base_only = x @ Wq.T rel = lambda y: ((y - ref).norm() / ref.norm()).item() print(f"adapter off rel. error {rel(base_only):.4f}") print(f"unmerged adapter rel. error {rel(unmerged):.4f}") print(f"merge + requant rel. error {rel(merged):.4f}") d_ref = x @ (scale * B @ A).T # the adapter's own contribution kept = lambda y: ((y - base_only) * d_ref).sum().item() / (d_ref * d_ref).sum().item() print(f"share of adapter effect kept: unmerged {kept(unmerged):.3f}, merged {kept(merged):.3f}") # many adapters, one batch: each request picks its adapter by id (the idea behind multi-LoRA serving) K = 4 As = torch.randn(K, r, d_in) / d_in ** 0.5 Bs = torch.randn(K, d_out, r) * 0.02 ids = torch.tensor([0, 2, 2, 1, 3, 0, 1, 3]) # adapter id per request xb = torch.randn(len(ids), d_in) y_batched = xb @ Wq.T + scale * torch.einsum("bor,bri,bi->bo", Bs[ids], As[ids], xb) y_loop = torch.stack([xb[i] @ Wq.T + scale * Bs[k] @ (As[k] @ xb[i]) for i, k in enumerate(ids.tolist())]) print("batched == loop:", torch.allclose(y_batched, y_loop, atol=1e-5)) print(f"adapter size {2 * r * (d_in + d_out) / 1e3:.1f} kB in fp16 vs base {d_in * d_out * 4.5 / 8 / 1e3:.1f} kB in 4.5-bit")
Output (PyTorch 2.8 on a CPU):
adapter off rel. error 0.1858 unmerged adapter rel. error 0.0958 merge + requant rel. error 0.0973 share of adapter effect kept: unmerged 1.000, merged 1.004 batched == loop: True adapter size 32.8 kB in fp16 vs base 147.5 kB in 4.5-bit
Read it this way. Without the adapter, the output is 18.6% away from the fine-tuned layer. Both ways of applying it bring the error down to about 9.6 to 9.7%, and what remains is the 4-bit rounding of the base, not the adapter. Both keep the whole adapter effect (the share is 1.000 and 1.004). So in this test, merging into a quantized base does not lose accuracy; the real reasons to choose are operational: storage, switching between tasks, and the small runtime cost of formula (198.10). The batched gather gives exactly the loop’s result, which is how one engine can serve many adapters in one batch.
Where this is used
| Application | Typical setup | What decides the design |
|---|---|---|
| Chat assistant with photo upload | vLLM or TensorRT-LLM, one engine per GPU group, prefix cache on | Image limits and max_pixels set concurrency (formula 198.3); ITL under mixed traffic |
| Document AI (invoices, forms, contracts) | High max_pixels, few pages per prompt; separate encoder pool for high volume | Large images per request; encoder time per page |
| Video QA, action recognition, key-moment detection | Fixed frame count and frame size in the server config; video limit 1 per prompt | Frames × tokens per frame (formula 198.2); encoder load per clip, where E→PD helps most |
| Video retrieval, Video-RAG, fingerprinting | Offline encoder jobs in large batches; embeddings stored; online engine only for the final answer | Encoder throughput, not latency; the encoder fleet scales on its own |
| Phone and laptop assistants (screen reading, photo questions) | llama.cpp, MLX, or ONNX Runtime with a 4-bit LLM and 16-bit or 8-bit vision part | Memory (formula 198.6) and TTFT on the device (198.8); battery |
| In-browser demos and private tools | transformers.js on WebGPU with a small VLM (for example SmolVLM) | Download size; per-graph precision |
| Robots, drones and cameras (Jetson-class) | llama.cpp or TensorRT on the device; fixed low resolution; few frames | Shared memory of 8 to 64 GB; real-time TTFT |
| Many customer-specific variants | One base plus a LoRA per customer: vLLM multi-LoRA on servers, --lora on devices | Adapter size and rank (formula 198.9); per-request adapter selection |
Common mistakes
- Leaving the image limits at their defaults. On a 24 GB GPU, the defaults for Qwen2.5-VL-7B (several images at up to 12.8 MP) do not even fit one worst-case request. Set
limit_mm_per_prompt,max_pixelsand the frame count for your task, and turn off modalities you do not use. - Lowering max_pixels without testing accuracy. The limit changes what the model sees. Test OCR and small-object tasks at the limit you deploy.
- Tuning only the token budget for ITL. Chunked prefill cannot split the encoder. A large image still puts its whole encoder into one step; watch the p99 ITL when large images arrive, not the average.
- Disaggregating too early. A separate encoder pool costs GPUs and a network hop. For small fixed encoders or text-heavy traffic it adds cost with no gain. Measure the encoder share first.
- Counting only the 4-bit LLM on the edge. An F16 vision file can be as large as the 4-bit LLM (1.34 GB vs 1.74 GB for Qwen2.5-VL-3B). Include it, and the KV cache at the real context length, in the memory budget.
- Trusting edge preprocessing. Reimplemented resize and tiling can differ from the reference processor. Compare tokens or embeddings on sample images.
- Using an adapter with a different base. A LoRA trained on one base (or one quantization of it) is not guaranteed to work on another. Retest after conversion to GGUF or MLX.
- Setting max-lora-rank far above the ranks you serve. The engine reserves memory for the largest rank. Set it to the largest rank you actually use.
Interview questions
Why does vLLM ask for a limit on images per prompt?basic
How many tokens can one image and one video produce in Qwen2.5-VL?core
Chunked prefill keeps ITL low. Why can a large image still cause a long pause for other users?core
What is E→PD disaggregation, and what does it cost?core
When would you use mm_encoder_tp_mode="data" instead of a separate encoder pool?advanced
What are TensorRT-LLM’s and vLLM’s similarities for VLM serving?basic
How do you run a VLM with llama.cpp, and why are there two files?basic
Estimate the memory and decode speed of Qwen2.5-VL-3B on a laptop with 100 GB/s.core
For a small VLM on a phone, why is the encoder often the bottleneck for time to first token?advanced
You must serve 50 customer-specific fine-tunes of one VLM. Merge or not?core
Does merging a LoRA into a 4-bit base lose the fine-tune?advanced
Exercises
- Qwen2.5-VL-7B on an H100 80 GB at \(u = 0.9\), 2 images per prompt at max_pixels 1,003,520, no video, 512 text tokens. How many worst-case requests fit? (Use the 2 GB reserve.)
- A video limit of 1 with 128 frames at 448 × 448 is added to exercise 1. How many tokens is the worst case now?
- How large are the embeddings of a 640 × 480 image (391 tokens) for Qwen2.5-VL-7B, and how long does sending them take at 25 GB/s?
- Compute the size of the Qwen2.5-VL-7B files at Q4_0 (LLM) and Q8_0 (vision). Parameters: 6,525.6 M layers, 545.0 M embedding, 545.0 M output head, 676.6 M vision and merger.
- With formula (198.7), what is the decode bound of the model in exercise 4 on a device with 273 GB/s at \(\eta = 0.7\), ignoring KV?
- A LoRA with \(r = 8\) on the attention matrices only of Qwen2.5-VL-3B: how many parameters and how many MB in FP16?
Answerscheck
Key takeaways
- A serving engine must plan for the worst request it accepts. Image count, video frames and max_pixels set that worst case (198.1, 198.2) and so the KV pool and the number of concurrent requests (198.3). On small GPUs the defaults may not fit at all.
- vLLM and TensorRT-LLM handle images in the same places: a CPU front end with a processor cache, a scheduler with an encoder budget, an encoder cache, and a prefix cache with image hashes.
- Chunked prefill cannot split the vision encoder. A large image stretches one shared step for every user (198.4). A separate encoder pool (E→PD) removes this at the price of extra GPUs and a cheap embedding transfer (198.5); data-parallel encoders are a cheaper middle step.
- On the edge, every runtime splits the VLM into a vision part and a language part. Count both and the KV cache (198.6). Decode speed depends only on the LLM bytes (198.7); TTFT is often dominated by the encoder for small models (198.8).
- LoRA gives one base and small task files (198.9). Unmerged adapters cost a few percent per token on a device (198.10) and allow switching; merging costs no runtime and, in our test, no accuracy, but needs a full copy per task.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- vLLM project (docs). vLLM documentation
- NVIDIA (code). TensorRT-LLM
- ggml-org (code). llama.cpp
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- vLLM project (docs). vLLM documentation
- NVIDIA (code). TensorRT-LLM
- ggml-org (code). llama.cpp
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
- Singh et al. (2025). Efficiently Serving Large Multimodal Models Using EPD Disaggregation
- vLLM team (2025). vLLM docs: Conserving memory (limit_mm_per_prompt, mm_processor_kwargs) and Optimization (mm_processor_cache_gb, mm_encoder_tp_mode)
- vLLM team (2025). vLLM docs: Disaggregated Encoder
- vLLM team (2025). vLLM docs: LoRA Adapters
- NVIDIA (2025). TensorRT-LLM docs: Multimodal support
- NVIDIA (2025). NVIDIA Dynamo docs: Encoder Disaggregation
- ggml-org (2025). llama.cpp docs: Multimodal (libmtmd, mmproj) and server options
- Blaizzy et al. (2025). MLX-VLM: vision language models on Apple silicon
- Hugging Face (2025). Transformers.js docs: Using quantized models (dtypes), per-module dtypes
- Chen et al. (2023). Punica: Multi-Tenant LoRA Serving
- Sheng et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters