A latency budget on an NVIDIA L4: worked example
Can a 7B vision-language model answer photo questions in under 4 seconds on one NVIDIA L4? This session answers with a pencil: count the visual tokens, estimate the vision encoder, prefill and decode with the roofline, then choose resolution, frames, quantization and batch size to meet the budget. It ends by showing how one real measurement corrects the estimate.
What you will learn
- The inputs of a VLM latency budget: resized image size, frames, visual tokens, prompt and output length, load and targets
- How to count Qwen2-VL visual tokens and estimate the vision encoder and prefill FLOPs, including the quadratic attention terms
- Why the encoder and prefill are compute-bound and decode is bandwidth-bound on an L4, with TTFT, TPOT and end-to-end estimates
- Which knob moves which term: resolution and frames, 4-bit weights against FP8, output length, and batch size
- A closed-loop batching model to choose the batch size and the number of GPUs for a peak load
- How to measure TTFT and TPOT, compute the implied efficiencies and re-plan
Why this matters
A product manager asks: “Can we answer photo questions with a 7B vision-language model on one NVIDIA L4, in under 4 seconds?” You could rent the GPU, install a server and try. Or you could answer in ten minutes with a pencil. This session does the pencil version, one step at a time, with real numbers from the model’s config and the GPU’s datasheet. Then it shows how to check the pencil against a real measurement.
A latency budget is a target time for one request, split into parts. Each part gets an estimate. If the sum is over the target, you change something and estimate again. This is how serving teams plan GPU purchases and choose model settings before they write code. [Likely]
The previous sessions gave us all the pieces. Session 196 followed one request through the vision encoder, the connector, prefill and decode. Session 197 showed that the vision encoder and prefill are limited by compute, while decode is limited by memory bandwidth. Session 198 put a model behind a server such as vLLM. Here we join them into one calculation for one cheap GPU. In the next session, session 200, you run the server and measure the real numbers.
Why the L4? It is a small, low-power data-center GPU (72 W, one slot) that is common in clouds for inference. It has a lot of compute for its price but little memory bandwidth. That makes the difference between compute-bound and bandwidth-bound stages very clear, so it is a good GPU for learning to budget.
The inputs: what you must know before you estimate
Write down six numbers before you compute anything. Most estimation mistakes come from a wrong input, not from wrong math.
| Input | Our example | Why it matters |
|---|---|---|
| Image size (after the model’s resize) | 1280×960 photo, resized to 1288×952 | sets the number of patches, so the encoder and prefill cost |
| Number of images or video frames | 1 image | multiplies the visual tokens |
| Visual tokens | 1,564 | the LLM sees these in prefill; they also fill the KV cache |
| Prompt text tokens | 60 (system prompt, chat template, question) | small next to the image here |
| Output tokens | 128 | each one is a full decode step |
| Concurrency and load | 2 requests per second at peak | sets the batch size and the number of GPUs |
And the targets. We use a common pair of metrics (see session 81):
- TTFT (time to first token): from the moment the request arrives until the first output token. The user sees “the model has started”. Target: at most 1.0 s, even in a burst.
- E2E (end-to-end latency): until the last token. Target: at most 4.0 s for a 128-token answer.
- TPOT (time per output token): the average time between output tokens after the first. It is not a target here, but E2E depends on it.
The hardware: an NVIDIA L4
From NVIDIA’s datasheet [Certain]: 24 GB of memory, 300 GB/s memory bandwidth, 121 TFLOP/s for dense BF16 or FP16 on the tensor cores and 242 TFLOP/s for dense FP8. (The datasheet prints 242 and 485, which assume 2:4 sparsity; dense numbers are half of that.) The L4 is an Ada Lovelace GPU, so it has FP8 tensor cores, and vLLM supports FP8 W8A8 on it. [Certain]
Nobody reaches the peak. We use two efficiency factors: \(\eta_c\), the fraction of peak compute that real kernels reach, and \(\eta_m\), the fraction of peak bandwidth. We start with \(\eta_c = 0.5\) and \(\eta_m = 0.8\). These are assumptions, not measurements. [Likely] Large matrix multiplies often reach 50–70% of peak and memory-bound kernels 70–90%; the L4’s low power limit may lower the compute number further. Section “Check the estimate” shows how to replace them with your own measured values.
The model: Qwen2-VL-7B-Instruct
We use the same model as in session 197. From its config.json [Certain]: the LLM has 28 layers, width 3,584, MLP width 18,944, 28 query heads and 4 KV heads of size 128, and a vocabulary of 152,064. The vision encoder is a ViT with 32 layers of width 1,280, patch size 14, and a merger that joins 2×2 patches into one token. For video it also joins 2 frames into one (temporal patch size 2). Counting the weights from these numbers:
- LLM linear layers: \(N_{\text{lin}} = 6.525\) B weights. Output layer (lm_head): \(N_{\text{head}} = 0.545\) B. Input embedding: another 0.545 B (not tied).
- Vision encoder linear layers: \(N_v = 0.631\) B. Merger: 0.045 B.
- KV cache: \(2 \times 28 \times 4 \times 128 \times 2 = 57{,}344\) bytes per token in BF16.
Step 1: from pixels to visual tokens
Qwen2-VL first resizes each image so that both sides are multiples of 28 (patch 14 times merge 2). It rounds each side to the nearest multiple, within a minimum and maximum pixel count. [Certain] Then it cuts the image into 14×14 patches and the merger joins each 2×2 block into one visual token:
- Read the symbols
- \(H', W'\)
- height and width after the resize, both multiples of 28
- \(n_p\)
- number of patches the vision encoder processes
- \(n_{\text{vis}}\)
- number of visual tokens the LLM sees, after the 2×2 merge
- \(G\)
- number of temporal groups. Two video frames become one group, so a 32-frame video gives 16 groups
For our photo: 1280×960 becomes 1288×952 (46 and 34 blocks of 28). So \(n_p = 92 \times 68 = 6{,}256\) patches and \(n_{\text{vis}} = 1{,}564\) visual tokens. With 60 text tokens, the prompt has \(T = 1{,}624\) tokens. The image is 96% of the prompt. This is normal for VLMs, and it is why image size is the first knob of a VLM budget.
Step 2: estimate each stage with the roofline
Every stage does some arithmetic and reads some bytes. Its time is at least the larger of the two limits (the roofline model, see session 82):
- Read the symbols
- \(F\)
- floating-point operations (FLOPs) of the stage
- \(Q\)
- bytes the stage reads from GPU memory: weights, and the KV cache in the LLM
- \(P,\ B_w\)
- peak compute (121 TFLOP/s in BF16) and peak bandwidth (300 GB/s) of the L4
- \(\eta_c,\ \eta_m\)
- efficiency factors: the fraction of each peak that real kernels reach
The vision encoder
A linear layer with \(N\) weights costs \(2N\) FLOPs per token (one multiply and one add per weight). Attention adds a term that grows with the square of the sequence length. In the Qwen2-VL encoder, each temporal group attends only to its own patches (the code builds one attention segment per group). [Certain] So:
- Read the symbols
- \(N_v\)
- weights in the encoder’s linear layers: 0.631 B
- \(L_v,\ d_v\)
- encoder layers (32) and width (1,280)
- \(4 L_v d_v (\cdot)^2\)
- attention scores \(QK^\top\) and the weighted sum with \(V\): 2 FLOPs each per pair of patches per channel, in every layer
For our photo: linear \(2 \times 0.631\text{ B} \times 6{,}256 = 7.89\) TFLOP, attention \(4 \times 32 \times 1{,}280 \times 6{,}256^2 = 6.41\) TFLOP, merger 0.14 TFLOP. Total 14.44 TFLOP. At 60.5 TFLOP/s: 239 ms. The encoder reads only 1.35 GB of weights (5.6 ms), so it is compute-bound by a factor of about 40. Note the attention term: it grows with \(n_p^2\), so doubling both sides of the image multiplies it by 16.
Prefill
Prefill runs all \(T\) prompt tokens through the LLM in one pass:
- Read the symbols
- \(T\)
- prompt length in tokens: visual tokens plus text tokens
- \(L,\ d\)
- LLM layers (28) and width (3,584)
- \(2 L d T^2\)
- causal attention: like the encoder term, but each token only looks back, so half the pairs
- \(2 N_{\text{head}}\)
- the lm_head runs only for the last prompt token, to produce the first output token
For \(T = 1{,}624\): linear 21.19 TFLOP, attention 0.53 TFLOP, total 21.72 TFLOP. At 60.5 TFLOP/s: 359 ms. Compute-bound again: the 14.1 GB of weights take 59 ms to read. Here attention is small, because the LLM is wide and the prompt is short.
So the first token appears after:
(We leave out the image download, JPEG decoding and resize on the CPU, and the network. They are real, so measure them; see the last section.)
Decode
Each new token needs one pass through the LLM. At batch size \(b\), the step reads all weights once and the KV cache of every request, and does \(2N\) FLOPs per weight for each of the \(b\) tokens:
- Read the symbols
- \(\beta\)
- bits per LLM linear weight: 16 in BF16, 8 in FP8, 4.25 for 4-bit AWQ with group size 128 (the scales add 0.25 bit)
- \(2 N_{\text{head}}\)
- bytes of the lm_head, kept in BF16
- \(k\)
- KV-cache bytes per token: 57,344 for this model
- \(\bar T\)
- average context length during decode: the prompt plus half the output, \(1{,}624 + 64 = 1{,}688\)
In BF16 at \(b = 1\): the step reads \(14.14 + 0.10 = 14.24\) GB, so 59.3 ms per token. The compute side is only 0.24 ms. Decode is bandwidth-bound by a factor of about 250. The whole answer:
Over the 4.0 s target by a factor of two. And 93% of the time is decode. The image, which looked like the big cost, is not the problem here.
How to read Figure 1
Time runs from left to right. The two dashed lines are the targets: TTFT at 1.0 s and E2E at 4.0 s. Steps 1–3 build the BF16 estimate: the first token comes at 0.60 s (fine), but the answer ends at 8.19 s. Step 4 switches the LLM weights to 4-bit: only the green decode bar shrinks, to 3.08 s in total. Step 5 lowers the image to 896×672: now the orange (encoder) and dark red (prefill) bars shrink, and the total is 2.73 s. Each change moves the stage it targets, and only that stage.
Step 3: choose the knobs
Now we have four knobs. Each one acts on a different term of the budget:
| Knob | Moves | Effect in our example | Cost |
|---|---|---|---|
| Image resolution | encoder (\(\propto n_p\) and \(n_p^2\)), prefill (\(\propto T\)), KV cache | \(1288{\times}952 \to 896{\times}672\): TTFT \(598 \to 272\) ms | small text and small objects get harder to read [Likely] |
| Number of frames (video) | same as resolution, times \(G\) | 32 frames at 448×448: 4,096 visual tokens, TTFT 1.35 s | short events can fall between frames |
| LLM weight precision | decode bytes; FP8 also doubles prefill compute | BF16 \(\to\) 4-bit: TPOT \(59.3 \to 19.4\) ms | a small quality risk; test it (session 197) |
| Batch size | throughput; adds waiting time per request | see Step 4 | each request waits for the others’ prefill |
| Output length | number of decode steps | \(128 \to 96\) tokens: E2E \(2.73 \to 2.11\) s | shorter answers; set by the prompt or max_tokens |
The precision options behave very differently on the L4, and this is the most useful lesson of the whole page:
- 4-bit weights, 16-bit activations (W4A16, e.g. AWQ). Decode reads 4.25 bits per weight: 4.56 GB per step instead of 14.14 GB, so TPOT drops from 59.3 to 19.4 ms. The multiplications still run in FP16, so prefill does not get faster. (Real 4-bit kernels can even make prefill a little slower, because they also unpack the weights. [Likely])
- FP8 weights and activations (W8A8). Decode reads 1 byte per weight: 7.62 GB, so TPOT is 32.1 ms. But the tensor cores run FP8 at twice the BF16 rate, so the LLM part of prefill halves: \(359 \to 184\) ms.
- So: W4A16 wins when decode dominates (long answers, few images). FP8 wins when prefill dominates (many images, video, long documents, short answers). With our 128-token answers, W4A16 is the right choice.
Our first plan: 4-bit LLM weights (vision encoder stays BF16), images resized to 896×672. That gives \(n_{\text{vis}} = 32 \times 24 = 768\) tokens, \(T = 828\), \(F_{\text{enc}} = 5.49\) TFLOP (91 ms), \(F_{\text{pre}} = 10.94\) TFLOP (181 ms), TTFT 272 ms, TPOT 19.2 ms, E2E 2.73 s. It passes, for one user at a time. Try the knobs in Figure 2.
How to read Figure 2
The top bar is the time of one request, split into encoder (orange), prefill (dark red) and decode (green); the dashed lines are the 1.0 s TTFT and 4.0 s E2E targets. The bottom bar is GPU memory. Start at the defaults (BF16, 1288×952): 8.19 s. Click 4-bit: 3.08 s. Lower the resolution to 896: 2.73 s. Then set the long side to 448 and choose 32 frames (square 448×448 frames): TTFT jumps over 1 s, because 4,096 visual tokens must be encoded and prefilled; now click FP8 and see TTFT fall to 0.90 s while decode gets slower. Try the 2B model too: its decode is fast, and the encoder (the same 0.63 B ViT) becomes the largest part of TTFT.
The 2B model has the same vision encoder, so \(t_{\text{enc}}\) is still 238 ms at 1288×952. Its LLM is much smaller: width 1,536, MLP width 8,960, 2 KV heads, tied embeddings. So \(N_{\text{lin}} = 1.310\) B and \(N_{\text{head}} = 0.233\) B. Prefill: 4.48 TFLOP, 74 ms. TTFT = 312 ms. Decode in BF16: \(2 \times (1.310 + 0.233) = 3.09\) GB plus 0.05 GB of KV per step, so 13.1 ms per token. E2E = \(0.31 + 128 \times 0.0131 = 1.98\) s, even in BF16 and at full resolution. For a small VLM, the vision encoder is 76% of TTFT. This is why small VLMs try hard to cut visual tokens or use smaller encoders (session 183).
- input: model config, GPU peaks \(P, B_w\), factors \(\eta_c, \eta_m\), image size, frames, text tokens, output tokens, targets
- resize as the model does; \(n_p, n_{\text{vis}} \leftarrow\) formula (199.1); \(T \leftarrow n_{\text{vis}} + n_{\text{text}}\)
- \(t_{\text{enc}} \leftarrow \max(F_{\text{enc}}/\eta_c P,\ Q_{\text{enc}}/\eta_m B_w)\) // (199.3); compute-bound
- \(t_{\text{pre}} \leftarrow \max(F_{\text{pre}}/\eta_c P,\ Q_{\text{pre}}/\eta_m B_w)\) // (199.4); use the FP8 rate for FP8 layers
- \(\text{TPOT} \leftarrow\) formula (199.6) at the planned batch size // bandwidth-bound at small b
- \(\text{TTFT} \leftarrow t_{\text{enc}} + t_{\text{pre}}\); \(\text{E2E} \leftarrow \text{TTFT} + N_{\text{out}}\,\text{TPOT}\)
- check memory: weights + KV cache \(\le\) usable memory (199.9)
- while a target fails: find the largest term; apply the knob for that term (resolution or frames for encoder and prefill; precision or output length for decode; FP8 for compute-bound prefill); go to step 2
- return the plan and its estimated TTFT, TPOT, E2E and memory
Step 4: batch size, throughput and the number of GPUs
One request at a time wastes the GPU. During decode at \(b = 1\), the tensor cores are idle more than 99% of the time, waiting for weights. If 3 requests decode together, the GPU reads the weights once for all 3. So the step costs almost the same, and we get 3 tokens instead of 1. Figure 3 shows this on the L4’s roofline. The arithmetic intensity (FLOPs per byte read) of a decode step is about:
Decode becomes compute-bound only when \(I_{\text{dec}}\) passes the ridge point of 403. For BF16 weights (\(\beta = 16\)), that needs \(b \approx 403\); for 4-bit weights, \(b \approx 107\), ignoring the KV cache. Long before that, the KV-cache term in the denominator grows and holds the intensity down. So for a VLM on an L4, decode is bandwidth-bound at every batch size you will use.
How to read Figure 3
The sloped line is the bandwidth limit, the flat line the compute limit; they meet at the ridge point (403 FLOPs per byte in BF16). A point left of the ridge is bandwidth-bound: moving it right (more FLOPs per byte) makes it faster in FLOP/s. The encoder and prefill sit far to the right: compute-bound. Decode at \(b = 1\) sits at intensity 1.0 in BF16 (3.1 with 4-bit weights). Raise \(b\): the decode point moves up the slope, so the GPU does more useful work per second, but even at \(b = 256\) it stays left of the ridge.
What does batching cost each user? With continuous batching (session 135), a new request’s prefill runs between decode steps, and while it runs, everyone else’s decode waits. A simple model is a closed loop: \(b\) requests are always in flight, and each one, during its life, sees the prefill of all \(b\) requests (its own and the others’) and all \(N_{\text{out}}\) shared decode steps:
- Read the symbols
- \(t_{\text{pf}}\)
- GPU time for the encoder plus prefill of one request: the TTFT at \(b = 1\)
- \(X(b)\)
- throughput: finished requests per second on one GPU
- \(\text{TTFT}_{\text{burst}}\)
- worst case: \(b\) requests arrive at the same moment, and the last one waits for all prefills
This is a rough, pessimistic model. [Likely] Chunked prefill (Sarathi-Serve) mixes prefill pieces into decode steps, and because decode leaves the tensor cores mostly idle, part of the prefill work hides there. Real E2E at \(b \gt 1\) is often somewhat better. But the model has the right shape, and it is easy to compute. The number of GPUs for a peak load \(\lambda\) is then:
How to read Figure 4
The green line (E2E) and the dark red line (burst TTFT) rise with \(b\); the dashed lines are their targets (4 s and 1 s). The orange bars are throughput, which rises with \(b\) because decode steps are shared. The best batch size is the largest one where both lines are under their targets. For the default plan, \(b^{*} = 3\): E2E 3.33 s, burst TTFT 0.81 s, 0.90 requests per second per L4, so 3 L4s for 2 requests per second. Here the TTFT target, not E2E, stops the batch from growing. Switch to 1288×952 and the prefill is so long that only \(b = 1\) fits, which needs 7 L4s.
Plan: 4-bit LLM, 896×672 images, so \(t_{\text{pf}} = 0.272\) s. TPOT grows slowly with \(b\), because only the KV cache adds bytes: 19.20 ms at \(b = 1\), 19.63 ms at \(b = 3\), 19.84 ms at \(b = 4\).
- \(b = 3\): E2E \(= 3 \times 0.272 + 128 \times 0.01963 = 3.33\) s (pass); burst TTFT \(= 0.81\) s (pass); \(X = 3 / 3.33 = 0.90\) requests per second.
- \(b = 4\): E2E 3.63 s (pass), but burst TTFT \(= 1.09\) s (fail).
- So \(b^{*} = 3\), and \(n_{\text{GPU}} = \lceil 2 / 0.90 \rceil = 3\) L4s.
Memory is not the limit here. Weights: \(3.47\) GB (4-bit LLM) + \(2.18\) GB (embedding and lm_head in BF16) + \(1.35\) GB (vision) = 7.00 GB. KV cache per request at the end of the answer: \((828 + 128) \times 57{,}344 = 0.055\) GB. With 21.6 GB usable, about 266 requests would fit. It is a different story for long video: a 64-frame request at 448×448 has 8,192 visual tokens and needs 0.48 GB of KV, so a BF16 model (16.6 GB of weights) fits only 10 such requests.
The factor 0.9 is vLLM’s default gpu_memory_utilization; the rest is left for activations, the CUDA context and fragmentation. [Likely] vLLM also reserves activation memory for the largest batch it profiles, so the real KV capacity is a bit smaller than this formula says.
Step 5: check the estimate with a real measurement
An estimate is a hypothesis. You test it with one measurement, and then you use the measurement to fix the two numbers you guessed: \(\eta_c\) and \(\eta_m\). Measure TTFT and TPOT for one request at \(b = 1\), with the exact plan (same resolution, same precision, same prompt). Then:
- Read the symbols
- \(\hat\eta_c\)
- measured compute efficiency of the encoder plus prefill (valid because both are compute-bound)
- \(\hat\eta_m\)
- measured bandwidth efficiency of a decode step (valid because decode is bandwidth-bound)
- \(Q_{\text{dec}}\)
- bytes per decode step: 4.61 GB for the 4-bit plan at \(\bar T = 892\)
Then put \(\hat\eta_c\) and \(\hat\eta_m\) back into formulas (199.2)–(199.10) and redo the plan. If one measured number is far from every reasonable efficiency (for example \(\hat\eta_m = 0.2\)), the model of that stage is wrong, not just the factor: look for a missing cost.
The numbers in this box are made-up example inputs to show the arithmetic, not a real L4 measurement. Suppose the lab measures TTFT = 420 ms and TPOT = 24.0 ms for the 4-bit, 896×672 plan at \(b = 1\).
- \(\hat\eta_c = 16.43\text{ TFLOP} / (121\text{ TFLOP/s} \times 0.420\text{ s}) = 0.32\) (we assumed 0.5).
- \(\hat\eta_m = 4.61\text{ GB} / (300\text{ GB/s} \times 0.0240\text{ s}) = 0.64\) (we assumed 0.8).
- New E2E at \(b = 1\): \(0.42 + 128 \times 0.024 = 3.49\) s. At \(b = 2\): burst TTFT 0.84 s and E2E 3.95 s, both pass. At \(b = 3\): burst TTFT 1.26 s, fail.
- So \(b^{*} = 2\), \(X = 0.507\) requests per second, and the plan now needs \(\lceil 2 / 0.507 \rceil = 4\) L4s instead of 3.
One measurement changed the purchase by one GPU. That is why you always measure before you commit.
How to read Figure 5
Each pair of bars compares the estimate with \(\eta_c = 0.5\), \(\eta_m = 0.8\) (grey) and your measurement (colored). The text below the bars shows the efficiency each measurement implies. Values between about 0.3 and 0.9 are plausible; outside that range, suspect a missing cost (CPU preprocessing, a slow kernel, a wrong precision) rather than a slow GPU. The last line is the new plan: \(b^{*}\), E2E and the number of L4s. Move TTFT up to 600 ms and watch \(b^{*}\) fall to 1.
How to measure, and what usually differs
Two rules make the measurement useful. First, measure the same thing you estimated: one request, \(b = 1\), fixed resolution and output length (set max_tokens and ignore the end-of-sequence token, so every run writes exactly 128 tokens). Second, measure each part separately, so you know which estimate was wrong. The next session, session 200, does this with vLLM’s benchmark tool, which reports TTFT, TPOT and their percentiles. [Likely] The code below shows the idea with plain PyTorch.
Where the estimate and the measurement usually differ:
- CPU preprocessing. JPEG decoding, resizing and normalizing a large photo run on the CPU, before the GPU sees anything. They add directly to TTFT. Measure them alone.
- Lower compute efficiency for small or odd shapes. 768 tokens is a small matrix multiply for a 72 W GPU; \(\eta_c\) for the encoder and for prefill can be quite different. Time them separately.
- Kernel launch overhead in decode. A decode step launches hundreds of small kernels. Without CUDA graphs, launch time can be a large part of a 20 ms step. [Likely] vLLM uses CUDA graphs for decode by default. [Likely]
- Warm-up and clocks. The first request compiles kernels and allocates memory. Run a few warm-up requests and report the median of many.
- Mixed precision you did not plan. A 4-bit checkpoint whose vision tower is BF16 is expected; one whose lm_head is also 4-bit, or whose KV cache is FP8, changes \(Q_{\text{dec}}\).
- input: a running server or model with the planned settings; the estimate of Algorithm 1
- run 5 warm-up requests; discard them
- for 30 requests at \(b = 1\): record CPU preprocessing time, TTFT and the time of each later token
- take medians: \(\text{TTFT}_{\text{meas}}\) (GPU part only), \(\text{TPOT}_{\text{meas}}\)
- \(\hat\eta_c, \hat\eta_m \leftarrow\) formula (199.12) // sanity check: both between about 0.3 and 0.9
- redo Algorithm 1 and formula (199.10) with \(\hat\eta_c, \hat\eta_m\) and the measured CPU time
- run a load test at the chosen \(b^{*}\) and peak load; compare p50 and p95 E2E with the targets // the closed-loop model is only a first guess
- return the calibrated plan and the number of GPUs
The runnable code below has two parts. Part 1 is the budget estimator of this page in plain Python; it prints the numbers of the worked examples and runs anywhere. Part 2 measures the two peaks your own GPU actually reaches (a large BF16 matrix multiply and a large memory copy), which gives you a first \(\eta_c\) and \(\eta_m\) before you even load a model. On a machine without a CUDA GPU it runs on the CPU, so the printed numbers are only meaningful on the GPU you plan to use.
import math, time, torch # ---------- Part 1: the budget estimator (formulas 199.1-199.10) ---------- P16, P8, BW = 121e12, 242e12, 300e9 # NVIDIA L4: dense BF16, dense FP8, bytes/s VIS = dict(L=32, d=1280, lin=32 * (1280*3840 + 1280*1280 + 2*1280*5120) + 3*2*14*14*1280) LLM = dict(L=28, d=3584, ff=18944, kvh=4, hd=128, vocab=152064) # Qwen2-VL-7B config.json d, L = LLM["d"], LLM["L"] N_LIN = L * (2*d*d + 2*d*LLM["kvh"]*LLM["hd"] + 3*d*LLM["ff"]) N_HEAD = LLM["vocab"] * d N_MER = 5120*5120 + 5120*d KV = 2 * L * LLM["kvh"] * LLM["hd"] * 2 # bytes per token BITS = {"bf16": 16, "fp8": 8, "w4": 4.25} def resize(h, w, f=28): return max(f, round(h / f) * f), max(f, round(w / f) * f) def budget(h, w, frames=1, n_text=60, n_out=128, prec="bf16", b=1, ec=0.5, em=0.8): H, W = resize(h, w) G = frames // 2 if frames > 1 else 1 per = (H // 14) * (W // 14) # patches per temporal group n_p = G * per; n_vis = n_p // 4; T = n_vis + n_text # (199.1) F_enc = 2*VIS["lin"]*n_p + 4*VIS["L"]*VIS["d"]*G*per**2 + 2*N_MER*n_vis # (199.3) t_enc = max(F_enc / (ec*P16), 2*(VIS["lin"] + N_MER) / (em*BW)) P = P8 if prec == "fp8" else P16 w_bytes = BITS[prec] / 8 * N_LIN + 2 * N_HEAD t_pre = max(2*N_LIN*T / (ec*P) + (2*L*d*T*T + 2*N_HEAD) / (ec*P16), (w_bytes + KV*T) / (em*BW)) # (199.4) Tb = T + n_out / 2 tpot = max(b*2*N_LIN / (ec*P) + b*(2*N_HEAD + 4*L*d*Tb) / (ec*P16), (w_bytes + b*KV*Tb) / (em*BW)) # (199.6) t_pf = t_enc + t_pre e2e = b * t_pf + n_out * tpot # (199.9) return dict(n_vis=n_vis, T=T, t_enc=t_enc, t_pre=t_pre, ttft=t_pf, tpot=tpot, e2e=e2e, burst=b * t_pf, X=b / e2e) def plan(lam=2.0, ttft_max=1.0, e2e_max=4.0, **kw): best = None for b in range(1, 65): r = budget(b=b, **kw) if r["e2e"] <= e2e_max and r["burst"] <= ttft_max: best = (b, r) if best is None: return None b, r = best return b, round(r["e2e"], 2), round(r["X"], 3), math.ceil(lam / r["X"]) # (199.10) for name, kw in [("BF16 1288x952", dict(h=1280, w=960)), ("4-bit 1288x952", dict(h=1280, w=960, prec="w4")), ("4-bit 896x672", dict(h=896, w=672, prec="w4"))]: r = budget(**kw) print(f"{name:15s} vis={r['n_vis']:5d} TTFT={r['ttft']*1e3:6.1f} ms " f"TPOT={r['tpot']*1e3:5.1f} ms E2E={r['e2e']:5.2f} s plan(b*, E2E, X, GPUs)={plan(**kw)}") # BF16 1288x952 vis= 1564 TTFT= 597.8 ms TPOT= 59.3 ms E2E= 8.19 s plan(b*, E2E, X, GPUs)=None # 4-bit 1288x952 vis= 1564 TTFT= 597.8 ms TPOT= 19.4 ms E2E= 3.08 s plan(b*, E2E, X, GPUs)=(1, 3.08, 0.325, 7) # 4-bit 896x672 vis= 768 TTFT= 271.6 ms TPOT= 19.2 ms E2E= 2.73 s plan(b*, E2E, X, GPUs)=(3, 3.33, 0.902, 3) # ---------- Part 2: what does YOUR GPU reach? (first guess for eta_c, eta_m) ---------- def timed(fn, reps=20): sync = torch.cuda.synchronize if torch.cuda.is_available() else (lambda: None) for _ in range(3): # warm-up fn() sync(); t0 = time.perf_counter() for _ in range(reps): fn() sync() return (time.perf_counter() - t0) / reps torch.manual_seed(0) dev = "cuda" if torch.cuda.is_available() else "cpu" n = 4096 if dev == "cuda" else 1024 dt = torch.bfloat16 if dev == "cuda" else torch.float32 # BF16 matmul is slow on many CPUs a = torch.randn(n, n, device=dev, dtype=dt) bm = torch.randn(n, n, device=dev, dtype=dt) t_mm = timed(lambda: a @ bm) x = torch.zeros(256 * 2**20 // 2, device=dev, dtype=torch.bfloat16) # 256 MB y = torch.empty_like(x) t_cp = timed(lambda: y.copy_(x)) flops = 2 * n**3 / t_mm bw = 2 * x.numel() * x.element_size() / t_cp # a copy reads and writes print(f"{dev}: matmul {flops/1e12:.1f} TFLOP/s, copy {bw/1e9:.0f} GB/s") if dev == "cuda": print(f"if this is an L4: eta_c ~ {flops/P16:.2f}, eta_m ~ {bw/BW:.2f}")
Part 1 prints exactly the numbers used on this page (they are shown as comments). Part 2 gives upper bounds for \(\eta_c\) and \(\eta_m\): a single large, square matrix multiply is the best case for the tensor cores, and the encoder and prefill shapes reach less. For the full check, time the real model: for example, with Hugging Face transformers, time generate with max_new_tokens=1 (TTFT) and with max_new_tokens=129, min_new_tokens=129; the difference divided by 128 is TPOT. Synchronize the GPU before reading the clock.
Where this is used
| Application | What dominates the budget | First knob to try |
|---|---|---|
| Photo or screenshot question answering (chat) | decode, for answers of 100+ tokens | 4-bit LLM weights; shorter answers |
| Document and invoice extraction | encoder and prefill (high resolution for small text), short JSON output | FP8 for prefill; crop to the useful region instead of lowering resolution |
| Video question answering and Video-RAG | visual tokens from many frames: encoder, prefill and KV memory | fewer frames, lower frame resolution, token merging; FP8 |
| Key-moment detection on live video | steady encoder load per second of video, plus a strict TTFT | frame rate and resolution per stream; streams per GPU from the encoder cost |
| Video classification and action recognition with a VLM | encoder and prefill; outputs are a few tokens | frames and resolution; FP8; decode hardly matters |
| Video retrieval and fingerprinting (encoder only) | the vision encoder alone, compute-bound | batch many frames; INT8 or FP8 encoder; no decode at all |
| Capacity planning and cost per request | throughput \(X(b^{*})\) | formula (199.10): \(\text{GPUs} = \lceil \text{load} / \text{throughput} \rceil\) |
Common mistakes
- Using the sparsity number from the datasheet. The L4’s “242 TFLOPS BF16” assumes 2:4 sparse weights. Dense models get at most 121.
- Counting visual tokens from the original image. The model resizes first. 1280×960 becomes 1288×952 in Qwen2-VL; other models tile or pad, and the count can change a lot (session 180).
- Forgetting that the encoder’s attention is quadratic. At 6,256 patches, attention is 44% of the encoder FLOPs. Doubling both image sides gives 4× the linear cost but 16× the attention cost.
- Quantizing to speed up TTFT with 4-bit weights. W4A16 makes decode faster, not prefill. For a prefill-heavy request, use FP8 (W8A8) or fewer visual tokens.
- Reporting TPOT from a run that stopped early. If the model emits the end-of-sequence token after 20 tokens, your “128-token” E2E is wrong. Force a fixed output length when you benchmark.
- Budgeting with \(b = 1\) and serving with \(b = 16\). At higher batch, each request waits for others’ prefill (199.9). Check TTFT in a burst, not only alone.
- Ignoring CPU time. Image download, decoding and resizing are not in the GPU roofline but are in the user’s TTFT.
- Trusting the estimate without a measurement. The efficiency factors are guesses until you measure them (199.12); in the example, one measurement changed the plan from 3 to 4 GPUs.
Interview questions
Why is decode bandwidth-bound and prefill compute-bound, using numbers?basic
Estimate TTFT for Qwen2-VL-7B on an L4 with one 1280×960 photo.core
The E2E target is missed by 2×. Which knob do you turn first?core
FP8 or 4-bit AWQ on an L4: which is faster?core
How do you choose the batch size for a latency target?core
Why is the vision encoder a bigger share of TTFT for a 2B VLM than for a 7B one?basic
Your measured TPOT is 2× the estimate. What do you check?advanced
A 32-frame video request misses the 1 s TTFT target. What are your options?advanced
Can a 7B VLM ever reach 15 ms per token on an L4?advanced
How do you turn one measurement into a better plan?core
Exercises
- A 1920×1080 screenshot goes into Qwen2-VL. After the resize, how many visual tokens does it give? (Round each side to the nearest multiple of 28.)
- Using Figure 2 or the formulas, estimate E2E for Qwen2-VL-2B in BF16 with the 1288×952 photo, 60 text tokens and 128 output tokens.
- For the 4-bit, 896×672 plan, the product team allows answers of only 96 tokens. What is the new E2E at \(b = 1\)?
- With formula (199.8), at what batch size would decode become compute-bound on an L4 for 4-bit weights, if you ignore the KV cache and the lm_head?
- A 64-frame video at 448×448 goes to the BF16 7B model. How much KV cache does one request need at the end of a 128-token answer, and how many such requests fit in 21.6 GB?
- Required TPOT is 15 ms for the 4-bit 7B plan. What bandwidth would that need? Is it possible on an L4?
Answerscheck
Key takeaways
- Write down the inputs first: resized image size, frames, visual tokens, text tokens, output tokens, load and targets. The image is usually most of the prompt (199.1).
- Each stage takes the larger of compute time and memory time (199.2). The vision encoder and prefill are compute-bound; decode is bandwidth-bound at every practical batch size on an L4 (199.8).
- \(\text{TTFT} \approx t_{\text{enc}} + t_{\text{pre}}\); \(\text{E2E} \approx \text{TTFT} + N_{\text{out}} \times \text{TPOT}\) (199.5, 199.7). For a chat answer, decode usually dominates; for video and documents, prefill does.
- Match the knob to the term: resolution and frames for encoder and prefill; 4-bit weights or shorter answers for decode; FP8 for compute-bound prefill.
- Batching raises throughput but adds prefill waiting; pick the largest batch that meets both targets, then \(\text{GPUs} = \lceil \text{load} / \text{throughput} \rceil\) (199.9, 199.10).
- The efficiency factors are guesses. Measure TTFT and TPOT once, compute the implied efficiencies (199.12), and re-plan.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- NVIDIA (2023). NVIDIA L4 Tensor Core GPU (product page and datasheet)
- vLLM project (docs). vLLM documentation: FP8 W8A8 quantization