LLM & VLM courseBlog
199Track 7: Vision and VLMs · session 37 of 38

A latency budget on an NVIDIA L4: worked example

Can a 7B vision-language model answer photo questions in under 4 seconds on one NVIDIA L4? This session answers with a pencil: count the visual tokens, estimate the vision encoder, prefill and decode with the roofline, then choose resolution, frames, quantization and batch size to meet the budget. It ends by showing how one real measurement corrects the estimate.

Time: 2 to 3 hours (reading 1.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions on inference metrics (81), the roofline (82), batching (84, 135), the VLM inference path (196) and quantizing VLMs (197)

What you will learn

  • The inputs of a VLM latency budget: resized image size, frames, visual tokens, prompt and output length, load and targets
  • How to count Qwen2-VL visual tokens and estimate the vision encoder and prefill FLOPs, including the quadratic attention terms
  • Why the encoder and prefill are compute-bound and decode is bandwidth-bound on an L4, with TTFT, TPOT and end-to-end estimates
  • Which knob moves which term: resolution and frames, 4-bit weights against FP8, output length, and batch size
  • A closed-loop batching model to choose the batch size and the number of GPUs for a peak load
  • How to measure TTFT and TPOT, compute the implied efficiencies and re-plan

Why this matters

A product manager asks: “Can we answer photo questions with a 7B vision-language model on one NVIDIA L4, in under 4 seconds?” You could rent the GPU, install a server and try. Or you could answer in ten minutes with a pencil. This session does the pencil version, one step at a time, with real numbers from the model’s config and the GPU’s datasheet. Then it shows how to check the pencil against a real measurement.

A latency budget is a target time for one request, split into parts. Each part gets an estimate. If the sum is over the target, you change something and estimate again. This is how serving teams plan GPU purchases and choose model settings before they write code. [Likely]

The previous sessions gave us all the pieces. Session 196 followed one request through the vision encoder, the connector, prefill and decode. Session 197 showed that the vision encoder and prefill are limited by compute, while decode is limited by memory bandwidth. Session 198 put a model behind a server such as vLLM. Here we join them into one calculation for one cheap GPU. In the next session, session 200, you run the server and measure the real numbers.

Why the L4? It is a small, low-power data-center GPU (72 W, one slot) that is common in clouds for inference. It has a lot of compute for its price but little memory bandwidth. That makes the difference between compute-bound and bandwidth-bound stages very clear, so it is a good GPU for learning to budget.

The inputs: what you must know before you estimate

Write down six numbers before you compute anything. Most estimation mistakes come from a wrong input, not from wrong math.

InputOur exampleWhy it matters
Image size (after the model’s resize)1280×960 photo, resized to 1288×952sets the number of patches, so the encoder and prefill cost
Number of images or video frames1 imagemultiplies the visual tokens
Visual tokens1,564the LLM sees these in prefill; they also fill the KV cache
Prompt text tokens60 (system prompt, chat template, question)small next to the image here
Output tokens128each one is a full decode step
Concurrency and load2 requests per second at peaksets the batch size and the number of GPUs

And the targets. We use a common pair of metrics (see session 81):

  • TTFT (time to first token): from the moment the request arrives until the first output token. The user sees “the model has started”. Target: at most 1.0 s, even in a burst.
  • E2E (end-to-end latency): until the last token. Target: at most 4.0 s for a 128-token answer.
  • TPOT (time per output token): the average time between output tokens after the first. It is not a target here, but E2E depends on it.

The hardware: an NVIDIA L4

From NVIDIA’s datasheet [Certain]: 24 GB of memory, 300 GB/s memory bandwidth, 121 TFLOP/s for dense BF16 or FP16 on the tensor cores and 242 TFLOP/s for dense FP8. (The datasheet prints 242 and 485, which assume 2:4 sparsity; dense numbers are half of that.) The L4 is an Ada Lovelace GPU, so it has FP8 tensor cores, and vLLM supports FP8 W8A8 on it. [Certain]

Nobody reaches the peak. We use two efficiency factors: \(\eta_c\), the fraction of peak compute that real kernels reach, and \(\eta_m\), the fraction of peak bandwidth. We start with \(\eta_c = 0.5\) and \(\eta_m = 0.8\). These are assumptions, not measurements. [Likely] Large matrix multiplies often reach 50–70% of peak and memory-bound kernels 70–90%; the L4’s low power limit may lower the compute number further. Section “Check the estimate” shows how to replace them with your own measured values.

The model: Qwen2-VL-7B-Instruct

We use the same model as in session 197. From its config.json [Certain]: the LLM has 28 layers, width 3,584, MLP width 18,944, 28 query heads and 4 KV heads of size 128, and a vocabulary of 152,064. The vision encoder is a ViT with 32 layers of width 1,280, patch size 14, and a merger that joins 2×2 patches into one token. For video it also joins 2 frames into one (temporal patch size 2). Counting the weights from these numbers:

  • LLM linear layers: \(N_{\text{lin}} = 6.525\) B weights. Output layer (lm_head): \(N_{\text{head}} = 0.545\) B. Input embedding: another 0.545 B (not tied).
  • Vision encoder linear layers: \(N_v = 0.631\) B. Merger: 0.045 B.
  • KV cache: \(2 \times 28 \times 4 \times 128 \times 2 = 57{,}344\) bytes per token in BF16.

Step 1: from pixels to visual tokens

Qwen2-VL first resizes each image so that both sides are multiples of 28 (patch 14 times merge 2). It rounds each side to the nearest multiple, within a minimum and maximum pixel count. [Certain] Then it cuts the image into 14×14 patches and the merger joins each 2×2 block into one visual token:

\[ n_{p} \;=\; G \cdot \frac{H'}{14} \cdot \frac{W'}{14}, \qquad n_{\text{vis}} \;=\; \frac{n_p}{4}, \qquad G = \begin{cases} 1 & \text{one image} \\ F/2 & \text{video of } F \text{ frames} \end{cases} \tag{199.1} \]
Read the symbols
\(H', W'\)
height and width after the resize, both multiples of 28
\(n_p\)
number of patches the vision encoder processes
\(n_{\text{vis}}\)
number of visual tokens the LLM sees, after the 2×2 merge
\(G\)
number of temporal groups. Two video frames become one group, so a 32-frame video gives 16 groups

For our photo: 1280×960 becomes 1288×952 (46 and 34 blocks of 28). So \(n_p = 92 \times 68 = 6{,}256\) patches and \(n_{\text{vis}} = 1{,}564\) visual tokens. With 60 text tokens, the prompt has \(T = 1{,}624\) tokens. The image is 96% of the prompt. This is normal for VLMs, and it is why image size is the first knob of a VLM budget.

Step 2: estimate each stage with the roofline

Every stage does some arithmetic and reads some bytes. Its time is at least the larger of the two limits (the roofline model, see session 82):

\[ t_{\text{stage}} \;\approx\; \max\!\left( \frac{F}{\eta_c\, P},\; \frac{Q}{\eta_m\, B_w} \right) \tag{199.2} \]
Read the symbols
\(F\)
floating-point operations (FLOPs) of the stage
\(Q\)
bytes the stage reads from GPU memory: weights, and the KV cache in the LLM
\(P,\ B_w\)
peak compute (121 TFLOP/s in BF16) and peak bandwidth (300 GB/s) of the L4
\(\eta_c,\ \eta_m\)
efficiency factors: the fraction of each peak that real kernels reach

The vision encoder

A linear layer with \(N\) weights costs \(2N\) FLOPs per token (one multiply and one add per weight). Attention adds a term that grows with the square of the sequence length. In the Qwen2-VL encoder, each temporal group attends only to its own patches (the code builds one attention segment per group). [Certain] So:

\[ F_{\text{enc}} \;\approx\; 2\, N_v\, n_p \;+\; 4\, L_v\, d_v\, G \left(\frac{n_p}{G}\right)^{2} \tag{199.3} \]
Read the symbols
\(N_v\)
weights in the encoder’s linear layers: 0.631 B
\(L_v,\ d_v\)
encoder layers (32) and width (1,280)
\(4 L_v d_v (\cdot)^2\)
attention scores \(QK^\top\) and the weighted sum with \(V\): 2 FLOPs each per pair of patches per channel, in every layer

For our photo: linear \(2 \times 0.631\text{ B} \times 6{,}256 = 7.89\) TFLOP, attention \(4 \times 32 \times 1{,}280 \times 6{,}256^2 = 6.41\) TFLOP, merger 0.14 TFLOP. Total 14.44 TFLOP. At 60.5 TFLOP/s: 239 ms. The encoder reads only 1.35 GB of weights (5.6 ms), so it is compute-bound by a factor of about 40. Note the attention term: it grows with \(n_p^2\), so doubling both sides of the image multiplies it by 16.

Prefill

Prefill runs all \(T\) prompt tokens through the LLM in one pass:

\[ F_{\text{pre}} \;\approx\; 2\, N_{\text{lin}}\, T \;+\; 2\, L\, d\, T^{2} \;+\; 2\, N_{\text{head}} \tag{199.4} \]
Read the symbols
\(T\)
prompt length in tokens: visual tokens plus text tokens
\(L,\ d\)
LLM layers (28) and width (3,584)
\(2 L d T^2\)
causal attention: like the encoder term, but each token only looks back, so half the pairs
\(2 N_{\text{head}}\)
the lm_head runs only for the last prompt token, to produce the first output token

For \(T = 1{,}624\): linear 21.19 TFLOP, attention 0.53 TFLOP, total 21.72 TFLOP. At 60.5 TFLOP/s: 359 ms. Compute-bound again: the 14.1 GB of weights take 59 ms to read. Here attention is small, because the LLM is wide and the prompt is short.

So the first token appears after:

\[ \text{TTFT} \;\approx\; t_{\text{enc}} + t_{\text{pre}} \;=\; 239 + 359 \;=\; 598 \text{ ms} \tag{199.5} \]

(We leave out the image download, JPEG decoding and resize on the CPU, and the network. They are real, so measure them; see the last section.)

Decode

Each new token needs one pass through the LLM. At batch size \(b\), the step reads all weights once and the KV cache of every request, and does \(2N\) FLOPs per weight for each of the \(b\) tokens:

\[ \text{TPOT}(b) \;\approx\; \max\!\left( \frac{b \left(2 N_{\text{lin}} + 2 N_{\text{head}} + 4 L d \bar T\right)}{\eta_c\, P},\; \frac{\tfrac{\beta}{8} N_{\text{lin}} + 2 N_{\text{head}} + b\, k\, \bar T}{\eta_m\, B_w} \right) \tag{199.6} \]
Read the symbols
\(\beta\)
bits per LLM linear weight: 16 in BF16, 8 in FP8, 4.25 for 4-bit AWQ with group size 128 (the scales add 0.25 bit)
\(2 N_{\text{head}}\)
bytes of the lm_head, kept in BF16
\(k\)
KV-cache bytes per token: 57,344 for this model
\(\bar T\)
average context length during decode: the prompt plus half the output, \(1{,}624 + 64 = 1{,}688\)

In BF16 at \(b = 1\): the step reads \(14.14 + 0.10 = 14.24\) GB, so 59.3 ms per token. The compute side is only 0.24 ms. Decode is bandwidth-bound by a factor of about 250. The whole answer:

\[ \text{E2E} \;\approx\; \text{TTFT} + N_{\text{out}} \cdot \text{TPOT} \;=\; 0.60 + 128 \times 0.0593 \;=\; 8.19 \text{ s} \tag{199.7} \]

Over the 4.0 s target by a factor of two. And 93% of the time is decode. The image, which looked like the big cost, is not the problem here.

vision encoderprefilldecode (128 tokens)
Figure 1. The worked example as a timeline. One request to Qwen2-VL-7B on one L4: a 1280×960 photo, 60 text tokens, 128 output tokens, batch size 1, with \(\eta_c = 0.5\) and \(\eta_m = 0.8\). Each step adds one stage or applies one change. All bars are estimates from formulas (199.2)–(199.7).
How to read Figure 1

Time runs from left to right. The two dashed lines are the targets: TTFT at 1.0 s and E2E at 4.0 s. Steps 1–3 build the BF16 estimate: the first token comes at 0.60 s (fine), but the answer ends at 8.19 s. Step 4 switches the LLM weights to 4-bit: only the green decode bar shrinks, to 3.08 s in total. Step 5 lowers the image to 896×672: now the orange (encoder) and dark red (prefill) bars shrink, and the total is 2.73 s. Each change moves the stage it targets, and only that stage.

Step 3: choose the knobs

Now we have four knobs. Each one acts on a different term of the budget:

KnobMovesEffect in our exampleCost
Image resolutionencoder (\(\propto n_p\) and \(n_p^2\)), prefill (\(\propto T\)), KV cache\(1288{\times}952 \to 896{\times}672\): TTFT \(598 \to 272\) mssmall text and small objects get harder to read [Likely]
Number of frames (video)same as resolution, times \(G\)32 frames at 448×448: 4,096 visual tokens, TTFT 1.35 sshort events can fall between frames
LLM weight precisiondecode bytes; FP8 also doubles prefill computeBF16 \(\to\) 4-bit: TPOT \(59.3 \to 19.4\) msa small quality risk; test it (session 197)
Batch sizethroughput; adds waiting time per requestsee Step 4each request waits for the others’ prefill
Output lengthnumber of decode steps\(128 \to 96\) tokens: E2E \(2.73 \to 2.11\) sshorter answers; set by the prompt or max_tokens

The precision options behave very differently on the L4, and this is the most useful lesson of the whole page:

  • 4-bit weights, 16-bit activations (W4A16, e.g. AWQ). Decode reads 4.25 bits per weight: 4.56 GB per step instead of 14.14 GB, so TPOT drops from 59.3 to 19.4 ms. The multiplications still run in FP16, so prefill does not get faster. (Real 4-bit kernels can even make prefill a little slower, because they also unpack the weights. [Likely])
  • FP8 weights and activations (W8A8). Decode reads 1 byte per weight: 7.62 GB, so TPOT is 32.1 ms. But the tensor cores run FP8 at twice the BF16 rate, so the LLM part of prefill halves: \(359 \to 184\) ms.
  • So: W4A16 wins when decode dominates (long answers, few images). FP8 wins when prefill dominates (many images, video, long documents, short answers). With our 128-token answers, W4A16 is the right choice.

Our first plan: 4-bit LLM weights (vision encoder stays BF16), images resized to 896×672. That gives \(n_{\text{vis}} = 32 \times 24 = 768\) tokens, \(T = 828\), \(F_{\text{enc}} = 5.49\) TFLOP (91 ms), \(F_{\text{pre}} = 10.94\) TFLOP (181 ms), TTFT 272 ms, TPOT 19.2 ms, E2E 2.73 s. It passes, for one user at a time. Try the knobs in Figure 2.

Model
LLM weights
Figure 2. A budget calculator for an L4 (formulas 199.1–199.11), with \(\eta_c = 0.5\), \(\eta_m = 0.8\). The vision encoder always runs in BF16. Images keep a 4:3 shape; video frames are square. With \(b \gt 1\), the bars show the closed-loop model of Step 4: each request also waits for the prefills of the other \(b - 1\) requests. Memory is weights plus the KV cache at the end of all \(b\) answers, against 90% of 24 GB.
How to read Figure 2

The top bar is the time of one request, split into encoder (orange), prefill (dark red) and decode (green); the dashed lines are the 1.0 s TTFT and 4.0 s E2E targets. The bottom bar is GPU memory. Start at the defaults (BF16, 1288×952): 8.19 s. Click 4-bit: 3.08 s. Lower the resolution to 896: 2.73 s. Then set the long side to 448 and choose 32 frames (square 448×448 frames): TTFT jumps over 1 s, because 4,096 visual tokens must be encoded and prefilled; now click FP8 and see TTFT fall to 0.90 s while decode gets slower. Try the 2B model too: its decode is fast, and the encoder (the same 0.63 B ViT) becomes the largest part of TTFT.

Worked example: the same photo on Qwen2-VL-2B

The 2B model has the same vision encoder, so \(t_{\text{enc}}\) is still 238 ms at 1288×952. Its LLM is much smaller: width 1,536, MLP width 8,960, 2 KV heads, tied embeddings. So \(N_{\text{lin}} = 1.310\) B and \(N_{\text{head}} = 0.233\) B. Prefill: 4.48 TFLOP, 74 ms. TTFT = 312 ms. Decode in BF16: \(2 \times (1.310 + 0.233) = 3.09\) GB plus 0.05 GB of KV per step, so 13.1 ms per token. E2E = \(0.31 + 128 \times 0.0131 = 1.98\) s, even in BF16 and at full resolution. For a small VLM, the vision encoder is 76% of TTFT. This is why small VLMs try hard to cut visual tokens or use smaller encoders (session 183).

Algorithm 1: a latency budget for one VLM request
  1. input: model config, GPU peaks \(P, B_w\), factors \(\eta_c, \eta_m\), image size, frames, text tokens, output tokens, targets
  2. resize as the model does; \(n_p, n_{\text{vis}} \leftarrow\) formula (199.1); \(T \leftarrow n_{\text{vis}} + n_{\text{text}}\)
  3. \(t_{\text{enc}} \leftarrow \max(F_{\text{enc}}/\eta_c P,\ Q_{\text{enc}}/\eta_m B_w)\) // (199.3); compute-bound
  4. \(t_{\text{pre}} \leftarrow \max(F_{\text{pre}}/\eta_c P,\ Q_{\text{pre}}/\eta_m B_w)\) // (199.4); use the FP8 rate for FP8 layers
  5. \(\text{TPOT} \leftarrow\) formula (199.6) at the planned batch size // bandwidth-bound at small b
  6. \(\text{TTFT} \leftarrow t_{\text{enc}} + t_{\text{pre}}\); \(\text{E2E} \leftarrow \text{TTFT} + N_{\text{out}}\,\text{TPOT}\)
  7. check memory: weights + KV cache \(\le\) usable memory (199.9)
  8. while a target fails: find the largest term; apply the knob for that term (resolution or frames for encoder and prefill; precision or output length for decode; FP8 for compute-bound prefill); go to step 2
  9. return the plan and its estimated TTFT, TPOT, E2E and memory

Step 4: batch size, throughput and the number of GPUs

One request at a time wastes the GPU. During decode at \(b = 1\), the tensor cores are idle more than 99% of the time, waiting for weights. If 3 requests decode together, the GPU reads the weights once for all 3. So the step costs almost the same, and we get 3 tokens instead of 1. Figure 3 shows this on the L4’s roofline. The arithmetic intensity (FLOPs per byte read) of a decode step is about:

\[ I_{\text{dec}}(b) \;\approx\; \frac{2\, b\, N}{\tfrac{\beta}{8} N + b\, k\, \bar T} \;\;\xrightarrow{\;k \bar T \,\ll\, N\;}\;\; \frac{16\, b}{\beta} \tag{199.8} \]

Decode becomes compute-bound only when \(I_{\text{dec}}\) passes the ridge point of 403. For BF16 weights (\(\beta = 16\)), that needs \(b \approx 403\); for 4-bit weights, \(b \approx 107\), ignoring the KV cache. Long before that, the KV-cache term in the denominator grows and holds the intensity down. So for a VLM on an L4, decode is bandwidth-bound at every batch size you will use.

LLM weights
Figure 3. Roofline of the L4 (log–log), at peak: \(\text{attainable FLOP/s} = \min(P,\ I \cdot B_w)\). Points: the vision encoder and prefill of the 896×672 plan, and one decode step of Qwen2-VL-7B at batch size \(b\) (context \(\bar T = 892\) tokens per request). The FP8 roof (242 TFLOP/s) is drawn for the FP8 setting.
How to read Figure 3

The sloped line is the bandwidth limit, the flat line the compute limit; they meet at the ridge point (403 FLOPs per byte in BF16). A point left of the ridge is bandwidth-bound: moving it right (more FLOPs per byte) makes it faster in FLOP/s. The encoder and prefill sit far to the right: compute-bound. Decode at \(b = 1\) sits at intensity 1.0 in BF16 (3.1 with 4-bit weights). Raise \(b\): the decode point moves up the slope, so the GPU does more useful work per second, but even at \(b = 256\) it stays left of the ridge.

What does batching cost each user? With continuous batching (session 135), a new request’s prefill runs between decode steps, and while it runs, everyone else’s decode waits. A simple model is a closed loop: \(b\) requests are always in flight, and each one, during its life, sees the prefill of all \(b\) requests (its own and the others’) and all \(N_{\text{out}}\) shared decode steps:

\[ \text{E2E}(b) \;\approx\; b\, t_{\text{pf}} + N_{\text{out}}\, \text{TPOT}(b), \qquad X(b) \;=\; \frac{b}{\text{E2E}(b)}, \qquad \text{TTFT}_{\text{burst}} \;\approx\; b\, t_{\text{pf}} \tag{199.9} \]
Read the symbols
\(t_{\text{pf}}\)
GPU time for the encoder plus prefill of one request: the TTFT at \(b = 1\)
\(X(b)\)
throughput: finished requests per second on one GPU
\(\text{TTFT}_{\text{burst}}\)
worst case: \(b\) requests arrive at the same moment, and the last one waits for all prefills

This is a rough, pessimistic model. [Likely] Chunked prefill (Sarathi-Serve) mixes prefill pieces into decode steps, and because decode leaves the tensor cores mostly idle, part of the prefill work hides there. Real E2E at \(b \gt 1\) is often somewhat better. But the model has the right shape, and it is easy to compute. The number of GPUs for a peak load \(\lambda\) is then:

\[ n_{\text{GPU}} \;=\; \left\lceil \frac{\lambda}{X(b^{*})} \right\rceil, \qquad b^{*} = \max\{\, b : \text{E2E}(b) \le 4\text{ s},\ \text{TTFT}_{\text{burst}}(b) \le 1\text{ s},\ \text{memory fits} \,\} \tag{199.10} \]
Plan
\(\text{E2E}(b)\)\(\text{TTFT}_{\text{burst}}(b)\), TTFT in a burstthroughput \(X(b)\), right axis
Figure 4. The closed-loop model (199.9) for Qwen2-VL-7B on one L4, 60 text tokens and 128 output tokens, \(\eta_c = 0.5\), \(\eta_m = 0.8\). Filled dots meet both targets. The readout picks \(b^{*}\) with formula (199.10) and the number of L4s for the chosen peak load.
How to read Figure 4

The green line (E2E) and the dark red line (burst TTFT) rise with \(b\); the dashed lines are their targets (4 s and 1 s). The orange bars are throughput, which rises with \(b\) because decode steps are shared. The best batch size is the largest one where both lines are under their targets. For the default plan, \(b^{*} = 3\): E2E 3.33 s, burst TTFT 0.81 s, 0.90 requests per second per L4, so 3 L4s for 2 requests per second. Here the TTFT target, not E2E, stops the batch from growing. Switch to 1288×952 and the prefill is so long that only \(b = 1\) fits, which needs 7 L4s.

Worked example: how many L4s for 2 requests per second?

Plan: 4-bit LLM, 896×672 images, so \(t_{\text{pf}} = 0.272\) s. TPOT grows slowly with \(b\), because only the KV cache adds bytes: 19.20 ms at \(b = 1\), 19.63 ms at \(b = 3\), 19.84 ms at \(b = 4\).

  • \(b = 3\): E2E \(= 3 \times 0.272 + 128 \times 0.01963 = 3.33\) s (pass); burst TTFT \(= 0.81\) s (pass); \(X = 3 / 3.33 = 0.90\) requests per second.
  • \(b = 4\): E2E 3.63 s (pass), but burst TTFT \(= 1.09\) s (fail).
  • So \(b^{*} = 3\), and \(n_{\text{GPU}} = \lceil 2 / 0.90 \rceil = 3\) L4s.

Memory is not the limit here. Weights: \(3.47\) GB (4-bit LLM) + \(2.18\) GB (embedding and lm_head in BF16) + \(1.35\) GB (vision) = 7.00 GB. KV cache per request at the end of the answer: \((828 + 128) \times 57{,}344 = 0.055\) GB. With 21.6 GB usable, about 266 requests would fit. It is a different story for long video: a 64-frame request at 448×448 has 8,192 visual tokens and needs 0.48 GB of KV, so a BF16 model (16.6 GB of weights) fits only 10 such requests.

\[ M \;=\; \underbrace{\sum_{c} \tfrac{\beta_c}{8} N_c}_{\text{weights}} \;+\; b\, k\, (T + N_{\text{out}}) \;\le\; 0.9 \times 24 \text{ GB} \tag{199.11} \]

The factor 0.9 is vLLM’s default gpu_memory_utilization; the rest is left for activations, the CUDA context and fragmentation. [Likely] vLLM also reserves activation memory for the largest batch it profiles, so the real KV capacity is a bit smaller than this formula says.

Step 5: check the estimate with a real measurement

An estimate is a hypothesis. You test it with one measurement, and then you use the measurement to fix the two numbers you guessed: \(\eta_c\) and \(\eta_m\). Measure TTFT and TPOT for one request at \(b = 1\), with the exact plan (same resolution, same precision, same prompt). Then:

\[ \hat\eta_c \;=\; \frac{F_{\text{enc}} + F_{\text{pre}}}{P \cdot \text{TTFT}_{\text{meas}}}, \qquad \hat\eta_m \;=\; \frac{Q_{\text{dec}}}{B_w \cdot \text{TPOT}_{\text{meas}}} \tag{199.12} \]
Read the symbols
\(\hat\eta_c\)
measured compute efficiency of the encoder plus prefill (valid because both are compute-bound)
\(\hat\eta_m\)
measured bandwidth efficiency of a decode step (valid because decode is bandwidth-bound)
\(Q_{\text{dec}}\)
bytes per decode step: 4.61 GB for the 4-bit plan at \(\bar T = 892\)

Then put \(\hat\eta_c\) and \(\hat\eta_m\) back into formulas (199.2)–(199.10) and redo the plan. If one measured number is far from every reasonable efficiency (for example \(\hat\eta_m = 0.2\)), the model of that stage is wrong, not just the factor: look for a missing cost.

Worked example: re-planning after a measurement

The numbers in this box are made-up example inputs to show the arithmetic, not a real L4 measurement. Suppose the lab measures TTFT = 420 ms and TPOT = 24.0 ms for the 4-bit, 896×672 plan at \(b = 1\).

  • \(\hat\eta_c = 16.43\text{ TFLOP} / (121\text{ TFLOP/s} \times 0.420\text{ s}) = 0.32\) (we assumed 0.5).
  • \(\hat\eta_m = 4.61\text{ GB} / (300\text{ GB/s} \times 0.0240\text{ s}) = 0.64\) (we assumed 0.8).
  • New E2E at \(b = 1\): \(0.42 + 128 \times 0.024 = 3.49\) s. At \(b = 2\): burst TTFT 0.84 s and E2E 3.95 s, both pass. At \(b = 3\): burst TTFT 1.26 s, fail.
  • So \(b^{*} = 2\), \(X = 0.507\) requests per second, and the plan now needs \(\lceil 2 / 0.507 \rceil = 4\) L4s instead of 3.

One measurement changed the purchase by one GPU. That is why you always measure before you commit.

Figure 5. Calibrating the budget (formula 199.12) for the plan “Qwen2-VL-7B, 4-bit LLM, 896×672, 128 output tokens” on one L4. Move the sliders to the numbers you measure; the default values are the made-up example of the box above. The readout re-plans with formulas (199.9) and (199.10) for a peak load of 2 requests per second.
How to read Figure 5

Each pair of bars compares the estimate with \(\eta_c = 0.5\), \(\eta_m = 0.8\) (grey) and your measurement (colored). The text below the bars shows the efficiency each measurement implies. Values between about 0.3 and 0.9 are plausible; outside that range, suspect a missing cost (CPU preprocessing, a slow kernel, a wrong precision) rather than a slow GPU. The last line is the new plan: \(b^{*}\), E2E and the number of L4s. Move TTFT up to 600 ms and watch \(b^{*}\) fall to 1.

How to measure, and what usually differs

Two rules make the measurement useful. First, measure the same thing you estimated: one request, \(b = 1\), fixed resolution and output length (set max_tokens and ignore the end-of-sequence token, so every run writes exactly 128 tokens). Second, measure each part separately, so you know which estimate was wrong. The next session, session 200, does this with vLLM’s benchmark tool, which reports TTFT, TPOT and their percentiles. [Likely] The code below shows the idea with plain PyTorch.

Where the estimate and the measurement usually differ:

  • CPU preprocessing. JPEG decoding, resizing and normalizing a large photo run on the CPU, before the GPU sees anything. They add directly to TTFT. Measure them alone.
  • Lower compute efficiency for small or odd shapes. 768 tokens is a small matrix multiply for a 72 W GPU; \(\eta_c\) for the encoder and for prefill can be quite different. Time them separately.
  • Kernel launch overhead in decode. A decode step launches hundreds of small kernels. Without CUDA graphs, launch time can be a large part of a 20 ms step. [Likely] vLLM uses CUDA graphs for decode by default. [Likely]
  • Warm-up and clocks. The first request compiles kernels and allocates memory. Run a few warm-up requests and report the median of many.
  • Mixed precision you did not plan. A 4-bit checkpoint whose vision tower is BF16 is expected; one whose lm_head is also 4-bit, or whose KV cache is FP8, changes \(Q_{\text{dec}}\).
Algorithm 2: measure, calibrate and re-plan
  1. input: a running server or model with the planned settings; the estimate of Algorithm 1
  2. run 5 warm-up requests; discard them
  3. for 30 requests at \(b = 1\): record CPU preprocessing time, TTFT and the time of each later token
  4. take medians: \(\text{TTFT}_{\text{meas}}\) (GPU part only), \(\text{TPOT}_{\text{meas}}\)
  5. \(\hat\eta_c, \hat\eta_m \leftarrow\) formula (199.12) // sanity check: both between about 0.3 and 0.9
  6. redo Algorithm 1 and formula (199.10) with \(\hat\eta_c, \hat\eta_m\) and the measured CPU time
  7. run a load test at the chosen \(b^{*}\) and peak load; compare p50 and p95 E2E with the targets // the closed-loop model is only a first guess
  8. return the calibrated plan and the number of GPUs

The runnable code below has two parts. Part 1 is the budget estimator of this page in plain Python; it prints the numbers of the worked examples and runs anywhere. Part 2 measures the two peaks your own GPU actually reaches (a large BF16 matrix multiply and a large memory copy), which gives you a first \(\eta_c\) and \(\eta_m\) before you even load a model. On a machine without a CUDA GPU it runs on the CPU, so the printed numbers are only meaningful on the GPU you plan to use.

import math, time, torch

# ---------- Part 1: the budget estimator (formulas 199.1-199.10) ----------
P16, P8, BW = 121e12, 242e12, 300e9          # NVIDIA L4: dense BF16, dense FP8, bytes/s
VIS = dict(L=32, d=1280, lin=32 * (1280*3840 + 1280*1280 + 2*1280*5120) + 3*2*14*14*1280)
LLM = dict(L=28, d=3584, ff=18944, kvh=4, hd=128, vocab=152064)   # Qwen2-VL-7B config.json
d, L = LLM["d"], LLM["L"]
N_LIN = L * (2*d*d + 2*d*LLM["kvh"]*LLM["hd"] + 3*d*LLM["ff"])
N_HEAD = LLM["vocab"] * d
N_MER = 5120*5120 + 5120*d
KV = 2 * L * LLM["kvh"] * LLM["hd"] * 2                # bytes per token
BITS = {"bf16": 16, "fp8": 8, "w4": 4.25}

def resize(h, w, f=28):
    return max(f, round(h / f) * f), max(f, round(w / f) * f)

def budget(h, w, frames=1, n_text=60, n_out=128, prec="bf16", b=1, ec=0.5, em=0.8):
    H, W = resize(h, w)
    G = frames // 2 if frames > 1 else 1
    per = (H // 14) * (W // 14)                # patches per temporal group
    n_p = G * per; n_vis = n_p // 4; T = n_vis + n_text           # (199.1)
    F_enc = 2*VIS["lin"]*n_p + 4*VIS["L"]*VIS["d"]*G*per**2 + 2*N_MER*n_vis   # (199.3)
    t_enc = max(F_enc / (ec*P16), 2*(VIS["lin"] + N_MER) / (em*BW))
    P = P8 if prec == "fp8" else P16
    w_bytes = BITS[prec] / 8 * N_LIN + 2 * N_HEAD
    t_pre = max(2*N_LIN*T / (ec*P) + (2*L*d*T*T + 2*N_HEAD) / (ec*P16),
                (w_bytes + KV*T) / (em*BW))                         # (199.4)
    Tb = T + n_out / 2
    tpot = max(b*2*N_LIN / (ec*P) + b*(2*N_HEAD + 4*L*d*Tb) / (ec*P16),
               (w_bytes + b*KV*Tb) / (em*BW))                       # (199.6)
    t_pf = t_enc + t_pre
    e2e = b * t_pf + n_out * tpot                                   # (199.9)
    return dict(n_vis=n_vis, T=T, t_enc=t_enc, t_pre=t_pre, ttft=t_pf, tpot=tpot,
                e2e=e2e, burst=b * t_pf, X=b / e2e)

def plan(lam=2.0, ttft_max=1.0, e2e_max=4.0, **kw):
    best = None
    for b in range(1, 65):
        r = budget(b=b, **kw)
        if r["e2e"] <= e2e_max and r["burst"] <= ttft_max:
            best = (b, r)
    if best is None:
        return None
    b, r = best
    return b, round(r["e2e"], 2), round(r["X"], 3), math.ceil(lam / r["X"])  # (199.10)

for name, kw in [("BF16 1288x952", dict(h=1280, w=960)),
                 ("4-bit 1288x952", dict(h=1280, w=960, prec="w4")),
                 ("4-bit 896x672", dict(h=896, w=672, prec="w4"))]:
    r = budget(**kw)
    print(f"{name:15s} vis={r['n_vis']:5d} TTFT={r['ttft']*1e3:6.1f} ms "
          f"TPOT={r['tpot']*1e3:5.1f} ms E2E={r['e2e']:5.2f} s  plan(b*, E2E, X, GPUs)={plan(**kw)}")
# BF16 1288x952   vis= 1564 TTFT= 597.8 ms TPOT= 59.3 ms E2E= 8.19 s  plan(b*, E2E, X, GPUs)=None
# 4-bit 1288x952  vis= 1564 TTFT= 597.8 ms TPOT= 19.4 ms E2E= 3.08 s  plan(b*, E2E, X, GPUs)=(1, 3.08, 0.325, 7)
# 4-bit 896x672   vis=  768 TTFT= 271.6 ms TPOT= 19.2 ms E2E= 2.73 s  plan(b*, E2E, X, GPUs)=(3, 3.33, 0.902, 3)

# ---------- Part 2: what does YOUR GPU reach? (first guess for eta_c, eta_m) ----------
def timed(fn, reps=20):
    sync = torch.cuda.synchronize if torch.cuda.is_available() else (lambda: None)
    for _ in range(3):                  # warm-up
        fn()
    sync(); t0 = time.perf_counter()
    for _ in range(reps):
        fn()
    sync()
    return (time.perf_counter() - t0) / reps

torch.manual_seed(0)
dev = "cuda" if torch.cuda.is_available() else "cpu"
n = 4096 if dev == "cuda" else 1024
dt = torch.bfloat16 if dev == "cuda" else torch.float32   # BF16 matmul is slow on many CPUs
a = torch.randn(n, n, device=dev, dtype=dt)
bm = torch.randn(n, n, device=dev, dtype=dt)
t_mm = timed(lambda: a @ bm)
x = torch.zeros(256 * 2**20 // 2, device=dev, dtype=torch.bfloat16)   # 256 MB
y = torch.empty_like(x)
t_cp = timed(lambda: y.copy_(x))
flops = 2 * n**3 / t_mm
bw = 2 * x.numel() * x.element_size() / t_cp    # a copy reads and writes
print(f"{dev}: matmul {flops/1e12:.1f} TFLOP/s, copy {bw/1e9:.0f} GB/s")
if dev == "cuda":
    print(f"if this is an L4: eta_c ~ {flops/P16:.2f}, eta_m ~ {bw/BW:.2f}")

Part 1 prints exactly the numbers used on this page (they are shown as comments). Part 2 gives upper bounds for \(\eta_c\) and \(\eta_m\): a single large, square matrix multiply is the best case for the tensor cores, and the encoder and prefill shapes reach less. For the full check, time the real model: for example, with Hugging Face transformers, time generate with max_new_tokens=1 (TTFT) and with max_new_tokens=129, min_new_tokens=129; the difference divided by 128 is TPOT. Synchronize the GPU before reading the clock.

Where this is used

ApplicationWhat dominates the budgetFirst knob to try
Photo or screenshot question answering (chat)decode, for answers of 100+ tokens4-bit LLM weights; shorter answers
Document and invoice extractionencoder and prefill (high resolution for small text), short JSON outputFP8 for prefill; crop to the useful region instead of lowering resolution
Video question answering and Video-RAGvisual tokens from many frames: encoder, prefill and KV memoryfewer frames, lower frame resolution, token merging; FP8
Key-moment detection on live videosteady encoder load per second of video, plus a strict TTFTframe rate and resolution per stream; streams per GPU from the encoder cost
Video classification and action recognition with a VLMencoder and prefill; outputs are a few tokensframes and resolution; FP8; decode hardly matters
Video retrieval and fingerprinting (encoder only)the vision encoder alone, compute-boundbatch many frames; INT8 or FP8 encoder; no decode at all
Capacity planning and cost per requestthroughput \(X(b^{*})\)formula (199.10): \(\text{GPUs} = \lceil \text{load} / \text{throughput} \rceil\)

Common mistakes

  • Using the sparsity number from the datasheet. The L4’s “242 TFLOPS BF16” assumes 2:4 sparse weights. Dense models get at most 121.
  • Counting visual tokens from the original image. The model resizes first. 1280×960 becomes 1288×952 in Qwen2-VL; other models tile or pad, and the count can change a lot (session 180).
  • Forgetting that the encoder’s attention is quadratic. At 6,256 patches, attention is 44% of the encoder FLOPs. Doubling both image sides gives 4× the linear cost but 16× the attention cost.
  • Quantizing to speed up TTFT with 4-bit weights. W4A16 makes decode faster, not prefill. For a prefill-heavy request, use FP8 (W8A8) or fewer visual tokens.
  • Reporting TPOT from a run that stopped early. If the model emits the end-of-sequence token after 20 tokens, your “128-token” E2E is wrong. Force a fixed output length when you benchmark.
  • Budgeting with \(b = 1\) and serving with \(b = 16\). At higher batch, each request waits for others’ prefill (199.9). Check TTFT in a burst, not only alone.
  • Ignoring CPU time. Image download, decoding and resizing are not in the GPU roofline but are in the user’s TTFT.
  • Trusting the estimate without a measurement. The efficiency factors are guesses until you measure them (199.12); in the example, one measurement changed the plan from 3 to 4 GPUs.

Interview questions

Why is decode bandwidth-bound and prefill compute-bound, using numbers?basic
A linear layer does 2 FLOPs per weight per token. Prefill pushes \(T\) tokens through each weight it reads, so its intensity is about \(2T/2 = T\) FLOPs per byte in BF16: with 828 tokens, about 828, above the L4’s ridge of \(121/0.3 = 403\). Decode at batch 1 pushes one token per weight: about 1 FLOP per byte, far below the ridge. So decode time is bytes divided by bandwidth: \(14.1\text{ GB} / 240\text{ GB/s} = 59\) ms per token for Qwen2-VL-7B in BF16.
Estimate TTFT for Qwen2-VL-7B on an L4 with one 1280×960 photo.core
Resize to 1288×952: 6,256 patches, 1,564 visual tokens, plus 60 text tokens: \(1{,}564 + 60 = 1{,}624\). Encoder: \(2 \times 0.631\text{ B} \times 6{,}256 = 7.9\) TFLOP of linear layers plus 6.4 TFLOP of attention, about 14.4 TFLOP. Prefill: \(2 \times 6.525\text{ B} \times 1{,}624 = 21.2\) TFLOP plus a little attention, 21.7 TFLOP. At 50% of 121 TFLOP/s: \(239 + 359 \approx 600\) ms, plus CPU preprocessing. Both stages are compute-bound, so the estimate depends on \(\eta_c\), which I would measure.
The E2E target is missed by 2×. Which knob do you turn first?core
The one that acts on the largest term. In our example decode is 93% of 8.2 s, so I cut decode bytes: 4-bit weight-only quantization of the LLM gives \(14.1 \to 4.6\) GB per step and \(59 \to 19\) ms per token, and E2E drops to 3.1 s. Lowering the resolution would only cut the 0.6 s TTFT part. If the request were prefill-heavy (video, short output), I would cut visual tokens or use FP8 instead.
FP8 or 4-bit AWQ on an L4: which is faster?core
It depends on which stage dominates. 4-bit AWQ reads 4.25 bits per weight in decode (19.4 ms per token for the 7B model, against 32.1 ms with FP8), but it computes in FP16, so prefill is not faster. FP8 W8A8 uses the FP8 tensor cores at 242 TFLOP/s dense, so the LLM part of prefill halves (\(359 \to 184\) ms in the example). Long answers: AWQ. Many images, video or short answers: FP8. The L4 is an Ada GPU, so it has FP8 tensor cores; an A100 does not.
How do you choose the batch size for a latency target?core
Throughput rises with batch because decode reads the weights once for all requests: TPOT goes only from 19.2 to 19.8 ms between \(b = 1\) and 4, while tokens per step grow 4×. But each request now waits for the other requests’ prefills. With the closed-loop model, \(\text{E2E}(b) \approx b\,t_{\text{pf}} + N_{\text{out}}\,\text{TPOT}(b)\) and the burst TTFT \(\text{TTFT}_{\text{burst}} \approx b\,t_{\text{pf}}\). I pick the largest \(b\) that meets both targets and fits in memory, then \(\text{GPUs} = \lceil \text{load} / \text{throughput} \rceil\). In the example, \(b^{*} = 3\), 0.90 requests per second, 3 L4s for 2 requests per second. Then I confirm with a load test, because chunked prefill usually does somewhat better than this model.
Why is the vision encoder a bigger share of TTFT for a 2B VLM than for a 7B one?basic
Both Qwen2-VL sizes use the same 0.63 B vision encoder, so its time is the same: about 238 ms for a 1288×952 image on an L4. The 7B LLM’s prefill takes 359 ms, the 2B’s only 74 ms. So the encoder is 40% of TTFT for the 7B model and 76% for the 2B model. For small VLMs, reducing visual tokens or using a smaller encoder matters most.
Your measured TPOT is 2× the estimate. What do you check?advanced
First, the implied bandwidth efficiency: \(Q_{\text{dec}} / (B_w \cdot \text{TPOT})\). If it is 0.4, something is wrong, because memory-bound kernels usually reach well above that. Then: is the model really 4-bit, or did it load in BF16? Is the lm_head or embedding larger than expected? Are CUDA graphs on, or is kernel launch overhead filling the step? Is the batch really 1, or are other requests sharing the GPU? Is the run stopping and restarting (not streaming)? Is the GPU throttled by power or temperature? Each of these changes either the bytes or the overhead of a step.
A 32-frame video request misses the 1 s TTFT target. What are your options?advanced
32 frames at 448×448 give 16 temporal groups of 1,024 patches, so 4,096 visual tokens. Estimated encoder 393 ms and prefill 954 ms, so TTFT 1.35 s. Options: (1) FP8 W8A8 for the LLM: prefill halves, TTFT about 0.90 s, but decode gets slower than with 4-bit. (2) Fewer frames or lower frame resolution: tokens fall linearly, encoder attention quadratically per group. (3) Token merging or pooling after the encoder (session 183). (4) Cache the video encoding if users ask several questions about the same video. (5) Prefix caching of the system prompt. I would pick by which costs the least quality on the target task.
Can a 7B VLM ever reach 15 ms per token on an L4?advanced
Not with 4-bit weights at batch 1 and normal decoding. The step must read about 4.6 GB (4.25-bit LLM, BF16 lm_head, KV cache). \(4.6\text{ GB} / 15\text{ ms} = 307\) GB/s, more than the L4’s peak of 300 GB/s. Options: quantize the lm_head too or use 3-bit weights (fewer bytes), speculative decoding (several tokens per weight read, see session 128), a smaller model, or a GPU with more bandwidth.
How do you turn one measurement into a better plan?core
Measure TTFT and TPOT at \(b = 1\) with the exact settings and a fixed output length, after warm-up, and take medians. Compute the implied efficiencies: \(\hat\eta_c = (F_{\text{enc}} + F_{\text{pre}}) / (P \cdot \text{TTFT})\) and \(\hat\eta_m = Q_{\text{dec}} / (B_w \cdot \text{TPOT})\). Check that they are plausible; if not, the model is missing a cost. Put them back into the formulas and redo the batch and GPU count. Finally, run a load test at the chosen batch and look at p95, not only the median.

Exercises

  1. A 1920×1080 screenshot goes into Qwen2-VL. After the resize, how many visual tokens does it give? (Round each side to the nearest multiple of 28.)
  2. Using Figure 2 or the formulas, estimate E2E for Qwen2-VL-2B in BF16 with the 1288×952 photo, 60 text tokens and 128 output tokens.
  3. For the 4-bit, 896×672 plan, the product team allows answers of only 96 tokens. What is the new E2E at \(b = 1\)?
  4. With formula (199.8), at what batch size would decode become compute-bound on an L4 for 4-bit weights, if you ignore the KV cache and the lm_head?
  5. A 64-frame video at 448×448 goes to the BF16 7B model. How much KV cache does one request need at the end of a 128-token answer, and how many such requests fit in 21.6 GB?
  6. Required TPOT is 15 ms for the 4-bit 7B plan. What bandwidth would that need? Is it possible on an L4?
Answerscheck
1. \(1920/28 = 68.6 \to 69 \times 28 = 1{,}932\); \(1080/28 = 38.6 \to 39 \times 28 = 1{,}092\). So \(69 \times 39 = 2{,}691\) visual tokens (10,764 patches). 2. \(\text{TTFT} = 238 + 74 = 312\) ms; \(\text{TPOT} = 13.1\) ms; \(\text{E2E} = 0.31 + 128 \times 0.0131 = 1.98\) s. 3. \(0.272 + 96 \times 0.0192 = 2.11\) s. 4. \(16b/\beta = 403\) with \(\beta = 4.25\) gives \(b \approx 107\). With the KV cache in the denominator it never gets there at a context of 892 tokens: at \(b = 256\), the intensity is only about 210 (Figure 3, which also counts the attention FLOPs of decode). 5. \(T = 8{,}192 + 60 = 8{,}252\) tokens; at the end, \(8{,}252 + 128 = 8{,}380\) tokens, and \(8{,}380 \times 57{,}344\) bytes \(= 0.48\) GB. Free memory: \(21.6 - 16.58 = 5.02\) GB, so about 10 requests. 6. \(Q_{\text{dec}} = 4.61\) GB, so \(4.61 / 0.015 = 307\) GB/s at 100% efficiency: above the L4’s 300 GB/s peak, so not possible at batch 1 without fewer bytes or speculative decoding.

Key takeaways

  • Write down the inputs first: resized image size, frames, visual tokens, text tokens, output tokens, load and targets. The image is usually most of the prompt (199.1).
  • Each stage takes the larger of compute time and memory time (199.2). The vision encoder and prefill are compute-bound; decode is bandwidth-bound at every practical batch size on an L4 (199.8).
  • \(\text{TTFT} \approx t_{\text{enc}} + t_{\text{pre}}\); \(\text{E2E} \approx \text{TTFT} + N_{\text{out}} \times \text{TPOT}\) (199.5, 199.7). For a chat answer, decode usually dominates; for video and documents, prefill does.
  • Match the knob to the term: resolution and frames for encoder and prefill; 4-bit weights or shorter answers for decode; FP8 for compute-bound prefill.
  • Batching raises throughput but adds prefill waiting; pick the largest batch that meets both targets, then \(\text{GPUs} = \lceil \text{load} / \text{throughput} \rceil\) (199.9, 199.10).
  • The efficiency factors are guesses. Measure TTFT and TPOT once, compute the implied efficiencies (199.12), and re-plan.

Key formulas

\[ n_{\text{vis}} = \frac{G H' W'}{4 \cdot 14^2}, \qquad t \approx \max\!\left(\frac{F}{\eta_c P}, \frac{Q}{\eta_m B_w}\right), \qquad F \approx 2 N n + \text{attention} \]
\[ \text{TTFT} \approx t_{\text{enc}} + t_{\text{pre}}, \qquad \text{TPOT} \approx \frac{\tfrac{\beta}{8} N_{\text{lin}} + 2 N_{\text{head}} + b\, k\, \bar T}{\eta_m B_w}, \qquad \text{E2E} \approx \text{TTFT} + N_{\text{out}}\, \text{TPOT} \]
\[ \text{E2E}(b) \approx b\, t_{\text{pf}} + N_{\text{out}} \text{TPOT}(b), \quad X(b) = \frac{b}{\text{E2E}(b)}, \quad n_{\text{GPU}} = \left\lceil \frac{\lambda}{X(b^{*})} \right\rceil, \quad \hat\eta_m = \frac{Q_{\text{dec}}}{B_w\, \text{TPOT}_{\text{meas}}} \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  2. Austin et al., Google DeepMind (2025). How to Scale Your Model
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  4. Austin et al., Google DeepMind (2025). How to Scale Your Model
  5. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  6. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  7. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  8. Micikevicius et al. (2022). FP8 Formats for Deep Learning
  9. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  10. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  11. NVIDIA (2023). NVIDIA L4 Tensor Core GPU (product page and datasheet)
  12. vLLM project (docs). vLLM documentation: FP8 W8A8 quantization