LLM & VLM courseBlog
141Track 5: LLM inference and serving · session 64 of 65

Scenarios and capacity planning: agents, long context, edge, cost

Real traffic is not one chat request. Agents send many calls that repeat the same long prefix, long-context requests make prefill and the KV cache explode, and edge devices have little memory and bandwidth. This session turns a description of the traffic (requests per second, prompt and output lengths, a latency target) into a number of GPUs and a cost per million tokens, and works it out for one 8B model on an L4 and on an H100.

Time: 3 to 4 hours (reading 2 hours, exercises and code 1 to 2 hours)Level: IntermediateBefore this: Sessions 81, 82 and 133 (metrics, the roofline, the cost model of a server); sessions 134 and 135 (prefix caching, chunked prefill)

What you will learn

  • Why an agent episode of k calls costs about k²/2 prompts without a prefix cache, and only one prompt plus the new tokens with it
  • Where the time and memory of long context go: the T² attention term, the crossover length N/(Ld), 128 KiB of KV cache per token, and what sparse attention saves (bandwidth, not capacity)
  • A simple upper bound for edge decoding speed: memory bandwidth divided by model size
  • A closed-form model of one server under load: TPOT as a function of request rate, and the three limits (bandwidth, compute, KV memory) that set the capacity of one GPU
  • How to turn traffic and an SLO into a number of GPUs and a cost per million output tokens
  • A worked plan for Llama 3.1 8B on an L4 and an H100: why the L4 cannot meet 50 ms in BF16, why the H100 is cheaper per token at high traffic, and when the L4 wins
  • Runnable PyTorch code that checks prefix-cache reuse gives the same logits, and a planner that reproduces the worked example

Why this matters

So far we have studied one request at a time, or a uniform stream of chat requests. Real products are messier. A coding agent sends 30 calls that all start with the same 5,000-token system prompt. A legal assistant reads a 100,000-token contract. A phone runs a 3B model on a few GB of memory. And somebody must answer the question every team asks before launch: how many GPUs do we need, and what does one million tokens cost us?

The efficient-inference survey of Zhou et al. ends with exactly these scenarios (its Section 7): agent and multi-model frameworks, which “naturally introduce a new optimization level, i.e., pipeline-level”; long-context models, where self-attention is quadratic in the context length; and edge deployment on devices with very limited resources. [Certain] This session takes each scenario, finds which resource it stresses (compute, memory bandwidth, or memory size), and then builds one capacity plan from traffic to dollars.

We use one model throughout, so that every number can be checked: Llama 3.1 8B. From its config: 32 layers, hidden size 4096, 8 KV heads of size 128, maximum context 131,072 tokens. [Certain] As in session 133, we count \(N = 8.03 \times 10^9\) parameters, \(M_W = 16.06\) GB of weights in BF16 (8.03 GB in FP8), and \(s_{\text{kv}} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB) of KV cache per token in BF16 (half in FP8). The two GPUs are an L4 (24 GB, 300 GB/s, 121 TFLOP/s dense BF16, 242 dense FP8, 72 W) and an H100 SXM (80 GB, 3.35 TB/s, 989 TFLOP/s dense BF16, 1,979 dense FP8). [Certain] We also keep the efficiency assumptions of session 133: matmuls reach \(u = 50\%\) of peak FLOP/s, memory reads reach \(\eta = 80\%\) of peak bandwidth, each step has \(t_0 = 1\) ms of fixed overhead, and 90% of GPU memory is usable. [Likely] These assumptions are rough; measure your own engine (session 142).

Scenario 1: agents and multi-model pipelines

An agent is a program that calls an LLM in a loop. Each call reads the whole history so far, writes a short output (a thought and a tool call, for example in the ReAct style), then a tool runs (a search, a code interpreter, a test suite), and its result is appended to the history. [Certain] Benchmarks such as SWE-bench, where a model must fix real GitHub issues, are solved this way, with tens of calls per task. [Likely]

From the server’s point of view, an agent episode is a sequence of requests with three special properties:

  1. Shared prefixes. Call \(i\) contains everything that call \(i-1\) contained, plus a little more. Many episodes also share the same system prompt and tool descriptions.
  2. Sequential dependency. Call \(i+1\) cannot start before call \(i\) and its tool have finished. Latencies add up.
  3. Idle gaps. While a tool runs (often seconds), the request is not in the GPU batch, but its KV cache is worth keeping.

How many tokens does an episode prefill?

Let the shared prefix (system prompt plus tool list) have \(S\) tokens, the task \(q\) tokens, and let each call write \(o\) tokens and each tool return \(r\) tokens. Then the prompt of call \(i\) (counting from 1) has

\[ T_i = S + q + (i - 1)(o + r) \tag{141.1} \]

tokens. Without any cache, every call prefills its whole prompt. With a prefix cache that keeps the KV of the episode between calls (session 134), call 1 prefills \(S + q\) and every later call prefills only the new tool result: the output tokens were already written into the cache when they were decoded. Summing over \(k\) calls:

\[ \text{no cache: } \sum_{i=1}^{k} T_i = k(S + q) + (o + r)\,\frac{k(k-1)}{2}, \qquad \text{prefix cache: } (S + q) + (k - 1)\, r \tag{141.2} \]
Read the symbols
\(k\)
number of LLM calls in one episode
\(S\), \(q\)
tokens of the shared prefix (system prompt, tool list) and of the user’s task
\(o\), \(r\)
tokens the model writes per call, and tokens a tool returns per call

Without a cache the cost grows like \(k^2\); with it, linearly. If the shared prefix \(S\) is also cached across users (the same system prompt for everybody), call 1 only prefills \(q\). The latency of the whole episode is a sum along the chain:

\[ E2E_{\text{episode}} = \sum_{i=1}^{k} \text{TTFT}_i \;+\; k\, o \cdot \text{TPOT} \;+\; (k - 1)\, t_{\text{tool}} \tag{141.3} \]
Worked example: a 10-call coding agent on one H100

Take \(k = 10\), \(S = 3{,}000\), \(q = 200\), \(o = 150\), \(r = 600\). By (141.2), without a cache the episode prefills \(10 \cdot 3{,}200 + 750 \cdot 45 = 65{,}750\) tokens. With an episode cache: \(3{,}200 + 9 \cdot 600 = 8{,}600\) tokens, 7.6 times fewer. With the system prompt cached too: \(200 + 5{,}400 = 5{,}600\).

Counting attention as well (a chunk of \(n\) new tokens after \(j\) cached ones costs \(2Nn + 4Ld\,n(j + n/2)\) FLOPs, as in session 84), the prefill work is 1.18 PFLOP without a cache and 0.160 PFLOP with it, 7.4 times less. At \(u = 50\%\) on an H100 that is 2.39 s against 0.32 s of GPU time. The TTFT of the last call falls from 376 ms to 26 ms.

But look at the latency (141.3) with \(\text{TPOT} = 10\) ms and 1 s per tool: decode is \(10 \cdot 150 \cdot 0.01 = 15\) s, tools are 9 s, and the whole episode takes 26.4 s without a cache and 24.3 s with it. The cache is a big saving in GPU cost and a small one in latency; for agents, latency is dominated by decoding and tools.

The price of keeping the cache: at the end the context holds \(3{,}200 + 1{,}500 + 5{,}400 = 10{,}100\) tokens, which is \(10{,}100 \times 131{,}072\ \text{bytes} = 1.32\ \text{GB}\) of KV cache per agent. The 55.9 GB left on an H100 after the weights keep only about 42 such agents. Agents are hungry for memory, not for compute.

tokens prefilled in this calltokens read from the prefix cacheoutput tokens decoded
Figure 1. One agent episode of Llama 3.1 8B on one H100 (BF16, \(u = 50\%\), TPOT fixed at 10 ms, task \(q = 200\) tokens). Each column is one call. The height is the prompt of that call, formula (141.1); the dark part must be prefilled, the light part comes from the cache, and the thin top part is the output.
How to read Figure 1

Start with “No cache”: the dark (prefilled) columns grow like a staircase and their total area grows like \(k^2\). Switch to “Cache within the episode”: only a thin dark band remains on each column (the tool result). Now raise the number of calls to 20 and the tool result to 2,000 tokens: without a cache the prefill time becomes comparable to the decode time; with a cache it stays small. Watch the KV size of the finished episode: that is the memory you must keep reserved during every tool call.

Serving agents well

Three system ideas follow from the numbers. First, cache-aware routing: send all calls of one episode to the replica that already holds its prefix; a random load balancer destroys the hit rate. SGLang’s RadixAttention was built for these “LLM programs” with many calls and shared prefixes [Certain], and its router balances load by cache hits [Likely]. Second, keep or move the cache during tool calls: keep it in GPU memory with a time limit, or move it to CPU memory or SSD and bring it back. Mooncake, the serving system of the Kimi chatbot, is organized around a large pool of KV cache in CPU memory and SSD across machines. [Certain] Third, run independent work in parallel: several tool calls, or several branches of a plan, can run at the same time (SGLang’s fork, Skeleton-of-Thought in session 91); they share the prefix, so they cost little extra prefill.

Algorithm 1: serving one agent episode
  1. input: episode id \(e\), shared prefix \(S\), task \(q\), cache time limit \(\tau_{\text{keep}}\)
  2. pick replica \(R_e\) by longest cached prefix (or a hash of \(e\)); remember it for the episode
  3. for each call \(i = 1, 2, \dots\)
  4.   send the full prompt \(T_i\) to \(R_e\); the engine matches the cached prefix and prefills only the new tokens // formula (141.2)
  5.   decode \(o\) tokens; their KV stays in the cache
  6.   if the output is a tool call: pin the cache for at most \(\tau_{\text{keep}}\); if memory is short, offload it to CPU instead of freeing it
  7.   run the tool (in parallel with other tools when they do not depend on each other); append its result to the prompt
  8.   if the output is a final answer: release the cache (keep only the shared prefix)

Multi-model pipelines

Many products are not one model but a chain: a small router or classifier, an embedding model and a reranker for retrieval, the main LLM, a guard model that checks the answer, and sometimes an LLM judge. Two rules organize the cost. Latency adds along the critical path, so a 30 ms guard model before the main call adds 30 ms to TTFT for everybody; run checks in parallel with the main call when you can, or stream the answer and check it in pieces. GPU cost adds over all models, weighted by how often each one runs: a router that sends 70% of easy queries to a small model (a model cascade, session 91) can save more than any kernel optimization. Small models in the chain (embedders, rerankers, guards) usually share one GPU, since one of them alone would leave it mostly idle. [Likely]

Scenario 2: long context

Llama 3.1 8B accepts 131,072 tokens. [Certain] What does it cost to use them? Three things change as the context \(T\) grows: prefill compute, KV memory, and the bytes each decode step reads.

Prefill: the quadratic term

The linear layers cost \(2N\) FLOPs per token. Causal attention adds, per layer, about \(2 d T^2\) FLOPs (the score matrix \(QK^\top\) and the product with \(V\), each \(2dT^2\), halved by the causal mask):

\[ F_{\text{prefill}}(T) = 2NT + 2LdT^2, \qquad \text{attention is larger when } T \gt T^{*} = \frac{N}{Ld} \tag{141.4} \]
Read the symbols
\(N\)
parameters (8.03 billion)
\(L\), \(d\)
layers (32) and model width (4096); \(L \cdot d = 131{,}072\)
\(T^{*}\)
the context length where attention FLOPs equal the FLOPs of all linear layers

For our model \(T^{*} = 8.03 \times 10^9 / 131{,}072 = 61{,}264\) tokens. Below about 8K tokens attention is a small part of prefill (12% at 8,192); at 32K it is 35%; at 128K it is 68%. On an H100 at \(u = 50\%\), the TTFT of a single prompt is 0.30 s at 8K, 1.63 s at 32K and 13.4 s at 128K. On an L4 in FP8 the 128K prefill takes 54.6 s.

A check against a real measurement: for a 1M-token prompt, formula (141.4) gives \(2.78 \times 10^{17}\) FLOPs. On an A100 (312 TFLOP/s dense BF16) at 50% that is 29.7 minutes. The MInference paper reports that an 8B model takes 30 minutes to prefill 1M tokens on one A100. [Certain] The simple formula is in the right range.

Memory: the KV cache per sequence

The KV cache of one sequence is \(T \cdot s_{\text{kv}}\). At 128K tokens in BF16 that is 17.2 GB (16 GiB): more than the 16.06 GB of weights. An H100 has room for only 3 such sequences, an L4 with FP8 weights and an FP8 cache for 1. Long context is first a memory-size problem: it kills batching, and without batching each decode step reads all weights for one user.

Decode: bytes per step, and what sparse attention saves

Each decode step reads the weights once and the KV cache of every sequence in the batch. Sparse attention methods for decoding read only a budget of \(K\) tokens per sequence. Quest, for example, keeps the whole cache in memory but, for each query, loads only the top-\(K\) pages whose keys can matter most; it reports up to 2.23 times faster self-attention. [Certain] With a batch of \(B\) sequences:

\[ \underbrace{M_W + B\,T\,s_{\text{kv}}}_{\text{memory held}} \le 0.9\,C, \qquad Q_{\text{step}} = M_W + B \cdot \min(T, K) \cdot s_{\text{kv}} \tag{141.5} \]

The second formula is the bytes read per step. The first is unchanged: a query-aware method like Quest saves bandwidth, not capacity. Methods that evict tokens (StreamingLLM keeps the first tokens and a recent window; H2O and SnapKV keep the tokens that received the most attention) save capacity too, but a dropped token can never be read again. [Certain] For prefill, MInference finds sparse patterns in each head and reports up to 10 times faster prefill at 1M tokens on an A100. [Certain] Other tools for long context are FP8 or 4-bit KV caches (session 110), prompt compression (session 88), and context parallelism such as Ring Attention, which spreads one long prefill over several GPUs (session 49).

Worked example: 128K tokens on an H100

One sequence: decode step \(= 1 + (16.06 + 17.18)/(0.8 \times 3.35)\) ms \(= 13.4\) ms. With a sparse budget \(K = 4{,}096\): \(1 + (16.06 + 0.54)/2.68 = 7.2\) ms, a 1.9 times speed-up. With the full batch of 3 sequences: 26.2 ms without sparsity and 7.6 ms with it, 3.5 times. The more of the step is KV reading, the more sparse decoding helps.

prefill time, linear layers onlyprefill time, with attentionKV cache of the sequences that fitweights
Figure 2. Long context for Llama 3.1 8B. Left: TTFT of one prompt of length \(T\) from formula (141.4), on log axes, with the crossover \(T^{*}\) marked. Right: GPU memory with the largest batch of length-\(T\) sequences that fits (141.5). The readout gives the decode step for that batch, with and without a sparse budget \(K\).
How to read Figure 2

On log axes a linear cost is a straight line with slope 1 and a quadratic cost has slope 2. The “with attention” curve leaves the “linear layers only” line near \(T^{*}\) and becomes twice as steep. Move \(T\) from 1K to 128K: TTFT grows by much more than 128 times, and the number of sequences that fit falls from hundreds to a few. Then switch the KV cache to FP8: capacity doubles. Choose a sparse budget: capacity does not change, but the decode step gets shorter once \(T\) is larger than \(K\).

Scenario 3: edge devices

On a laptop or a phone the batch is 1, so every decode step reads all the weights. The memory bandwidth sets a hard upper bound on speed:

\[ \text{tokens per second} \;\le\; \frac{\eta \cdot \text{BW}}{M_W} \tag{141.6} \]

Take a device with 100 GB/s of memory bandwidth (an example value; phones and laptops range widely). Our 8B model in 4-bit weights is about 4.0 GB, so at \(\eta = 0.8\) it runs at most 20 tokens per second; a 3B model in 4 bits (1.5 GB) at most 53. Prefill on the device is compute-bound: at 10 TFLOP/s of usable compute (again an example), a 2,000-token prompt of the 8B model takes \(2 \cdot 8.03 \times 10^9 \cdot 2000 / 10^{13} = 3.2\) s. That is why on-device assistants keep prompts short, cache the system prompt, and use 1B to 3B models. The survey notes that a performance gap still exists between small and large models, and points to cloud–edge collaboration: easy requests on the device, hard ones in the cloud. [Certain] Power matters too: an L4 uses at most 72 W, a phone a few watts. Session 140 covers the runtimes (llama.cpp, MLX) and the hardware.

From traffic to GPUs

Now the main question. A traffic description has four parts: the arrival rate \(\lambda\) (requests per second at peak), the distribution of prompt lengths \(n_{\text{in}}\), the distribution of output lengths \(n_{\text{out}}\), and the latency targets (an SLO such as “p95 TTFT under 1 s and TPOT under 50 ms”, session 136). We use the mean lengths here; later we discuss tails.

Work per request

Each request needs prefill and decode FLOPs, and it holds KV memory while it runs. With \(\bar L = n_{\text{in}} + n_{\text{out}}/2\), the average context during decoding:

\[ C_{\text{req}} = 2N\,(n_{\text{in}} + n_{\text{out}}) + 2Ld\,\big(n_{\text{in}}^2 + 2\, n_{\text{out}}\, \bar L\big) \tag{141.7} \]

The first term is the linear layers for every token; the second is attention (prefill from (141.4), and \(4Ld\) FLOPs per decoded token per token of context). For \(n_{\text{in}} = 2{,}000\) and \(n_{\text{out}} = 300\): \(C_{\text{req}} = 38.3\) TFLOP, of which prefill is 33.2 TFLOP. Prompts dominate the compute even though decode dominates the wall-clock time.

One GPU under load

Assume continuous batching with chunked prefill (session 84): in each step of length \(t\), the GPU decodes one token for each of the \(B\) running requests and also processes a chunk of waiting prompt tokens. In a steady state at rate \(\lambda\) per GPU, Little’s law (session 81) gives \(B = \lambda\, n_{\text{out}}\, t\): requests decode for \(n_{\text{out}}\) steps each. The FLOPs done in one step are \(\lambda\, t\, C_{\text{req}}\) (one second’s work times the step length). The step costs the larger of its memory time and its compute time, as in (133.4):

\[ t = t_0 + \max\!\left( \frac{M_W + B\,\bar L\, s_{\text{kv}}}{\eta\,\text{BW}},\; \frac{\lambda\, t\, C_{\text{req}}}{u\,F} \right) \quad\Longrightarrow\quad \text{TPOT}(\lambda) = \max\!\left( \frac{t_0 + M_W/(\eta\,\text{BW})}{1 - b},\; \frac{t_0}{1 - c} \right) \tag{141.8} \]
Read the symbols
\(b = \lambda\, n_{\text{out}}\, \bar L\, s_{\text{kv}} / (\eta\,\text{BW})\)
the share of each step spent reading KV cache: it grows with load
\(c = \lambda\, C_{\text{req}} / (u F)\)
the compute utilization: the share of the GPU’s usable FLOP/s that the traffic needs
\(t_0 + M_W/(\eta\,\text{BW})\)
the TPOT floor: the time to read the weights once, which no batch size can beat

To get the closed form, put \(B = \lambda n_{\text{out}} t\) into the step-time formula. Each branch of the max is then linear in \(t\) and has its own solution, and the larger one is the answer. Three things can stop a GPU, and each gives a limit on \(\lambda\):

  • Bandwidth: the first branch reaches the TPOT target \(\tau\). This happens early when the TPOT floor is already close to \(\tau\), as on an L4.
  • Compute: \(c\) approaches 1. Then TPOT shoots up (the step must also carry more and more prompt chunks), which is the sharp knee you saw in session 81.
  • KV memory: the running requests \(B\) need \(B\,(n_{\text{in}} + n_{\text{out}})\, s_{\text{kv}}\) bytes, which must fit next to the weights.

The capacity of one GPU for this traffic is the largest rate that respects all three:

\[ \mu^{*} = \max\Big\{ \lambda \;:\; \text{TPOT}(\lambda) \le \tau \ \text{ and } \ \lambda\, n_{\text{out}}\, \text{TPOT}(\lambda)\, (n_{\text{in}} + n_{\text{out}})\, s_{\text{kv}} \le 0.9\,C - M_W \Big\} \tag{141.9} \]

Both conditions get harder as \(\lambda\) grows, so \(\mu^{*}\) is found by bisection in a few microseconds. Then the plan and the price follow:

\[ R = \left\lceil \frac{\lambda}{h\, \mu^{*}} \right\rceil, \qquad \$ \text{ per } 10^6 \text{ output tokens} = \frac{R \cdot p}{3600\, \lambda\, n_{\text{out}}} \cdot 10^6 \;\ge\; \frac{p}{3600\, \mu^{*} n_{\text{out}}} \cdot 10^6 \tag{141.10} \]
Read the symbols
\(h\)
headroom: plan each GPU for at most a share \(h\) (for example 0.8) of its capacity, to absorb bursts and long-tail requests
\(R\)
number of GPUs (replicas of the model)
\(p\)
price of one GPU-hour in dollars
right side
the cost at full load (\(\lambda = \mu^{*}\) per GPU): the best this hardware can do for this traffic
TPOT from (141.8)TPOT target \(\tau\)KV memory fullcapacity \(\mu^{*}\)
Figure 3. How one GPU behaves under load, from formula (141.8), for Llama 3.1 8B (FP8 means FP8 weights, FP8 math and an FP8 KV cache). The x-axis is requests per second sent to this GPU. Capacity \(\mu^{*}\) (141.9) is where the curve crosses the target or the KV memory runs out, whichever comes first.
How to read Figure 3

The curve starts at the TPOT floor (reading the weights once) and rises slowly while there is spare compute, then turns up sharply where compute runs out. Choose “L4 BF16”: the floor (67.9 ms) is above a 50 ms target, so no load at all is possible; raise the target to 100 ms and a small capacity appears, limited by KV memory. Choose “H100 BF16”: the floor is 7 ms and the curve stays flat until about 12.6 requests per second. Now make the prompt longer: the knee moves left, because each request needs more compute and more memory.

Algorithm 2: a capacity plan
  1. input: peak rate \(\lambda\), prompt and output lengths (mean and p95), targets \(\tau_{\text{TPOT}}\) and \(\tau_{\text{TTFT}}\), headroom \(h\), candidate configurations (GPU, weight and KV format), prices \(p\)
  2. for each configuration
  3.   if \(0.9\,C \lt M_W\): reject (weights do not fit; consider a smaller format or tensor parallelism)
  4.   if the TPOT floor \(t_0 + M_W/(\eta\,\text{BW}) \ge \tau_{\text{TPOT}}\): reject // no batch size can help
  5.   if the p95 prompt’s prefill time \(F_{\text{prefill}}/(uF) \gt \tau_{\text{TTFT}}\): reject, or plan chunked prefill on more GPUs
  6.   \(\mu^{*} \leftarrow\) bisection on (141.9) // which limit binds: bandwidth, compute or KV
  7.   \(R \leftarrow \lceil \lambda / (h \mu^{*}) \rceil\); cost \(\leftarrow R\,p\) per hour, and per million tokens by (141.10)
  8. pick the cheapest configuration that passes; then measure it with a load test at \(\lambda\) and correct \(u\), \(\eta\), \(t_0\)

Worked example: one model on an L4 and an H100

The traffic: a chat assistant with \(\lambda = 8\) requests per second at peak, prompts of 2,000 tokens (system prompt, history and retrieved documents), answers of 300 tokens, and an SLO of TPOT at most 50 ms (20 tokens per second per user, faster than people read). Headroom \(h = 0.8\). For prices we use round example values of $0.80 per hour for an L4 and $3.00 per hour for an H100. These are assumptions for the calculation, not quotes: real prices differ by provider, region and contract, and change often. Put your own prices into Figure 4.

Four configurations, step by step

L4, BF16 weights. The weights fit (16.06 of 21.6 usable GB), but the TPOT floor is \(1 + 16.06/(0.8 \times 0.3) = 67.9\) ms. That is above 50 ms: this configuration cannot meet the SLO at any load. If the product accepted 100 ms, one L4 would carry 0.68 requests per second, limited by KV memory (only 5.5 GB are left for the cache); 15 L4s would be needed.

L4, FP8. The floor halves to 34.5 ms. The bandwidth limit binds: \(\mu^{*} = (1 - 34.46/50) \cdot 0.8 \cdot 300 \times 10^9 / (300 \cdot 2{,}150 \cdot 65{,}536) = 1.76\) requests per second, or 529 output tokens per second. (The compute limit would be 3.1, the KV limit about 6.) Then \(R = \lceil 8 / (0.8 \cdot 1.76) \rceil = \lceil 5.67 \rceil = 6\) GPUs, $4.80 per hour. TTFT is at least 274 ms.

H100, BF16. The floor is 7.0 ms, far below the target. Here compute and memory run out together: at 12.6 requests per second the compute utilization \(c\) reaches 0.98 and the 186 running requests fill the 55.9 GB of KV memory. \(\mu^{*} = 12.64\) requests per second, 3,792 output tokens per second. \(R = \lceil 8 / 10.1 \rceil = 1\) GPU, $3.00 per hour. At the planned load the TPOT is only 9.4 ms with about 22 requests running, and TTFT is at least 67 ms.

H100, FP8. Twice the compute and half the memory per token: \(\mu^{*} = 25.3\) requests per second. Still 1 GPU, now running at 32% of capacity.

Cost per million output tokens at full load (the right side of 141.10): L4 FP8 \(0.80 / (3600 \cdot 529) \times 10^6 = \$0.42\); H100 BF16 $0.22; H100 FP8 $0.11. As planned for 8 requests per second (the left side): L4 FP8 $0.56, H100 BF16 or FP8 $0.35. The H100 costs 3.75 times more per hour but carries 7.2 times more traffic of this kind, so per token the expensive GPU is cheaper.

When the L4 wins. GPUs come in whole units. At low traffic, one L4 ($0.80) is enough where an H100 ($3.00) would be mostly idle: at 0.5 requests per second the cost per million output tokens is $1.48 on the L4 and $5.56 on the H100. With these prices the L4 FP8 plan is cheaper up to about 4.2 requests per second (3 L4s for $2.40 per hour), and the H100 is cheaper above that. The L4 also uses 72 W and fits in small servers. Its users, though, see a TTFT 4 times longer and a TPOT near 45 ms instead of 9.

Long prompts change everything. With 32,000-token prompts and 500-token answers (document QA), one H100 in BF16 carries only 0.60 requests per second, TTFT is at least 1.6 s, and the full-load cost rises to $2.76 per million output tokens, 12.5 times the chat case. The L4 FP8 needs 6.5 s just for one prefill.

$ per 1M output tokens at full load$ per 1M output tokens as planned (whole GPUs)
Figure 4. A capacity planner for Llama 3.1 8B, using Algorithm 2 with formulas (141.7) to (141.10). The green row is the cheapest configuration that meets the TPOT target. Prices are example inputs; set your own.
How to read Figure 4

At the defaults the H100 plan is cheapest. Lower the rate to 2 requests per second: one H100 is mostly idle and the L4 FP8 plan wins. Raise the rate to 50: the H100 wins clearly, and FP8 halves the number of GPUs. Make the prompt 32K: every configuration gets much more expensive, and the L4 drops out on latency. Finally loosen the TPOT target to 100 ms: L4 BF16 becomes possible, and the L4 FP8 capacity roughly doubles.

What the simple model leaves out

Formula (141.8) uses mean lengths and a steady stream. Real traffic has tails: if 5% of prompts are 30K tokens, they take most of the prefill compute and most of the KV memory, so plan with the p95 lengths for memory and check TTFT with the longest prompts. It has bursts, which is what the headroom \(h\) is for (session 81 showed how sharp the knee is). Prefix caching lowers the effective \(n_{\text{in}}\) in (141.7): with a 50% hit rate, chat traffic with long shared system prompts needs much less compute. Speculative decoding lowers TPOT at small batches but helps less when the GPU is already compute-bound (session 128). And a model that does not fit one GPU needs tensor parallelism, which changes \(F\), \(\text{BW}\) and \(C\) (all multiplied by the number of GPUs) and adds communication time (session 137). Always end a plan with a load test.

The code

Part 1 runs a small agent episode on a tiny random causal model with a hand-written KV cache, and checks that prefilling only the new tokens gives exactly the same logits as recomputing the whole history. Part 2 is the planner of (141.8) to (141.10); it prints the numbers of the worked example.

import math, torch
torch.manual_seed(0)
torch.set_default_dtype(torch.float64)

# ---- Part 1: an agent episode on a tiny causal model, with and without a prefix cache ----
V, D, H, NL = 50, 32, 4, 2
emb = torch.randn(V, D) * 0.5
Wqkv = [torch.randn(D, 3 * D) / math.sqrt(D) for _ in range(NL)]
Wo = [torch.randn(D, D) / math.sqrt(D) for _ in range(NL)]
head = torch.randn(D, V) / math.sqrt(D)

def forward(ids, cache=None):
    """Run new tokens `ids` after the tokens already in `cache`. Returns logits and the new cache."""
    x = emb[ids]                                    # (n, D)
    n, past = len(ids), (0 if cache is None else cache[0][0].shape[1])
    new_cache = []
    for l in range(NL):
        q, k, v = (x @ Wqkv[l]).split(D, dim=-1)
        q, k, v = (t.view(n, H, D // H).transpose(0, 1) for t in (q, k, v))
        if cache is not None:                       # reuse the keys and values of earlier tokens
            k = torch.cat([cache[l][0], k], 1); v = torch.cat([cache[l][1], v], 1)
        new_cache.append((k, v))
        s = q @ k.transpose(1, 2) / math.sqrt(D // H)
        qpos = torch.arange(past, past + n)[:, None]; kpos = torch.arange(past + n)[None, :]
        s = s.masked_fill(kpos > qpos, float("-inf"))
        x = x + (torch.softmax(s, -1) @ v).transpose(0, 1).reshape(n, D) @ Wo[l]
    return x @ head, new_cache

g = torch.Generator().manual_seed(1)
system = torch.randint(0, V, (300,), generator=g)    # shared system prompt + tool list
task = torch.randint(0, V, (20,), generator=g)
k_calls, out_len, obs_len = 6, 15, 60

history, cache = torch.cat([system, task]), None
cost_full, cost_cached, fed = 0, 0, 0               # tokens prefilled; tokens already in the cache
for call in range(k_calls):
    # (a) no cache: prefill the whole history again
    logits_full, _ = forward(history); cost_full += len(history)
    # (b) prefix cache: prefill only the tokens not yet in the cache
    new = history[fed:]
    logits_c, cache = forward(new, cache); cost_cached += len(new); fed = len(history)
    assert torch.allclose(logits_full[-1], logits_c[-1], atol=1e-9)
    # greedy decode out_len tokens with the cache; decoded tokens stay in it
    tok = int(logits_c[-1].argmax()); gen = [tok]
    for _ in range(out_len - 1):
        lg, cache = forward(torch.tensor([tok]), cache); tok = int(lg[-1].argmax()); gen.append(tok)
    fed += out_len - 1                              # the last token has no KV yet
    obs = torch.randint(0, V, (obs_len,), generator=g)   # the tool result
    history = torch.cat([history, torch.tensor(gen), obs])
print("prefill tokens without cache:", cost_full, " with cache:", cost_cached,
      f" ratio {cost_full / cost_cached:.2f}x")
# prefill tokens without cache: 3045  with cache: 625  ratio 4.87x

# ---- Part 2: the capacity model (141.7)-(141.10) ----
N, L, d = 8.03e9, 32, 4096
GPUS = {"L4 FP8":    dict(F=242e12, BW=300e9,  C=24e9, wb=1, kvb=1, price=0.80),
        "H100 BF16": dict(F=989e12, BW=3.35e12, C=80e9, wb=2, kvb=2, price=3.00)}
u, eta, t0, usable = 0.5, 0.8, 1e-3, 0.9

def tpot(g, lam, nin, nout):                      # formula (141.8)
    s_kv = 2 * L * 8 * 128 * g["kvb"]; Lb = nin + nout / 2
    C_req = 2 * N * (nin + nout) + 2 * L * d * (nin**2 + 2 * nout * Lb)   # (141.7)
    b = lam * nout * Lb * s_kv / (eta * g["BW"]); c = lam * C_req / (u * g["F"])
    if b >= 1 or c >= 1: return math.inf, s_kv
    return max((t0 + N * g["wb"] / (eta * g["BW"])) / (1 - b), t0 / (1 - c)), s_kv

def max_rate(g, nin, nout, tau):                  # formula (141.9), by bisection
    free = g["C"] * usable - N * g["wb"]
    def ok(lam):
        t, s_kv = tpot(g, lam, nin, nout)
        return t <= tau and lam * nout * t * (nin + nout) * s_kv <= free
    lo, hi = 0.0, 1e4
    for _ in range(100):
        mid = (lo + hi) / 2; lo, hi = (mid, hi) if ok(mid) else (lo, mid)
    return lo

lam, nin, nout, tau, headroom = 8, 2000, 300, 0.050, 0.8
for name, g in GPUS.items():
    mu = max_rate(g, nin, nout, tau)
    R = math.ceil(lam / (headroom * mu))            # formula (141.10)
    usd_full = g["price"] / 3600 / (mu * nout) * 1e6
    print(f"{name:9s}  max {mu:5.2f} req/s per GPU  -> {R} GPU(s), "
          f"${R * g['price']:.2f}/h, ${usd_full:.2f} per 1M output tokens at full load")
# L4 FP8     max  1.76 req/s per GPU  -> 6 GPU(s), $4.80/h, $0.42 per 1M output tokens at full load
# H100 BF16  max 12.64 req/s per GPU  -> 1 GPU(s), $3.00/h, $0.22 per 1M output tokens at full load

In the episode, the cached run prefills the 320-token start once and then \(60 + 1\) tokens per call (the tool result, plus the last generated token, whose KV is written only when it is fed back). That is \(320 + 5 \cdot 61 = 625\) tokens against \(6 \cdot 320 + 75 \cdot 15 = 3{,}045\), exactly formula (141.2) with \(r = 61\). The assertion shows that the cache changes the cost, never the result.

Where this is used

ScenarioWhat it stressesWhat helps most
Coding and research agents (SWE-bench style, tool loops)Prefill of a growing history; KV memory held during tool calls; sequential latencyPrefix caching with cache-aware routing, KV offload to CPU or SSD, parallel tool calls
Chat with RAGLong prompts (retrieved documents) with short answers: compute in prefillPrefix caching of the system prompt, prompt compression, chunked prefill, FP8
Long-document QA, contract review, codebase questions\(T^2\) prefill, KV capacity, low batchSeparate long-context pool, FP8 KV cache, sparse prefill (MInference) and decode (Quest), context parallelism
Video understanding and Video-RAGVery long visual contexts: for example 10 minutes at 1 frame per second and 196 tokens per frame is 117,600 tokensFewer frames or token merging before the LLM, retrieval of key moments instead of the whole video, then the long-context tools above
Video retrieval and classification at scale (embeddings, fingerprinting, action recognition)Throughput of an encoder, no decode at allLarge batches, offline scheduling, cheapest GPU per frame; the decode-side limits here do not apply
On-device assistantsMemory size and bandwidth at batch 1 (141.6), power1B to 3B models in 4-bit, short prompts, cached system prompt, cloud fallback
Offline batch jobs (nightly summarization, labeling, synthetic data)Only cost; no TPOT targetLargest batch that fits, cheapest $ per token (141.10, right side), spot or idle capacity

Common mistakes

  • Sizing from tokens per second alone. “The engine does 4,000 tokens/s, we need 2,400, so one GPU” ignores the latency target, the prompt length and the KV memory. Use all three limits (141.9).
  • Using a GPU whose TPOT floor is above the target. No batch size and no number of replicas can beat \(t_0 + M_W/(\eta\,\text{BW})\). Check the floor first; then quantize the weights or pick a faster GPU.
  • Planning with mean lengths only. A few very long prompts can take most of the compute and memory. Check p95 and maximum lengths for TTFT and for KV memory.
  • Random load balancing for agents. It spreads the calls of one episode over replicas and throws away the prefix cache. Route by episode or by longest cached prefix.
  • Expecting sparse attention to fit more users. Query-aware methods like Quest only cut the bytes read; the full cache still sits in memory. To fit more sequences, compress (FP8, INT4 KV) or evict.
  • Planning at 100% of capacity. The knee of Figure 3 is sharp; plan with headroom (\(h \approx 0.7\) to 0.8) and autoscale on queue length (session 133).
  • Comparing GPUs by price per hour. Compare by price per million tokens at your traffic and SLO; and remember that GPUs come in whole units, so at low traffic the smaller GPU can win.
  • Treating the model’s outputs as the only tokens. In agents and RAG, input tokens are often many times the output tokens (44 times in the agent example without a cache), and they dominate compute. Count both.

Interview questions

An agent makes 10 LLM calls per task. Why can it cost far more than 10 chat requests, and how do you fix it?core
Each call’s prompt contains the whole history, so without caching the prefill tokens grow like \(k(S+q) + (o+r)k(k-1)/2\) (141.2), quadratic in the number of calls. With a prefix cache that keeps the episode’s KV between calls, each call prefills only the new tool output, so the cost becomes linear. In the example of this session (10 calls, 3,000-token prefix) that is 65,750 versus 8,600 tokens. To make it work in production you need cache-aware routing (same replica for the episode), a policy for keeping or offloading the cache during tool calls, and caching of the shared system prompt across users.
At what context length does attention dominate prefill compute for Llama 3.1 8B?core
Linear layers cost \(2NT\), causal attention about \(2LdT^2\). They are equal at \(T^{*} = N/(Ld) = 8.03 \times 10^9 / (32 \cdot 4096) \approx 61{,}000\) tokens. Below 8K attention is around 12% of prefill FLOPs; at 128K it is about 68%. In practice attention kernels run at lower efficiency than big GEMMs, so the time share of attention is often higher than the FLOP share. [Likely]
Does sparse attention let you serve more long-context users at once?advanced
It depends on the type. Query-aware sparse decoding (Quest) keeps the full KV cache and only reads the top-\(K\) pages per query: it shortens decode steps (less bandwidth) but does not free memory, so the number of sequences that fit is unchanged (141.5). Eviction methods (StreamingLLM, H2O, SnapKV) drop tokens and do free memory, at the risk of losing information that a later question needs. Compression of the cache (FP8, INT4) frees memory without dropping tokens. Sparse prefill (MInference) reduces TTFT, not memory.
We expect 8 requests per second, 2K-token prompts, 300-token answers, \(\text{TPOT} \le 50\) ms, with an 8B model. How many GPUs?core
Compute the work per request (about 38 TFLOP, mostly prefill) and the KV per request (about 0.3 GB in BF16). For each GPU, check the TPOT floor (weights read time), then find the largest rate where TPOT stays under target and the running batch fits in memory. With the assumptions of this session: one H100 in BF16 carries about 12.6 requests per second (compute and KV run out together), so with 80% headroom one H100 is enough; an L4 in BF16 cannot meet 50 ms at all (floor 68 ms), and in FP8 it carries about 1.8 requests per second, so 6 L4s. Then validate with a load test and correct the efficiency assumptions.
Why can an L4 not serve an 8B BF16 model at 50 ms per token, even for one user?basic
Each decode step must read all 16 GB of weights. At 300 GB/s peak, and about 80% of that in practice, this takes about 67 ms, plus overhead. Batching does not help the per-user TPOT, it only shares that read among users. The fixes are smaller weights (FP8 halves the floor to about 34 ms, INT4 to about 18 ms), a faster GPU, or splitting the model over several GPUs.
An H100 costs about four times more per hour than an L4. How can it be cheaper per token?core
Because it carries more than four times the traffic. It has about 11 times the bandwidth, 8 times the compute and 3.3 times the memory, so it reaches much larger batches before any limit binds. For 2K/300 chat traffic at 50 ms TPOT the model of this session gives 12.6 requests per second on an H100 BF16 against 1.76 on an L4 FP8, 7.2 times more, while the example price ratio is 3.75. Cost per token is price per hour divided by tokens per hour, not price per hour alone.
When would you still choose the smaller GPU?advanced
When traffic is low, because GPUs come in whole units: at 0.5 requests per second one L4 is enough and one H100 would be mostly idle, so the L4 is several times cheaper per token. Also for power- or space-limited sites (an L4 is 72 W), for many small models that each need their own GPU, and for regions where large GPUs are not available. You pay with longer TTFT and TPOT, so the SLO must allow it.
Which of bandwidth, compute and KV memory limits a GPU, and when?advanced
Bandwidth binds when the time to read the weights is already close to the TPOT target, as on an L4: little room is left for KV reads, so the batch stays small. Compute binds when prompts are long and the GPU is fast and large, as on an H100 with 2K prompts: prefill chunks fill the steps and TPOT shoots up near full utilization. KV memory binds with long contexts or long outputs: the running batch \(B = \lambda n_{\text{out}} \text{TPOT}\) times the tokens per request no longer fits. Formula (141.8) shows the first two, and the memory condition in (141.9) the third.
How fast can a 7B–8B model decode on a laptop?basic
At batch 1 every token reads all the weights, so the speed is at most usable bandwidth divided by model size (141.6). In 4-bit an 8B model is about 4 GB; on a device with 100 GB/s at 80% efficiency that is at most 20 tokens per second. Prefill is compute-bound on such devices and can take seconds for a few thousand tokens. That is why on-device models are small and quantized and prompts are kept short.
A few users send 100K-token prompts and everybody else’s latency gets worse. What do you do?advanced
A 100K prefill holds an H100 for many seconds and takes a large part of its KV memory. Options: chunked prefill with a token budget so decodes keep running; a limit on prompt tokens in flight; a separate pool of replicas (or prefill–decode disaggregation) for long-context traffic with its own SLO; FP8 KV cache to fit more; prefix caching if the same document is asked about repeatedly; and sparse prefill or prompt compression to cut the prefill work. Charging long prompts separately is also a product decision.
How do the latency and the cost of a multi-model pipeline combine?basic
Latency adds along the critical path: a router, a retriever, a reranker, the main model and a guard model in sequence each add their time to TTFT, so run independent steps in parallel and keep small models fast. Cost adds over all models, weighted by how often each is called: a router that sends easy requests to a small model can reduce the main model’s load a lot. Small models are often packed together on one GPU.

Exercises

  1. An agent episode has \(k = 20\) calls, \(S = 4{,}000\), \(q = 100\), \(o = 100\), \(r = 400\). How many tokens are prefilled without a cache and with an episode cache? What is the ratio?
  2. Compute \(T^{*}\) for a model with \(N = 70 \times 10^9\), \(L = 80\) and \(d = 8192\). Is attention a large part of prefill at 32K tokens?
  3. How many 64K-token sequences fit on an H100 next to Llama 3.1 8B in BF16, with a BF16 KV cache? With an FP8 KV cache?
  4. What is the TPOT floor of Llama 3.1 8B with INT4 weights (0.5 bytes per parameter, ignore scales) on an L4, with the assumptions of this session?
  5. One GPU costs $2.00 per hour and carries 2,000 output tokens per second at full load. What is the cost per million output tokens? What is it if the traffic only fills 40% of the GPU?
  6. In Figure 4, find the smallest request rate at which the H100 BF16 plan becomes cheaper than the L4 FP8 plan with the default prices. Explain the result with formula (141.10).
Answerscheck
1. No cache: \(20 \cdot 4{,}100 + 500 \cdot 190 = 82{,}000 + 95{,}000 = 177{,}000\). Cache: \(4{,}100 + 19 \cdot 400 = 11{,}700\). Ratio about 15.1: longer episodes gain more. 2. \(T^{*} = 70 \times 10^9 / (80 \cdot 8192) = 106{,}812\) tokens. At 32K, attention FLOPs are \(2LdT^2 = 1.41 \times 10^{15}\) against \(2NT = 4.59 \times 10^{15}\), about 23% of the total: noticeable, but the linear layers still dominate. 3. One 64K sequence holds \(65{,}536 \times 131{,}072\ \text{bytes} = 8.59\ \text{GB}\); free memory is \(72 - 16.06 = 55.94\) GB, so 6 sequences; with FP8 (4.29 GB each) 13. 4. \(1 + 4.015/(0.8 \cdot 0.3) = 1 + 16.7 = 17.7\) ms. 5. \(2.00 / (3600 \cdot 2000) \times 10^6 = \$0.28\); at 40% load \(\$0.69\) (divide by 0.4). 6. About 4.5 requests per second (the slider moves in steps of 0.5). Up to about 4.2 requests per second, 3 L4s ($2.40 per hour) are enough; above it 4 L4s cost $3.20, more than one H100 at $3.00, while one H100 lasts until about 10.1 requests per second. The left side of (141.10) is a step function of \(\lambda\), and the steps of the two GPUs cross there.

Key takeaways

  • Agents repeat their history in every call: without a prefix cache prefill grows like \(k^2\) (141.2); with it, linearly. Their latency is mostly decode and tools (141.3); their real cost is KV memory held between calls.
  • Long context: attention passes the linear layers at \(T^{*} = N/(Ld)\), about 61K tokens for Llama 3.1 8B (141.4). The KV cache of one 128K sequence (17 GB) is larger than the weights, which kills batching. Query-aware sparse attention saves bandwidth, not capacity (141.5).
  • At the edge, decode speed is at most usable bandwidth divided by model size (141.6).
  • One GPU under load: TPOT as a function of request rate has a floor (reading the weights) and a sharp knee (compute); KV memory is the third limit (141.8, 141.9).
  • Plan with whole GPUs and headroom; compare cost per million tokens, not price per hour (141.10). For 2K/300 chat at 50 ms, the H100 is cheaper per token at high traffic, the L4 at low traffic, and the L4 cannot run the 8B model in BF16 at all.

Key formulas

\[ \textstyle\sum_i T_i = k(S + q) + (o + r)\frac{k(k-1)}{2}, \qquad F_{\text{prefill}} = 2NT + 2LdT^2, \qquad T^{*} = \frac{N}{Ld} \]
\[ \text{TPOT}(\lambda) = \max\!\left( \frac{t_0 + M_W/(\eta\,\text{BW})}{1 - b},\; \frac{t_0}{1 - c} \right), \qquad R = \left\lceil \frac{\lambda}{h\,\mu^{*}} \right\rceil, \qquad \frac{\$}{10^6 \text{ tok}} = \frac{R\,p \cdot 10^6}{3600\,\lambda\, n_{\text{out}}} \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Austin et al., Google DeepMind (2025). How to Scale Your Model
  3. Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  4. Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
  5. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  6. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  7. Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
  8. Zhang et al. (2023). H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  9. Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
  10. Liu et al. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context
  11. Jiang, Li, Zhang et al. (2024). MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
  12. Tang, Zhao, Zhu, Xiao, Kasikci, Han (2024). Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  13. Li et al. (2024). SnapKV: LLM Knows What You are Looking for Before Generation
  14. Qin, Li, He et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
  15. Meta (mirrored by Unsloth on Hugging Face) (2024). Llama 3.1 8B Instruct: config.json
  16. NVIDIA (docs). NVIDIA L4 Tensor Core GPU: product specifications
  17. NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
  18. NVIDIA (docs). NVIDIA A100 Tensor Core GPU: product specifications
  19. Yao et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models
  20. Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?