Scenarios and capacity planning: agents, long context, edge, cost
Real traffic is not one chat request. Agents send many calls that repeat the same long prefix, long-context requests make prefill and the KV cache explode, and edge devices have little memory and bandwidth. This session turns a description of the traffic (requests per second, prompt and output lengths, a latency target) into a number of GPUs and a cost per million tokens, and works it out for one 8B model on an L4 and on an H100.
What you will learn
- Why an agent episode of k calls costs about k²/2 prompts without a prefix cache, and only one prompt plus the new tokens with it
- Where the time and memory of long context go: the T² attention term, the crossover length N/(Ld), 128 KiB of KV cache per token, and what sparse attention saves (bandwidth, not capacity)
- A simple upper bound for edge decoding speed: memory bandwidth divided by model size
- A closed-form model of one server under load: TPOT as a function of request rate, and the three limits (bandwidth, compute, KV memory) that set the capacity of one GPU
- How to turn traffic and an SLO into a number of GPUs and a cost per million output tokens
- A worked plan for Llama 3.1 8B on an L4 and an H100: why the L4 cannot meet 50 ms in BF16, why the H100 is cheaper per token at high traffic, and when the L4 wins
- Runnable PyTorch code that checks prefix-cache reuse gives the same logits, and a planner that reproduces the worked example
Why this matters
So far we have studied one request at a time, or a uniform stream of chat requests. Real products are messier. A coding agent sends 30 calls that all start with the same 5,000-token system prompt. A legal assistant reads a 100,000-token contract. A phone runs a 3B model on a few GB of memory. And somebody must answer the question every team asks before launch: how many GPUs do we need, and what does one million tokens cost us?
The efficient-inference survey of Zhou et al. ends with exactly these scenarios (its Section 7): agent and multi-model frameworks, which “naturally introduce a new optimization level, i.e., pipeline-level”; long-context models, where self-attention is quadratic in the context length; and edge deployment on devices with very limited resources. [Certain] This session takes each scenario, finds which resource it stresses (compute, memory bandwidth, or memory size), and then builds one capacity plan from traffic to dollars.
We use one model throughout, so that every number can be checked: Llama 3.1 8B. From its config: 32 layers, hidden size 4096, 8 KV heads of size 128, maximum context 131,072 tokens. [Certain] As in session 133, we count \(N = 8.03 \times 10^9\) parameters, \(M_W = 16.06\) GB of weights in BF16 (8.03 GB in FP8), and \(s_{\text{kv}} = 2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB) of KV cache per token in BF16 (half in FP8). The two GPUs are an L4 (24 GB, 300 GB/s, 121 TFLOP/s dense BF16, 242 dense FP8, 72 W) and an H100 SXM (80 GB, 3.35 TB/s, 989 TFLOP/s dense BF16, 1,979 dense FP8). [Certain] We also keep the efficiency assumptions of session 133: matmuls reach \(u = 50\%\) of peak FLOP/s, memory reads reach \(\eta = 80\%\) of peak bandwidth, each step has \(t_0 = 1\) ms of fixed overhead, and 90% of GPU memory is usable. [Likely] These assumptions are rough; measure your own engine (session 142).
Scenario 1: agents and multi-model pipelines
An agent is a program that calls an LLM in a loop. Each call reads the whole history so far, writes a short output (a thought and a tool call, for example in the ReAct style), then a tool runs (a search, a code interpreter, a test suite), and its result is appended to the history. [Certain] Benchmarks such as SWE-bench, where a model must fix real GitHub issues, are solved this way, with tens of calls per task. [Likely]
From the server’s point of view, an agent episode is a sequence of requests with three special properties:
- Shared prefixes. Call \(i\) contains everything that call \(i-1\) contained, plus a little more. Many episodes also share the same system prompt and tool descriptions.
- Sequential dependency. Call \(i+1\) cannot start before call \(i\) and its tool have finished. Latencies add up.
- Idle gaps. While a tool runs (often seconds), the request is not in the GPU batch, but its KV cache is worth keeping.
How many tokens does an episode prefill?
Let the shared prefix (system prompt plus tool list) have \(S\) tokens, the task \(q\) tokens, and let each call write \(o\) tokens and each tool return \(r\) tokens. Then the prompt of call \(i\) (counting from 1) has
tokens. Without any cache, every call prefills its whole prompt. With a prefix cache that keeps the KV of the episode between calls (session 134), call 1 prefills \(S + q\) and every later call prefills only the new tool result: the output tokens were already written into the cache when they were decoded. Summing over \(k\) calls:
- Read the symbols
- \(k\)
- number of LLM calls in one episode
- \(S\), \(q\)
- tokens of the shared prefix (system prompt, tool list) and of the user’s task
- \(o\), \(r\)
- tokens the model writes per call, and tokens a tool returns per call
Without a cache the cost grows like \(k^2\); with it, linearly. If the shared prefix \(S\) is also cached across users (the same system prompt for everybody), call 1 only prefills \(q\). The latency of the whole episode is a sum along the chain:
Take \(k = 10\), \(S = 3{,}000\), \(q = 200\), \(o = 150\), \(r = 600\). By (141.2), without a cache the episode prefills \(10 \cdot 3{,}200 + 750 \cdot 45 = 65{,}750\) tokens. With an episode cache: \(3{,}200 + 9 \cdot 600 = 8{,}600\) tokens, 7.6 times fewer. With the system prompt cached too: \(200 + 5{,}400 = 5{,}600\).
Counting attention as well (a chunk of \(n\) new tokens after \(j\) cached ones costs \(2Nn + 4Ld\,n(j + n/2)\) FLOPs, as in session 84), the prefill work is 1.18 PFLOP without a cache and 0.160 PFLOP with it, 7.4 times less. At \(u = 50\%\) on an H100 that is 2.39 s against 0.32 s of GPU time. The TTFT of the last call falls from 376 ms to 26 ms.
But look at the latency (141.3) with \(\text{TPOT} = 10\) ms and 1 s per tool: decode is \(10 \cdot 150 \cdot 0.01 = 15\) s, tools are 9 s, and the whole episode takes 26.4 s without a cache and 24.3 s with it. The cache is a big saving in GPU cost and a small one in latency; for agents, latency is dominated by decoding and tools.
The price of keeping the cache: at the end the context holds \(3{,}200 + 1{,}500 + 5{,}400 = 10{,}100\) tokens, which is \(10{,}100 \times 131{,}072\ \text{bytes} = 1.32\ \text{GB}\) of KV cache per agent. The 55.9 GB left on an H100 after the weights keep only about 42 such agents. Agents are hungry for memory, not for compute.
How to read Figure 1
Start with “No cache”: the dark (prefilled) columns grow like a staircase and their total area grows like \(k^2\). Switch to “Cache within the episode”: only a thin dark band remains on each column (the tool result). Now raise the number of calls to 20 and the tool result to 2,000 tokens: without a cache the prefill time becomes comparable to the decode time; with a cache it stays small. Watch the KV size of the finished episode: that is the memory you must keep reserved during every tool call.
Serving agents well
Three system ideas follow from the numbers. First, cache-aware routing: send all calls of one episode to the replica that already holds its prefix; a random load balancer destroys the hit rate. SGLang’s RadixAttention was built for these “LLM programs” with many calls and shared prefixes [Certain], and its router balances load by cache hits [Likely]. Second, keep or move the cache during tool calls: keep it in GPU memory with a time limit, or move it to CPU memory or SSD and bring it back. Mooncake, the serving system of the Kimi chatbot, is organized around a large pool of KV cache in CPU memory and SSD across machines. [Certain] Third, run independent work in parallel: several tool calls, or several branches of a plan, can run at the same time (SGLang’s fork, Skeleton-of-Thought in session 91); they share the prefix, so they cost little extra prefill.
- input: episode id \(e\), shared prefix \(S\), task \(q\), cache time limit \(\tau_{\text{keep}}\)
- pick replica \(R_e\) by longest cached prefix (or a hash of \(e\)); remember it for the episode
- for each call \(i = 1, 2, \dots\)
- send the full prompt \(T_i\) to \(R_e\); the engine matches the cached prefix and prefills only the new tokens // formula (141.2)
- decode \(o\) tokens; their KV stays in the cache
- if the output is a tool call: pin the cache for at most \(\tau_{\text{keep}}\); if memory is short, offload it to CPU instead of freeing it
- run the tool (in parallel with other tools when they do not depend on each other); append its result to the prompt
- if the output is a final answer: release the cache (keep only the shared prefix)
Multi-model pipelines
Many products are not one model but a chain: a small router or classifier, an embedding model and a reranker for retrieval, the main LLM, a guard model that checks the answer, and sometimes an LLM judge. Two rules organize the cost. Latency adds along the critical path, so a 30 ms guard model before the main call adds 30 ms to TTFT for everybody; run checks in parallel with the main call when you can, or stream the answer and check it in pieces. GPU cost adds over all models, weighted by how often each one runs: a router that sends 70% of easy queries to a small model (a model cascade, session 91) can save more than any kernel optimization. Small models in the chain (embedders, rerankers, guards) usually share one GPU, since one of them alone would leave it mostly idle. [Likely]
Scenario 2: long context
Llama 3.1 8B accepts 131,072 tokens. [Certain] What does it cost to use them? Three things change as the context \(T\) grows: prefill compute, KV memory, and the bytes each decode step reads.
Prefill: the quadratic term
The linear layers cost \(2N\) FLOPs per token. Causal attention adds, per layer, about \(2 d T^2\) FLOPs (the score matrix \(QK^\top\) and the product with \(V\), each \(2dT^2\), halved by the causal mask):
- Read the symbols
- \(N\)
- parameters (8.03 billion)
- \(L\), \(d\)
- layers (32) and model width (4096); \(L \cdot d = 131{,}072\)
- \(T^{*}\)
- the context length where attention FLOPs equal the FLOPs of all linear layers
For our model \(T^{*} = 8.03 \times 10^9 / 131{,}072 = 61{,}264\) tokens. Below about 8K tokens attention is a small part of prefill (12% at 8,192); at 32K it is 35%; at 128K it is 68%. On an H100 at \(u = 50\%\), the TTFT of a single prompt is 0.30 s at 8K, 1.63 s at 32K and 13.4 s at 128K. On an L4 in FP8 the 128K prefill takes 54.6 s.
A check against a real measurement: for a 1M-token prompt, formula (141.4) gives \(2.78 \times 10^{17}\) FLOPs. On an A100 (312 TFLOP/s dense BF16) at 50% that is 29.7 minutes. The MInference paper reports that an 8B model takes 30 minutes to prefill 1M tokens on one A100. [Certain] The simple formula is in the right range.
Memory: the KV cache per sequence
The KV cache of one sequence is \(T \cdot s_{\text{kv}}\). At 128K tokens in BF16 that is 17.2 GB (16 GiB): more than the 16.06 GB of weights. An H100 has room for only 3 such sequences, an L4 with FP8 weights and an FP8 cache for 1. Long context is first a memory-size problem: it kills batching, and without batching each decode step reads all weights for one user.
Decode: bytes per step, and what sparse attention saves
Each decode step reads the weights once and the KV cache of every sequence in the batch. Sparse attention methods for decoding read only a budget of \(K\) tokens per sequence. Quest, for example, keeps the whole cache in memory but, for each query, loads only the top-\(K\) pages whose keys can matter most; it reports up to 2.23 times faster self-attention. [Certain] With a batch of \(B\) sequences:
The second formula is the bytes read per step. The first is unchanged: a query-aware method like Quest saves bandwidth, not capacity. Methods that evict tokens (StreamingLLM keeps the first tokens and a recent window; H2O and SnapKV keep the tokens that received the most attention) save capacity too, but a dropped token can never be read again. [Certain] For prefill, MInference finds sparse patterns in each head and reports up to 10 times faster prefill at 1M tokens on an A100. [Certain] Other tools for long context are FP8 or 4-bit KV caches (session 110), prompt compression (session 88), and context parallelism such as Ring Attention, which spreads one long prefill over several GPUs (session 49).
One sequence: decode step \(= 1 + (16.06 + 17.18)/(0.8 \times 3.35)\) ms \(= 13.4\) ms. With a sparse budget \(K = 4{,}096\): \(1 + (16.06 + 0.54)/2.68 = 7.2\) ms, a 1.9 times speed-up. With the full batch of 3 sequences: 26.2 ms without sparsity and 7.6 ms with it, 3.5 times. The more of the step is KV reading, the more sparse decoding helps.
How to read Figure 2
On log axes a linear cost is a straight line with slope 1 and a quadratic cost has slope 2. The “with attention” curve leaves the “linear layers only” line near \(T^{*}\) and becomes twice as steep. Move \(T\) from 1K to 128K: TTFT grows by much more than 128 times, and the number of sequences that fit falls from hundreds to a few. Then switch the KV cache to FP8: capacity doubles. Choose a sparse budget: capacity does not change, but the decode step gets shorter once \(T\) is larger than \(K\).
Scenario 3: edge devices
On a laptop or a phone the batch is 1, so every decode step reads all the weights. The memory bandwidth sets a hard upper bound on speed:
Take a device with 100 GB/s of memory bandwidth (an example value; phones and laptops range widely). Our 8B model in 4-bit weights is about 4.0 GB, so at \(\eta = 0.8\) it runs at most 20 tokens per second; a 3B model in 4 bits (1.5 GB) at most 53. Prefill on the device is compute-bound: at 10 TFLOP/s of usable compute (again an example), a 2,000-token prompt of the 8B model takes \(2 \cdot 8.03 \times 10^9 \cdot 2000 / 10^{13} = 3.2\) s. That is why on-device assistants keep prompts short, cache the system prompt, and use 1B to 3B models. The survey notes that a performance gap still exists between small and large models, and points to cloud–edge collaboration: easy requests on the device, hard ones in the cloud. [Certain] Power matters too: an L4 uses at most 72 W, a phone a few watts. Session 140 covers the runtimes (llama.cpp, MLX) and the hardware.
From traffic to GPUs
Now the main question. A traffic description has four parts: the arrival rate \(\lambda\) (requests per second at peak), the distribution of prompt lengths \(n_{\text{in}}\), the distribution of output lengths \(n_{\text{out}}\), and the latency targets (an SLO such as “p95 TTFT under 1 s and TPOT under 50 ms”, session 136). We use the mean lengths here; later we discuss tails.
Work per request
Each request needs prefill and decode FLOPs, and it holds KV memory while it runs. With \(\bar L = n_{\text{in}} + n_{\text{out}}/2\), the average context during decoding:
The first term is the linear layers for every token; the second is attention (prefill from (141.4), and \(4Ld\) FLOPs per decoded token per token of context). For \(n_{\text{in}} = 2{,}000\) and \(n_{\text{out}} = 300\): \(C_{\text{req}} = 38.3\) TFLOP, of which prefill is 33.2 TFLOP. Prompts dominate the compute even though decode dominates the wall-clock time.
One GPU under load
Assume continuous batching with chunked prefill (session 84): in each step of length \(t\), the GPU decodes one token for each of the \(B\) running requests and also processes a chunk of waiting prompt tokens. In a steady state at rate \(\lambda\) per GPU, Little’s law (session 81) gives \(B = \lambda\, n_{\text{out}}\, t\): requests decode for \(n_{\text{out}}\) steps each. The FLOPs done in one step are \(\lambda\, t\, C_{\text{req}}\) (one second’s work times the step length). The step costs the larger of its memory time and its compute time, as in (133.4):
- Read the symbols
- \(b = \lambda\, n_{\text{out}}\, \bar L\, s_{\text{kv}} / (\eta\,\text{BW})\)
- the share of each step spent reading KV cache: it grows with load
- \(c = \lambda\, C_{\text{req}} / (u F)\)
- the compute utilization: the share of the GPU’s usable FLOP/s that the traffic needs
- \(t_0 + M_W/(\eta\,\text{BW})\)
- the TPOT floor: the time to read the weights once, which no batch size can beat
To get the closed form, put \(B = \lambda n_{\text{out}} t\) into the step-time formula. Each branch of the max is then linear in \(t\) and has its own solution, and the larger one is the answer. Three things can stop a GPU, and each gives a limit on \(\lambda\):
- Bandwidth: the first branch reaches the TPOT target \(\tau\). This happens early when the TPOT floor is already close to \(\tau\), as on an L4.
- Compute: \(c\) approaches 1. Then TPOT shoots up (the step must also carry more and more prompt chunks), which is the sharp knee you saw in session 81.
- KV memory: the running requests \(B\) need \(B\,(n_{\text{in}} + n_{\text{out}})\, s_{\text{kv}}\) bytes, which must fit next to the weights.
The capacity of one GPU for this traffic is the largest rate that respects all three:
Both conditions get harder as \(\lambda\) grows, so \(\mu^{*}\) is found by bisection in a few microseconds. Then the plan and the price follow:
- Read the symbols
- \(h\)
- headroom: plan each GPU for at most a share \(h\) (for example 0.8) of its capacity, to absorb bursts and long-tail requests
- \(R\)
- number of GPUs (replicas of the model)
- \(p\)
- price of one GPU-hour in dollars
- right side
- the cost at full load (\(\lambda = \mu^{*}\) per GPU): the best this hardware can do for this traffic
How to read Figure 3
The curve starts at the TPOT floor (reading the weights once) and rises slowly while there is spare compute, then turns up sharply where compute runs out. Choose “L4 BF16”: the floor (67.9 ms) is above a 50 ms target, so no load at all is possible; raise the target to 100 ms and a small capacity appears, limited by KV memory. Choose “H100 BF16”: the floor is 7 ms and the curve stays flat until about 12.6 requests per second. Now make the prompt longer: the knee moves left, because each request needs more compute and more memory.
- input: peak rate \(\lambda\), prompt and output lengths (mean and p95), targets \(\tau_{\text{TPOT}}\) and \(\tau_{\text{TTFT}}\), headroom \(h\), candidate configurations (GPU, weight and KV format), prices \(p\)
- for each configuration
- if \(0.9\,C \lt M_W\): reject (weights do not fit; consider a smaller format or tensor parallelism)
- if the TPOT floor \(t_0 + M_W/(\eta\,\text{BW}) \ge \tau_{\text{TPOT}}\): reject // no batch size can help
- if the p95 prompt’s prefill time \(F_{\text{prefill}}/(uF) \gt \tau_{\text{TTFT}}\): reject, or plan chunked prefill on more GPUs
- \(\mu^{*} \leftarrow\) bisection on (141.9) // which limit binds: bandwidth, compute or KV
- \(R \leftarrow \lceil \lambda / (h \mu^{*}) \rceil\); cost \(\leftarrow R\,p\) per hour, and per million tokens by (141.10)
- pick the cheapest configuration that passes; then measure it with a load test at \(\lambda\) and correct \(u\), \(\eta\), \(t_0\)
Worked example: one model on an L4 and an H100
The traffic: a chat assistant with \(\lambda = 8\) requests per second at peak, prompts of 2,000 tokens (system prompt, history and retrieved documents), answers of 300 tokens, and an SLO of TPOT at most 50 ms (20 tokens per second per user, faster than people read). Headroom \(h = 0.8\). For prices we use round example values of $0.80 per hour for an L4 and $3.00 per hour for an H100. These are assumptions for the calculation, not quotes: real prices differ by provider, region and contract, and change often. Put your own prices into Figure 4.
L4, BF16 weights. The weights fit (16.06 of 21.6 usable GB), but the TPOT floor is \(1 + 16.06/(0.8 \times 0.3) = 67.9\) ms. That is above 50 ms: this configuration cannot meet the SLO at any load. If the product accepted 100 ms, one L4 would carry 0.68 requests per second, limited by KV memory (only 5.5 GB are left for the cache); 15 L4s would be needed.
L4, FP8. The floor halves to 34.5 ms. The bandwidth limit binds: \(\mu^{*} = (1 - 34.46/50) \cdot 0.8 \cdot 300 \times 10^9 / (300 \cdot 2{,}150 \cdot 65{,}536) = 1.76\) requests per second, or 529 output tokens per second. (The compute limit would be 3.1, the KV limit about 6.) Then \(R = \lceil 8 / (0.8 \cdot 1.76) \rceil = \lceil 5.67 \rceil = 6\) GPUs, $4.80 per hour. TTFT is at least 274 ms.
H100, BF16. The floor is 7.0 ms, far below the target. Here compute and memory run out together: at 12.6 requests per second the compute utilization \(c\) reaches 0.98 and the 186 running requests fill the 55.9 GB of KV memory. \(\mu^{*} = 12.64\) requests per second, 3,792 output tokens per second. \(R = \lceil 8 / 10.1 \rceil = 1\) GPU, $3.00 per hour. At the planned load the TPOT is only 9.4 ms with about 22 requests running, and TTFT is at least 67 ms.
H100, FP8. Twice the compute and half the memory per token: \(\mu^{*} = 25.3\) requests per second. Still 1 GPU, now running at 32% of capacity.
Cost per million output tokens at full load (the right side of 141.10): L4 FP8 \(0.80 / (3600 \cdot 529) \times 10^6 = \$0.42\); H100 BF16 $0.22; H100 FP8 $0.11. As planned for 8 requests per second (the left side): L4 FP8 $0.56, H100 BF16 or FP8 $0.35. The H100 costs 3.75 times more per hour but carries 7.2 times more traffic of this kind, so per token the expensive GPU is cheaper.
When the L4 wins. GPUs come in whole units. At low traffic, one L4 ($0.80) is enough where an H100 ($3.00) would be mostly idle: at 0.5 requests per second the cost per million output tokens is $1.48 on the L4 and $5.56 on the H100. With these prices the L4 FP8 plan is cheaper up to about 4.2 requests per second (3 L4s for $2.40 per hour), and the H100 is cheaper above that. The L4 also uses 72 W and fits in small servers. Its users, though, see a TTFT 4 times longer and a TPOT near 45 ms instead of 9.
Long prompts change everything. With 32,000-token prompts and 500-token answers (document QA), one H100 in BF16 carries only 0.60 requests per second, TTFT is at least 1.6 s, and the full-load cost rises to $2.76 per million output tokens, 12.5 times the chat case. The L4 FP8 needs 6.5 s just for one prefill.
How to read Figure 4
At the defaults the H100 plan is cheapest. Lower the rate to 2 requests per second: one H100 is mostly idle and the L4 FP8 plan wins. Raise the rate to 50: the H100 wins clearly, and FP8 halves the number of GPUs. Make the prompt 32K: every configuration gets much more expensive, and the L4 drops out on latency. Finally loosen the TPOT target to 100 ms: L4 BF16 becomes possible, and the L4 FP8 capacity roughly doubles.
What the simple model leaves out
Formula (141.8) uses mean lengths and a steady stream. Real traffic has tails: if 5% of prompts are 30K tokens, they take most of the prefill compute and most of the KV memory, so plan with the p95 lengths for memory and check TTFT with the longest prompts. It has bursts, which is what the headroom \(h\) is for (session 81 showed how sharp the knee is). Prefix caching lowers the effective \(n_{\text{in}}\) in (141.7): with a 50% hit rate, chat traffic with long shared system prompts needs much less compute. Speculative decoding lowers TPOT at small batches but helps less when the GPU is already compute-bound (session 128). And a model that does not fit one GPU needs tensor parallelism, which changes \(F\), \(\text{BW}\) and \(C\) (all multiplied by the number of GPUs) and adds communication time (session 137). Always end a plan with a load test.
The code
Part 1 runs a small agent episode on a tiny random causal model with a hand-written KV cache, and checks that prefilling only the new tokens gives exactly the same logits as recomputing the whole history. Part 2 is the planner of (141.8) to (141.10); it prints the numbers of the worked example.
import math, torch torch.manual_seed(0) torch.set_default_dtype(torch.float64) # ---- Part 1: an agent episode on a tiny causal model, with and without a prefix cache ---- V, D, H, NL = 50, 32, 4, 2 emb = torch.randn(V, D) * 0.5 Wqkv = [torch.randn(D, 3 * D) / math.sqrt(D) for _ in range(NL)] Wo = [torch.randn(D, D) / math.sqrt(D) for _ in range(NL)] head = torch.randn(D, V) / math.sqrt(D) def forward(ids, cache=None): """Run new tokens `ids` after the tokens already in `cache`. Returns logits and the new cache.""" x = emb[ids] # (n, D) n, past = len(ids), (0 if cache is None else cache[0][0].shape[1]) new_cache = [] for l in range(NL): q, k, v = (x @ Wqkv[l]).split(D, dim=-1) q, k, v = (t.view(n, H, D // H).transpose(0, 1) for t in (q, k, v)) if cache is not None: # reuse the keys and values of earlier tokens k = torch.cat([cache[l][0], k], 1); v = torch.cat([cache[l][1], v], 1) new_cache.append((k, v)) s = q @ k.transpose(1, 2) / math.sqrt(D // H) qpos = torch.arange(past, past + n)[:, None]; kpos = torch.arange(past + n)[None, :] s = s.masked_fill(kpos > qpos, float("-inf")) x = x + (torch.softmax(s, -1) @ v).transpose(0, 1).reshape(n, D) @ Wo[l] return x @ head, new_cache g = torch.Generator().manual_seed(1) system = torch.randint(0, V, (300,), generator=g) # shared system prompt + tool list task = torch.randint(0, V, (20,), generator=g) k_calls, out_len, obs_len = 6, 15, 60 history, cache = torch.cat([system, task]), None cost_full, cost_cached, fed = 0, 0, 0 # tokens prefilled; tokens already in the cache for call in range(k_calls): # (a) no cache: prefill the whole history again logits_full, _ = forward(history); cost_full += len(history) # (b) prefix cache: prefill only the tokens not yet in the cache new = history[fed:] logits_c, cache = forward(new, cache); cost_cached += len(new); fed = len(history) assert torch.allclose(logits_full[-1], logits_c[-1], atol=1e-9) # greedy decode out_len tokens with the cache; decoded tokens stay in it tok = int(logits_c[-1].argmax()); gen = [tok] for _ in range(out_len - 1): lg, cache = forward(torch.tensor([tok]), cache); tok = int(lg[-1].argmax()); gen.append(tok) fed += out_len - 1 # the last token has no KV yet obs = torch.randint(0, V, (obs_len,), generator=g) # the tool result history = torch.cat([history, torch.tensor(gen), obs]) print("prefill tokens without cache:", cost_full, " with cache:", cost_cached, f" ratio {cost_full / cost_cached:.2f}x") # prefill tokens without cache: 3045 with cache: 625 ratio 4.87x # ---- Part 2: the capacity model (141.7)-(141.10) ---- N, L, d = 8.03e9, 32, 4096 GPUS = {"L4 FP8": dict(F=242e12, BW=300e9, C=24e9, wb=1, kvb=1, price=0.80), "H100 BF16": dict(F=989e12, BW=3.35e12, C=80e9, wb=2, kvb=2, price=3.00)} u, eta, t0, usable = 0.5, 0.8, 1e-3, 0.9 def tpot(g, lam, nin, nout): # formula (141.8) s_kv = 2 * L * 8 * 128 * g["kvb"]; Lb = nin + nout / 2 C_req = 2 * N * (nin + nout) + 2 * L * d * (nin**2 + 2 * nout * Lb) # (141.7) b = lam * nout * Lb * s_kv / (eta * g["BW"]); c = lam * C_req / (u * g["F"]) if b >= 1 or c >= 1: return math.inf, s_kv return max((t0 + N * g["wb"] / (eta * g["BW"])) / (1 - b), t0 / (1 - c)), s_kv def max_rate(g, nin, nout, tau): # formula (141.9), by bisection free = g["C"] * usable - N * g["wb"] def ok(lam): t, s_kv = tpot(g, lam, nin, nout) return t <= tau and lam * nout * t * (nin + nout) * s_kv <= free lo, hi = 0.0, 1e4 for _ in range(100): mid = (lo + hi) / 2; lo, hi = (mid, hi) if ok(mid) else (lo, mid) return lo lam, nin, nout, tau, headroom = 8, 2000, 300, 0.050, 0.8 for name, g in GPUS.items(): mu = max_rate(g, nin, nout, tau) R = math.ceil(lam / (headroom * mu)) # formula (141.10) usd_full = g["price"] / 3600 / (mu * nout) * 1e6 print(f"{name:9s} max {mu:5.2f} req/s per GPU -> {R} GPU(s), " f"${R * g['price']:.2f}/h, ${usd_full:.2f} per 1M output tokens at full load") # L4 FP8 max 1.76 req/s per GPU -> 6 GPU(s), $4.80/h, $0.42 per 1M output tokens at full load # H100 BF16 max 12.64 req/s per GPU -> 1 GPU(s), $3.00/h, $0.22 per 1M output tokens at full load
In the episode, the cached run prefills the 320-token start once and then \(60 + 1\) tokens per call (the tool result, plus the last generated token, whose KV is written only when it is fed back). That is \(320 + 5 \cdot 61 = 625\) tokens against \(6 \cdot 320 + 75 \cdot 15 = 3{,}045\), exactly formula (141.2) with \(r = 61\). The assertion shows that the cache changes the cost, never the result.
Where this is used
| Scenario | What it stresses | What helps most |
|---|---|---|
| Coding and research agents (SWE-bench style, tool loops) | Prefill of a growing history; KV memory held during tool calls; sequential latency | Prefix caching with cache-aware routing, KV offload to CPU or SSD, parallel tool calls |
| Chat with RAG | Long prompts (retrieved documents) with short answers: compute in prefill | Prefix caching of the system prompt, prompt compression, chunked prefill, FP8 |
| Long-document QA, contract review, codebase questions | \(T^2\) prefill, KV capacity, low batch | Separate long-context pool, FP8 KV cache, sparse prefill (MInference) and decode (Quest), context parallelism |
| Video understanding and Video-RAG | Very long visual contexts: for example 10 minutes at 1 frame per second and 196 tokens per frame is 117,600 tokens | Fewer frames or token merging before the LLM, retrieval of key moments instead of the whole video, then the long-context tools above |
| Video retrieval and classification at scale (embeddings, fingerprinting, action recognition) | Throughput of an encoder, no decode at all | Large batches, offline scheduling, cheapest GPU per frame; the decode-side limits here do not apply |
| On-device assistants | Memory size and bandwidth at batch 1 (141.6), power | 1B to 3B models in 4-bit, short prompts, cached system prompt, cloud fallback |
| Offline batch jobs (nightly summarization, labeling, synthetic data) | Only cost; no TPOT target | Largest batch that fits, cheapest $ per token (141.10, right side), spot or idle capacity |
Common mistakes
- Sizing from tokens per second alone. “The engine does 4,000 tokens/s, we need 2,400, so one GPU” ignores the latency target, the prompt length and the KV memory. Use all three limits (141.9).
- Using a GPU whose TPOT floor is above the target. No batch size and no number of replicas can beat \(t_0 + M_W/(\eta\,\text{BW})\). Check the floor first; then quantize the weights or pick a faster GPU.
- Planning with mean lengths only. A few very long prompts can take most of the compute and memory. Check p95 and maximum lengths for TTFT and for KV memory.
- Random load balancing for agents. It spreads the calls of one episode over replicas and throws away the prefix cache. Route by episode or by longest cached prefix.
- Expecting sparse attention to fit more users. Query-aware methods like Quest only cut the bytes read; the full cache still sits in memory. To fit more sequences, compress (FP8, INT4 KV) or evict.
- Planning at 100% of capacity. The knee of Figure 3 is sharp; plan with headroom (\(h \approx 0.7\) to 0.8) and autoscale on queue length (session 133).
- Comparing GPUs by price per hour. Compare by price per million tokens at your traffic and SLO; and remember that GPUs come in whole units, so at low traffic the smaller GPU can win.
- Treating the model’s outputs as the only tokens. In agents and RAG, input tokens are often many times the output tokens (44 times in the agent example without a cache), and they dominate compute. Count both.
Interview questions
An agent makes 10 LLM calls per task. Why can it cost far more than 10 chat requests, and how do you fix it?core
At what context length does attention dominate prefill compute for Llama 3.1 8B?core
Does sparse attention let you serve more long-context users at once?advanced
We expect 8 requests per second, 2K-token prompts, 300-token answers, \(\text{TPOT} \le 50\) ms, with an 8B model. How many GPUs?core
Why can an L4 not serve an 8B BF16 model at 50 ms per token, even for one user?basic
An H100 costs about four times more per hour than an L4. How can it be cheaper per token?core
When would you still choose the smaller GPU?advanced
Which of bandwidth, compute and KV memory limits a GPU, and when?advanced
How fast can a 7B–8B model decode on a laptop?basic
A few users send 100K-token prompts and everybody else’s latency gets worse. What do you do?advanced
How do the latency and the cost of a multi-model pipeline combine?basic
Exercises
- An agent episode has \(k = 20\) calls, \(S = 4{,}000\), \(q = 100\), \(o = 100\), \(r = 400\). How many tokens are prefilled without a cache and with an episode cache? What is the ratio?
- Compute \(T^{*}\) for a model with \(N = 70 \times 10^9\), \(L = 80\) and \(d = 8192\). Is attention a large part of prefill at 32K tokens?
- How many 64K-token sequences fit on an H100 next to Llama 3.1 8B in BF16, with a BF16 KV cache? With an FP8 KV cache?
- What is the TPOT floor of Llama 3.1 8B with INT4 weights (0.5 bytes per parameter, ignore scales) on an L4, with the assumptions of this session?
- One GPU costs $2.00 per hour and carries 2,000 output tokens per second at full load. What is the cost per million output tokens? What is it if the traffic only fills 40% of the GPU?
- In Figure 4, find the smallest request rate at which the H100 BF16 plan becomes cheaper than the L4 FP8 plan with the default prices. Explain the result with formula (141.10).
Answerscheck
Key takeaways
- Agents repeat their history in every call: without a prefix cache prefill grows like \(k^2\) (141.2); with it, linearly. Their latency is mostly decode and tools (141.3); their real cost is KV memory held between calls.
- Long context: attention passes the linear layers at \(T^{*} = N/(Ld)\), about 61K tokens for Llama 3.1 8B (141.4). The KV cache of one 128K sequence (17 GB) is larger than the weights, which kills batching. Query-aware sparse attention saves bandwidth, not capacity (141.5).
- At the edge, decode speed is at most usable bandwidth divided by model size (141.6).
- One GPU under load: TPOT as a function of request rate has a floor (reading the weights) and a sharp knee (compute); KV memory is the third limit (141.8, 141.9).
- Plan with whole GPUs and headroom; compare cost per million tokens, not price per hour (141.10). For 2K/300 chat at 50 ms, the H100 is cheaper per token at high traffic, the L4 at low traffic, and the L4 cannot run the 8B model in BF16 at all.
Key formulas
References for this session
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
- Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
- Zhang et al. (2023). H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
- Liu et al. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context
- Jiang, Li, Zhang et al. (2024). MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
- Tang, Zhao, Zhu, Xiao, Kasikci, Han (2024). Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Li et al. (2024). SnapKV: LLM Knows What You are Looking for Before Generation
- Qin, Li, He et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- Meta (mirrored by Unsloth on Hugging Face) (2024). Llama 3.1 8B Instruct: config.json
- NVIDIA (docs). NVIDIA L4 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA A100 Tensor Core GPU: product specifications
- Yao et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models
- Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?