the whole LLM system on one whiteboard
This checkpoint puts the whole LLM system on one whiteboard: tokenizer, model, training, serving and application. You compute the numbers that decide whether a design works (parameters, FLOPs, memory, KV cache, latency) for one reference model, Llama 3 8B, and you find the bottleneck in five short case studies. It ends with a self-check, so you can fix what is still unclear before the vision part of the course.
What you will learn
- How to draw the full path of an LLM system, tokenizer → model → training → serving → application, with one key number on each box
- How to count parameters from a config.json, and why Llama 3 8B has 8,030,261,248 of them
- How to estimate training compute (6ND), GPU-hours and the 16 bytes per parameter of AdamW training
- How to compute the KV cache per token and check whether weights plus cache fit on a GPU
- How to bound decode time by memory bandwidth and prefill time by FLOP/s, and find the batch where decode turns compute-bound
- How to find the bottleneck in a trace, and name the branch of the efficient-inference survey that fixes it
- Which topics you should review before the vision track
Why this matters
You have now seen every part of an LLM system, one session at a time: how text becomes tokens, how a transformer turns tokens into a next-token distribution, how a model is trained and aligned, how it is served to many users, and how applications such as RAG and agents are built on top. This checkpoint puts all of it on one whiteboard. You draw the full path, you compute the numbers that decide whether a design works, and you find the bottleneck in five short case studies.
This is also the shape of a real interview. “Design a chat service for an 8B model” or “Our RAG system is slow, what do you do?” are not questions about one paper. The interviewer wants to see that you can draw the system, put a number on each box (memory, FLOPs, KV cache, latency), and point to the box that limits the whole thing. An engineer who can do this in ten minutes with a pen is worth more than one who knows fifty method names.
The lab has four parts, one per key point of this session:
- Draw the full path: tokenizer → model → training → serving → application (Figure 1).
- Answer exam-style questions with numbers: memory, FLOPs, KV cache, latency. A calculator (Figure 2) and a drill with 13 questions (Figure 3).
- Find the bottleneck in five short case studies, and name the branch of the efficient-inference survey (Zhou et al., 2024) that fixes it (Figure 4).
- List what is still unclear with a self-check (Figure 5). Fix those gaps before the vision part of the course (Track 7), because every vision-language model is this same system with an image encoder in front.
The session uses one reference model throughout, so that all numbers connect: Llama 3 8B. The Session 161 lab was the last application you built; now we step back and look at the whole stack under that agent.
The whiteboard: one request, from text to answer
Start with the drawing. Five boxes, left to right. Each box has an input, an output, one number that matters most, and one typical failure. Click the boxes in Figure 1, or step through them.
How to read Figure 1
Each step highlights one box and shows five lines: what goes in, what comes out, the number to remember, the formula behind that number, and the failure you see most often in practice. The last line lists the sessions to reread if the box is not clear. Notice that the same model weights appear in three places: they are produced by training, read once per token by serving, and wrapped by the application.
Here is the same drawing in words, with the facts you should know for the reference model. The shape of Llama 3 8B, from its config.json [Certain]:
| Field | Symbol | Llama 3 8B | Meaning |
|---|---|---|---|
| vocab_size | \(V\) | 128,256 | number of token IDs |
| hidden_size | \(d\) | 4,096 | width of the residual stream |
| num_hidden_layers | \(L\) | 32 | number of transformer blocks |
| num_attention_heads | \(H\) | 32 | query heads |
| num_key_value_heads | \(n_{kv}\) | 8 | key and value heads (grouped-query attention, 4 query heads share one) |
| head_dim | \(d_h\) | 128 | size of one head |
| intermediate_size | \(d_{ff}\) | 14,336 | width of the SwiGLU feed-forward layer |
| tie_word_embeddings | false | input embedding and output layer are two separate matrices |
- Tokenizer. Text in, a list of integer IDs out. Llama 3 uses a byte-level BPE vocabulary of 128K tokens (100K taken from OpenAI’s tiktoken plus 28K extra, mostly for other languages) [Certain]. A chat template wraps the messages in special tokens before tokenization. Most bugs here are silent: a wrong template or a missing end-of-turn token makes a good model look bad (session 9).
- Model. IDs in, one probability distribution over 128,256 tokens out, for every position. An embedding lookup, 32 blocks of attention plus feed-forward with residual connections and RMSNorm, then the output layer and a softmax. About \(2N\) FLOPs per token in the forward pass.
- Training. Text in, weights out. Pretraining on next-token prediction (Llama 3: over 15 trillion tokens [Certain]), then supervised fine-tuning, then preference tuning or RL. The compute is about \(6ND\) FLOPs; the memory is about 16 bytes per parameter with AdamW.
- Serving. Requests in, streamed tokens out. Prefill reads the prompt in one parallel pass (compute-bound); decode makes one token per step (memory-bound). The KV cache stores keys and values of past tokens; the scheduler batches many users together.
- Application. A user goal in, an answer out. Prompting, structured output, RAG, agents with tools, and the evaluation that tells you whether it works. Most latency here is often not in the GPU: retrieval, tool calls and repeated calls to the model add up.
The numbers you must be able to write down
A small set of formulas covers almost every exam question. You have seen all of them in earlier sessions; here they are in one place, in the order of the whiteboard.
Parameters
One block of a Llama-style model has four attention matrices (query, key, value, output), three feed-forward matrices (gate, up, down), and two RMSNorm weight vectors:
- Read the symbols
- \(N\)
- total number of parameters (weights)
- \(d, L, H, n_{kv}, d_h, d_{ff}, V\)
- the config fields in the table above
- \(2Vd\)
- input embedding plus output layer; it is \(Vd\) if they are tied
- last \(d\)
- the final RMSNorm before the output layer
For Llama 3 8B, \(P_{\text{layer}} = 218{,}112{,}000\), the 32 blocks hold 6.98 B, the two vocabulary matrices hold 1.05 B, and \(N = 8{,}030{,}261{,}248\). The feed-forward layers are 81% of each block. (The same formula gives exactly 7,615,616,512 for Qwen2.5-7B once you add its query, key and value biases, which matches the count Hugging Face reports.) [Certain: checked in Python]
Compute
A forward pass does one multiply and one add per weight per token, so about \(2N\) FLOPs per token, plus the attention scores that grow with the context. Training adds the backward pass, which costs twice the forward (session 4):
- Read the symbols
- \(t\)
- the position of the token, so the number of earlier tokens it attends to
- \(D\)
- number of training tokens
- peak
- the GPU’s dense FLOP/s for the number format (989 TFLOP/s in BF16 on an H100 SXM)
- MFU
- model FLOPs utilization: the fraction of peak that training really reaches (often 35% to 50% for large dense runs [Likely])
Memory
The 16 bytes are BF16 weights (2), BF16 gradients (2), an FP32 master copy (4) and the two AdamW moments in FP32 (4 + 4) (session 38). For inference you only need the weights, \(b_w\) bytes each: 2 for BF16, 1 for FP8, about 0.5 for INT4.
The KV cache keeps one key and one value vector per KV head, per layer, per token:
For Llama 3 8B in BF16: \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes, exactly 128 KiB per token. Without grouped-query attention (32 KV heads) it would be 512 KiB. A serving GPU must hold the weights and the cache of every running sequence:
- Read the symbols
- \(B\)
- number of sequences served at the same time (the batch)
- \(T\)
- tokens held per sequence (prompt plus output so far)
- \(u\)
- the fraction of GPU memory you let the engine use; we take 0.9, the default memory fraction in vLLM [Likely]
- \(M_{\text{GPU}}, G\)
- memory of one GPU, and the number of GPUs the model is split across
Latency
In decode, each step makes one token for each of the \(B\) sequences. It must read all weights and all cached keys and values from memory, and it does \(2N\) FLOPs per sequence. The step can be no faster than the slower of the two:
In prefill, the whole prompt of \(P\) tokens goes through in one pass. Many tokens share each weight read, so prefill is compute-bound. With causal attention over the prompt:
- Read the symbols
- BW
- memory bandwidth in bytes per second (3.35 TB/s on an H100 SXM)
- \(\eta\)
- the fraction of peak FLOP/s the prefill kernels reach; we use 0.5 when a question says so [assumption]
- TTFT
- time to first token, ignoring queueing and network
Which side of the roofline?
The arithmetic intensity of a step is FLOPs done per byte read. For decode with weights dominating the bytes, it is about \(2NB / (b_w N) = 2B / b_w\). The GPU becomes compute-bound only when this exceeds its ridge point:
On an H100, \(989 / 3.35 = 295\) FLOP/byte. With BF16 weights (\(b_w = 2\)), decode stays memory-bound until about 295 sequences share each weight read. With INT4 weights the crossover is near 74, because each byte now feeds four times more math. Below the crossover, a bigger batch is almost free; above it, every extra sequence costs extra time.
How many requests are in the system?
Finally, the link between traffic and memory is Little’s law (session 141): on average, the number of requests in the system equals the arrival rate times the time each one stays.
20 requests per second that each take 5 seconds means about 100 sequences in flight. Put that \(B\) into formula (162.7), and you know whether the KV cache fits.
Parameters (162.2): \(N = 8.03\) B. Weights in BF16: \(2N = 16.06\) GB. Training state (162.5): \(16N = 128.5\) GB, more than one 80 GB GPU before any activation, which is why full fine-tuning needs sharding or LoRA.
Pretraining compute (162.3) for 15 T tokens: \(6 \times 8.03 \times 10^9 \times 15 \times 10^{12} = 7.23 \times 10^{23}\) FLOPs. At 40% MFU on H100s (162.4): \(7.23 \times 10^{23} / (989 \times 10^{12} \times 0.4 \times 3600) \approx 507{,}000\) GPU-hours.
KV cache (162.6): 128 KiB per token, so 1.07 GB for one 8,192-token sequence. Capacity (162.7): \(0.9 \times 80 - 16.06 = 55.9\) GB is left, enough for 52 sequences of 8,192 tokens.
Decode (162.8) for one user, ignoring the cache: \(16.06 \text{ GB} / 3.35 \text{ TB/s} = 4.79\) ms per token, at most 209 tokens per second. With an 8,192-token cache, the step reads 17.13 GB and takes 5.11 ms.
Prefill (162.9) of a 2,000-token prompt at 50% of peak: \(2NP = 3.21 \times 10^{13}\) FLOPs plus \(1.05 \times 10^{12}\) for attention (3%), so TTFT is about 67 ms.
Figure 2 puts all of these formulas into one calculator. Change the model, the GPU, the precision, the batch and the context, and watch which limit you hit first.
How to read Figure 2
Start with Llama 3 8B on one H100, batch 1: the memory bar is mostly free, and the step time is 5.1 ms, set by memory bandwidth. Move the batch up: the step time barely changes until the KV cache starts to dominate the bytes read, so aggregate throughput grows almost linearly. Now pick Llama 3 70B: it does not fit on one GPU in BF16 (141 GB). Choose 4 GPUs or INT4 weights. Pick the L4: the same 8B model fits, but every step is 11 times slower because the L4 reads 300 GB/s instead of 3,350. Switch the KV cache to FP8 at long contexts and watch the maximum batch double.
Exam-style questions with numbers
Now practice. The drill in Figure 3 has 13 questions of the kind you will meet in interviews and exams. Type a number and press Check. An answer within 5% counts as correct, because whiteboard math should be fast, not exact to the last digit. Each question has a full solution.
How to read Figure 3
Try each question on paper first, with the formulas above. The score at the top right counts questions you got right on the first try (it is saved in this browser only). If you are wrong, the readout tells you how far off you are, which often points at the mistake: a factor of 2 is usually bytes per parameter or keys plus values; a factor of 1,000 is units.
Two of these questions deserve to be done slowly, because they combine several boxes of the whiteboard.
Each request has about 3,000 tokens of prompt and history and lives for 5 seconds. By Little’s law (162.11), \(\bar B = 20 \times 5 = 100\) sequences are in flight on average.
KV cache (162.6, 162.7): \(100 \times 3{,}000 \times 131{,}072 \text{ bytes} = 39.3\) GB. Weights are 16.06 GB, so the total is 55.4 GB, under the 72 GB usable. It fits.
Decode step (162.8): bytes read \(= 16.06 + 39.3 = 55.4\) GB, so the step takes at least \(55.4 / 3.35 = 16.5\) ms. FLOPs \(= 2N \times 100 = 1.6 \times 10^{12}\), which take 1.6 ms at peak. Still memory-bound, now because of the cache, not the weights. Each user sees at most about 60 tokens per second; the GPU produces about 6,000 tokens per second in total. If the product needs 100 tokens per second per user, you need FP8 KV cache, a second GPU, or shorter histories.
From (162.2) with \(d = 8{,}192\), \(L = 80\), \(H = 64\), \(n_{kv} = 8\), \(d_{ff} = 28{,}672\): \(N = 70.55\) B, so 141.1 GB in BF16. Two H100s (144 GB usable at 90%) would hold the weights and almost no cache. The KV cache is 320 KiB per token (162.6). With tensor parallelism over 4 GPUs, each holds 35.3 GB of weights and has 36.7 GB left for the cache; together that is about 448,000 tokens of BF16 cache. With INT4 weights (about 35 GB), one 80 GB GPU can serve it at short contexts.
Finding the bottleneck
A system is as fast as its slowest part. To make it faster you must first know which part that is. Guessing is the most common and most expensive mistake: a team spends a month on a faster attention kernel, and the users see no change, because 50% of the time was spent in tool calls.
- input: model config, GPU (peak, BW, memory), number formats \(b_w, b_{kv}\), traffic (\(\lambda\), prompt length \(P\), output length \(O\))
- \(N \leftarrow\) formula (162.2); \(\text{kv}_{\text{tok}} \leftarrow\) formula (162.6)
- \(G \leftarrow\) smallest number of GPUs with \(b_w N \lt u \cdot M_{\text{GPU}} \cdot G\) // weights must fit, with room left
- \(W \leftarrow\) first guess of the request lifetime: TTFT (162.9) \(+\; O \times t_{\text{step}}\) at batch 1
- repeat until \(W\) stops changing:
- \(B \leftarrow \lambda W\) // Little’s law (162.11)
- if \(b_w N + B (P + O)\, \text{kv}_{\text{tok}} \gt u M_{\text{GPU}} G\): add GPUs, lower \(b_{kv}\), or reject the design
- \(t_{\text{step}} \leftarrow\) formula (162.8) at this \(B\); \(W \leftarrow \text{TTFT} + O \cdot t_{\text{step}}\)
- return \(G\), \(B\), TTFT, \(t_{\text{step}}\), aggregate throughput \(B / t_{\text{step}}\)
The estimate tells you where the system should be. The diagnosis compares that with what you observe:
- input: a trace of real requests (timestamps of arrival, first token, each token, tool calls), GPU metrics
- split the end-to-end time of a typical request into: queueing, prefill, decode, work outside the model (retrieval, tools, network)
- take the largest part. If it is outside the model: change the call pattern first (fewer calls, parallel calls, caching) // data-level: output organization
- if it is prefill: compare the prompt length with what the task needs // data-level: input compression; system-level: prefix caching
- if it is decode at small batch: compare \(t_{\text{step}}\) with weight bytes / BW // model-level: quantization; system-level: speculative decoding
- if the batch cannot grow because memory is full: compare \(B\) with formula (162.7) // KV-cache quantization, efficient attention, paged memory
- if it is queueing while the GPU is underused: look at how requests are batched and scheduled // system-level: continuous batching, scheduling
- apply one fix, measure again, and go back to step 2: the bottleneck moves
The last column of Algorithm 2 uses the map of the efficient-inference survey (Zhou et al., arXiv 2404.14294), which you followed through Track 5. Its top levels are [Certain, from the paper]:
| Level | Branch | Sub-branches | What it attacks |
|---|---|---|---|
| Data-level | Input compression | prompt pruning, prompt summary, soft prompt-based compression, retrieval-augmented generation | too many prompt tokens |
| Output organization | Skeleton-of-Thought, parallel calls, SGLang-style programs | sequential generation and sequential calls | |
| Model-level | Efficient structure design | efficient FFN (MoE), efficient attention (MQA, GQA, low-complexity attention), transformer alternates | FLOPs and KV bytes built into the architecture |
| Model compression | quantization, sparsification, structure optimization, knowledge distillation, dynamic inference | bytes and FLOPs of a trained model | |
| System-level | Inference engine | graph and operator optimization, speculative decoding, offloading | how fast one batch runs |
| Serving system | memory management, continuous batching, scheduling strategy, distributed systems | how many requests share the hardware, and in what order |
Figure 4 gives you five case studies. Read the symptoms, pick the bottleneck, and then see the computation and the survey branches that fix it.
How to read Figure 4
Do not look at the bar first; read the story and make a guess. Then check the bar: it is the whiteboard estimate of each part, so it tells you which part is large. A wrong answer shows why that part is not the problem here. The survey map has three columns (data-level, model-level, system-level). Highlighted boxes are the fixes worth trying first, not the only ones.
A summary of the five answers, so you can check your reasoning afterwards:
| Case | Bottleneck | Key number | Survey branch that fixes it |
|---|---|---|---|
| 1. One user, slow tokens (A100) | memory bandwidth in decode | 15.0 GB read per token, 7.4 ms at 2.04 TB/s; observed 8.5 ms | model compression: quantization (INT4: 2.3 ms); inference engine: speculative decoding |
| 2. RAG with 24,000-token prompts | prefill compute | 1.08 s of prefill at 50% of peak; 28% of it is attention | input compression (fewer, better chunks: 4,800 tokens give 0.17 s); memory management (prefix caching) |
| 3. 32K-token contexts, small batch | KV-cache capacity | 4.29 GB per sequence, only 13 fit; about 606 tokens/s in total | quantization of the KV cache (26 fit, about 1,210 tokens/s); efficient attention; memory management |
| 4. Static batches, long waits | batching and scheduling | slot use 25% (mean output 200, longest 800) | serving system: continuous batching, scheduling strategy |
| 5. Agent with 10 steps, 19 s | work outside the GPU and the call pattern | tools 10.0 s (52%), decode 7.2 s, prefill 1.9 s | output organization (parallel tool calls, fewer and shorter steps); prefix caching cuts prefill to 0.27 s |
The whiteboard in code
The code below has two parts. First, formulas (162.1) to (162.9) as one Python function, applied to Llama 3 8B on an H100. Second, a tiny Llama-shaped model in PyTorch (RMSNorm, RoPE, grouped-query attention, SwiGLU, untied output layer), used to check that the parameter formula and the KV-cache formula match real tensors, and that cached decoding gives the same logits as recomputing everything. It runs on a CPU in a few seconds.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # ---- 1) the whiteboard formulas (162.1) to (162.9) as one function ---- def whiteboard(V, d, L, H, KV, dh, dff, gpu, B=1, T=8192, P=2000, bw=2, bkv=2, eff=1.0): peak, bwidth, mem = gpu layer = 2*d*H*dh + 2*d*KV*dh + 3*d*dff + 2*d # (162.1) N = L*layer + 2*V*d + d # (162.2) untied embeddings kv_tok = 2*L*KV*dh*bkv # (162.6) used = bw*N + B*T*kv_tok # (162.7) t_mem = used / (eff*bwidth); t_cmp = 2*N*B / (eff*peak) t_step = max(t_mem, t_cmp) # (162.8) ttft = (2*N*P + 2*L*P*P*H*dh) / (eff*peak) # (162.9) return dict(N=N, weights_GB=bw*N/1e9, kv_KiB=kv_tok/1024, used_GB=used/1e9, fits=used <= 0.9*mem, tpot_ms=1e3*t_step, ttft_ms=1e3*ttft, train_flops=6*N*15e12, adam_GB=16*N/1e9) # (162.3), (162.5) H100 = (989e12, 3.35e12, 80e9) llama3_8b = dict(V=128256, d=4096, L=32, H=32, KV=8, dh=128, dff=14336) for k, v in whiteboard(**llama3_8b, gpu=H100).items(): print(f"{k:12s} {v:.4g}" if not isinstance(v, bool) else f"{k:12s} {v}") print("TTFT at 50% of peak:", round(whiteboard(**llama3_8b, gpu=H100, eff=0.5)["ttft_ms"], 1), "ms") # ---- 2) a tiny Llama-shaped model, to check the formulas against real tensors ---- class RMSNorm(nn.Module): def __init__(s, d): super().__init__(); s.w = nn.Parameter(torch.ones(d)) def forward(s, x): return s.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-5) def rope(x, pos): # x: (B, heads, T, dh) dh = x.shape[-1]; inv = 1.0 / (10000 ** (torch.arange(0, dh, 2) / dh)) ang = pos[:, None].float() * inv[None] # (T, dh/2) c, s = ang.cos(), ang.sin(); x1, x2 = x[..., 0::2], x[..., 1::2] return torch.stack([x1*c - x2*s, x1*s + x2*c], -1).flatten(-2) class Block(nn.Module): def __init__(s, d, H, KV, dh, dff): super().__init__(); s.H, s.KV, s.dh = H, KV, dh s.n1, s.n2 = RMSNorm(d), RMSNorm(d) s.q = nn.Linear(d, H*dh, bias=False); s.k = nn.Linear(d, KV*dh, bias=False) s.v = nn.Linear(d, KV*dh, bias=False); s.o = nn.Linear(H*dh, d, bias=False) s.gate = nn.Linear(d, dff, bias=False); s.up = nn.Linear(d, dff, bias=False) s.down = nn.Linear(dff, d, bias=False) def forward(s, x, cache, start): B, T, _ = x.shape; h = s.n1(x); pos = torch.arange(start, start + T) q = rope(s.q(h).view(B, T, s.H, s.dh).transpose(1, 2), pos) k = rope(s.k(h).view(B, T, s.KV, s.dh).transpose(1, 2), pos) v = s.v(h).view(B, T, s.KV, s.dh).transpose(1, 2) if cache is not None: k = torch.cat([cache[0], k], 2); v = torch.cat([cache[1], v], 2) rep = s.H // s.KV # grouped-query attention a = F.scaled_dot_product_attention(q, k.repeat_interleave(rep, 1), v.repeat_interleave(rep, 1), is_causal=(cache is None)) x = x + s.o(a.transpose(1, 2).reshape(B, T, -1)) h = s.n2(x); x = x + s.down(F.silu(s.gate(h)) * s.up(h)) return x, (k, v) class TinyLM(nn.Module): def __init__(s, V, d, L, H, KV, dh, dff): super().__init__(); s.emb = nn.Embedding(V, d) s.blocks = nn.ModuleList(Block(d, H, KV, dh, dff) for _ in range(L)) s.norm = RMSNorm(d); s.head = nn.Linear(d, V, bias=False) def forward(s, ids, caches=None, start=0): x = s.emb(ids); new = [] for i, b in enumerate(s.blocks): x, c = b(x, None if caches is None else caches[i], start); new.append(c) return s.head(s.norm(x)), new cfg = dict(V=1000, d=256, L=4, H=8, KV=2, dh=32, dff=688) m = TinyLM(**cfg).eval() real = sum(p.numel() for p in m.parameters()) pred = whiteboard(**cfg, gpu=H100)["N"] print("tiny model parameters:", real, "formula:", pred) ids = torch.randint(0, cfg["V"], (1, 48)) with torch.no_grad(): logits, caches = m(ids[:, :40]) # prefill 40 tokens for t in range(40, 48): # decode 8 tokens, one at a time step, caches = m(ids[:, t:t+1], caches, start=t) full, _ = m(ids) # recompute everything kv_bytes = sum(k.numel()*k.element_size() + v.numel()*v.element_size() for k, v in caches) print("KV bytes:", kv_bytes, "formula (FP32, 48 tokens):", 48 * 2*cfg["L"]*cfg["KV"]*cfg["dh"]*4) print("cached decode matches full recompute:", torch.allclose(step[0, -1], full[0, -1], atol=1e-4))
Output of our run (PyTorch on a CPU) [Certain, our run]: N 8.03e+09, weights_GB 16.06, kv_KiB 128, used_GB 17.13, fits True, tpot_ms 5.115, ttft_ms 33.54 (at 100% of peak), train_flops 7.227e+23, adam_GB 128.5, then TTFT at 50% of peak: 67.1 ms, tiny model parameters: 3283200 formula: 3283200, KV bytes: 98304 formula (FP32, 48 tokens): 98304 and cached decode matches full recompute: True.
What is still unclear? A self-check
The last part of the lab is honest bookkeeping. Go through the list in Figure 5 and mark each item as clear, shaky or unclear. Be strict: “clear” means you could explain it at a whiteboard with a number, without notes. The readout collects the sessions to reread. Do this before Track 7: a vision-language model adds an image encoder and a connector, but every box of this whiteboard stays, and visual tokens make the prefill and KV-cache numbers much larger.
- Why byte-level BPE never fails on unknown text, and what a chat template adds (session 8, session 9)
- Embedding lookup, the output layer, and tied vs untied weights (session 11)
- How RoPE encodes position, and why the cache needs the position offset (session 14)
- Attention shapes, the causal mask, and why GQA shrinks the KV cache (session 17, session 26)
- Counting parameters and FLOPs from a config.json, formulas (162.1) to (162.3) (session 25)
- Scaling laws: \(C \approx 6ND\) and compute-optimal tokens per parameter (session 35)
- Number formats and the 16 bytes per parameter of AdamW training (session 38)
- DDP, ZeRO/FSDP and tensor parallelism: what each one shards (session 45)
- SFT, LoRA, DPO and GRPO: what data each needs and what it changes (session 54, session 68)
- Prefill vs decode, the KV cache, and the roofline, formulas (162.6) to (162.10) (session 82)
- Weight-only vs weight-activation quantization, and KV-cache quantization (session 104)
- Why speculative decoding is lossless and when it speeds things up (session 128)
- Paged KV memory, continuous batching, chunked prefill and scheduling (session 134, session 135)
- Capacity planning with Little’s law, formula (162.11) (session 141)
- Embeddings, retrieval, reranking, and how to evaluate a RAG system (session 153)
- The agent loop, tool calls, and prompt injection (session 154, session 158)
- Release gates, confidence intervals, judges and hallucination checks (session 155)
How to read Figure 5
Aim for zero “unclear” and at most three “shaky” before you start the vision track. For each shaky item, reread the session and then redo the matching question in Figure 3 or the matching case in Figure 4. If an item is unclear after rereading, write down the exact sentence you cannot explain; that sentence is usually one missing definition.
Where this is used
The whiteboard is not only for exams. The same ten-minute estimate is the first step of most real decisions.
| Situation | Whiteboard question | Formulas |
|---|---|---|
| Choosing a GPU for a chat product | Do weights and the KV cache of the expected traffic fit, and is the per-user token rate fast enough? | (162.7), (162.8), (162.11) |
| Planning a pretraining or fine-tuning run | How many GPU-hours, and does the training state fit or must it be sharded? | (162.3) to (162.5) |
| A RAG or long-document product | How much TTFT do the retrieved chunks cost, and is it worth retrieving fewer? | (162.9) |
| An agent product | Which part of the end-to-end time is model, which is tools, and which calls can run in parallel? | (162.8), (162.9) plus the trace |
| Video retrieval and Video-RAG | A one-hour video gives thousands of frames and many thousand visual tokens: which ones go into the prompt, and what does that do to TTFT and KV memory? | (162.6), (162.9) |
| Live key-moment detection on broadcast video | Can the model keep up with frames per second, given a fixed latency budget per clip? | (162.8), (162.9), (162.11) |
| Action recognition or shot classification at archive scale | How many GPU-hours to label the archive once, and is a smaller distilled model enough? | (162.3), (162.9) |
| Interviews (ML system design) | Draw the system, give the numbers, name the bottleneck, propose a fix and its cost | all of them (session 238) |
Common mistakes
- Mixing GB and GiB, or bits and bytes. 128 KiB is 131,072 bytes, not 128,000. INT4 is half a byte per weight. Write the unit at every step.
- Forgetting the factor 2 in the KV cache. Keys and values. And using the number of query heads instead of KV heads, which overestimates Llama 3 8B by 4×.
- Using \(2N\) FLOPs for training. Training is about \(6N\) per token (forward plus a backward that costs twice as much); inference is about \(2N\).
- Calling decode compute-bound. At small batch, decode does about 1 FLOP per byte (BF16), far below a ridge point of 295. Faster math does not help; fewer bytes do.
- Counting only the weights in serving memory. At long contexts and large batches, the KV cache is bigger than the weights (case 3: 13 sequences of 32K tokens hold 56 GB of cache next to 16 GB of weights).
- Optimizing before measuring. In an agent, the GPU may be less than half of the latency (case 5). Split the trace first.
- Treating lower bounds as predictions. The formulas give the best case. If you measure 1.2× the bound, the system is well tuned; if 5×, something is wrong (small batches, no CUDA graphs, CPU overhead, or a different bottleneck).
- Ignoring the tokenizer in length estimates. Token counts differ between tokenizers and languages; the same document can be many more tokens in one model than in another. Count with the real tokenizer.
Interview questions
Draw an LLM system from user text to answer, and put one number on each box.basic
How much GPU memory does Llama 3 8B need to serve 32 users with 8K-token contexts?core
Why is decode memory-bound and prefill compute-bound?core
Estimate the pretraining cost of an 8B model on 15T tokens.core
Users complain the chatbot is slow. Walk me through how you find the cause.core
Name the three levels of the efficient-inference survey and one method per branch.basic
At what batch size does decoding an 8B BF16 model on an H100 become compute-bound? And with INT4 weights?advanced
Why does grouped-query attention matter more for serving than for training?advanced
A RAG system has TTFT of over a second. What are your options, in order?core
How would you serve Llama 3 70B, and what decides the number of GPUs?advanced
An agent takes 19 seconds per task. The team wants a faster GPU. What do you say?core
Exercises
- Compute \(N\) for Mistral 7B: \(V = 32{,}000\), \(d = 4{,}096\), \(L = 32\), \(H = 32\), \(n_{kv} = 8\), \(d_h = 128\), \(d_{ff} = 14{,}336\), untied embeddings.
- For Llama 3 8B on an L4 (24 GB, 300 GB/s), how many 4,096-token sequences fit with BF16 weights and BF16 KV cache, using 90% of memory?
- Same L4, batch 1, no cache: what is the lower bound on the time per token? How does INT4 weights change it?
- A service gets 5 requests per second; each request lasts 8 seconds and holds 6,000 tokens. Llama 3 8B on one H100: does the KV cache fit?
- Training Llama 3 8B for 1 T tokens on 256 H100s at 40% MFU: how many days?
- In case 4 of Figure 4, a new request arrives just after a static batch has started. Up to how long does it wait before it can start, at 5.4 ms per step?
Answerscheck
Key takeaways
- The whole system fits on one whiteboard: tokenizer → model → training → serving → application. Each box has one number that matters most.
- Parameters come from the config (162.1, 162.2): Llama 3 8B has 8.03 B. Inference is about \(2N\) FLOPs per token, training about \(6N\) (162.3); AdamW training needs about 16 bytes per parameter (162.5).
- The KV cache is \(2 L n_{kv} d_h b_{kv}\) bytes per token (162.6): 128 KiB for Llama 3 8B. Weights plus cache must fit (162.7), and Little’s law (162.11) tells you how many sequences to plan for.
- Decode is memory-bound until the batch reaches about \(b_w/2 \times\) peak/BW (162.8, 162.10); prefill is compute-bound (162.9).
- Find the bottleneck before you fix it: split the trace, compare with the bounds, then pick the survey branch (data-, model- or system-level). The bottleneck moves after each fix.
Key formulas
References for this session
- Zhao et al. (2023). A Survey of Large Language Models
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Zhao et al. (2023). A Survey of Large Language Models
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Kaplan et al. (2020). Scaling Laws for Neural Language Models
- Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Meta (2024). Llama 3.1 8B and 70B config.json files (Hugging Face mirrors of meta-llama/Meta-Llama-3.1-8B-Instruct and -70B-Instruct)
- NVIDIA (2023). NVIDIA H100 Tensor Core GPU (product page and data sheet)