LLM & VLM courseBlog
162Track 6: LLM applications · session 20 of 20

the whole LLM system on one whiteboard

This checkpoint puts the whole LLM system on one whiteboard: tokenizer, model, training, serving and application. You compute the numbers that decide whether a design works (parameters, FLOPs, memory, KV cache, latency) for one reference model, Llama 3 8B, and you find the bottleneck in five short case studies. It ends with a self-check, so you can fix what is still unclear before the vision part of the course.

Time: 3 to 4 hours (reading 1 hour, drill, case studies and exercises 2 to 3 hours)Level: Intermediate · review labBefore this: Sessions 1 to 161; at least sessions 25, 38, 80, 82, 134 to 136 and 141

What you will learn

  • How to draw the full path of an LLM system, tokenizer → model → training → serving → application, with one key number on each box
  • How to count parameters from a config.json, and why Llama 3 8B has 8,030,261,248 of them
  • How to estimate training compute (6ND), GPU-hours and the 16 bytes per parameter of AdamW training
  • How to compute the KV cache per token and check whether weights plus cache fit on a GPU
  • How to bound decode time by memory bandwidth and prefill time by FLOP/s, and find the batch where decode turns compute-bound
  • How to find the bottleneck in a trace, and name the branch of the efficient-inference survey that fixes it
  • Which topics you should review before the vision track

Why this matters

You have now seen every part of an LLM system, one session at a time: how text becomes tokens, how a transformer turns tokens into a next-token distribution, how a model is trained and aligned, how it is served to many users, and how applications such as RAG and agents are built on top. This checkpoint puts all of it on one whiteboard. You draw the full path, you compute the numbers that decide whether a design works, and you find the bottleneck in five short case studies.

This is also the shape of a real interview. “Design a chat service for an 8B model” or “Our RAG system is slow, what do you do?” are not questions about one paper. The interviewer wants to see that you can draw the system, put a number on each box (memory, FLOPs, KV cache, latency), and point to the box that limits the whole thing. An engineer who can do this in ten minutes with a pen is worth more than one who knows fifty method names.

The lab has four parts, one per key point of this session:

  1. Draw the full path: tokenizer → model → training → serving → application (Figure 1).
  2. Answer exam-style questions with numbers: memory, FLOPs, KV cache, latency. A calculator (Figure 2) and a drill with 13 questions (Figure 3).
  3. Find the bottleneck in five short case studies, and name the branch of the efficient-inference survey (Zhou et al., 2024) that fixes it (Figure 4).
  4. List what is still unclear with a self-check (Figure 5). Fix those gaps before the vision part of the course (Track 7), because every vision-language model is this same system with an image encoder in front.

The session uses one reference model throughout, so that all numbers connect: Llama 3 8B. The Session 161 lab was the last application you built; now we step back and look at the whole stack under that agent.

What is computed and what is measured on this pageEvery number on this page comes from formulas (162.1) to (162.11) and public specifications (model config files, GPU data sheets). Nothing here is a measured benchmark. The formulas give lower bounds and first estimates; real systems are slower by a factor you learn to expect (often 1.2× to 2× for well-tuned engines [Likely]). The five case studies are made-up but realistic scenarios; their “observed” numbers are chosen to match the formulas.

The whiteboard: one request, from text to answer

Start with the drawing. Five boxes, left to right. Each box has an input, an output, one number that matters most, and one typical failure. Click the boxes in Figure 1, or step through them.

Figure 1. The whole LLM system on one whiteboard. The top row is the path of one request at run time. Training (dashed arrow) happens once, before serving, and produces the weights the model box uses. All numbers are for Llama 3 8B.
How to read Figure 1

Each step highlights one box and shows five lines: what goes in, what comes out, the number to remember, the formula behind that number, and the failure you see most often in practice. The last line lists the sessions to reread if the box is not clear. Notice that the same model weights appear in three places: they are produced by training, read once per token by serving, and wrapped by the application.

Here is the same drawing in words, with the facts you should know for the reference model. The shape of Llama 3 8B, from its config.json [Certain]:

FieldSymbolLlama 3 8BMeaning
vocab_size\(V\)128,256number of token IDs
hidden_size\(d\)4,096width of the residual stream
num_hidden_layers\(L\)32number of transformer blocks
num_attention_heads\(H\)32query heads
num_key_value_heads\(n_{kv}\)8key and value heads (grouped-query attention, 4 query heads share one)
head_dim\(d_h\)128size of one head
intermediate_size\(d_{ff}\)14,336width of the SwiGLU feed-forward layer
tie_word_embeddingsfalseinput embedding and output layer are two separate matrices
  1. Tokenizer. Text in, a list of integer IDs out. Llama 3 uses a byte-level BPE vocabulary of 128K tokens (100K taken from OpenAI’s tiktoken plus 28K extra, mostly for other languages) [Certain]. A chat template wraps the messages in special tokens before tokenization. Most bugs here are silent: a wrong template or a missing end-of-turn token makes a good model look bad (session 9).
  2. Model. IDs in, one probability distribution over 128,256 tokens out, for every position. An embedding lookup, 32 blocks of attention plus feed-forward with residual connections and RMSNorm, then the output layer and a softmax. About \(2N\) FLOPs per token in the forward pass.
  3. Training. Text in, weights out. Pretraining on next-token prediction (Llama 3: over 15 trillion tokens [Certain]), then supervised fine-tuning, then preference tuning or RL. The compute is about \(6ND\) FLOPs; the memory is about 16 bytes per parameter with AdamW.
  4. Serving. Requests in, streamed tokens out. Prefill reads the prompt in one parallel pass (compute-bound); decode makes one token per step (memory-bound). The KV cache stores keys and values of past tokens; the scheduler batches many users together.
  5. Application. A user goal in, an answer out. Prompting, structured output, RAG, agents with tools, and the evaluation that tells you whether it works. Most latency here is often not in the GPU: retrieval, tool calls and repeated calls to the model add up.

The numbers you must be able to write down

A small set of formulas covers almost every exam question. You have seen all of them in earlier sessions; here they are in one place, in the order of the whiteboard.

Parameters

One block of a Llama-style model has four attention matrices (query, key, value, output), three feed-forward matrices (gate, up, down), and two RMSNorm weight vectors:

\[ P_{\text{layer}} = \underbrace{2\, d\, H d_h}_{\text{query, output}} + \underbrace{2\, d\, n_{kv} d_h}_{\text{key, value}} + \underbrace{3\, d\, d_{ff}}_{\text{SwiGLU}} + \underbrace{2d}_{\text{norms}} \tag{162.1} \]
\[ N = L \cdot P_{\text{layer}} + \underbrace{2\, V d}_{\text{embedding, output layer}} + d \tag{162.2} \]
Read the symbols
\(N\)
total number of parameters (weights)
\(d, L, H, n_{kv}, d_h, d_{ff}, V\)
the config fields in the table above
\(2Vd\)
input embedding plus output layer; it is \(Vd\) if they are tied
last \(d\)
the final RMSNorm before the output layer

For Llama 3 8B, \(P_{\text{layer}} = 218{,}112{,}000\), the 32 blocks hold 6.98 B, the two vocabulary matrices hold 1.05 B, and \(N = 8{,}030{,}261{,}248\). The feed-forward layers are 81% of each block. (The same formula gives exactly 7,615,616,512 for Qwen2.5-7B once you add its query, key and value biases, which matches the count Hugging Face reports.) [Certain: checked in Python]

Compute

A forward pass does one multiply and one add per weight per token, so about \(2N\) FLOPs per token, plus the attention scores that grow with the context. Training adds the backward pass, which costs twice the forward (session 4):

\[ F_{\text{fwd}}(t) \approx 2N + 4\, L\, t\, H d_h, \qquad C_{\text{train}} \approx 6\, N D \tag{162.3} \]
\[ \text{GPU-hours} = \frac{C_{\text{train}}}{\text{peak} \times \text{MFU} \times 3600} \tag{162.4} \]
Read the symbols
\(t\)
the position of the token, so the number of earlier tokens it attends to
\(D\)
number of training tokens
peak
the GPU’s dense FLOP/s for the number format (989 TFLOP/s in BF16 on an H100 SXM)
MFU
model FLOPs utilization: the fraction of peak that training really reaches (often 35% to 50% for large dense runs [Likely])

Memory

\[ M_{\text{train}} \approx 16\, N \text{ bytes} + \text{activations}, \qquad M_{\text{weights}} = b_w N \tag{162.5} \]

The 16 bytes are BF16 weights (2), BF16 gradients (2), an FP32 master copy (4) and the two AdamW moments in FP32 (4 + 4) (session 38). For inference you only need the weights, \(b_w\) bytes each: 2 for BF16, 1 for FP8, about 0.5 for INT4.

The KV cache keeps one key and one value vector per KV head, per layer, per token:

\[ \text{kv}_{\text{tok}} = 2 \cdot L \cdot n_{kv} \cdot d_h \cdot b_{kv} \tag{162.6} \]

For Llama 3 8B in BF16: \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes, exactly 128 KiB per token. Without grouped-query attention (32 KV heads) it would be 512 KiB. A serving GPU must hold the weights and the cache of every running sequence:

\[ b_w N + B \cdot T \cdot \text{kv}_{\text{tok}} \;\le\; u \cdot M_{\text{GPU}} \cdot G \tag{162.7} \]
Read the symbols
\(B\)
number of sequences served at the same time (the batch)
\(T\)
tokens held per sequence (prompt plus output so far)
\(u\)
the fraction of GPU memory you let the engine use; we take 0.9, the default memory fraction in vLLM [Likely]
\(M_{\text{GPU}}, G\)
memory of one GPU, and the number of GPUs the model is split across

Latency

In decode, each step makes one token for each of the \(B\) sequences. It must read all weights and all cached keys and values from memory, and it does \(2N\) FLOPs per sequence. The step can be no faster than the slower of the two:

\[ t_{\text{step}} \;\ge\; \max\!\left( \frac{b_w N + B\, T\, \text{kv}_{\text{tok}}}{\text{BW}},\; \frac{2 N B}{\text{peak}} \right) \tag{162.8} \]

In prefill, the whole prompt of \(P\) tokens goes through in one pass. Many tokens share each weight read, so prefill is compute-bound. With causal attention over the prompt:

\[ \text{TTFT} \approx \frac{2 N P + 2\, L\, P^2 H d_h}{\eta \cdot \text{peak}} \tag{162.9} \]
Read the symbols
BW
memory bandwidth in bytes per second (3.35 TB/s on an H100 SXM)
\(\eta\)
the fraction of peak FLOP/s the prefill kernels reach; we use 0.5 when a question says so [assumption]
TTFT
time to first token, ignoring queueing and network

Which side of the roofline?

The arithmetic intensity of a step is FLOPs done per byte read. For decode with weights dominating the bytes, it is about \(2NB / (b_w N) = 2B / b_w\). The GPU becomes compute-bound only when this exceeds its ridge point:

\[ I_{\text{decode}} \approx \frac{2B}{b_w} \quad \text{vs} \quad I_{\text{ridge}} = \frac{\text{peak}}{\text{BW}} \;\Rightarrow\; B^{*} \approx \frac{b_w}{2} \cdot \frac{\text{peak}}{\text{BW}} \tag{162.10} \]

On an H100, \(989 / 3.35 = 295\) FLOP/byte. With BF16 weights (\(b_w = 2\)), decode stays memory-bound until about 295 sequences share each weight read. With INT4 weights the crossover is near 74, because each byte now feeds four times more math. Below the crossover, a bigger batch is almost free; above it, every extra sequence costs extra time.

How many requests are in the system?

Finally, the link between traffic and memory is Little’s law (session 141): on average, the number of requests in the system equals the arrival rate times the time each one stays.

\[ \bar{B} = \lambda \cdot W \tag{162.11} \]

20 requests per second that each take 5 seconds means about 100 sequences in flight. Put that \(B\) into formula (162.7), and you know whether the KV cache fits.

Worked example: Llama 3 8B on one H100, end to end

Parameters (162.2): \(N = 8.03\) B. Weights in BF16: \(2N = 16.06\) GB. Training state (162.5): \(16N = 128.5\) GB, more than one 80 GB GPU before any activation, which is why full fine-tuning needs sharding or LoRA.

Pretraining compute (162.3) for 15 T tokens: \(6 \times 8.03 \times 10^9 \times 15 \times 10^{12} = 7.23 \times 10^{23}\) FLOPs. At 40% MFU on H100s (162.4): \(7.23 \times 10^{23} / (989 \times 10^{12} \times 0.4 \times 3600) \approx 507{,}000\) GPU-hours.

KV cache (162.6): 128 KiB per token, so 1.07 GB for one 8,192-token sequence. Capacity (162.7): \(0.9 \times 80 - 16.06 = 55.9\) GB is left, enough for 52 sequences of 8,192 tokens.

Decode (162.8) for one user, ignoring the cache: \(16.06 \text{ GB} / 3.35 \text{ TB/s} = 4.79\) ms per token, at most 209 tokens per second. With an 8,192-token cache, the step reads 17.13 GB and takes 5.11 ms.

Prefill (162.9) of a 2,000-token prompt at 50% of peak: \(2NP = 3.21 \times 10^{13}\) FLOPs plus \(1.05 \times 10^{12}\) for attention (3%), so TTFT is about 67 ms.

Figure 2 puts all of these formulas into one calculator. Change the model, the GPU, the precision, the batch and the context, and watch which limit you hit first.

Model
GPU
GPUs (TP)
WeightsKV cache
Figure 2. The whiteboard calculator. Left: memory per GPU, split into weights, KV cache and what is left of the usable 90%. Right: the decode step time (162.8) against the batch size, on log axes; the memory line (solid) rises slowly as the cache grows, the compute line (dashed) rises in proportion to the batch, and the dot is your batch. All values are lower bounds from formulas (162.1) to (162.10), with tensor parallelism assumed perfect (no communication cost).
How to read Figure 2

Start with Llama 3 8B on one H100, batch 1: the memory bar is mostly free, and the step time is 5.1 ms, set by memory bandwidth. Move the batch up: the step time barely changes until the KV cache starts to dominate the bytes read, so aggregate throughput grows almost linearly. Now pick Llama 3 70B: it does not fit on one GPU in BF16 (141 GB). Choose 4 GPUs or INT4 weights. Pick the L4: the same 8B model fits, but every step is 11 times slower because the L4 reads 300 GB/s instead of 3,350. Switch the KV cache to FP8 at long contexts and watch the maximum batch double.

Exam-style questions with numbers

Now practice. The drill in Figure 3 has 13 questions of the kind you will meet in interviews and exams. Type a number and press Check. An answer within 5% counts as correct, because whiteboard math should be fast, not exact to the last digit. Each question has a full solution.

Figure 3. The exam drill. All questions use Llama 3 8B in BF16 and an H100 SXM (989 TFLOP/s, 3.35 TB/s, 80 GB) unless they say otherwise. GB means \(10^9\) bytes; KiB means 1,024 bytes.
How to read Figure 3

Try each question on paper first, with the formulas above. The score at the top right counts questions you got right on the first try (it is saved in this browser only). If you are wrong, the readout tells you how far off you are, which often points at the mistake: a factor of 2 is usually bytes per parameter or keys plus values; a factor of 1,000 is units.

Two of these questions deserve to be done slowly, because they combine several boxes of the whiteboard.

Worked example: can one H100 serve a 20 requests-per-second chat service?

Each request has about 3,000 tokens of prompt and history and lives for 5 seconds. By Little’s law (162.11), \(\bar B = 20 \times 5 = 100\) sequences are in flight on average.

KV cache (162.6, 162.7): \(100 \times 3{,}000 \times 131{,}072 \text{ bytes} = 39.3\) GB. Weights are 16.06 GB, so the total is 55.4 GB, under the 72 GB usable. It fits.

Decode step (162.8): bytes read \(= 16.06 + 39.3 = 55.4\) GB, so the step takes at least \(55.4 / 3.35 = 16.5\) ms. FLOPs \(= 2N \times 100 = 1.6 \times 10^{12}\), which take 1.6 ms at peak. Still memory-bound, now because of the cache, not the weights. Each user sees at most about 60 tokens per second; the GPU produces about 6,000 tokens per second in total. If the product needs 100 tokens per second per user, you need FP8 KV cache, a second GPU, or shorter histories.

Worked example: how many GPUs for Llama 3 70B?

From (162.2) with \(d = 8{,}192\), \(L = 80\), \(H = 64\), \(n_{kv} = 8\), \(d_{ff} = 28{,}672\): \(N = 70.55\) B, so 141.1 GB in BF16. Two H100s (144 GB usable at 90%) would hold the weights and almost no cache. The KV cache is 320 KiB per token (162.6). With tensor parallelism over 4 GPUs, each holds 35.3 GB of weights and has 36.7 GB left for the cache; together that is about 448,000 tokens of BF16 cache. With INT4 weights (about 35 GB), one 80 GB GPU can serve it at short contexts.

Finding the bottleneck

A system is as fast as its slowest part. To make it faster you must first know which part that is. Guessing is the most common and most expensive mistake: a team spends a month on a faster attention kernel, and the users see no change, because 50% of the time was spent in tool calls.

Algorithm 1: whiteboard estimate for a deployment
  1. input: model config, GPU (peak, BW, memory), number formats \(b_w, b_{kv}\), traffic (\(\lambda\), prompt length \(P\), output length \(O\))
  2. \(N \leftarrow\) formula (162.2); \(\text{kv}_{\text{tok}} \leftarrow\) formula (162.6)
  3. \(G \leftarrow\) smallest number of GPUs with \(b_w N \lt u \cdot M_{\text{GPU}} \cdot G\) // weights must fit, with room left
  4. \(W \leftarrow\) first guess of the request lifetime: TTFT (162.9) \(+\; O \times t_{\text{step}}\) at batch 1
  5. repeat until \(W\) stops changing:
  6. \(B \leftarrow \lambda W\) // Little’s law (162.11)
  7. if \(b_w N + B (P + O)\, \text{kv}_{\text{tok}} \gt u M_{\text{GPU}} G\): add GPUs, lower \(b_{kv}\), or reject the design
  8. \(t_{\text{step}} \leftarrow\) formula (162.8) at this \(B\); \(W \leftarrow \text{TTFT} + O \cdot t_{\text{step}}\)
  9. return \(G\), \(B\), TTFT, \(t_{\text{step}}\), aggregate throughput \(B / t_{\text{step}}\)

The estimate tells you where the system should be. The diagnosis compares that with what you observe:

Algorithm 2: find the bottleneck, then name the fix
  1. input: a trace of real requests (timestamps of arrival, first token, each token, tool calls), GPU metrics
  2. split the end-to-end time of a typical request into: queueing, prefill, decode, work outside the model (retrieval, tools, network)
  3. take the largest part. If it is outside the model: change the call pattern first (fewer calls, parallel calls, caching) // data-level: output organization
  4. if it is prefill: compare the prompt length with what the task needs // data-level: input compression; system-level: prefix caching
  5. if it is decode at small batch: compare \(t_{\text{step}}\) with weight bytes / BW // model-level: quantization; system-level: speculative decoding
  6. if the batch cannot grow because memory is full: compare \(B\) with formula (162.7) // KV-cache quantization, efficient attention, paged memory
  7. if it is queueing while the GPU is underused: look at how requests are batched and scheduled // system-level: continuous batching, scheduling
  8. apply one fix, measure again, and go back to step 2: the bottleneck moves

The last column of Algorithm 2 uses the map of the efficient-inference survey (Zhou et al., arXiv 2404.14294), which you followed through Track 5. Its top levels are [Certain, from the paper]:

LevelBranchSub-branchesWhat it attacks
Data-levelInput compressionprompt pruning, prompt summary, soft prompt-based compression, retrieval-augmented generationtoo many prompt tokens
Output organizationSkeleton-of-Thought, parallel calls, SGLang-style programssequential generation and sequential calls
Model-levelEfficient structure designefficient FFN (MoE), efficient attention (MQA, GQA, low-complexity attention), transformer alternatesFLOPs and KV bytes built into the architecture
Model compressionquantization, sparsification, structure optimization, knowledge distillation, dynamic inferencebytes and FLOPs of a trained model
System-levelInference enginegraph and operator optimization, speculative decoding, offloadinghow fast one batch runs
Serving systemmemory management, continuous batching, scheduling strategy, distributed systemshow many requests share the hardware, and in what order

Figure 4 gives you five case studies. Read the symptoms, pick the bottleneck, and then see the computation and the survey branches that fix it.

What is the bottleneck?
Figure 4. Five bottleneck case studies on Llama 3 8B. The bar shows where the time (or memory) goes, computed with formulas (162.6) to (162.9). After you answer, the survey map below highlights the branches that fix the case.
How to read Figure 4

Do not look at the bar first; read the story and make a guess. Then check the bar: it is the whiteboard estimate of each part, so it tells you which part is large. A wrong answer shows why that part is not the problem here. The survey map has three columns (data-level, model-level, system-level). Highlighted boxes are the fixes worth trying first, not the only ones.

A summary of the five answers, so you can check your reasoning afterwards:

CaseBottleneckKey numberSurvey branch that fixes it
1. One user, slow tokens (A100)memory bandwidth in decode15.0 GB read per token, 7.4 ms at 2.04 TB/s; observed 8.5 msmodel compression: quantization (INT4: 2.3 ms); inference engine: speculative decoding
2. RAG with 24,000-token promptsprefill compute1.08 s of prefill at 50% of peak; 28% of it is attentioninput compression (fewer, better chunks: 4,800 tokens give 0.17 s); memory management (prefix caching)
3. 32K-token contexts, small batchKV-cache capacity4.29 GB per sequence, only 13 fit; about 606 tokens/s in totalquantization of the KV cache (26 fit, about 1,210 tokens/s); efficient attention; memory management
4. Static batches, long waitsbatching and schedulingslot use 25% (mean output 200, longest 800)serving system: continuous batching, scheduling strategy
5. Agent with 10 steps, 19 swork outside the GPU and the call patterntools 10.0 s (52%), decode 7.2 s, prefill 1.9 soutput organization (parallel tool calls, fewer and shorter steps); prefix caching cuts prefill to 0.27 s

The whiteboard in code

The code below has two parts. First, formulas (162.1) to (162.9) as one Python function, applied to Llama 3 8B on an H100. Second, a tiny Llama-shaped model in PyTorch (RMSNorm, RoPE, grouped-query attention, SwiGLU, untied output layer), used to check that the parameter formula and the KV-cache formula match real tensors, and that cached decoding gives the same logits as recomputing everything. It runs on a CPU in a few seconds.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# ---- 1) the whiteboard formulas (162.1) to (162.9) as one function ----
def whiteboard(V, d, L, H, KV, dh, dff, gpu, B=1, T=8192, P=2000, bw=2, bkv=2, eff=1.0):
    peak, bwidth, mem = gpu
    layer = 2*d*H*dh + 2*d*KV*dh + 3*d*dff + 2*d          # (162.1)
    N = L*layer + 2*V*d + d                               # (162.2) untied embeddings
    kv_tok = 2*L*KV*dh*bkv                                # (162.6)
    used = bw*N + B*T*kv_tok                              # (162.7)
    t_mem = used / (eff*bwidth); t_cmp = 2*N*B / (eff*peak)
    t_step = max(t_mem, t_cmp)                            # (162.8)
    ttft = (2*N*P + 2*L*P*P*H*dh) / (eff*peak)            # (162.9)
    return dict(N=N, weights_GB=bw*N/1e9, kv_KiB=kv_tok/1024, used_GB=used/1e9,
                fits=used <= 0.9*mem, tpot_ms=1e3*t_step, ttft_ms=1e3*ttft,
                train_flops=6*N*15e12, adam_GB=16*N/1e9)   # (162.3), (162.5)

H100 = (989e12, 3.35e12, 80e9)
llama3_8b = dict(V=128256, d=4096, L=32, H=32, KV=8, dh=128, dff=14336)
for k, v in whiteboard(**llama3_8b, gpu=H100).items():
    print(f"{k:12s} {v:.4g}" if not isinstance(v, bool) else f"{k:12s} {v}")
print("TTFT at 50% of peak:", round(whiteboard(**llama3_8b, gpu=H100, eff=0.5)["ttft_ms"], 1), "ms")

# ---- 2) a tiny Llama-shaped model, to check the formulas against real tensors ----
class RMSNorm(nn.Module):
    def __init__(s, d): super().__init__(); s.w = nn.Parameter(torch.ones(d))
    def forward(s, x): return s.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-5)

def rope(x, pos):                                         # x: (B, heads, T, dh)
    dh = x.shape[-1]; inv = 1.0 / (10000 ** (torch.arange(0, dh, 2) / dh))
    ang = pos[:, None].float() * inv[None]                # (T, dh/2)
    c, s = ang.cos(), ang.sin(); x1, x2 = x[..., 0::2], x[..., 1::2]
    return torch.stack([x1*c - x2*s, x1*s + x2*c], -1).flatten(-2)

class Block(nn.Module):
    def __init__(s, d, H, KV, dh, dff):
        super().__init__(); s.H, s.KV, s.dh = H, KV, dh
        s.n1, s.n2 = RMSNorm(d), RMSNorm(d)
        s.q = nn.Linear(d, H*dh, bias=False); s.k = nn.Linear(d, KV*dh, bias=False)
        s.v = nn.Linear(d, KV*dh, bias=False); s.o = nn.Linear(H*dh, d, bias=False)
        s.gate = nn.Linear(d, dff, bias=False); s.up = nn.Linear(d, dff, bias=False)
        s.down = nn.Linear(dff, d, bias=False)
    def forward(s, x, cache, start):
        B, T, _ = x.shape; h = s.n1(x); pos = torch.arange(start, start + T)
        q = rope(s.q(h).view(B, T, s.H, s.dh).transpose(1, 2), pos)
        k = rope(s.k(h).view(B, T, s.KV, s.dh).transpose(1, 2), pos)
        v = s.v(h).view(B, T, s.KV, s.dh).transpose(1, 2)
        if cache is not None: k = torch.cat([cache[0], k], 2); v = torch.cat([cache[1], v], 2)
        rep = s.H // s.KV                                 # grouped-query attention
        a = F.scaled_dot_product_attention(q, k.repeat_interleave(rep, 1), v.repeat_interleave(rep, 1),
                                           is_causal=(cache is None))
        x = x + s.o(a.transpose(1, 2).reshape(B, T, -1))
        h = s.n2(x); x = x + s.down(F.silu(s.gate(h)) * s.up(h))
        return x, (k, v)

class TinyLM(nn.Module):
    def __init__(s, V, d, L, H, KV, dh, dff):
        super().__init__(); s.emb = nn.Embedding(V, d)
        s.blocks = nn.ModuleList(Block(d, H, KV, dh, dff) for _ in range(L))
        s.norm = RMSNorm(d); s.head = nn.Linear(d, V, bias=False)
    def forward(s, ids, caches=None, start=0):
        x = s.emb(ids); new = []
        for i, b in enumerate(s.blocks):
            x, c = b(x, None if caches is None else caches[i], start); new.append(c)
        return s.head(s.norm(x)), new

cfg = dict(V=1000, d=256, L=4, H=8, KV=2, dh=32, dff=688)
m = TinyLM(**cfg).eval()
real = sum(p.numel() for p in m.parameters())
pred = whiteboard(**cfg, gpu=H100)["N"]
print("tiny model parameters:", real, "formula:", pred)

ids = torch.randint(0, cfg["V"], (1, 48))
with torch.no_grad():
    logits, caches = m(ids[:, :40])                       # prefill 40 tokens
    for t in range(40, 48):                               # decode 8 tokens, one at a time
        step, caches = m(ids[:, t:t+1], caches, start=t)
    full, _ = m(ids)                                      # recompute everything
kv_bytes = sum(k.numel()*k.element_size() + v.numel()*v.element_size() for k, v in caches)
print("KV bytes:", kv_bytes, "formula (FP32, 48 tokens):", 48 * 2*cfg["L"]*cfg["KV"]*cfg["dh"]*4)
print("cached decode matches full recompute:", torch.allclose(step[0, -1], full[0, -1], atol=1e-4))

Output of our run (PyTorch on a CPU) [Certain, our run]: N 8.03e+09, weights_GB 16.06, kv_KiB 128, used_GB 17.13, fits True, tpot_ms 5.115, ttft_ms 33.54 (at 100% of peak), train_flops 7.227e+23, adam_GB 128.5, then TTFT at 50% of peak: 67.1 ms, tiny model parameters: 3283200 formula: 3283200, KV bytes: 98304 formula (FP32, 48 tokens): 98304 and cached decode matches full recompute: True.

What is still unclear? A self-check

The last part of the lab is honest bookkeeping. Go through the list in Figure 5 and mark each item as clear, shaky or unclear. Be strict: “clear” means you could explain it at a whiteboard with a number, without notes. The readout collects the sessions to reread. Do this before Track 7: a vision-language model adds an image encoder and a connector, but every box of this whiteboard stays, and visual tokens make the prefill and KV-cache numbers much larger.

  • Why byte-level BPE never fails on unknown text, and what a chat template adds (session 8, session 9)
  • Embedding lookup, the output layer, and tied vs untied weights (session 11)
  • How RoPE encodes position, and why the cache needs the position offset (session 14)
  • Attention shapes, the causal mask, and why GQA shrinks the KV cache (session 17, session 26)
  • Counting parameters and FLOPs from a config.json, formulas (162.1) to (162.3) (session 25)
  • Scaling laws: \(C \approx 6ND\) and compute-optimal tokens per parameter (session 35)
  • Number formats and the 16 bytes per parameter of AdamW training (session 38)
  • DDP, ZeRO/FSDP and tensor parallelism: what each one shards (session 45)
  • SFT, LoRA, DPO and GRPO: what data each needs and what it changes (session 54, session 68)
  • Prefill vs decode, the KV cache, and the roofline, formulas (162.6) to (162.10) (session 82)
  • Weight-only vs weight-activation quantization, and KV-cache quantization (session 104)
  • Why speculative decoding is lossless and when it speeds things up (session 128)
  • Paged KV memory, continuous batching, chunked prefill and scheduling (session 134, session 135)
  • Capacity planning with Little’s law, formula (162.11) (session 141)
  • Embeddings, retrieval, reranking, and how to evaluate a RAG system (session 153)
  • The agent loop, tool calls, and prompt injection (session 154, session 158)
  • Release gates, confidence intervals, judges and hallucination checks (session 155)
Figure 5. The self-check. Mark each item; your marks are saved in this browser. The readout lists your shaky and unclear items with the sessions to reread, in course order.
How to read Figure 5

Aim for zero “unclear” and at most three “shaky” before you start the vision track. For each shaky item, reread the session and then redo the matching question in Figure 3 or the matching case in Figure 4. If an item is unclear after rereading, write down the exact sentence you cannot explain; that sentence is usually one missing definition.

Where this is used

The whiteboard is not only for exams. The same ten-minute estimate is the first step of most real decisions.

SituationWhiteboard questionFormulas
Choosing a GPU for a chat productDo weights and the KV cache of the expected traffic fit, and is the per-user token rate fast enough?(162.7), (162.8), (162.11)
Planning a pretraining or fine-tuning runHow many GPU-hours, and does the training state fit or must it be sharded?(162.3) to (162.5)
A RAG or long-document productHow much TTFT do the retrieved chunks cost, and is it worth retrieving fewer?(162.9)
An agent productWhich part of the end-to-end time is model, which is tools, and which calls can run in parallel?(162.8), (162.9) plus the trace
Video retrieval and Video-RAGA one-hour video gives thousands of frames and many thousand visual tokens: which ones go into the prompt, and what does that do to TTFT and KV memory?(162.6), (162.9)
Live key-moment detection on broadcast videoCan the model keep up with frames per second, given a fixed latency budget per clip?(162.8), (162.9), (162.11)
Action recognition or shot classification at archive scaleHow many GPU-hours to label the archive once, and is a smaller distilled model enough?(162.3), (162.9)
Interviews (ML system design)Draw the system, give the numbers, name the bottleneck, propose a fix and its costall of them (session 238)

Common mistakes

  • Mixing GB and GiB, or bits and bytes. 128 KiB is 131,072 bytes, not 128,000. INT4 is half a byte per weight. Write the unit at every step.
  • Forgetting the factor 2 in the KV cache. Keys and values. And using the number of query heads instead of KV heads, which overestimates Llama 3 8B by 4×.
  • Using \(2N\) FLOPs for training. Training is about \(6N\) per token (forward plus a backward that costs twice as much); inference is about \(2N\).
  • Calling decode compute-bound. At small batch, decode does about 1 FLOP per byte (BF16), far below a ridge point of 295. Faster math does not help; fewer bytes do.
  • Counting only the weights in serving memory. At long contexts and large batches, the KV cache is bigger than the weights (case 3: 13 sequences of 32K tokens hold 56 GB of cache next to 16 GB of weights).
  • Optimizing before measuring. In an agent, the GPU may be less than half of the latency (case 5). Split the trace first.
  • Treating lower bounds as predictions. The formulas give the best case. If you measure 1.2× the bound, the system is well tuned; if 5×, something is wrong (small batches, no CUDA graphs, CPU overhead, or a different bottleneck).
  • Ignoring the tokenizer in length estimates. Token counts differ between tokenizers and languages; the same document can be many more tokens in one model than in another. Count with the real tokenizer.

Interview questions

Draw an LLM system from user text to answer, and put one number on each box.basic
Tokenizer: text to IDs, vocabulary 128,256 for Llama 3. Model: 32 blocks, 8.03 B parameters, about \(2N = 16\) GFLOPs per token. Training: about \(6ND\), \(7.2 \times 10^{23}\) FLOPs for 15 T tokens, 16 bytes per parameter of AdamW state. Serving: 16 GB of BF16 weights, 128 KiB of KV cache per token, decode at least 4.8 ms per step on an H100 (memory-bound), prefill compute-bound. Application: prompt, retrieval or tools, structured output and evaluation; often most of the end-to-end time.
How much GPU memory does Llama 3 8B need to serve 32 users with 8K-token contexts?core
Weights: \(2 \times 8.03 \times 10^9 = 16.06\) GB. KV cache per token: \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes. For \(32 \times 8{,}192\) tokens: 34.4 GB. Total about 50.4 GB plus a few GB of activations and workspace: one 80 GB GPU is fine; a 48 GB GPU is not. With FP8 KV cache the cache halves to 17.2 GB.
Why is decode memory-bound and prefill compute-bound?core
Every step reads all weights once. In prefill, those bytes are used by all \(P\) prompt tokens, so FLOPs per byte is about \(P\) (for BF16) and quickly passes the ridge point (295 on an H100). In decode, each step makes one token per sequence, so FLOPs per byte is about the batch size \(B\). Below \(B \approx 295\), the GPU waits for memory. That is why batching, quantization and speculative decoding help decode, while prefill is helped by fewer prompt tokens and faster kernels.
Estimate the pretraining cost of an 8B model on 15T tokens.core
\(C \approx 6ND = 6 \times 8.03 \times 10^9 \times 1.5 \times 10^{13} = 7.2 \times 10^{23}\) FLOPs. An H100 gives 989 TFLOP/s in dense BF16; at 40% MFU that is about 396 TFLOP/s, so \(7.2 \times 10^{23} / 3.96 \times 10^{14} / 3600 \approx 507{,}000\) GPU-hours, about 21 days on 1,000 GPUs. Mention that this ignores failures and restarts, and that MFU is the big uncertainty.
Users complain the chatbot is slow. Walk me through how you find the cause.core
First define slow: TTFT or tokens per second, at which percentile, and under which load. Take a trace and split the end-to-end time into queueing, prefill, decode and work outside the model. Compare each part with its whiteboard bound: prefill with (162.9), decode with (162.8). If decode is near its memory bound at small batch, reduce bytes (quantization) or tokens per weight read (speculative decoding). If queueing dominates while the GPU is underused, fix batching and scheduling. If prompts are huge, compress them or cache the shared prefix. If tools dominate, change the call pattern. Fix one thing, measure again.
Name the three levels of the efficient-inference survey and one method per branch.basic
Data-level: input compression (prompt pruning such as LLMLingua, or RAG) and output organization (Skeleton-of-Thought). Model-level: efficient structure design (GQA, MoE, state space models) and model compression (quantization such as GPTQ or AWQ, sparsification such as Wanda, distillation, early exit). System-level: inference engine (FlashAttention and other operator optimization, speculative decoding, offloading) and serving system (PagedAttention memory management, continuous batching, scheduling, distributed serving).
At what batch size does decoding an 8B BF16 model on an H100 become compute-bound? And with INT4 weights?advanced
Decode intensity is about \(2NB / (b_w N) = 2B/b_w\) FLOPs per byte, ignoring the KV cache. The H100 ridge is \(989/3.35 \approx 295\). With \(b_w = 2\): \(B \approx 295\). With \(b_w = 0.5\): \(B \approx 74\). In practice the KV cache adds bytes that grow with \(B\) and context, so long-context decode may never become compute-bound; and INT4 kernels dequantize, which costs extra compute, so the real crossover is lower.
Why does grouped-query attention matter more for serving than for training?advanced
GQA reduces the number of key and value heads (8 instead of 32 in Llama 3 8B). In training the main costs are the weight matrices and activations, and the saving is small. In serving, the KV cache sets how many sequences fit (162.7) and how many bytes each decode step reads (162.8). Four times fewer KV heads means a 4× smaller cache: 128 KiB instead of 512 KiB per token, so about 4× more concurrent long-context users.
A RAG system has TTFT of over a second. What are your options, in order?core
Compute the prompt length first: 40 chunks of 600 tokens is 24,000 tokens, about 1.08 s of prefill on an H100 at 50% of peak for an 8B model. Options: retrieve fewer and better chunks with a reranker (8 chunks: about 0.17 s); compress prompts (LLMLingua-style pruning); cache the system prompt and frequent documents as a shared prefix; use chunked prefill so long prompts do not block other users; only then faster kernels or more GPUs. Check that answer quality holds with a RAG evaluation after each change.
How would you serve Llama 3 70B, and what decides the number of GPUs?advanced
Weights are 141 GB in BF16 (70.55 B parameters), so at least two 80 GB GPUs, and in practice four with tensor parallelism to leave room for the KV cache (320 KiB per token) and to speed up decode (each GPU reads a quarter of the weights). Alternatives: FP8 weights (71 GB, two GPUs with room for cache), or INT4 (about 35 GB, one GPU for short contexts). The number of GPUs is set by memory first (weights plus the KV cache of the expected concurrency from Little’s law), then by the per-user token rate target, then by cost.
An agent takes 19 seconds per task. The team wants a faster GPU. What do you say?core
Ask for the trace first. In a typical 10-step agent with 150 output tokens per step and 1-second tool calls, tools take 10 s, decode about 7 s and prefill about 2 s. A GPU twice as fast saves at most about 4.5 s; running independent tool calls in parallel, cutting steps, or asking for shorter outputs can save more, and costs nothing. Prefix caching cuts the repeated prefill from 1.9 to 0.27 s. Then measure success rate again, because fewer steps can lower quality.

Exercises

  1. Compute \(N\) for Mistral 7B: \(V = 32{,}000\), \(d = 4{,}096\), \(L = 32\), \(H = 32\), \(n_{kv} = 8\), \(d_h = 128\), \(d_{ff} = 14{,}336\), untied embeddings.
  2. For Llama 3 8B on an L4 (24 GB, 300 GB/s), how many 4,096-token sequences fit with BF16 weights and BF16 KV cache, using 90% of memory?
  3. Same L4, batch 1, no cache: what is the lower bound on the time per token? How does INT4 weights change it?
  4. A service gets 5 requests per second; each request lasts 8 seconds and holds 6,000 tokens. Llama 3 8B on one H100: does the KV cache fit?
  5. Training Llama 3 8B for 1 T tokens on 256 H100s at 40% MFU: how many days?
  6. In case 4 of Figure 4, a new request arrives just after a static batch has started. Up to how long does it wait before it can start, at 5.4 ms per step?
Answerscheck
1. \(P_{\text{layer}} = 2 \cdot 4096 \cdot 4096 + 2 \cdot 4096 \cdot 1024 + 3 \cdot 4096 \cdot 14336 + 8192 = 218{,}112{,}000\) (the same block as Llama 3 8B); \(N = 32 \times 218{,}112{,}000 + 2 \times 32{,}000 \times 4{,}096 + 4{,}096 = 7{,}241{,}732{,}096\), about 7.24 B. 2. Usable \(0.9 \times 24 = 21.6\) GB, minus 16.06 GB of weights, leaves 5.54 GB. One sequence of 4,096 tokens needs \(4096 \times 131{,}072 = 0.537\) GB, so 10 sequences fit. 3. \(16.06 / 0.3 = 53.5\) ms per token (about 19 tokens per second). With INT4 weights (about 4.0 GB) the bound is about 13.4 ms. 4. \(\bar B = 5 \times 8 = 40\) sequences, \(40 \times 6{,}000 \times 131{,}072 = 31.5\) GB of cache plus 16.06 GB of weights is 47.5 GB, under 72 GB: it fits. 5. \(6 \times 8.03 \times 10^9 \times 10^{12} = 4.82 \times 10^{22}\) FLOPs; \(256 \times 989 \times 10^{12} \times 0.4 = 1.013 \times 10^{17}\) FLOP/s; \(4.76 \times 10^5\) s, about 5.5 days. 6. The batch runs until its longest request finishes: 800 steps \(\times\) 5.4 ms \(\approx\) 4.3 s, before any prefill of the new request.

Key takeaways

  • The whole system fits on one whiteboard: tokenizer → model → training → serving → application. Each box has one number that matters most.
  • Parameters come from the config (162.1, 162.2): Llama 3 8B has 8.03 B. Inference is about \(2N\) FLOPs per token, training about \(6N\) (162.3); AdamW training needs about 16 bytes per parameter (162.5).
  • The KV cache is \(2 L n_{kv} d_h b_{kv}\) bytes per token (162.6): 128 KiB for Llama 3 8B. Weights plus cache must fit (162.7), and Little’s law (162.11) tells you how many sequences to plan for.
  • Decode is memory-bound until the batch reaches about \(b_w/2 \times\) peak/BW (162.8, 162.10); prefill is compute-bound (162.9).
  • Find the bottleneck before you fix it: split the trace, compare with the bounds, then pick the survey branch (data-, model- or system-level). The bottleneck moves after each fix.

Key formulas

\[ N = L\,(2dHd_h + 2dn_{kv}d_h + 3dd_{ff} + 2d) + 2Vd + d, \qquad C \approx 6ND, \qquad M_{\text{train}} \approx 16N \]
\[ \text{kv}_{\text{tok}} = 2Ln_{kv}d_hb_{kv}, \qquad t_{\text{step}} \ge \max\!\left(\frac{b_wN + BT\,\text{kv}_{\text{tok}}}{\text{BW}}, \frac{2NB}{\text{peak}}\right), \qquad \bar B = \lambda W \]

References for this session

  1. Zhao et al. (2023). A Survey of Large Language Models
  2. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  3. Zhao et al. (2023). A Survey of Large Language Models
  4. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  5. Llama Team, Meta (2024). The Llama 3 Herd of Models
  6. Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
  7. Kaplan et al. (2020). Scaling Laws for Neural Language Models
  8. Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
  9. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  10. Meta (2024). Llama 3.1 8B and 70B config.json files (Hugging Face mirrors of meta-llama/Meta-Llama-3.1-8B-Instruct and -70B-Instruct)
  11. NVIDIA (2023). NVIDIA H100 Tensor Core GPU (product page and data sheet)