LLM & VLM courseBlog
238Track 10: Capstone and interviews · session 5 of 6

Interview drills: ML system design for LLM, VLM and video services

System design interviews for LLM, VLM and video services test one skill: turning a vague product request into numbers, a design, and a clear view of what breaks first. This session gives you one answer frame and runs it on three drills: a chat API with a latency target, natural-language search over a sports video archive, and real-time key-moment alerts on a live broadcast.

Time: 3 to 4 hours (reading 2 hours, drills and exercises 1 to 2 hours)Level: Advanced · capstoneBefore this: Inference metrics, KV cache, batching, text-to-video retrieval and key-moment detection (sessions 80, 81, 135, 208, 207)

What you will learn

  • A five-step answer frame for any ML system design question: requirements, numbers, design, bottleneck, failure modes
  • Back-of-envelope formulas you can do on a whiteboard: Little's law, KV bytes per token, TPOT floor, compute utilization, replicas
  • Drill 1: size a chat API for a TTFT and TPOT target, and find out that the prompt, not the answer, is the main cost
  • Drill 2: design natural-language search over 10,000 hours of sports video: index size, backfill cost, two-stage retrieval and re-ranking
  • Drill 3: design real-time key-moment alerts and reason about the trade-off between alert delay and false alarms
  • How to name the bottleneck and the failure modes before the interviewer asks, and how to handle follow-up questions such as “now 10 times the traffic”

Why this matters

In a system design interview you get one vague sentence, such as “design a chat API” or “let fans search old matches by describing a moment”. You have about 45 minutes. The interviewer does not want the one correct design; there is none. They want to see how you think.

Most weak answers fail in the same way. The candidate draws boxes at once (“a load balancer, then vLLM, then a vector database”) and never says how big anything is. Then the interviewer asks “how many GPUs?” or “what happens at the final whistle when everyone searches at once?”, and the design has no numbers to answer with. Strong answers turn the vague sentence into numbers early, let the numbers choose the design, and name the part that breaks first before anyone asks.

This session is a drill book. It builds on almost every track of the course: the inference metrics and serving ideas of Track 5, the retrieval ideas of session 208, and the key-moment work of session 207 and session 235. We first fix one answer frame. Then we run it three times, each time with real numbers that you can check with a calculator:

  • Drill 1. A chat API with a latency target. The numbers tell you how many GPUs you need and that the prompt is the main cost.
  • Drill 2. Natural-language search over a sports archive. The numbers tell you how big the index is, how long the backfill takes and where the query time goes.
  • Drill 3. Real-time key-moment alerts on a live broadcast. The numbers tell you the price of every second of delay in false alarms.

The previous session, session 237, measured where a real system fails. This session is the spoken version of the same skill: explain a system, its numbers and its failures to another engineer in a short time.

The answer frame

We use five steps, always in the same order. Each step has one job.

StepJobWhat you produceTime in a 45-minute round
1. RequirementsTurn the vague request into things you can measureUsers and traffic, inputs and outputs, latency targets, quality target, cost limit, what is out of scopeabout 5 min
2. NumbersSize the problem before choosing partsRequests per second, tokens or frames per second, bytes in memory, FLOPs per second, number of GPUsabout 8 min
3. DesignChoose the parts that the numbers allowA diagram of the online path and the offline path, with the model and hardware choice for each boxabout 15 min
4. BottleneckSay which resource runs out first, and the fix“Compute is the limit, because …; we fix it with …”about 7 min
5. Failure modesShow that you have run real systemsWhat breaks, how you detect it, what the system does then; how you evaluate qualityabout 7 min

The times are a suggestion, not a rule. The order matters more: numbers come before the design, because the numbers decide the design. A system with 1 request per second and one with 1,000 requests per second are different systems.

Figure 1 shows the frame for each of the three drills of this session. Use it as a summary card now, and come back to it after you have read each drill.

Figure 1. The five-step answer frame, filled in for the three drills of this session. Choose a drill, then step through it or click a box. The grey box at the bottom of each step is a sentence you could say in the interview.
How to read Figure 1

The five boxes are the five steps, left to right. The highlighted box is the current step; the boxes before it are done. The panel shows what you produce in this step for the chosen drill. Notice that the same five steps fit three very different systems: a text service, a search service and a streaming service. Notice also that the “Numbers” step of every drill already points at the bottleneck.

Step 1: requirements as numbers

Ask, or state as an assumption, these five things. If the interviewer says “you decide”, choose a round number and write it down.

  1. Load. Requests per second at peak (not on average), and how “bursty” it is.
  2. Size of each request. For text: prompt tokens \(n_{\text{in}}\) and output tokens \(n_{\text{out}}\). For video: hours of video, frames per second, clips per query.
  3. Latency targets. A service level objective (SLO) is a target such as “95% of requests get their first token in under 1 s”. For streaming text we use two numbers from session 81: TTFT (time to first token) and TPOT (time per output token, the gap between two tokens).
  4. Quality. How you will measure it: recall@10 for search, precision and recall of alerts, a judge score for chat.
  5. Cost. A budget in GPUs or dollars, or “as cheap as possible for this SLO”.

Step 2: the five formulas you need

Almost every number in this session comes from a handful of formulas. The first is Little’s law from queueing theory: in a stable system, the average number of items inside equals the arrival rate times the average time each item stays.

\[ B = \lambda \cdot W, \qquad W \approx \text{TTFT} + n_{\text{out}} \cdot \text{TPOT} \tag{238.1} \]
Read the symbols
\(B\)
the average number of requests inside the system at one time (for an LLM server: the running batch)
\(\lambda\)
the arrival rate, in requests per second
\(W\)
the average time one request stays, in seconds
\(n_{\text{out}}\)
output tokens per request

Little’s law needs no assumption about the arrival pattern, only a stable system [Certain]. It is the bridge from “requests per second” (what the product team says) to “requests in memory at once” (what the GPU sees). In the formulas below we use the part of \(W\) spent decoding, \(n_{\text{out}}\cdot\text{TPOT}\), because that is when a request holds a slot in the decode batch.

The second formula is the size of the KV cache (session 80): every token in the context keeps one key and one value vector in every layer.

\[ s_{\text{kv}} = 2 \cdot L \cdot n_{\text{kv}} \cdot d_h \cdot b \tag{238.2} \]
Read the symbols
\(s_{\text{kv}}\)
bytes of KV cache per token of context
\(2\)
one key and one value
\(L\)
number of layers
\(n_{\text{kv}}, d_h\)
number of key/value heads (fewer than query heads with GQA) and the size of each head
\(b\)
bytes per number: 2 for BF16, 1 for FP8

The other three formulas (memory, step time, compute) come in Drill 1, where we need them. Before that, here are the facts you should know by heart. Everything else you compute.

FactValueSource
NVIDIA H100 SXM80 GB, about 3.35 TB/s, about 989 TFLOP/s dense BF16NVIDIA datasheet [Certain]
NVIDIA L424 GB, 300 GB/s, 121 TFLOP/s dense BF16NVIDIA datasheet (242 is “with sparsity”) [Certain]
Llama 3.1 8B8.03 B parameters, 32 layers, 8 KV heads of size 128Llama 3 paper and model config [Certain]
Llama 3.1 70B70.6 B parameters, 80 layers, 8 KV heads of size 128Llama 3 paper and model config [Certain]
Forward pass costabout \(2N\) FLOPs per token for a model with \(N\) parameterssession 1 [Certain]
CLIP image encodersViT-B/16 gives 512-d vectors, ViT-L/14 gives 768-d vectorsCLIP paper and released models [Certain]

We also need assumptions about how well real software uses the hardware. We reuse the ones from session 141: matrix multiplies reach \(u = 50\%\) of peak FLOP/s, memory reads reach \(\eta = 80\%\) of peak bandwidth, every decode step has \(t_0 = 1\) ms of fixed overhead, and 90% of GPU memory is usable. These are planning assumptions, not measurements. Say them out loud in the interview; a good interviewer cares more that you know they exist than about their exact values.

Steps 3 to 5: design, bottleneck, failure modes

For the design, always draw two paths. The online path is what happens while a user waits; it has a latency budget. The offline path is everything that can run in the background: indexing, training, evaluation. Moving work from the online path to the offline path is the most common trick in system design.

For the bottleneck, name one resource: GPU compute, GPU memory bandwidth, GPU memory size, network, CPU, or a human (labels, review). Then say the fix and what it costs. For the failure modes, give three to five concrete ones, each with how you detect it and what the system does then.

Algorithm 1: the answer frame
  1. input: a vague product request, about 45 minutes
  2. write down load \(\lambda\) at peak, request size, SLOs (p95 TTFT, TPOT, or end-to-end), quality metric, cost limit // ask or assume
  3. compute per-request work (tokens, frames, FLOPs) and per-request memory (KV bytes, index bytes)
  4. multiply by \(\lambda\) and use Little’s law (238.1) to get totals that must exist at the same time
  5. divide totals by one GPU’s usable limits to get a first GPU count // compute, bandwidth, memory: the largest wins
  6. draw the online path (within the latency budget) and the offline path (no user waiting)
  7. name the binding limit, and one fix that moves it (cache, batch, quantize, cascade, move work offline)
  8. list failure modes with detection and fallback; state how you measure quality before and after launch
  9. when the interviewer changes a number (“10× traffic”, “half the budget”) do go back to line 3, not to line 6

Drill 1: a chat API with a latency target

The prompt. “We want a chat assistant API for our product, built on an open model. It must feel fast. Design the serving system.”

Requirements

We fix these numbers and say them out loud: peak load \(\lambda = 20\) requests per second; on average \(n_{\text{in}} = 1{,}000\) prompt tokens (system prompt plus chat history) and \(n_{\text{out}} = 300\) output tokens; p95 TTFT under 1 s; TPOT under 50 ms, which is 20 tokens per second, faster than people read [Likely]. Model: Llama 3.1 8B in BF16 for a first plan, with the 70B as a follow-up. Out of scope: training, safety filters, billing.

Numbers

We need three more formulas. Memory: the running requests must fit next to the weights. With \(M_W\) the weight bytes and \(C\) the GPU memory:

\[ B \cdot (n_{\text{in}} + n_{\text{out}}) \cdot s_{\text{kv}} \;\le\; 0.9\,C - M_W \tag{238.3} \]

Step time: one decode step reads all weights and the KV cache of the batch, and also does some FLOPs. It costs the larger of the two, plus a fixed overhead. With continuous batching and chunked prefill (session 135), the same steps also process waiting prompt tokens. Putting \(B = \lambda\, n_{\text{out}}\,\text{TPOT}\) into the step time and solving (this is formula 141.8 of session 141) gives:

\[ \text{TPOT} = \max\!\left( \frac{t_0 + M_W/(\eta\,\text{BW})}{1 - \beta},\; \frac{t_0}{1 - c} \right), \qquad \beta = \frac{\lambda\, n_{\text{out}}\, \bar L\, s_{\text{kv}}}{\eta\,\text{BW}} \tag{238.4} \]
Read the symbols
\(t_0 + M_W/(\eta\,\text{BW})\)
the TPOT floor: the time to read all weights once. No batch size can beat it.
\(\beta\)
the share of each step spent reading KV cache; it grows with load
\(\bar L = n_{\text{in}} + n_{\text{out}}/2\)
the average context length while a request is decoding
\(c\)
compute utilization, formula (238.5)
\(\text{BW}\)
memory bandwidth of the GPU, bytes per second

Compute: each request costs about \(2N\) FLOPs per token it pushes through the model. If a share \(h\) of the prompt is already in a prefix cache (the KV of a shared system prompt or of earlier chat turns, kept from before, session 134), those tokens need no compute.

\[ C_{\text{req}} = 2N\left( (1-h)\, n_{\text{in}} + n_{\text{out}} \right), \qquad c = \frac{\lambda\, C_{\text{req}}}{u\,F}, \qquad \text{TTFT} \approx \frac{t_0 + 2N(1-h)\,n_{\text{in}}/(uF)}{1 - c} \tag{238.5} \]
Read the symbols
\(N\)
model parameters; \(2N\) FLOPs per token for the linear layers (attention FLOPs are small at this context length: under 2% here)
\(h\)
prefix-cache hit rate: the share of prompt tokens whose KV is already stored
\(F, u\)
peak FLOP/s of the GPU and the share we really reach (50%)
\(1/(1-c)\)
a simple queueing factor: waiting time grows without bound as the GPU gets full. It is the same shape as the M/M/1 queue, used here as a rough rule

Finally the plan. Let \(\mu^{*}\) be the largest load one replica can carry while meeting every target. We do not plan to run at 100% of \(\mu^{*}\): traffic has bursts and long requests. With headroom \(H\) (we use 0.8):

\[ \mu^{*} = \max\{\lambda : \text{TPOT} \le \tau_{\text{TPOT}},\ \text{TTFT} \le \tau_{\text{TTFT}},\ (238.3) \text{ holds}\}, \qquad R = \left\lceil \frac{\lambda}{H\,\mu^{*}} \right\rceil \tag{238.6} \]
Worked example: Llama 3.1 8B on H100s

Memory. Weights \(M_W = 8.03\times10^9 \times 2 = 16.06\) GB. KV per token (238.2): \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes = 128 KiB. One request holds up to \(1{,}300\) tokens, so about 170 MB.

Compute. \(C_{\text{req}} = 2 \times 8.03\times10^9 \times 1{,}300 = 20.9\) TFLOP. At 20 requests per second that is 417.6 TFLOP/s. One H100 gives \(uF = 0.5 \times 989 = 494.5\) TFLOP/s, so on one GPU \(c = 0.84\).

Latency on one GPU. TPOT floor \(= 1 + 16.06/(0.8 \times 3.35) = 7.0\) ms. KV share \(\beta = 20 \times 300 \times 1{,}150 \times 131{,}072 / (0.8 \times 3.35\times10^{12}) = 0.34\). So TPOT \(= \max(7.0/0.66,\ 1/0.16) = 10.6\) ms, far under 50 ms. But TTFT \(= (1 + 32.5)/(1 - 0.84) = 215\) ms, and it explodes as \(c \to 1\). Little’s law: \(B = 20 \times 300 \times 0.0106 = 63\) requests in the batch, holding 10.8 GB of KV, far under the 55.9 GB free.

Plan. Bisection on (238.6) gives \(\mu^{*} = 22.9\) requests per second per H100, limited by TTFT (that is, by compute). With headroom 0.8: \(R = \lceil 20 / (0.8 \times 22.9) \rceil = 2\) H100s. Each then runs at 10 requests per second: compute 42%, TPOT 8.4 ms, TTFT 58 ms, 25 requests in the batch, 4.3 GB of KV.

Two follow-ups. With 50% of prompt tokens in the prefix cache, \(\mu^{*} = 37.2\) and one H100 is enough. With the 70B model on groups of 4 H100s (tensor parallel, communication ignored), \(\mu^{*} = 10.0\) per group, so 3 groups = 12 GPUs. On an L4 the 8B plan is impossible: the TPOT floor is \(1 + 16.06/(0.8 \times 0.3) = 67.9\) ms, above the 50 ms target, for any load.

The surprise for many candidates: the answer has 300 tokens and the prompt 1,000, so 77% of the compute is prompt processing (prefill). Decode takes most of the wall-clock time of a request, but prefill takes most of the FLOPs. This is why prefix caching and short system prompts are the cheapest speed-ups for chat, and why some systems put prefill and decode on different GPUs (session 137). Try it in Figure 2.

Figure 2. A whiteboard capacity plan for Drill 1, from formulas (238.1) to (238.6), with \(u = 0.5\), \(\eta = 0.8\), \(t_0 = 1\) ms, 90% usable memory and headroom \(H = 0.8\). The bars show one replica at the planned load \(\lambda / R\). For “70B on 4×H100” the four GPUs are treated as one big GPU (tensor parallel, no communication cost), so it is a best case.
How to read Figure 2

The big line is the answer to “how many GPUs?”. Each bar is one limit of one replica at its planned load, as a share of what is allowed: compute busy (of 100%), KV memory used (of the free memory), TPOT (of its target) and TTFT (of its target). A bar near 100% is the bottleneck. Start with the defaults: 2 H100s, and compute is the fullest bar. Raise the prefix-cache hit rate to 0.5: one GPU is enough. Raise the prompt to 8,000 tokens: the GPU count jumps, because prefill compute grows with the prompt. Choose the L4: no load is possible at a 50 ms TPOT, because reading 16 GB of weights at 240 GB/s takes longer than 50 ms; move the TPOT target to 70 ms or more to see it work.

Design

Now the numbers allow a simple design. Online path: an API gateway (authentication, rate limits per customer) sends requests to a router, which picks one of \(R\) model replicas. Each replica runs an engine with continuous batching, chunked prefill and a paged KV cache with prefix caching (vLLM, SGLang or TensorRT-LLM, session 139). Tokens stream back to the client as they are made. The router should be prefix-aware: send the next turn of a chat to the replica that already holds its history in the cache, or the prefix cache hit rate drops by about a factor \(R\) [Likely]. Offline path: logs of TTFT, TPOT, queue length and cache hit rate per replica; load tests with a realistic mix of prompt lengths; an evaluation set to check quality after every model or quantization change.

ComponentChoiceWhy, from the numbers
Model replicaLlama 3.1 8B, BF16, one H100 eachWeights 16 GB fit with room for KV; TPOT floor 7 ms ≪ 50 ms
Replicas2 at peak (autoscale down at night)\(\mu^{*} = 22.9\) req/s per GPU, headroom 0.8
BatchingContinuous, chunked prefillPrefill is 77% of FLOPs; chunking stops long prompts from freezing decode
KV cachePaged, with prefix cachingChat history is reused every turn; each 10% of hits cuts prefill by 10%
RouterLeast-loaded, with session or prefix affinityKeeps cache hits high; spreads bursts

Bottleneck

Compute, through prefill. Our numbers show TPOT has a lot of slack (8 ms against 50 ms), while TTFT grows fast as compute fills up. Fixes, in order of cost: prefix caching and shorter system prompts (free); FP8 weights and activations, which roughly doubles peak FLOP/s on an H100 [Likely]; more replicas; and, at large scale, separate prefill and decode GPUs so that each can be sized for its own limit (session 137). Speculative decoding (session 128) would not help much here: it trades extra compute for lower TPOT, and TPOT is not our problem.

Failure modes

  • Burst at peak (a product launch). Detect: queue length and TTFT p95 rising. Act: admission control (return “busy, retry” for the lowest tier), scale out; do not let queues grow without limit.
  • KV memory full. Detect: preemptions per minute in the engine. Act: the engine swaps or recomputes some requests (higher latency); cap max context per request.
  • A replica dies. Detect: health checks. Act: the router stops sending to it; streams in progress are lost, so the client retries. With \(R = 2\) and headroom 0.8, one replica alone cannot carry the full peak, so \(R = 3\) is the honest answer if the SLO must hold during a failure (“N+1”).
  • Quality drop after a change (new quantization, new model). Detect: offline evaluation set before launch, and a small share of live traffic (canary) compared with the old version.
  • Very long outputs (a loop that repeats). Detect: output length distribution. Act: a max-token limit and repetition checks.

Drill 2: natural-language video search over a sports archive

The prompt. “We have 10,000 hours of football matches. Editors want to type ‘header goal from a corner in the last minute’ and get the right moments. Design it.”

Requirements

Users: a few hundred editors, so the query load is small (say under 10 queries per second at peak). Each query returns the top 20 segments with timestamps. Latency: under 1 s for the first results; editors accept 2 to 3 s if results get better. Quality: recall@20 on a labeled test set of queries; the right moment should usually be in the first page. Archive: 10,000 hours now, plus new matches every day. Cost: indexing can be slow (days are fine); the query path must be cheap.

The first insight comes from this list: the load is tiny, but the archive is big. So the hard part is not serving queries; it is turning 36 million seconds of video into something we can search in milliseconds.

Numbers

We cut every match into short segments (for example 4 s), sample frames at \(f\) frames per second, encode them with an image–text encoder such as CLIP, and pool the frame vectors into one vector per segment (session 208). The index size and the one-time encoding cost (the backfill) are:

\[ N_{\text{seg}} = \frac{3600\, H_v}{\Delta}, \qquad S_{\text{index}} = N_{\text{seg}} \cdot d \cdot b_v \tag{238.7} \]
\[ T_{\text{backfill}} = \frac{3600\, H_v \cdot f \cdot G_{\text{frame}}}{u\,F} \tag{238.8} \]
Read the symbols
\(H_v\)
hours of video in the archive
\(\Delta\)
segment length in seconds
\(d, b_v\)
vector size and bytes per number in the index (4 for FP32, 2 for FP16, 1 for INT8); with product quantization (PQ) every vector becomes a fixed code, here 64 bytes
\(f\)
frames sampled and encoded per second of video
\(G_{\text{frame}}\)
FLOPs to encode one frame

What is \(G_{\text{frame}}\)? A ViT with \(L\) layers, width \(d\) and \(T\) tokens costs about \(2L\,(12\,T d^2 + 2T^2 d)\) FLOPs. For ViT-B/16 at 224 pixels (\(L = 12\), \(d = 768\), \(T = 197\)) that is 34.9 GFLOPs per frame; for ViT-L/14 (\(L = 24\), \(d = 1024\), \(T = 257\)) it is 161.7 GFLOPs. These match the usual published figures of about 17.6 and 81 billion multiply-adds [Likely].

Worked example: 10,000 hours, 4-second segments

Index. \(N_{\text{seg}} = 3600 \times 10{,}000 / 4 = 9{,}000{,}000\) segments. With 512-d CLIP ViT-B/16 vectors: FP32 18.4 GB, FP16 9.2 GB, INT8 4.6 GB, PQ with 64-byte codes 0.58 GB. All of these fit in one server’s RAM; FP16 even fits on one L4.

Backfill. At 1 frame per second: \(3.6\times10^{7}\) frames \(\times\) 34.9 GFLOPs \(= 1.26\times10^{18}\) FLOPs. One L4 at \(u = 0.5\) gives 60.5 TFLOP/s, so about 1,734 frames per second and 5.8 GPU-hours. With ViT-L/14 it is 26.7 GPU-hours. Video decoding on the CPU, not the GPU, is often the real limit of this job [Likely].

Query. A brute-force (“flat”) scan over 9 M FP16 vectors of size 512 reads 9.2 GB; on an L4 at 80% of 300 GB/s that is 38 ms. An approximate nearest neighbor (ANN) index such as HNSW or IVF-PQ reads only a small part of it (session 150).

Re-ranking. Re-ranking 20 candidates with an 8B VLM, each with 4 frames of 256 visual tokens plus 50 text tokens (1,074 tokens), costs \(20 \times 2 \times 8.03\times10^9 \times 1{,}074 = 345\) TFLOP: about 0.7 s on one H100 at \(u = 0.5\).

So the numbers say: the index is small and cheap; the backfill is a one-day batch job; and the expensive part of a query is the re-ranker. Figure 3 lets you change the archive and the choices.

Figure 3. Sizing the video index for Drill 2 with formulas (238.7) and (238.8). Bars: index size for each storage format at the chosen archive size and segment length; the dashed lines are the memory of one L4 (24 GB) and one H100 (80 GB). The readout gives the backfill time on L4 GPUs and the time for one brute-force scan on an L4.
How to read Figure 3

The chosen format is the highlighted bar. Move the archive slider (it is logarithmic: 100 to 100,000 hours). At 100,000 hours with 2-second segments, even FP16 no longer fits in one H100: you need INT8, PQ or a CPU index. Make the segments longer: the index shrinks, but one segment now mixes more events, and search gets less precise in time. Change the encoder to ViT-L/14: the index grows by 1.5× (768 against 512 numbers) but the backfill grows by about 4.6×, because each frame costs 161.7 instead of 34.9 GFLOPs. Encoding more frames per second costs backfill time but does not change the index size, because frames are pooled into one vector per segment.

Design

Offline path: (1) decode each match and sample frames; (2) encode frames with the image encoder and pool them per segment; (3) also store cheap metadata per segment: match, teams, minute of play, and the transcribed commentary from speech recognition (Whisper), because commentators say “goal!” and player names; (4) build an ANN index (HNSW or IVF-PQ in FAISS) plus a text index over the commentary. New matches are added each night.

Online path, in three stages, each smaller and more expensive than the one before:

Algorithm 2: one search query
  1. input: query text \(q\), filters (team, season), \(k = 20\)
  2. \(\mathbf{e} \leftarrow \text{TextEncoder}(q)\) // a few ms on a GPU
  3. \(A \leftarrow\) ANN top 500 segments by \(\cos(\mathbf{e}, \mathbf{v}_s)\), only those that pass the filters // stage 1: about 10 ms
  4. \(A' \leftarrow\) top 500 by keyword score on commentary (BM25) // catches names and rare words
  5. \(C \leftarrow\) top 50 of \(A \cup A'\) by fused rank // stage 2: reciprocal-rank fusion
  6. merge neighboring segments of the same match into one moment with start and end times
  7. \(S \leftarrow\) VLM re-ranker scores the top 20 moments of \(C\) with a few frames each // stage 3: the costly step, about 0.7 s
  8. return top \(k\) moments of \(S\), with thumbnails and timestamps; show stage-2 results first if stage 3 is slow

Bottleneck

Not the index and not the load: the re-ranker. It costs about 0.7 s of an H100 per query in our example, and it grows linearly with the number of candidates and frames. Fixes: re-rank fewer candidates; use a smaller re-ranker (a 2B VLM is about 4× cheaper than the 8B); cache results of popular queries; stream stage-2 results first. The second real limit is quality in time: a 4-second segment either contains the header or it does not, and the pooled vector of a busy segment is a blur. That is why we merge neighbors and why a temporal grounding model (session 207) can refine the boundaries of the final few moments.

Failure modes

  • Football-specific words. CLIP was not trained on “offside trap” or “nutmeg”. Detect: low recall on those query types in the test set. Act: the commentary text index, and fine-tuning the encoder on sports clip–caption pairs.
  • Replays and highlights packages. The same goal appears 5 times (live, 3 replays, a studio clip). Act: near-duplicate grouping with video fingerprints (session 209) so that one goal is one result.
  • Index and model mismatch. If you update the encoder, old vectors are no longer comparable with new query vectors. Act: version the index, and re-run the backfill (we know it costs about 6 GPU-hours, so this is affordable).
  • Recall measured only on easy queries. Act: build the test set from real editor queries, by type (event, player, tactic, emotion), and report recall per type (session 213).

Drill 3: real-time key-moment alerts on live broadcast

The prompt. “During live matches, send an alert to editors (and later to fans) within seconds of a goal, a red card or a big save. Design it.”

Requirements

Load: up to 50 live matches at the same time on a busy day. Events: about 10 alert-worthy moments per 90-minute match (our assumption for the drill). Latency: the alert should arrive within about 10 s of the moment; social media is often faster than that, so every second counts. Quality: almost no missed goals, and very few false alerts, because each false alert to millions of fans is visible and costly. In this drill the key quality number is false alerts per match, not accuracy per frame.

Numbers

Two budgets matter: time and false alarms. First, the time. A detector looks at a window of the last \(w\) seconds and fires when its score passes a threshold \(\theta\). For an event that raises the score by \(A\) from the moment it starts, the windowed mean passes \(\theta\) after about \(\theta w / A\) seconds. The full delay is:

\[ D_{\text{alert}} \approx \underbrace{\frac{\theta\, w}{A}}_{\text{evidence}} + \underbrace{\Delta_{\text{hop}}}_{\text{wait for next check}} + t_{\text{model}} + t_{\text{verify}} + t_{\text{send}} \tag{238.9} \]

Second, the false alarms. If the score is pure noise with standard deviation \(\sigma\) per second, the mean of \(w\) seconds has standard deviation \(\sigma/\sqrt{w}\). With one check per second over a match of \(T\) seconds, the expected number of noise crossings is at most:

\[ \E[\text{noise alarms}] \;\lesssim\; T \cdot Q\!\left( \frac{\theta \sqrt{w}}{\sigma} \right), \qquad Q(z) = P(Z \gt z),\ Z \sim \mathcal{N}(0, 1) \tag{238.10} \]
Read the symbols
\(w\)
the length of the evidence window, in seconds
\(\theta\)
the alert threshold on the windowed mean score
\(A\)
how much a real event raises the per-second score
\(\Delta_{\text{hop}}\)
time between two checks of the detector (1 s here)
\(Q(z)\)
the tail probability of a standard normal variable

Read the two formulas together: a longer window \(w\) and a higher threshold \(\theta\) push the noise term down very fast (it is a Gaussian tail), but they add delay linearly. This is the core trade-off of the whole drill. Formula (238.10) only covers random noise. Real false alarms often come from look-alikes: a replay of an earlier goal, a shot that hits the post, a loud crowd. Those are not noise, so no threshold on noise statistics removes them. Duration helps (a replay may be shorter than the real moment), and a second, smarter model helps more.

Third, compute. With one GPU giving \(uF\) FLOP/s and a frame encoder costing \(G_{\text{frame}}\) FLOPs:

\[ \text{streams per GPU} = \left\lfloor \frac{u\,F}{f \cdot G_{\text{frame}}} \right\rfloor \tag{238.11} \]

For ViT-B/16 at \(f = 2\) frames per second on an L4: \(\lfloor 60.5\times10^{12} / (2 \times 34.9\times10^{9}) \rfloor = 866\) streams. Even ViT-L/14 at 2 frames per second gives 187. So with 50 matches, the cheap always-on detector is not the cost. The cost is the heavy model that checks candidates, and that runs only a few times per minute.

Worked example: the delay–false-alarm trade-off

Take \(\sigma = 0.3\), \(A = 1\), a 90-minute match (\(T = 5{,}400\) s). With \(w = 1\), \(\theta = 0.7\): \(z = 0.7 / 0.3 = 2.33\), \(Q(2.33) \approx 0.0098\), so up to \(5{,}400 \times 0.0098 = 53\) noise crossings per match: useless. With \(w = 4\), \(\theta = 0.7\): \(z = 0.7 \times 2 / 0.3 = 4.67\), so 0.008 noise alarms per match; the evidence delay is \(\theta w / A = 2.8\) s. With \(w = 8\), \(\theta = 0.6\): \(z = 5.66\) and an evidence delay of 4.8 s. Figure 4 runs these settings on a simulated match, where look-alikes also exist.

Figure 4. A simulated 90-minute match with a per-second detector score: Gaussian noise (\(\sigma = 0.3\)), 10 real key moments (score +1.0 for 8 s) and 20 look-alikes such as replays (+0.7 for 5 s). The detector fires when the mean of the last \(w\) seconds is above \(\theta\), then stays quiet for 30 s. An alert within 30 s after a moment starts counts as a hit; any other alert is a false alarm. The random numbers are seeded, so every run is the same.
How to read Figure 4

Top strip, the whole match: the line is the score (the highest value in each 6-second bin), green bands are the real moments, grey bands are the look-alikes, and the triangles above are alerts (green: hit, red: false). Bottom panel, one moment up close: bars are the per-second scores, the thick line is the windowed mean, the dashed line is \(\theta\), and the marker shows when the alert fires. Start with \(w = 1\): many red triangles from noise. At \(w = 4\) the noise alarms are gone, but the look-alikes still fire. At \(w = 8\), \(\theta = 0.6\) a look-alike can no longer reach \(\theta\) (its 5 s at +0.7 average to at most 0.44 over 8 s), and there are no false alarms, but the delay grows to about 5 s. Push \(w\) to 15 and \(\theta\) to 0.6: real moments are missed too, because an 8-second moment averages to at most 0.53 over 15 s.

The simulation shows the three regimes of formula (238.9) and (238.10): noise-limited (short windows), look-alike-limited (medium windows), and miss-limited (windows longer than the event). In a real design, we do not ask one threshold to do everything. We use a cascade: a cheap, always-on detector tuned for high recall (it may fire on look-alikes), and a heavy verifier that runs only on its candidates and is tuned for precision.

Design

Per live match: ingest the broadcast stream; decode and sample 2 frames per second, plus audio; run a light video model and an audio model (crowd noise and commentator excitement are strong, fast signals) every second over a sliding window; fuse their scores. When the fused score passes a low threshold, send a candidate (the last 20 s of frames and audio transcript) to the verifier: a VLM prompted to answer “was there a goal, red card or big save, and is this live play or a replay?”. Only verified events become alerts. An editor can approve alerts for the fan channel in the first weeks, which also creates labels.

Algorithm 3: streaming alert loop for one match
  1. input: live stream, window \(w\), low threshold \(\theta_1\), refractory time \(r\)
  2. every \(\Delta_{\text{hop}} = 1\) s do
  3. \(s_t \leftarrow \text{fuse}(\text{VideoModel}(\text{last } w \text{ s of frames}),\ \text{AudioModel}(\text{last } w \text{ s}))\)
  4. if \(s_t \gt \theta_1\) and no alert in the last \(r\) seconds then
  5. clip \(\leftarrow\) last 20 s of frames + transcript // small: send to the verifier queue
  6. \((y, \text{type}) \leftarrow \text{VLMVerifier}(\text{clip})\) // 1 to 2 s; also says “replay or live”
  7. if \(y\) = yes then send alert(type, \(t\)) // at-most-once, with an event id
  8. log \(s_t\), candidates and verifier answers // for threshold tuning and labels

Bottleneck

Not compute (one L4 can serve hundreds of streams for the light detector, by (238.11)). The bottleneck is the delay–false-alarm trade-off, plus the verifier’s latency at peak: when a goal is scored in 10 matches at the same minute, 10 verifier calls arrive at once. Give the verifier its own GPU pool, sized for bursts, and a time limit; if it times out, send the alert to the editor queue instead of to fans.

Failure modes

  • Replays fire again after the real moment. Act: the refractory window, a “replay” output in the verifier, and a broadcast graphics detector (replays often start with a logo wipe [Likely]).
  • Stream problems: the feed freezes or lags. Detect: frame timestamps against wall-clock time. Act: pause alerts for that match; never alert on a frozen frame.
  • Disallowed goals (offside, video review). Act: send a correction alert linked to the first one; for fans, consider waiting for the restart of play.
  • Duplicates when the service restarts. Act: an event id per match and moment, and at-most-once delivery.
  • Drift: a new competition with different graphics or camera style. Detect: the rate of candidates and verifier rejections per match moves. Act: re-tune thresholds on the new data.

Handling follow-up questions

After your design, the interviewer changes one number to see whether your design was built on numbers or on habit. Always go back to the numbers step. The common follow-ups and the first thing to say:

Follow-upFirst moveDrill example
“10× the traffic”Which limit grows linearly, which does not?Drill 1: 10× the replicas, but prefix caching and the router matter more. Drill 2: load is still small; the index does not change.
“Half the budget”Remove the biggest cost, not the easiestDrill 1: FP8 or a smaller model, after checking quality. Drill 2: fewer re-ranked candidates.
“Latency target halved”Is the new target below a floor?Drill 1: TPOT floor is 7 ms on an H100, fine; on an L4 it was already impossible.
“Add images or video to the chat”Visual tokens are prompt tokensDrill 1: a 1,000-token image adds prefill compute and KV like 1,000 text tokens.
“How do you know it works?”Offline metric, online metric, and a canaryDrill 2: recall@20 by query type; Drill 3: false alerts per match and median delay.
“What would you build first?”The smallest end-to-end version, then measureDrill 3: audio-only detector plus editor approval, before any VLM.

Code: the three drills in a few lines

The whiteboard numbers above come from these three programs. The first is the capacity plan of Drill 1, written in plain Python (it is just arithmetic).

import math

def chat_capacity(lam, n_in, n_out, hit=0.0, tpot_max=0.050, ttft_max=1.0,
                  N=8.03e9, L=32, n_kv=8, d_h=128, b=2,
                  mem=80e9, bw=3.35e12, flops=989e12, u=0.5, eta=0.8, t0=1e-3):
    """Back-of-envelope plan for one model replica (formulas 238.1 to 238.6)."""
    M_W = N * b                                   # weight bytes
    s_kv = 2 * L * n_kv * d_h * b                 # KV bytes per token (238.2)
    L_bar = n_in + n_out / 2                      # mean context while decoding
    C_req = 2 * N * (n_in * (1 - hit) + n_out)    # FLOPs per request (238.5)
    c = lam * C_req / (u * flops)                 # compute utilization
    kv_read = lam * n_out * L_bar * s_kv / (eta * bw)
    if c >= 1 or kv_read >= 1:
        return None                               # this replica is overloaded
    floor = t0 + M_W / (eta * bw)                 # read all weights once
    tpot = max(floor / (1 - kv_read), t0 / (1 - c))          # (238.4)
    ttft = (t0 + 2 * N * n_in * (1 - hit) / (u * flops)) / (1 - c)
    B = lam * n_out * tpot                        # Little's law (238.1)
    kv_bytes = B * (n_in + n_out) * s_kv          # (238.3)
    ok = tpot <= tpot_max and ttft <= ttft_max and kv_bytes <= 0.9 * mem - M_W
    return dict(ok=ok, c=c, tpot=tpot, ttft=ttft, B=B, kv_gb=kv_bytes / 1e9)

def replicas(lam, n_in, n_out, headroom=0.8, **kw):
    lo, hi = 0.0, 1000.0                          # bisection for mu* (238.6)
    for _ in range(60):
        mid = (lo + hi) / 2
        r = chat_capacity(mid, n_in, n_out, **kw)
        lo, hi = (mid, hi) if (r and r["ok"]) else (lo, mid)
    return lo, math.ceil(lam / (headroom * lo))

mu, R = replicas(20, 1000, 300)
print(f"one H100: mu* = {mu:.1f} req/s, replicas for 20 req/s: {R}")
r = chat_capacity(20 / R, 1000, 300)
print({k: round(v, 3) for k, v in r.items() if k != "ok"})
mu, R = replicas(20, 1000, 300, hit=0.5)
print(f"with 50% prefix-cache hits: mu* = {mu:.1f}, replicas: {R}")
# one H100: mu* = 22.9 req/s, replicas for 20 req/s: 2
# {'c': 0.422, 'tpot': 0.008, 'ttft': 0.058, 'B': 25.236, 'kv_gb': 4.3}
# with 50% prefix-cache hits: mu* = 37.2, replicas: 1

The second is the two-stage search of Drill 2 in PyTorch: a coarse scan over a 4× smaller INT8 index, then an exact re-rank of the top 100. The “ground truth” here is what an exact FP32 scan would return.

import torch
import torch.nn.functional as F

torch.manual_seed(0)
n, d, q = 50_000, 256, 200
db = F.normalize(torch.randn(n, d), dim=1)            # segment embeddings (fp32)
target = torch.randint(0, n, (q,))
queries = F.normalize(db[target] + 1.5 * torch.randn(q, d) / d**0.5, dim=1)

# exact search: the ground truth for "what the fp32 index would return"
exact = (queries @ db.T).topk(10, dim=1).indices

# stage 1: int8 index (4x smaller than fp32), coarse scores, keep top 100
scale = db.abs().amax(dim=1, keepdim=True) / 127
db_q = torch.round(db / scale).to(torch.int8)         # what we store
coarse = queries @ (db_q.float() * scale).T
cand = coarse.topk(100, dim=1).indices                 # (q, 100)

# stage 2: rerank the 100 candidates with exact fp32 vectors
fine = torch.einsum("qd,qkd->qk", queries, db[cand])
top10 = cand.gather(1, fine.topk(10, dim=1).indices)

def recall(a, b):  # share of the exact top-10 that we also returned
    return sum(len(set(x.tolist()) & set(y.tolist())) for x, y in zip(a, b)) / a.numel()

print("index MB fp32 / int8:", db.numel() * 4 / 1e6, db_q.numel() / 1e6)
print("recall@10, int8 only:", recall(exact, coarse.topk(10, dim=1).indices))
print("recall@10, int8 + rerank:", recall(exact, top10))
# index MB fp32 / int8: 51.2 12.8
# recall@10, int8 only: 0.984
# recall@10, int8 + rerank: 1.0

The third is the streaming detector of Drill 3: a windowed mean (a 1-D convolution), a threshold, and a refractory time. It contains one real moment at second 200 and one look-alike at second 400.

import torch
import torch.nn.functional as F

torch.manual_seed(0)
T, sigma = 600, 0.3                       # 10 minutes of per-second scores
score = sigma * torch.randn(T)
score[200:208] += 1.0                     # a real key moment (8 s)
score[400:405] += 0.8                     # a replay that looks similar (5 s)

def alerts(score, w, theta, refractory=30):
    kernel = torch.full((1, 1, w), 1.0 / w)
    mean = F.conv1d(score.view(1, 1, -1), kernel).view(-1)  # mean[i] covers i..i+w-1
    out, last = [], -10**9
    for i, m in enumerate(mean.tolist()):
        t = i + w                         # time when this window is complete
        if m > theta and t - last >= refractory:
            out.append(t)
            last = t
    return out

for w, theta in [(1, 0.7), (4, 0.7), (8, 0.6)]:
    print(w, theta, alerts(score, w, theta))
# 1 0.7 [60, 201, 338, 394, 453, 539]   noise alarms everywhere
# 4 0.7 [203, 405]                      the replay fires too
# 8 0.6 [205]                           only the real moment, 5 s after it starts

Note the last line: the alert comes at second 205, five seconds after the moment starts. Formula (238.9) predicts \(\theta w / A = 0.6 \times 8 / 1 = 4.8\) s of evidence delay, and the windows are checked once per second, so the first window that passes is the one ending at 205.

Where this is used

SettingWhat the frame gives youSession to go deeper
ML system design interviews (LLM, search, ads, video)A structure that keeps you on time and shows numbers earlythis session
Capacity planning for a chat or agent productGPUs per SLO, the binding limit, the cost of long promptssession 141
Design reviews and launch documentsRequirements, numbers, risks and fallbacks in one pagesession 237
Video retrieval and Video-RAG productsIndex size, backfill cost, re-ranker budgetsession 208, session 210
Live sports and broadcast monitoringDelay–false-alarm trade-off, cascades, stream failure handlingsession 207, session 211
Content safety and copyright on video platformsThe same cascade: cheap always-on detector, costly verifiersession 209
VLM serving on edge or small GPUsFloors (TPOT floor, memory) decide feasibility before anything elsesession 199

Common mistakes

  • Drawing boxes before numbers. The design for 1 request per second and for 1,000 is not the same. Do the numbers step first, even if roughly.
  • Using average load. Systems fail at peak. Plan for the peak and for p95 request sizes, and keep headroom.
  • Forgetting the prompt. Many candidates size an LLM service by output tokens only. In Drill 1, prompts are 77% of the compute.
  • Ignoring floors. No batch size makes TPOT lower than the time to read the weights once. If the floor is above the target, the hardware or the model must change.
  • “Accuracy” as the only quality number. For search it is recall@k by query type; for alerts it is false alerts per match and delay; for chat it is a judge score plus user signals.
  • No failure modes, or only “the GPU dies”. The interesting failures are data failures: replays, new vocabulary, drift, duplicates.
  • One big model for everything. Most real systems are cascades: cheap and wide first, expensive and narrow last.
  • Invented precision. “The verifier takes 1.37 s” sounds false. Give ranges and say how you would measure.

Interview questions

What is Little’s law and why is it useful for LLM serving?basic
In a stable system, the average number of items inside equals the arrival rate times the average time in the system: \(B = \lambda W\) (238.1). It needs no assumption about the arrival distribution. For LLM serving it turns product numbers (requests per second, latency) into the number of concurrent requests, which sets the batch size and the KV-cache memory. Example: 20 req/s, 300 output tokens at 10.6 ms each gives about 63 requests decoding at the same time.
Compute the KV cache per token for Llama 3.1 8B in BF16. How much for a 1,300-token request?basic
\(2 \times 32 \text{ layers} \times 8 \text{ KV heads} \times 128 \times 2 \text{ bytes} = 131{,}072\) bytes = 128 KiB per token (238.2). For 1,300 tokens: about 170 MB. Without GQA (32 KV heads) it would be 4× more, which is why GQA matters so much for serving.
Your chat service misses its TTFT target but has a lot of TPOT slack. What do you do?core
TTFT is set by prefill compute and queueing; TPOT slack means decode is not the problem. Cut prefill work: prefix caching (with a prefix-aware router so hits stay high), shorter system prompts, chunked prefill so that long prompts do not block others, FP8 compute, more replicas. At scale, disaggregate prefill and decode so prefill GPUs can be added alone. Do not reach for speculative decoding: it improves TPOT at the cost of extra compute.
Why can no batch size make an 8B BF16 model reach 50 ms TPOT on an L4?core
Each decode step must read all 16.06 GB of weights. At 80% of 300 GB/s that takes 67 ms, plus overhead: the TPOT floor is about 67.9 ms. Batching shares that read across more requests, so it raises throughput, but it cannot make one step faster than one read of the weights. Options: quantize weights (INT4 is about 4 GB, so a floor of about 18 ms), a smaller model, or a GPU with more bandwidth.
Size the index for natural-language search over 10,000 hours of video.core
State the segment length and vector size. With 4-second segments, \(3600 \times 10{,}000 / 4 = 9\)M vectors (238.7). At 512-d: FP32 18.4 GB, FP16 9.2 GB, INT8 4.6 GB, PQ 64-byte codes 0.58 GB. All fit on one server, so the index is not the hard part; recall per query type and the cost of re-ranking are.
Why use a two-stage (retrieve then re-rank) design for video search?core
A dual encoder (text vector against pre-computed segment vectors) can score millions of segments in milliseconds, but it compresses each segment into one vector and misses fine details such as “from a corner” or “last minute”. A cross-encoder or VLM that sees query and frames together is much more accurate but costs about 17 TFLOP per candidate for an 8B VLM in our example, so it can only look at tens of candidates. Retrieve wide and cheap, then re-rank narrow and expensive. Add keyword search on commentary for names.
Explain the trade-off between alert delay and false alarms in live key-moment detection.core
The detector needs evidence. A window of \(w\) seconds and threshold \(\theta\) add about \(\theta w / A\) seconds of delay (238.9), linear in \(w\). The noise false-alarm rate falls like a Gaussian tail in \(\theta\sqrt{w}/\sigma\) (238.10), much faster. So a few seconds of delay buys a huge drop in noise alarms. Look-alikes (replays, near misses) are not noise and need a different fix: duration, a replay detector, or a verifier model. If the window becomes longer than the event, real events get missed.
How would you evaluate the live alert system before and after launch?advanced
Before: replay recorded full matches through the real streaming code (not offline clips) and measure recall per event type, false alerts per match and the delay distribution (median and p95) against human-labeled event times; check by competition and broadcaster. Tune thresholds on some matches and report on others. After: shadow mode (alerts go to editors only), then a canary. Track editor rejections, corrections, verifier rejection rate and stream lag as drift signals. The metrics must match the product promise: “no false goal alerts” is per match, not per frame.
One of your two chat replicas fails at peak. What happens and how should you have planned?advanced
The other replica gets all 20 req/s. One H100 can meet the targets up to about 22.9 req/s in our model, so it survives but at about 87% of its capacity with no headroom: TTFT rises from about 58 ms toward about 215 ms and any burst breaks the SLO. In-flight streams of the failed replica are lost; clients must retry. If the SLO must hold during one failure, plan N+1 (here 3 replicas), or allow graceful degradation: admission control for low tiers, a shorter max output, or a smaller fallback model.
The interviewer says: “now add image input to the chat API”. What changes in your numbers?advanced
An image becomes visual tokens (hundreds to thousands per image, depending on the encoder and resolution), and those are prompt tokens: they add prefill FLOPs (\(2N\) per token in the LLM) and KV-cache memory exactly like text tokens, plus the vision encoder’s own cost. So TTFT and compute grow, and the cache hit rate may drop (images are rarely repeated, but the same image is reused across turns, so cache it). Options: fewer tokens per image (pooling, lower resolution), a separate encoder pool, and caching the image embeddings and their KV across turns.
How do you choose between more replicas and a quantized model to cut cost?advanced
First find the binding limit. If it is compute (prefill), FP8 weight and activation quantization raises usable FLOP/s; weight-only INT4 helps bandwidth (TPOT floor) more than compute. If it is memory bandwidth (TPOT floor near target), weight quantization helps directly. Then check quality on your own evaluation set, because quantization errors are task-dependent. More replicas always work but cost linearly; quantization is a one-time engineering cost with a quality risk. Often you do both: quantize, then size replicas with the new numbers.

Exercises

  1. Drill 1 with Llama 3.1 70B in BF16: compute the weight bytes, the KV bytes per token, and the TPOT floor on a group of 4 H100s treated as one GPU (\(\eta = 0.8\), \(t_0 = 1\) ms).
  2. In Drill 1 on one H100 at 20 req/s, what share of the request FLOPs is prefill? What does it become with a prefix-cache hit rate of 0.5?
  3. Drill 2: the archive grows to 100,000 hours with 2-second segments and 768-d vectors. What is the FP16 index size? Does it fit on one H100?
  4. Drill 3: with \(\sigma = 0.3\), \(w = 4\) and \(\theta = 0.5\), compute \(z\) and the expected noise alarms in a 90-minute match from (238.10). Then compare the evidence delay with \(w = 4, \theta = 0.7\).
  5. In Figure 4, find the setting with no false alarms and all 10 moments detected that has the smallest median delay. Explain with formulas (238.9) and (238.10) why it sits where it does.
  6. In Figure 2, choose “8B on 1×L4” and find the smallest TPOT target for which a plan exists. Why does it not depend on the load?
Answerscheck
1. Weights \(70.6\times10^9 \times 2 = 141.2\) GB. KV \(2 \times 80 \times 8 \times 128 \times 2 = 327{,}680\) bytes = 320 KiB per token. Floor \(= 1 + 141.2 / (0.8 \times 13.4) = 14.2\) ms. 2. Prefill \(= 1{,}000 / 1{,}300 = 77\%\). With \(h = 0.5\): \(500 / 800 = 62.5\%\), and the total FLOPs per request drop from 20.9 to 12.8 TFLOP. 3. \(3600 \times 100{,}000 / 2 = 1.8\times10^8\) vectors \(\times 768 \times 2 = 276\) GB: no, it does not fit on one H100 (80 GB); use INT8 (138 GB, still too big), PQ codes (11.5 GB at 64 bytes), or a CPU or sharded index. 4. \(z = 0.5 \times 2 / 0.3 = 3.33\), \(Q(3.33) \approx 4.3\times10^{-4}\), so about 2.3 noise alarms per match (an upper bound: close crossings merge into one alert). The evidence delay is \(0.5 \times 4 = 2.0\) s against 2.8 s at \(\theta = 0.7\). 5. In this simulation two settings tie with a median delay of 5.0 s and no false alarms: \(w = 6\), \(\theta = 0.75\) and \(w = 8\), \(\theta = 0.6\). Both are just above what a look-alike can reach: a 5-second look-alike averages at most \(0.7 \times 5/6 = 0.58\) over 6 s and \(0.7 \times 5/8 = 0.44\) over 8 s. Their evidence delays, \(0.75 \times 6 = 4.5\) s and \(0.6 \times 8 = 4.8\) s, both round up to the check at 5 s. Shorter windows need a higher \(\theta\) to stay above the look-alikes, so the delay \(\theta w\) cannot drop much; noise (238.10) is already negligible here. 6. About 70 ms (the floor is 67.9 ms, and the slider moves in 5 ms steps). The floor is the time to read all weights once, which every decode step needs whatever the batch size, so no load (not even one request) can go below it.

Key takeaways

  • Use one frame every time: requirements, numbers, design, bottleneck, failure modes. Numbers come before boxes.
  • A few formulas cover most questions: Little’s law (238.1), KV bytes (238.2), the TPOT floor and step time (238.4), compute utilization (238.5), replicas with headroom (238.6).
  • Drill 1: for chat, prompts are most of the compute, so prefix caching and chunked prefill matter more than faster decoding; 20 req/s of Llama 3.1 8B needs about 2 H100s with our assumptions.
  • Drill 2: a 10,000-hour archive is only 9 M segments and a few GB of index; the hard parts are recall on sports language and the cost of re-ranking.
  • Drill 3: delay grows linearly with the evidence window, noise alarms fall like a Gaussian tail; look-alikes need a cascade with a verifier, not a higher threshold.
  • Name the binding limit and the failure modes before you are asked, and answer follow-ups by changing the numbers, not the habits.

Key formulas

\[ B = \lambda W, \qquad s_{\text{kv}} = 2 L\, n_{\text{kv}}\, d_h\, b, \qquad \text{TPOT}_{\min} = t_0 + \frac{M_W}{\eta\,\text{BW}} \]
\[ c = \frac{\lambda \cdot 2N\left((1-h)n_{\text{in}} + n_{\text{out}}\right)}{uF}, \qquad R = \left\lceil \frac{\lambda}{H \mu^{*}} \right\rceil \]
\[ S_{\text{index}} = \frac{3600 H_v}{\Delta}\, d\, b_v, \qquad D_{\text{alert}} \approx \frac{\theta w}{A} + \Delta_{\text{hop}} + t_{\text{model}}, \qquad \E[\text{alarms}] \lesssim T\, Q\!\left(\frac{\theta\sqrt{w}}{\sigma}\right) \]

References for this session

  1. Austin et al., Google DeepMind (2025). How to Scale Your Model
  2. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  3. Austin et al., Google DeepMind (2025). How to Scale Your Model
  4. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  5. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  6. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  7. Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
  8. Llama Team, Meta (2024). The Llama 3 Herd of Models
  9. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  10. Johnson et al. (2017). Billion-scale similarity search with GPUs
  11. Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
  12. Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
  13. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  14. Zhang et al. (2022). ActionFormer: Localizing Moments of Actions with Transformers