LLM & VLM courseBlog
239Track 10: Capstone and interviews · session 6 of 6

Final review and what to read next

This last session puts the whole course in one place. We review its seven parts with one hard question each, and we solve each question with real numbers. Then we look at four open problems, build a six-month reading plan, and set up simple habits to keep up with a field that changes every week.

Time: 3 to 4 hours (reading 2 hours, exercises and planning 1 to 2 hours)Level: Advanced · course reviewBefore this: The rest of the course, or at least the big picture session and the track you care about most

What you will learn

  • The course as seven parts, and one hard interview question per part, solved with real numbers (parameters, KV cache, 6ND, GRPO advantages, the decode bound, RAG recall, visual tokens)
  • Why long video is still hard: count the tokens and the KV cache for a 90-minute match
  • Why “think longer” is not free: what majority voting can and cannot fix, with the exact binomial formula
  • What unified (any-to-any) models try to do, and where the tension between understanding and generation comes from
  • Why evaluation is an open problem: benchmark noise, contamination and judges, with the standard-error formula
  • A six-month reading plan in three versions (systems, post-training, video and multimodal) with a build task each month
  • How to keep up: surveys, conferences, arXiv listings and reading real code

Why this matters

You have reached the last session. The course had 238 sessions before this one. That is a lot of ideas, and many of them now sit in your head as separate facts. This session ties them together and points you forward.

We do three things. First, we review the course as seven parts. For each part we ask one hard question, the kind a senior interviewer asks, and we solve it with real numbers. If you can answer these seven questions without notes, you understand the core of the course. Second, we look at four open problems: long video, efficient reasoning, unified models and evaluation. These are the places where the field is still moving, and where your next job or project will likely be. Third, we build a reading plan for six months and a set of simple habits to keep up.

Why does this matter for interviews? A strong candidate does not list facts. A strong candidate connects a question to the right formula, puts real numbers in, and says what the numbers mean for the design. The previous session, session 238, gave you an answer frame: requirements, numbers, design, bottleneck, failure modes. Here we practice the “numbers” step for every part of the course, because that is the step most people skip.

The course in one picture: seven parts

The course has nine content tracks plus this capstone track. For the review we group them into seven parts. Each part answers one big question about a model’s life: how it computes, how it learns, how it is aligned, how it runs fast, how it is used, how it sees images, and how it handles video and generation.

Figure 1. The seven parts of the course. Click a part (or press Play) to see its sessions, its one core idea, its hard question, and the key number of the answer. The answers are worked out in the next section.
How to read Figure 1

Each box is one part with its session range. The grey panel shows the core idea in one sentence, then the hard question, then the numbers you should be able to produce. The line at the bottom counts how many sessions the part covers. Try to answer the question yourself before you read the worked answer below.

We use one method for every hard question. It is the same frame as in the interview drills, cut down to the parts that matter for a single question.

Algorithm 1: answer a hard question with numbers
  1. input: a question about some part of the LLM or VLM stack
  2. name the part of the stack (compute, training, alignment, serving, application, vision, video)
  3. write the one formula for that part // e.g. KV bytes, 6ND, roofline bound
  4. state your assumptions out loud // model size, precision, hardware, context length
  5. put numbers in and round to one or two digits
  6. say what the number means for the design // “this does not fit”, “this is memory-bound”
  7. name one way it can fail, and one fix
  8. output: a number, a design decision, and a risk

Seven parts, seven hard questions

We use one model through most of the answers: Llama 3 8B. Its config file says: hidden size \(d = 4096\), \(L = 32\) layers, 32 query heads, 8 key-value heads (so head size \(d_h = 128\)), MLP width 14,336, vocabulary \(|V| = 128{,}256\), and the input and output embeddings are not tied [Certain: from the model’s config.json].

Part 1. Foundations and transformers (sessions 1 to 30)

Core idea. A decoder-only transformer predicts the next token. Tokens become vectors, pass through \(L\) blocks of attention and MLP on a residual stream, and become scores over the vocabulary.

Hard question: “From the config alone, how many parameters does Llama 3 8B have, and how many bytes of KV cache does it need per token?”

Count the weight matrices in one block. Attention has \(W_Q\) and \(W_O\) of size \(d \times d\), and \(W_K\), \(W_V\) of size \(d \times H_{kv} d_h\) because of grouped-query attention (GQA). The MLP is SwiGLU with three matrices of size \(d \times d_{ff}\). Then add the two embedding tables. For the cache, each layer stores one key and one value vector per KV head:

\[ N \approx \underbrace{2|V|d}_{\text{embeddings}} + L\Big(\underbrace{2d^2 + 2d\,H_{kv}d_h}_{\text{attention}} + \underbrace{3\,d\,d_{ff}}_{\text{MLP}}\Big) \tag{239.1} \]
\[ \text{KV bytes per token} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot b \tag{239.2} \]
Read the symbols
\(N\)
number of parameters (we ignore the tiny normalization weights in the formula)
\(|V|, d\)
vocabulary size and hidden width
\(L\)
number of transformer blocks (layers)
\(H_{kv}, d_h\)
number of key-value heads and the size of one head
\(d_{ff}\)
the inner width of the MLP
\(b\)
bytes per number: 2 for BF16, 1 for FP8
the first 2 in (239.2)
one key and one value
Worked example: Llama 3 8B

Embeddings: \(128{,}256 \times 4096 = 525{,}336{,}576\) per table, two tables (not tied).

Attention per block: \(2 \cdot 4096^2 + 2 \cdot 4096 \cdot 8 \cdot 128 = 41{,}943{,}040\). MLP per block: \(3 \cdot 4096 \cdot 14{,}336 = 176{,}160{,}768\). With the two norm vectors, one block has \(218{,}112{,}000\) parameters.

Total: \(2 \cdot 525{,}336{,}576 + 32 \cdot 218{,}112{,}000 + 4096 = 8{,}030{,}261{,}248 \approx 8.03\)B. The name “8B” is honest.

KV cache: \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes \(= 128\) KiB per token. So 8,192 tokens need 1 GiB, and a full 128K context needs 16 GiB for one sequence.

What it means: the MLP holds about 70% of the parameters (\(32 \times 176\)M out of 8.03B), but the KV cache, not the weights, often decides how many users fit on a GPU. Revisit session 25 and session 26 if any step felt new.

Part 2. Pretraining and scale (sessions 31 to 52)

Core idea. Loss falls smoothly as a power law in model size and data. Training cost is about six FLOPs per parameter per token.

Hard question: “Llama 3 8B was trained on more than 15T tokens. The Chinchilla rule says about 20 tokens per parameter. Was Meta wrong?”

\[ C \approx 6\,N\,D \tag{239.3} \]
Read the symbols
\(C\)
total training compute in FLOPs
\(N, D\)
parameters and training tokens
6
2 FLOPs per parameter per token in the forward pass, 4 in the backward pass

With \(N = 8\times10^9\) and \(D = 15\times10^{12}\): \(C = 7.2\times10^{23}\) FLOPs, and \(D/N = 1875\) tokens per parameter, about 94 times the Chinchilla ratio (Chinchilla would pick 160B tokens for 8B parameters). Meta was not wrong. Chinchilla minimizes training compute for a target loss. It ignores inference. A small model trained far past the compute-optimal point is a bit worse per training FLOP but much cheaper to serve for its whole life, and small models keep improving with more data [Likely: Meta reports that its 8B and 70B models kept improving well past the Chinchilla point]. To turn \(C\) into GPU-hours, divide by real throughput: at an assumed 40% utilization of an H100’s 989 dense BF16 TFLOP/s, \(7.2\times10^{23}\) FLOPs is about 506,000 GPU-hours. This is our estimate from the formula, not a reported number. See session 35 and session 51.

Part 3. Post-training and RL (sessions 53 to 77)

Core idea. SFT teaches the format; preference learning (RLHF, DPO) and RL with verifiable rewards teach what “better” means.

Hard question: “In GRPO, a group of 8 answers to one math prompt gets rewards 1, 0, 0, 1, 1, 0, 0, 0. What advantage does each answer get, and what happens if all 8 are correct?”

GRPO (from DeepSeekMath) needs no value network. It compares each answer with the other answers to the same prompt:

\[ A_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)} \tag{239.4} \]
Read the symbols
\(r_i\)
reward of answer \(i\) (here 1 = correct, 0 = wrong)
\(G\)
group size, the number of answers sampled for one prompt
\(A_i\)
advantage: how much better than the group this answer was

Mean \(= 3/8 = 0.375\). Standard deviation (population form) \(= 0.484\). Correct answers get \(A = (1 - 0.375)/0.484 = 1.29\); wrong ones get \(A = -0.775\). The advantages sum to zero. If all 8 are correct, every \(r_i - \text{mean} = 0\), so every advantage is 0 (implementations add a small \(\varepsilon\) to the std to avoid 0/0). That prompt gives no learning signal at all. This is why RLVR pipelines filter out prompts that are always solved or never solved: only prompts of medium difficulty teach. Some code uses the sample std (divide by \(G-1\)); then the values are 1.21 and −0.72. Dr. GRPO argues for removing the std division entirely (session 70).

Part 4. Inference and serving (sessions 78 to 142)

Core idea. Prefill is compute-bound; decode is memory-bound. Most inference tricks either move fewer bytes per token or do more useful work per byte.

Hard question: “What is the fastest possible decode speed for Llama 3 8B on one NVIDIA L4 at batch size 1, and what changes it?”

At batch 1, each new token must read all weights once, plus this sequence’s KV cache. The memory bandwidth sets a ceiling:

\[ \text{tokens/s} \;\le\; \frac{\text{bandwidth}}{\text{weight bytes} + \text{KV bytes}} \tag{239.5} \]
Worked example: L4, 300 GB/s

BF16 weights: \(8\times10^9 \times 2 = 16\) GB. Bound: \(300/16 = 18.75\) tokens/s. With an 8,192-token context the KV cache adds 1 GiB \(= 1.074\) GB per step: \(300/17.07 = 17.6\) tokens/s.

4-bit weights: \(4\) GB, bound \(300/4 = 75\) tokens/s. Same hardware, four times the ceiling, because decode is about bytes, not FLOPs.

What changes it: batching (the weights are read once for many sequences, so throughput grows until the KV cache or compute becomes the limit), quantization (fewer bytes), speculative decoding (several tokens verified per weight read), and GQA or MLA (fewer KV bytes). See session 82 and session 128.

Part 5. LLM applications (sessions 143 to 162)

Core idea. Real systems wrap the model with retrieval, tools and evaluation. Most failures are in the wrapper, not in the model.

Hard question: “Your RAG system answers 74% of questions correctly. Should you buy a bigger LLM or fix retrieval?”

Split the end-to-end accuracy by whether the right passage was retrieved:

\[ P(\text{correct}) = R@k \cdot P(\text{correct} \mid \text{hit}) + (1 - R@k) \cdot P(\text{correct} \mid \text{miss}) \tag{239.6} \]
Read the symbols
\(R@k\)
recall at \(k\): the fraction of questions where a passage with the answer is in the top \(k\)
hit / miss
the right passage was / was not retrieved

Example numbers (made up to show the method): \(R@5 = 0.80\), \(P(\text{correct}\mid\text{hit}) = 0.90\), \(P(\text{correct}\mid\text{miss}) = 0.10\). Then \(0.8 \cdot 0.9 + 0.2 \cdot 0.1 = 0.74\). A perfect reader (1.0 on hits) gives at most \(0.80 + 0.02 = 0.82\). Perfect retrieval with the same reader gives \(0.90\). In this example, retrieval is the bigger lever. You only know this if you log retrieval hits per question; that is the main lesson of session 153 and session 155.

Part 6. Vision and VLMs (sessions 163 to 200)

Core idea. A VLM is a vision encoder, a connector and an LLM. The image becomes a sequence of visual tokens that the LLM reads like text.

Hard question: “How many visual tokens does a \(1344 \times 896\) image become in a Qwen2-VL-style model, and why do you care?”

The ViT cuts the image into \(14\times14\) patches, and an MLP then merges each \(2\times2\) group of patches into one token [Certain: Qwen2-VL paper]. So one LLM token covers \(28\times28\) pixels:

\[ n_{\text{vis}} = \frac{H}{p\,m}\cdot\frac{W}{p\,m}, \qquad p = 14,\; m = 2 \tag{239.7} \]

For \(1344\times896\): \(48 \times 32 = 1536\) tokens. A \(224\times224\) image gives \(8\times8 = 64\) (the paper reports 66 with the two special tokens around the image). LLaVA-1.5 uses a fixed \(336\times336\) input with patch 14 and no merge: \(24^2 = 576\) tokens. Why you care: visual tokens are prompt tokens. They cost prefill compute and KV cache like text, so one high-resolution image can cost more than a page of text. See session 180 and session 183.

Part 7. Video, multimodal and generative models (sessions 201 to 233)

Core idea. Video adds time: more tokens, plus the need to find when something happens. Generative models (diffusion, flow) learn to turn noise into data, and native multimodal models try to read and write all modalities in one network.

Hard question: “Why can’t you feed a whole 90-minute football match to a VLM and ask for the goals?”

\[ n_{\text{video}} = \text{seconds} \times \text{fps} \times \text{tokens per frame} \tag{239.8} \]

At 1 frame per second and 256 tokens per frame: \(5400 \times 1 \times 256 = 1{,}382{,}400\) tokens, which is 10.5 times a 128K context. With an LLM the size of Llama 3 8B, the KV cache alone would be \(1{,}382{,}400 \times 128\) KiB \(= 168.75\) GiB. And 1 fps can still miss a fast event. This is why real systems sample frames, compress tokens, retrieve clips first, or use agents (session 206, session 235). It is also our first open problem.

Open problems

An open problem is a question where nobody has a clean, accepted answer yet. The four below come up in research papers, in product teams, and in senior interviews. For each we give the numbers that make it hard, what people try today, and what is still missing. Statements about the future are tagged [Speculation].

Open problem 1: long video

Formula (239.8) is the whole problem in one line. Every term you lower costs something. Fewer frames per second miss short events (a goal kick lasts a second). Fewer tokens per frame lose small objects (the ball, a jersey number). And a longer context makes attention and the KV cache grow. Figure 2 lets you feel the budget.

Tokens compared with a 128K context window
KV cache (Llama-3-8B-sized LLM, BF16) compared with the 64 GB left on an 80 GB GPU after 16 GB of weights
Figure 2. The token budget of long video, formula (239.8). The bars turn orange when the video does not fit. The KV cache uses formula (239.2): 128 KiB per token.
How to read Figure 2

Move the sliders and watch the two bars. The first bar is the number of visual tokens divided by 131,072 (a 128K window). The second bar is the KV cache in GiB against 64 GB (59.6 GiB) of free GPU memory. Try a 90-minute match at 1 fps with 256 tokens per frame (1,382,400 tokens). Then try 0.5 fps with 64 tokens per frame: 172,800 tokens and 21.1 GiB. It fits in memory, but each frame now has only 64 tokens.

What people try today [Certain that these lines of work exist; see session 206]:

  • Compress tokens: merge similar tokens across neighbouring frames, or drop tokens of frames that look the same as the last one (LongVU does adaptive spatio-temporal compression).
  • Extend the context: train the language model on long text, then transfer to video (LongVA).
  • Retrieve first: cut the video into clips, embed them, retrieve only the clips that match the question, then let the VLM read those (Video-RAG).
  • Agents: let an LLM decide which part of the video to look at next (VideoAgent).
  • Streaming: keep a memory and answer while the video plays (VideoLLM-online).

What is missing: a model that keeps a compact memory of hours of video, finds a two-second event in it, and gives the timestamp, at a cost close to a text query. Benchmarks for hour-long video are also young (Video-MME has a long-video split with videos up to about an hour [Likely]). [Speculation] The answer will probably mix learned memory with retrieval, not one giant context.

Open problem 2: efficient reasoning

Reasoning models (o1-style, DeepSeek-R1) get better answers by writing long chains of thought before they answer. That is test-time compute: you pay for more tokens at inference to get more accuracy. Long thinking has a direct price: by (239.5), every token of thought is one more memory-bound decode step, and the KV cache grows with every token.

The simplest test-time method is self-consistency: sample \(k\) answers and take a majority vote. Assume the answers are independent, each correct with probability \(p\), and every wrong answer is the same wrong answer (the worst case for voting). For odd \(k\):

\[ P_{\text{maj}}(p, k) = \sum_{j = (k+1)/2}^{k} \binom{k}{j}\, p^{j}\,(1-p)^{k-j} \tag{239.9} \]
Read the symbols
\(p\)
chance that one sampled answer is correct
\(k\)
number of samples (odd, so there is no tie)
\(j\)
number of correct samples; the vote is right when \(j\) is more than half of \(k\)
\(\binom{k}{j}\)
the number of ways to choose which \(j\) samples are correct

Compare it with pass@k, \(1 - (1-p)^k\): the chance that at least one sample is correct. Pass@k only helps if you have a verifier (unit tests, a math checker) that can pick the correct one. Figure 3 shows both.

Figure 3. Majority vote (formula 239.9, solid) and pass@k (dashed) against the number of samples \(k\). Cost grows linearly with \(k\). Accuracy does not.
How to read Figure 3

The x-axis is \(k\), the y-axis is accuracy. The dot marks your chosen \(k\). With \(p = 0.6\), voting goes from 0.600 at \(k=1\) to 0.683 at \(k=5\) and 0.787 at \(k=15\): you pay 15 times the tokens for 19 more points. Now set \(p = 0.4\): the vote gets worse as \(k\) grows (0.317 at \(k=5\)), because voting makes the most common answer win, and here the common answer is wrong. Pass@k keeps rising, but you need a verifier to use it.

Lessons: sampling more is a weak lever unless the model is already right more often than wrong, and a verifier is worth a lot. The open questions are how to get reasoning quality with fewer tokens: shorter chains of thought, deciding when to stop thinking, thinking in hidden vectors instead of text, and distilling long reasoning into small models. Snell et al. found that spending test-time compute adaptively per prompt can beat a fixed budget [Certain: their paper’s main claim]. [Speculation] Expect “reasoning budget” to become a normal serving parameter, like max tokens. See session 72.

Open problem 3: unified models

Today most systems are pipelines: a VLM to understand images, a separate diffusion model to make images, a separate speech model. A unified (any-to-any) model reads and writes text, images, audio and video with one network. The hard part is that understanding and generation want different things from the visual representation: understanding wants semantic features (what is in the image), generation wants pixel-level detail (exact textures and positions).

ModelHow images enter and leaveThe trade-off it accepts
ChameleonImages become discrete tokens from a VQ tokenizer; one transformer predicts all tokens (early fusion)One loss and one model, but discrete tokens lose detail, and mixed-modal training needed extra stability tricks
Emu3Images and video as discrete tokens, trained only with next-token predictionVery simple recipe; generation needs many tokens per image
TransfusionText with next-token loss, image patches as continuous vectors with a diffusion loss, in one transformerKeeps continuous detail; two losses must be balanced
JanusTwo separate visual encoders: a semantic one (SigLIP) for understanding and a VQ tokenizer for generation, one shared transformerRemoves the conflict in the encoder; the model has two visual “languages”

Open questions: does training on generation make understanding better, or worse? Which tokenizer serves both? How do you evaluate one model on so many tasks? [Speculation] The field may settle on a shared backbone with task-specific visual encoders and decoders, as Janus suggests, but this is not decided. See session 232.

Open problem 4: evaluation

Evaluation is the problem under all other problems: if you cannot measure progress, you cannot make it. Four things make it hard.

  • Noise. A benchmark is a sample of questions. Its score has random error.
  • Contamination. Test questions leak into web-scale training data, so a high score may be memory.
  • Saturation. Good models score near the top, so the benchmark stops separating them.
  • Judges. For open-ended answers we use humans (Chatbot Arena) or LLM judges, and LLM judges have known biases, for example toward the first answer shown or toward longer answers [Certain: reported in the MT-Bench paper].

Noise is the easiest to compute and the most often ignored. With \(n\) independent questions and true accuracy \(p\), the standard error of the measured accuracy and the 95% interval are:

\[ \operatorname{SE} = \sqrt{\frac{p(1-p)}{n}}, \qquad \text{95\% interval} \approx \hat{p} \pm 1.96\,\operatorname{SE} \tag{239.10} \]
Read the symbols
\(n\)
number of test questions
\(p, \hat{p}\)
true accuracy and measured accuracy
1.96
the normal-distribution value that leaves 2.5% in each tail

On 500 questions at 70% accuracy, \(\operatorname{SE} = 2.05\) points and the interval is \(\pm 4.0\) points. Two models scored separately (unpaired) need a gap of about \(1.96\sqrt{2}\cdot\operatorname{SE} = 5.7\) points before you can trust the order. To see a 2-point gap you need about 4,000 questions per model. When both models answer the same questions, a paired test (look only at the questions where they disagree) is much tighter, so always keep per-question results.

Figure 4. Benchmark noise, formula (239.10). Each bar is a 95% interval around a measured accuracy. The axis shows a 30-point window around model A. If the intervals of A and B overlap a lot, the benchmark cannot tell the two models apart.
How to read Figure 4

Model A is the upper bar, model B (A plus the gap) the lower bar. The readout gives the standard error, the half-width of each interval, and the smallest gap an unpaired comparison can detect, \(1.96\sqrt{2}\cdot\operatorname{SE}\). Start at 500 questions and a 3-point gap: not enough. Move \(n\) to the right until the verdict changes. Then drop \(n\) to 100, the size of many small video or VLM test sets, and see how wide the bars become.

For video the problem is worse: test sets are small because videos are expensive to label, many questions can be answered from one frame (so they do not test time), and temporal answers need special metrics such as IoU of time spans (session 213). [Speculation] Expect more private, refreshed test sets and more “in-the-loop” evaluation on real product traffic.

Your reading list for the next six months

Reading papers is a skill. Most papers are not meant to be read from start to end. A good method is the three-pass approach from S. Keshav’s short note “How to Read a Paper”:

Algorithm 2: read a paper in three passes
  1. input: one paper, a question you want it to answer
  2. pass 1 (10 minutes): read title, abstract, introduction, section headings, conclusion; look at the figures // decide: is this worth more time?
  3. pass 2 (1 hour): read the whole paper but skip proofs; write down the main equation, the main table, and what is compared to what
  4. check the baselines: same data? same compute? same evaluation? // most “wins” hide here
  5. pass 3 (several hours, only for key papers): re-derive the main equation, or run the code, or re-implement the core in under 100 lines
  6. write five lines: problem, idea, key number, weakness, how it connects to the course
  7. output: a short note you can find again in six months

Figure 5 gives three six-month plans. Each month has three readings and one thing to build. All readings are real papers or books from the reference list of this course; the arXiv number is shown when we have it. Pick the plan that matches the job you want. About five hours a week is enough: two readings in pass 2, one in pass 3, and the build.

MonthReadBuild
Figure 5. Three six-month reading plans. Each one starts with a survey or a foundation paper, goes deep in the middle, and ends at the open problems from this session.
How to read Figure 5

Choose a plan with the buttons. Each row is one month. Read the papers in order: later ones assume the earlier ones. The “Build” column is a small project that forces you to use what you read; most of them match a lab in this course. The readout counts the readings and names the open problem the plan ends at.

Four surveys are good maps for any plan. Read their table of contents first and use them to find papers, not to learn details:

SurveyUse it for
Zhao et al., A Survey of Large Language Models (arXiv 2303.18223)The whole LLM pipeline: pretraining, adaptation, use, evaluation. Updated many times.
Yin et al., A Survey on Multimodal Large Language Models (arXiv 2306.13549)VLM architectures, training data and instruction tuning.
Tang et al., Video Understanding with Large Language Models: A SurveyVideo-LLMs, sorted by how the video is fed to the LLM.
Jin et al., Efficient Multimodal Large Language Models: A SurveySmall VLMs, visual token compression, efficient vision encoders.

How to keep up: surveys, conferences, code

The field publishes far more than anyone can read. The goal is not to read everything. The goal is a small, steady routine that catches the important ideas a few weeks after they appear.

Algorithm 3: a weekly routine (about five hours)
  1. Monday, 30 min: scan new titles in arXiv cs.CL, cs.CV and cs.LG (or a curated digest); save at most 5 papers
  2. Tuesday to Thursday, 3 h: pass 1 on the 5 saved papers; pass 2 on the best 2
  3. Friday, 1.5 h: read code: one pull request or one file of a major repo (vLLM, SGLang, TRL, transformers, torchtitan)
  4. once a month: pass 3 on one paper, and one small build
  5. once a quarter: skim the accepted-paper list and orals of the latest big conference
  6. keep one notes file per topic; add the five-line summary from Algorithm 2 // your future interview notes

The main conferences, and roughly when they happen [Likely: months move a little each year]:

VenueFocusUsually held
ICLRRepresentation learning, LLM training and theoryApril to May
CVPRComputer vision, VLMs, videoJune
ICMLGeneral machine learningJuly
ACL / EMNLP / NAACLNatural language processing, LLM evaluation and applicationsSummer (ACL), late autumn (EMNLP)
ICCV (odd years) / ECCV (even years)Computer vision, videoAutumn
COLMLanguage modeling onlyAutumn
NeurIPSGeneral machine learning, many datasets and benchmarksDecember
MLSysSystems for ML: serving, kernels, compilersSpring

Three more sources matter as much as papers. Technical reports of open models (Llama 3, DeepSeek-V3, Qwen3, Gemma 3) often hold more practical detail than conference papers. Code is the ground truth: a config file and a forward function tell you exactly what a model does, as you saw in session 234. And courses and books such as Stanford CS336, “How to Scale Your Model”, and Lambert’s RLHF book are updated more slowly but explain more carefully.

Code: check your review numbers

One short script reproduces every number in the review. Run it, then change the config to another model (for example a Qwen or Gemma config.json) and redo the answers. It needs only PyTorch and runs in under a second on a CPU.

import math
import torch

# Part 1: parameters and KV cache from a config (Llama 3 8B values)
cfg = dict(d=4096, L=32, H=32, Hkv=8, ff=14336, V=128256, tied=False)
dh = cfg["d"] // cfg["H"]
attn = 2 * cfg["d"] * cfg["d"] + 2 * cfg["d"] * cfg["Hkv"] * dh   # Wq, Wo, Wk, Wv
mlp = 3 * cfg["d"] * cfg["ff"]                                   # gate, up, down
block = attn + mlp + 2 * cfg["d"]                                # + two RMSNorm weights
emb = cfg["V"] * cfg["d"] * (1 if cfg["tied"] else 2)            # input table + LM head
N = emb + cfg["L"] * block + cfg["d"]                            # + final norm
kv_per_tok = 2 * cfg["L"] * cfg["Hkv"] * dh * 2                  # K and V, BF16
print(f"params {N/1e9:.2f}B, KV {kv_per_tok/1024:.0f} KiB/token")

# Part 2: training compute and the Chinchilla ratio
D = 15e12
print(f"C = {6*N*D:.2e} FLOPs, tokens per param = {D/N:.0f}")

# Part 3: GRPO group advantages, formula (239.4)
r = torch.tensor([1., 0, 0, 1, 1, 0, 0, 0])
A = (r - r.mean()) / r.std(unbiased=False)
print("advantages", [round(a, 3) for a in A.tolist()])

# Part 4: decode upper bound on an L4 (300 GB/s), batch 1, BF16
print(f"decode bound {300e9 / (2*N):.1f} tokens/s")

# Part 6: visual tokens, 14-px patches merged 2x2 (28 px per token)
print("visual tokens", (1344 // 28) * (896 // 28))

# Part 7: a 90-minute match at 1 fps, 256 tokens per frame
T = 90 * 60 * 1 * 256
print(f"video tokens {T:,}, KV {T*kv_per_tok/2**30:.1f} GiB")

# Open problem 2: majority vote with k independent samples, formula (239.9)
def majority(p, k):
    j = torch.arange(k // 2 + 1, k + 1)
    logc = torch.lgamma(torch.tensor(k + 1.)) - torch.lgamma(j + 1.) - torch.lgamma(k - j + 1.)
    return torch.exp(logc + j * math.log(p) + (k - j) * math.log(1 - p)).sum().item()
print("vote p=0.6:", [round(majority(0.6, k), 3) for k in (1, 5, 15)],
      "p=0.4:", [round(majority(0.4, k), 3) for k in (1, 5, 15)])

# Open problem 4: benchmark noise, 95% interval half-width, formula (239.10)
n, p = 500, 0.7
print(f"+/- {1.96*math.sqrt(p*(1-p)/n)*100:.1f} points on {n} questions")

Output (PyTorch 2.8, CPU):

params 8.03B, KV 128 KiB/token
C = 7.23e+23 FLOPs, tokens per param = 1868
advantages [1.291, -0.775, -0.775, 1.291, 1.291, -0.775, -0.775, -0.775]
decode bound 18.7 tokens/s
visual tokens 1536
video tokens 1,382,400, KV 168.8 GiB
vote p=0.6: [0.6, 0.683, 0.787] p=0.4: [0.4, 0.317, 0.213]
+/- 4.0 points on 500 questions

The script uses the exact \(N = 8.03\)B, so the compute is \(7.23\times10^{23}\), the ratio is 1868 and the decode bound is 18.7 tokens/s; in the text we rounded \(N\) to 8B (\(7.2\times10^{23}\), 1875, 18.75). The binomial in majority is computed with log-gamma so that large \(k\) does not overflow.

Where this is used

Each review question maps to a real decision in a real system. The table shows where.

Question typeFormulaReal decision it drivesVideo example
Parameters and KV cache(239.1), (239.2)Which GPU, how many users per GPU, max context to offerHow many concurrent video-QA sessions fit on one server
Training compute(239.3)Budget and schedule for pretraining or continued pretrainingCost of adapting a VLM to sports broadcasts
Group advantages(239.4)Which prompts to keep in an RLVR datasetRL on temporal grounding with an IoU reward
Decode bound(239.5)Quantize or not; batch size; whether to use speculative decodingReal-time captions for a live match
RAG decomposition(239.6)Spend effort on retrieval or on the reader modelVideo-RAG: is the right clip retrieved? Text-to-video retrieval recall
Visual tokens(239.7), (239.8)Input resolution, frame rate, token compressionKey-moment detection, action recognition on long footage
Majority vote and pass@k(239.9)How many samples to draw; whether a verifier is worth buildingVoting over several clip-level answers for a highlight
Benchmark noise(239.10)Test-set size; whether a new model really is betterIs a 2-point gain on a 300-clip video benchmark real? (No.)

Common mistakes

  • Treating the review as a list of facts. Interviewers ask “why” and “how much”. Practice each hard question out loud with numbers, not by rereading.
  • Forgetting the KV cache. People size a GPU by weights only. At long context or high batch, the cache is often larger than the weights (formula 239.2).
  • Using Chinchilla as a law for deployed models. It optimizes training compute only. Models meant to be served are trained far longer.
  • Thinking “more samples” always helps. Majority voting amplifies the most common answer, right or wrong (Figure 3 at \(p \lt 0.5\)).
  • Trusting small benchmark gaps. A 2-point gain on 500 questions is inside the noise (Figure 4). Report intervals and keep per-question results.
  • Counting images as “one input”. One high-resolution image can be 1,500 tokens; one minute of video can be tens of thousands.
  • Reading too many papers shallowly. Ten pass-1 reads teach less than one pass-3 read with a re-implementation.
  • Ignoring code. Papers leave out details (masking, normalization, the exact loss scaling). The repository is the real specification.

Interview questions

Walk me through what happens, end to end, when a user sends a prompt to a chat model.basic
The text is formatted with the chat template and tokenized. The server queues the request; the scheduler adds it to a running batch (continuous batching). Prefill runs all prompt tokens in parallel through the transformer and fills the KV cache; this is compute-bound and sets time to first token. Decode then produces one token per step: each step reads the weights and the KV cache, so it is memory-bound and sets the time per output token. Sampling (temperature, top-p) picks each token; tokens are streamed back and detokenized until a stop token or the length limit. Around this: safety filters, possibly retrieval or tools, and logging for evaluation.
How many parameters does Llama 3 8B have, and where are they?core
About 8.03B. Two embedding tables of \(128{,}256 \times 4096 = 525\)M each (input and output, not tied), so about 1.05B. Each of the 32 blocks has 41.9M attention parameters (GQA makes \(W_K, W_V\) only \(4096\times1024\)) and 176.2M MLP parameters (three SwiGLU matrices of \(4096\times14{,}336\)), 218.1M per block, 6.98B for all blocks. So the MLPs hold about 70% of all parameters.
A team wants to serve a 128K-context feature with an 8B model on 24 GB GPUs. What do you tell them?core
At 128 KiB per token, one full 128K sequence needs 16 GiB of KV cache in BF16, plus 16 GB of BF16 weights: it does not fit on 24 GB even for one user. Options: quantize weights to 4-bit (about 4 GB) and the KV cache to FP8 or INT4 (8 or 4 GiB), offload or page the cache, use a model with fewer KV heads or MLA, shard across GPUs, or cap context and use retrieval instead of full context. Also, decode speed falls as the cache grows, because every step reads it.
Why did Llama 3 train an 8B model on 15T tokens when Chinchilla suggests about 160B?core
Chinchilla finds the model size and data size that minimize loss for a fixed training compute. It does not count inference. A model that will serve billions of tokens is cheaper over its life if it is small, and small models keep improving when trained on more data, just more slowly per FLOP. So labs “overtrain” small models: 1,875 tokens per parameter here versus about 20 at the Chinchilla point. Training cost is \(6ND \approx 7.2\times10^{23}\) FLOPs.
In GRPO, what happens to prompts the model always solves? Why does that matter?core
All rewards in the group are equal, so every \(r_i - \text{mean} = 0\) and every advantage is zero (with an \(\varepsilon\) in the denominator). The prompt contributes no policy gradient, only cost. The same holds for prompts that are never solved. So RLVR datasets are filtered or re-sampled for medium difficulty, and curricula move harder prompts in as the model improves. It also means the effective batch can shrink a lot late in training.
Give the decode speed bound for an 8B model on an L4 and two ways to raise it.basic
At batch 1, each token reads all weights: 16 GB in BF16 at 300 GB/s gives at most about 19 tokens/s (about 17.6 with an 8K-token KV cache). Raise it with 4-bit weights (4 GB, bound 75 tokens/s), with speculative decoding (several tokens accepted per weight read), or raise throughput with batching, since one weight read then serves many sequences.
Your RAG system has 74% end-to-end accuracy. How do you decide what to improve?core
Log, per question, whether a gold passage was in the top \(k\) and whether the answer was correct. Then split accuracy as \(R@k \cdot P(\text{correct}\mid\text{hit}) + (1-R@k)\cdot P(\text{correct}\mid\text{miss})\). With \(R@5 = 0.8\), 0.9 on hits and 0.1 on misses, a perfect reader caps you at 0.82 while perfect retrieval gives 0.90, so work on retrieval first (hybrid search, re-ranking, better chunking, query rewriting). If hits are high but the reader fails, work on the prompt, the model, or context ordering.
Why is long video hard for VLMs? Give numbers and three ways to cope.advanced
Tokens grow as \(\text{seconds} \times \text{fps} \times \text{tokens per frame}\). A 90-minute match at 1 fps with 256 tokens per frame is 1,382,400 tokens, 10.5 times a 128K window, and the KV cache for an 8B-class LLM would be about 169 GiB. Lower fps misses short events, fewer tokens per frame lose small objects. Ways to cope: token compression (merge or drop redundant tokens across frames), retrieval of candidate clips before the VLM (Video-RAG), agents that choose where to look, streaming models with memory, and cheap specialist detectors that propose key moments for the VLM to verify.
Self-consistency with 15 samples: when does it help, and when does it hurt?advanced
If samples are independent and each is right with probability \(p\), majority voting pushes accuracy toward 1 when \(p \gt 0.5\) (0.6 → 0.787 at \(k = 15\)) and toward 0 when \(p \lt 0.5\) (0.4 → 0.213), in the worst case where all wrong answers agree. In practice wrong answers are often spread out, which helps voting, but samples are also correlated, which hurts it. Cost grows linearly with \(k\). If a verifier exists, pass@k \(= 1-(1-p)^k\) is the better target (best-of-k with the verifier).
A new model beats yours by 2 points on a 500-question benchmark. Do you switch?core
Not on that evidence alone. At 70% accuracy the standard error is \(\sqrt{0.7\cdot0.3/500} = 2.05\) points, so each score has a ±4.0-point 95% interval, and an unpaired difference needs about 5.7 points. Run a paired test on the same questions (count questions where only one model is right), check for contamination, test on your own task data, and weigh latency and cost. A 2-point gap needs roughly 4,000 questions per model to detect in the unpaired case.
What is a unified (any-to-any) model, and what is the main design tension?advanced
One network that reads and writes several modalities, e.g. text and images. Understanding wants semantic visual features (SigLIP-like), generation wants features that keep pixel detail (VQ tokens or VAE latents). Designs differ in how they handle this: Chameleon and Emu3 use discrete image tokens and one next-token loss; Transfusion keeps continuous patches with a diffusion loss next to the text loss; Janus uses two separate visual encoders for the two jobs with one shared transformer. Open questions are whether the two tasks help each other, and how to evaluate one model on everything.
How do you keep up with the field without drowning?basic
A fixed weekly routine: scan arXiv cs.CL, cs.CV and cs.LG titles once a week and save a few; do a quick first pass on those and a full read of the best two; read code of one major repo (vLLM, TRL, transformers). Once a month go deep on one paper and build something small. Use surveys as maps, technical reports of open models for practical detail, and conference orals once a quarter. Write a five-line note per paper. Be sceptical: check baselines and test-set sizes.

Exercises

  1. A model has \(L = 28\) layers, 4 KV heads and head size 128. How many KiB of KV cache does it need per token in BF16? How many GiB for 32,768 tokens?
  2. Estimate the training compute for a 3B model trained on 9T tokens. How many tokens per parameter is that?
  3. In a GRPO group of 4 answers, the rewards are 1, 0, 0, 0. Compute the advantages with the population std.
  4. A VLM uses 14-pixel patches with a \(2\times2\) merge. How many visual tokens does a \(448\times448\) image become? What about 8 frames of \(448\times448\) video, without any temporal merge?
  5. Single-sample accuracy is 0.7. What is majority-vote accuracy with \(k = 3\) (formula 239.9)? What is pass@3?
  6. How many questions do you need so that the 95% interval on a 50% accuracy is ±2 points?
  7. Pick one of the three plans in Figure 5. Write down the first paper you will read and the date you will finish its pass 2.
Answerscheck
1. \(2 \cdot 28 \cdot 4 \cdot 128 \cdot 2 = 57{,}344\) bytes \(= 56\) KiB per token; \(32{,}768 \times 56\) KiB \(= 1.75\) GiB. 2. \(6 \cdot 3\times10^9 \cdot 9\times10^{12} = 1.62\times10^{23}\) FLOPs; 3,000 tokens per parameter. 3. Mean 0.25, std \(\sqrt{0.25 \cdot 0.75} = 0.433\); advantages \(1.732, -0.577, -0.577, -0.577\) (they sum to 0). 4. \((448/28)^2 = 16^2 = 256\) tokens; 8 frames give 2,048 tokens. 5. \(P = 0.7^3 + 3 \cdot 0.7^2 \cdot 0.3 = 0.343 + 0.441 = 0.784\); pass@3 \(= 1 - 0.3^3 = 0.973\). 6. \(1.96\sqrt{0.25/n} = 0.02\) gives \(n = 0.25 \cdot (1.96/0.02)^2 = 2{,}401\) questions. 7. Your own answer; the point is a concrete date.

Key takeaways

  • Every part of the course has one formula that answers most hard questions: parameters and KV bytes (239.1, 239.2), \(6ND\) (239.3), group advantages (239.4), the decode bound (239.5), the RAG split (239.6), visual and video tokens (239.7, 239.8).
  • Llama 3 8B: 8.03B parameters, 128 KiB of KV cache per token, about 19 tokens/s bound on an L4 in BF16. Know these numbers by heart.
  • Long video is a token-budget problem: 90 minutes at 1 fps and 256 tokens per frame is 1.38M tokens.
  • Test-time compute is not free and majority voting only helps when the model is already right more often than wrong; verifiers change the game.
  • Unified models must reconcile semantic features for understanding with detailed features for generation.
  • Benchmarks are noisy: ±4 points on 500 questions at 70%. Report intervals, use paired tests, watch for contamination.
  • Keep up with a small weekly routine: scan, read two papers well, read code, build something each month.

Key formulas

\[ \text{KV/token} = 2 L H_{kv} d_h b, \qquad C \approx 6ND, \qquad \text{tokens/s} \le \frac{\text{bandwidth}}{\text{bytes read per step}} \]
\[ A_i = \frac{r_i - \bar r}{\operatorname{std}(r)}, \qquad n_{\text{vis}} = \frac{H}{pm}\cdot\frac{W}{pm}, \qquad n_{\text{video}} = \text{sec}\cdot\text{fps}\cdot\text{tok/frame} \]
\[ P_{\text{maj}} = \sum_{j \gt k/2} \binom{k}{j} p^j (1-p)^{k-j}, \qquad \operatorname{SE} = \sqrt{\frac{p(1-p)}{n}} \]
  1. Zhao et al. (2023). A Survey of Large Language Models
  2. Yin et al. (2023). A Survey on Multimodal Large Language Models
  3. Tang et al. (2023). Video Understanding with Large Language Models: A Survey
  4. Jin et al. (2024). Efficient Multimodal Large Language Models: A Survey
  5. Zhao et al. (2023). A Survey of Large Language Models
  6. Yin et al. (2023). A Survey on Multimodal Large Language Models
  7. Tang et al. (2023). Video Understanding with Large Language Models: A Survey
  8. Jin et al. (2024). Efficient Multimodal Large Language Models: A Survey
  9. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  10. Austin et al., Google DeepMind (2025). How to Scale Your Model
  11. Stanford (2025). CS336: Language Modeling from Scratch (course)
  12. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  13. Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
  14. Llama Team, Meta (2024). The Llama 3 Herd of Models
  15. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
  16. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  17. Snell et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
  18. Wang et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models
  19. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  20. Shen et al. (2024). LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
  21. Fu et al. (2024). Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  22. Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
  23. Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
  24. Wu et al. (2024). Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
  25. Wang et al. (2024). Emu3: Next-Token Prediction is All You Need
  26. Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
  27. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  28. Hsieh et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models?
  29. Keshav (2007). How to Read a Paper
  30. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 11: Advanced Topics and Cutting-Edge Research