LLM & VLM courseBlog
234Track 10: Capstone and interviews · session 1 of 6

Dissect a modern open model

A modern open model is a config.json file, a pile of weights and a technical report. This session shows a fixed method to take one apart: read the config, count the parameters exactly, compute FLOPs per token, KV cache per token and the memory to serve it, then explain every design choice in the report with what you learned in this course. We do it for Llama 3.1 8B, Qwen3, DeepSeek-V3, Gemma 3 and the vision tower of Qwen2.5-VL, and every count matches the real checkpoints.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: Advanced · capstoneBefore this: Counting parameters and FLOPs (session 25), attention variants and the KV cache (sessions 26 and 80), Mixture of Experts (session 29)

What you will learn

  • Read a Hugging Face config.json and know what every important field means (layers, heads, KV heads, head size, experts, vocabulary, RoPE, window, vision tower)
  • Count the parameters of dense, MoE and MLA models exactly, and check the count against the checkpoint size
  • Compute FLOPs per token, including the attention term that grows with context, and find where attention overtakes the weights
  • Compute KV cache per token for GQA, MLA and sliding-window models, and the memory and number of GPUs needed to serve them
  • Count visual tokens and vision-tower FLOPs for a native-resolution VLM (Qwen2.5-VL) and a fixed-resolution one (Gemma 3)
  • Read a technical report and explain each design choice (GQA, QK-norm, MLA, fine-grained MoE, auxiliary-loss-free balancing, local and global attention, window attention in the ViT) with the problem it solves

Why this matters

Every new open model arrives as three things: a config.json file, a folder of weights, and a technical report. If you can read those three, you can answer the questions that matter in practice: how big is it really, how fast can it run, how much memory does it need to serve 50 users with long prompts, and why did its authors build it this way.

This is the last step of the course before the capstone project. Almost every earlier session gave you one tool: counting parameters and FLOPs (session 25), GQA and MLA (session 26), Mixture of Experts (session 29), RoPE and its scaling (session 14), the KV cache (session 80), number formats (session 105), and VLM vision towers (session 182). Here we use all of them at once, on real models, and we check every count against the real checkpoint files.

We dissect five models that cover the main design families of 2024–2025 [Certain]:

  • Llama 3.1 8B: the plain dense baseline (GQA, SwiGLU, RoPE).
  • Qwen3-8B and Qwen3-235B-A22B: one family, dense and Mixture-of-Experts, with QK-norm.
  • DeepSeek-V3: MoE with 256 small experts plus Multi-head Latent Attention (MLA), the most memory-efficient KV cache here.
  • Gemma 3 27B: local (sliding-window) and global attention layers, plus a SigLIP vision encoder.
  • Qwen2.5-VL-7B: a vision-language model with a native-resolution ViT, for the vision-tower part.

The same method works for any newer model (Gemma 4, the next Qwen, the next DeepSeek). The last exercise gives you a fresh config to practice on.

The method: config, count, cost, report

A dissection has a fixed order. You first collect the shapes, then turn shapes into counts, then counts into costs, and only then read the report. Reading the report first is tempting, but then you only repeat the authors’ claims. Computing first lets you check them.

Algorithm 1: dissect a model
  1. input: config.json, model.safetensors.index.json, the technical report
  2. Read the shapes: \(d\), \(L\), \(h\), \(h_{kv}\), \(d_h\), \(d_{ff}\), vocabulary \(V\), tied or not, experts \(E\), top-\(k\), expert size \(f\), shared experts, attention type (GQA, MLA, sliding window), RoPE base and scaling, context length, vision tower
  3. Count parameters per block with formulas (234.1)–(234.3); multiply by \(L\); add embeddings
  4. Check: the checkpoint’s total_size in bytes \(\div\) bytes per value must equal your count. If not, find the missing piece (a bias, a norm, a vision tower)
  5. Split into total and active parameters (MoE uses only top-\(k\) experts per token)
  6. KV cache per token with (234.4)–(234.6); FLOPs per token with (234.7)
  7. Memory to serve with (234.9) for your target context and number of users
  8. Read the report. For each design choice, write: what problem it solves, what it costs, and which number above shows it

Step 1: read the config

The config.json file on Hugging Face is the source of truth for shapes. Here are the fields that matter, copied from the five real configs [Certain]. When a field is missing (Llama has no head_dim), it is \(d / h\).

FieldLlama 3.1 8BQwen3-8BQwen3-235B-A22BDeepSeek-V3Gemma 3 27B (text)
hidden_size \(d\)4,0964,0964,0967,1685,376
num_hidden_layers \(L\)3236946162
num_attention_heads \(h\)32326412832
num_key_value_heads \(h_{kv}\)884128, but MLA caches a 512 + 64 latent16
head_dim \(d_h\)128 (= 4096/32)128128q, k: 128 + 64 (RoPE part); v: 128128
intermediate_size \(d_{ff}\)14,33612,288experts: 1,536dense: 18,432; experts: 2,04821,504
Expertsnonenone128, top-8, no shared256 routed, top-8, plus 1 shared; first 3 layers densenone
vocab_size \(V\)128,256151,936151,936129,280262,208
tie_word_embeddingsfalsefalsefalsefalsetrue
max_position_embeddings131,07240,96040,960163,840131,072
rope_theta and scaling500,000, “llama3” scaling ×81,000,0001,000,00010,000, YaRN ×401M global, 10k local, linear ×8
SpecialnoneQK-normQK-normMLA, 1 MTP layer, FP8 weightswindow 1,024, 5 local : 1 global, QK-norm

A few fields need care:

  • max_position_embeddings is not always the advertised context. Qwen3-8B says 40,960. Its model card explains: 32,768 tokens for the output plus 8,192 for a typical prompt; the native context is 32,768, and 131,072 needs YaRN scaling turned on by hand [Certain]. DeepSeek-V3’s 163,840 is just 4,096 × 40, the YaRN factor [Certain].
  • MoE fields have model-specific names. Qwen3 uses num_experts, num_experts_per_tok, moe_intermediate_size. DeepSeek uses n_routed_experts, n_shared_experts, first_k_dense_replace. Read the modeling code once when a field is new.
  • Hybrid attention is a pattern field. Gemma 3 has sliding_window: 1024 and sliding_window_pattern: 6: every 6th layer is global, the rest are local. With 62 layers that is 10 global and 52 local layers.

Figure 1 turns each config into a parameter budget. Pick a model.

Figure 1. Where the parameters live, computed in your browser from the config fields with formulas (234.1)–(234.3). The readout compares the total with the real checkpoint size from model.safetensors.index.json.
How to read Figure 1

The bar is the full model. Dense models (Llama, Qwen3-8B) are mostly FFN, then attention, with a visible slice for embeddings because their vocabularies are large. In the MoE models, the “idle experts” part is experts that a given token does not use: it must be stored in GPU memory but costs no compute for that token. In DeepSeek-V3 more than 90% of the model is idle for any single token. For Gemma 3, the “vision tower” slice is SigLIP plus its projector: only about 1.5% of the parameters.

Step 2: count the parameters

A decoder block has an attention part and an FFN part, plus small norm vectors. Write each projection as a matrix and add up the sizes. For GQA attention with \(h\) query heads, \(h_{kv}\) key-value heads and head size \(d_h\):

\[ N_{\text{attn}} = \underbrace{2\,d\,h\,d_h}_{W_Q,\ W_O} + \underbrace{2\,d\,h_{kv}\,d_h}_{W_K,\ W_V} \;\big(+\,2 d_h \text{ for QK-norm}\big) \tag{234.1} \]

For a SwiGLU FFN with inner size \(d_{ff}\) there are three matrices (gate, up, down). An MoE layer has \(E\) such experts, each with a small inner size \(f\), plus a router; a token uses only \(k\) of them (plus \(n_s\) shared experts, if any):

\[ N_{\text{ffn}} = 3\,d\,d_{ff}, \qquad N_{\text{moe}}^{\text{total}} = (E + n_s)\, 3\,d\,f + d\,E, \qquad N_{\text{moe}}^{\text{active}} = (k + n_s)\, 3\,d\,f + d\,E \tag{234.2} \]

The whole model is \(L\) blocks plus the embedding table, the output head (the same matrix if tied) and the final norm:

\[ N = L\,\big(N_{\text{attn}} + N_{\text{ffn}} + n_{\text{norm}}\, d\big) + c\, V d + d, \qquad c = \begin{cases} 1 & \text{tied} \\ 2 & \text{untied} \end{cases} \tag{234.3} \]
Read the symbols
\(d\)
hidden size, the width of the residual stream
\(h, h_{kv}, d_h\)
number of query heads, number of key-value heads, size of one head
\(d_{ff}, f\)
inner size of a dense FFN, inner size of one expert
\(E, k, n_s\)
number of routed experts, how many each token uses, number of shared experts (always used)
\(n_{\text{norm}}\)
norm vectors per block: 2 for Llama and Qwen, 4 for Gemma (norms before and after each sub-layer)
\(V, c\)
vocabulary size; 1 if input and output embeddings share one matrix, else 2
Worked example: Qwen3-8B, to the last parameter

Attention per layer: \(W_Q\) and \(W_O\) are \(4096 \times 4096\) each, \(W_K\) and \(W_V\) are \(4096 \times 1024\) each (8 heads × 128). With the two QK-norm vectors of size 128: \(2 \cdot 16{,}777{,}216 + 2 \cdot 4{,}194{,}304 + 256 = 41{,}943{,}296\).

FFN per layer: \(3 \times 4096 \times 12{,}288 = 150{,}994{,}944\). Two RMSNorm vectors: 8,192. Block total: 192,946,432. Times 36 layers plus the final norm: 6,946,075,648. This is the “6.95B non-embedding parameters” on the model card [Certain].

Embeddings: \(151{,}936 \times 4096 = 622{,}329{,}856\), twice because they are not tied. Total: 8,190,735,360.

Check: the checkpoint index says total_size = 16,381,470,720 bytes. In BF16 (2 bytes) that is exactly 8,190,735,360 parameters. The count matches to the last digit [Certain].

The same formulas give exact matches for every model in Figure 1: Llama 3.1 8B has 8,030,261,248 parameters, Qwen3-235B-A22B has 235,093,634,560, and Gemma 3 27B has 27,432,406,640 including its vision tower [Certain]. Note what Qwen3-8B tells us: with a 152K vocabulary, the two embedding matrices are 15% of the model. That is why small models (Qwen3-0.6B to 4B) tie their embeddings [Certain, from the report’s table].

MoE: total versus active

Qwen3-235B-A22B has 94 layers, each with 128 experts of size \(3 \times 4096 \times 1536 = 18{,}874{,}368\). A token uses 8 of them. Formula (234.2) gives 235.09B total and 22.19B active, which is the “A22B” in the name [Certain]. So each token pays for 9.4% of the weights in compute, but all 100% must sit in GPU memory. This one fact explains most of MoE serving: memory is set by total parameters, compute by active parameters.

MLA: the attention that compresses itself

DeepSeek-V3 does not store keys and values per head. It projects the hidden state down to a small latent vector \(\mathbf{c}^{KV}\) of size \(d_c = 512\), and stores that, plus one shared RoPE key of size \(d_r = 64\). Keys and values for all 128 heads are rebuilt from the latent by an up-projection. Queries are also compressed (to 1,536) to save training memory [Certain]. Per layer, the matrices are:

MatrixShapeParameters
query down \(W^{DQ}\)7168 × 153611,010,048
query up \(W^{UQ}\) (128 heads × (128 + 64))1536 × 24,57637,748,736
KV down \(W^{DKV}\) plus RoPE key7168 × (512 + 64)4,128,768
KV up \(W^{UK}, W^{UV}\) (128 heads × (128 + 128))512 × 32,76816,777,216
output \(W_O\)16,384 × 7168117,440,512
two latent norms1536 + 5122,048
Total per layer187,107,328

Then add 3 dense layers (FFN 18,432), 58 MoE layers (256 routed + 1 shared expert of size 2,048, a router, and a 256-entry routing bias), and untied embeddings. Formula (234.3) gives 671,026,419,200 parameters, the 671B of the report, and 36.6B active per token without the input embedding (a lookup has no FLOPs) [Certain]. The Hugging Face checkpoint is 685B because it also stores the Multi-Token Prediction (MTP) module: 11.5B unique parameters plus a copy of the embedding and head [Certain].

Step 3: KV cache per token

During generation, every layer keeps the keys and values of all earlier tokens (the KV cache). Its size per token is often the number that decides how many users a GPU can serve. For standard attention or GQA, each layer stores \(h_{kv}\) keys and \(h_{kv}\) values of size \(d_h\):

\[ m_{\text{KV}} = 2 \cdot L \cdot h_{kv} \cdot d_h \cdot b \quad \text{bytes per token} \tag{234.4} \]

where \(b\) is bytes per value (2 for BF16, 1 for FP8). For MLA, each layer stores only the latent and the shared RoPE key, with no factor 2 and no heads:

\[ m_{\text{KV}}^{\text{MLA}} = L \cdot (d_c + d_r) \cdot b \quad \text{bytes per token} \tag{234.5} \]

For sliding-window attention, a local layer never keeps more than \(w\) tokens. With \(L_g\) global and \(L_l\) local layers, the cache for a context of \(T\) tokens is:

\[ M_{\text{KV}}(T) = 2\, h_{kv}\, d_h\, b \,\big(L_g\, T + L_l \min(T, w)\big) \tag{234.6} \]
Worked example: KV cache per token in BF16

Llama 3.1 8B: \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes = 128 KiB. Qwen3-8B: \(2 \cdot 36 \cdot 8 \cdot 128 \cdot 2\) = 144 KiB. Qwen3-235B-A22B: \(2 \cdot 94 \cdot 4 \cdot 128 \cdot 2\) = 188 KiB: a model 29 times bigger than Qwen3-8B needs only 31% more cache per token, because it has only 4 KV heads.

DeepSeek-V3: \(61 \cdot (512 + 64) \cdot 2 = 70{,}272\) bytes = 68.6 KiB. If the same 128 heads cached full keys (192) and values (128), it would be \(61 \cdot 128 \cdot 320 \cdot 2 = 4{,}997{,}120\) bytes: 71 times more. That factor is the reason MLA exists.

Gemma 3 27B: a global layer costs \(2 \cdot 16 \cdot 128 \cdot 2 = 8{,}192\) bytes per token. At \(T = 32{,}768\), formula (234.6) gives \(8192 \cdot (10 \cdot 32{,}768 + 52 \cdot 1024) = 3.12\) GB, against 16.6 GB if all 62 layers were global: 18.75%.

(1 KiB = 1,024 bytes. Totals below are in GB = \(10^9\) bytes.)

GQA, every layer globalMLA (latent cache)5 sliding-window layers per global layer
Figure 2. KV cache for the chosen context and batch, from formulas (234.4)–(234.6). GQA models grow in a straight line with \(T\); Gemma 3 grows slower once \(T\) passes its 1,024-token window; DeepSeek-V3 (MLA) stays small even though it is the largest model.
How to read Figure 2

Move the context slider from 1K to 128K. At 1K, Gemma 3 has the biggest cache (62 layers, 16 KV heads); at 128K it has one of the smallest, because 52 of its layers stop growing at 1,024 tokens. DeepSeek-V3 is a 671B model with a cache smaller than Qwen3-8B. Then raise the batch: the cache scales linearly with the number of sequences, and at 128K a batch of 8 Llama 3.1 8B requests already needs more memory than one H100 has.

Step 4: FLOPs per token

A forward pass costs about 2 FLOPs per weight that the token actually uses (one multiply, one add), plus attention, which grows with the context. For one new token at position \(T\), each layer computes scores against \(T\) keys and a weighted sum of \(T\) values:

\[ C(T) \approx 2\,N_{\text{act}} + \sum_{\ell=1}^{L} 2\, h\, (d_{qk} + d_v)\, T_\ell, \qquad T_\ell = \begin{cases} T & \text{global layer} \\ \min(T, w) & \text{local layer} \end{cases} \tag{234.7} \]
Read the symbols
\(N_{\text{act}}\)
active parameters used in matrix multiplies: everything except the input embedding (a lookup) and the idle experts
\(d_{qk}, d_v\)
size of a query-key head and of a value head (128 and 128 for most models; 192 and 128 for DeepSeek-V3)
\(T_\ell\)
how many tokens layer \(\ell\) attends to

For a uniform model (all layers global, same head sizes) the attention term equals the weight term at the crossover context:

\[ T^{*} = \frac{N_{\text{act}}}{L\, h\, (d_{qk} + d_v)} \tag{234.8} \]

Llama 3.1 8B: \(N_{\text{act}} = 7.505\)B (8.03B minus the 0.525B input table), so \(T^* = 7.505 \times 10^9 / (32 \cdot 32 \cdot 256) \approx 28{,}630\) tokens. Beyond that, a decoding step spends more FLOPs on attention than on all the weights. For MoE models the crossover comes much earlier, because \(N_{\text{act}}\) is small compared with their many attention heads: about 14,000 tokens for both Qwen3-235B-A22B and DeepSeek-V3. Gemma 3’s local layers push its crossover past 300,000 tokens.

weights: 2 × active parametersattention over the context
Figure 3. Forward FLOPs for one generated token at position \(T\), from formula (234.7). The weight part does not depend on \(T\); the attention part grows linearly (or stops growing, for local layers).
How to read Figure 3

At 4K, every model is dominated by its weights. At 32K, attention is already larger than the weights for Llama, Qwen3 and DeepSeek-V3. FLOPs are only half the story in decoding: a decode step is usually limited by reading the weights and the KV cache from memory, not by arithmetic. But this figure explains why long-context prefill gets slow, and why long-context models use windows, fewer KV heads or MLA.

The same count, times 3 for the backward pass, gives the training cost \(C_{\text{train}} \approx 6\,N_{\text{act}}\,D\) for \(D\) training tokens:

\[ C_{\text{train}} \approx 6\, N_{\text{act}}\, D \tag{234.9} \]

DeepSeek-V3: \(6 \times 36.6 \times 10^9 \times 14.8 \times 10^{12} \approx 3.25 \times 10^{24}\) FLOPs. Llama 3 405B (dense, 15.6T tokens): \(3.79 \times 10^{25}\), close to the \(3.8 \times 10^{25}\) the Llama 3 report gives [Certain]. A 671B MoE model trained with about 12 times less compute than a 405B dense one: that is the economic argument for MoE in one line.

Step 5: memory to serve

To serve a model you need the weights (all of them, including idle experts), the KV cache of every running request, and some room for activations, CUDA workspace and fragmentation:

\[ M_{\text{serve}} = N\, b_w + B\, T\, m_{\text{KV}} + M_{\text{other}}, \qquad n_{\text{GPU}} = \left\lceil \frac{M_{\text{serve}}}{u \cdot M_{\text{GPU}}} \right\rceil \tag{234.10} \]
Read the symbols
\(b_w\)
bytes per weight: 2 (BF16), 1 (FP8), about 0.5 (INT4, plus small scales)
\(B, T\)
number of concurrent sequences and their length (prompt plus output)
\(u\)
the usable fraction of GPU memory; serving engines reserve about 90% by default [Likely]; we use \(u = 0.9\)
Worked example: Qwen3-235B-A22B on H100s

Weights in BF16: \(235.09 \times 10^9 \times 2 = 470.19\) GB. KV cache for 8 requests of 32,768 tokens: \(8 \times 32{,}768 \times 192{,}512 = 50.47\) GB. Total 520.65 GB. One H100 gives \(0.9 \times 80 = 72\) GB, so we need \(520.65 / 72 = 7.23\), rounded up to 8 GPUs, one full 8-GPU node.

In FP8 the weights drop to 235 GB, and 4 GPUs are enough for the same load. Qwen3-8B in BF16 with 16 requests of 32K tokens: \(16.38 + 77.31 = 93.69\) GB: more than one H100, and 83% of it is KV cache. For small models with long contexts, the cache, not the weights, decides the hardware.

weightsKV cachefree
Figure 4. A serving calculator built on formula (234.10). Each tick is one GPU at 90% usable memory. Activations and fragmentation are ignored, so treat the answer as a lower bound.
How to read Figure 4

Start with Qwen3-235B at BF16: 8 H100s. Switch weights to FP8: the bar shrinks by half. Now pick DeepSeek-V3: even in FP8, its 671 GB of weights do not fit in 8 H100s (576 GB usable), but they fit in 8 H200s. Finally pick Llama 3.1 8B on an L4 (24 GB) with INT4 weights: the weights are only 4 GB, and the number of users you can serve is set almost entirely by the KV cache.

Algorithm 2: plan the hardware to serve a model
  1. input: counts from Algorithm 1, target context \(T\), concurrent requests \(B\), GPU memory \(M_{\text{GPU}}\), formats \(b_w, b_{kv}\)
  2. \(W \leftarrow N \cdot b_w\) // total parameters, not active
  3. \(K \leftarrow B \cdot M_{\text{KV}}(T)\) at \(b_{kv}\) bytes // formula (234.4), (234.5) or (234.6)
  4. \(n \leftarrow \lceil (W + K) / (0.9\, M_{\text{GPU}}) \rceil\); round up to a tensor-parallel size (1, 2, 4, 8)
  5. if \(W \gt 0.9\, n\, M_{\text{GPU}}\) with room for almost no cache: lower \(b_w\) (FP8, INT4) or add GPUs
  6. if \(K\) dominates: quantize the KV cache, use prefix caching, or limit \(T\)
  7. check speed: \(t_{\text{step}} \ge (W_{\text{act}} + K_{\text{read}}) \,/\, \text{BW}\) // decode time per step; bytes of active weights, bytes of KV read, memory bandwidth
  8. return \(n\), the formats, and the maximum batch that still fits

Step 6: the vision tower

A VLM adds a vision encoder and a connector. You dissect them the same way, but the important number is different: how many tokens one image or one second of video becomes, because each visual token costs as much LLM compute and KV cache as a text token.

Qwen2.5-VL-7B [Certain, config and report]: a 32-layer ViT, hidden size 1,280, 16 heads, patches of 14 × 14 pixels, RMSNorm and SwiGLU. Most layers use window attention over 112 × 112-pixel windows (8 × 8 patches); only 4 layers (indexes 7, 15, 23, 31) use full attention. Images keep their native size, rounded to a multiple of 28. A merger concatenates each 2 × 2 group of patches and maps it with a 2-layer MLP to the LLM width of 3,584. For video, two consecutive frames form one temporal patch. So:

\[ n_{\text{img}} = \frac{H'}{28} \cdot \frac{W'}{28}, \qquad H' = 28 \cdot \operatorname{round}\!\left(\frac{H}{28}\right), \qquad n_{\text{video}} = \left\lceil \frac{F}{2} \right\rceil \cdot \frac{H'}{28} \cdot \frac{W'}{28} \tag{234.11} \]

where \(F\) is the number of sampled frames. The vision tower has 632M parameters and the merger 45M; with the 7.62B LLM the checkpoint holds 8,292,166,656 parameters, again an exact match [Certain].

Gemma 3 27B [Certain, config and report]: a 400M SigLIP encoder (27 layers, width 1,152, patch 14) that always sees a 896 × 896 square: \(64 \times 64 = 4{,}096\) patches, pooled to 256 tokens per image. Non-square or detailed images can be cut into several crops (“pan and scan”) at inference, 256 tokens each.

Worked example: one 1280 × 720 frame

Qwen2.5-VL: \(H' = 728\), \(W' = 1288\), so \(52 \times 92 = 4{,}784\) patches and \(26 \times 46 = 1{,}196\) tokens. The ViT spends about 6.6 TFLOPs on it (linear layers \(2 \times 630\text{M} \times 4{,}784\) plus attention), and the LLM prefill of 1,196 tokens costs another \(2 \times 7.07\text{B} \times 1{,}196 = 16.9\) TFLOPs. In the KV cache, one such frame takes \(1{,}196 \times 56\) KiB = 68.6 MB.

Gemma 3: the frame is squashed to 896 × 896 and becomes 256 tokens, whatever its size. A minute of video at 2 frames per second and 448 × 448 is 15,360 tokens in Qwen2.5-VL. The fixed budget is cheap and predictable; the native resolution keeps small details such as text and far-away players.

Figure 5. Left: the image as Qwen2.5-VL sees it, each cell one LLM token (2 × 2 patches of 14 pixels). Right: Gemma 3 resizes everything to 896 × 896 and always outputs a 16 × 16 grid of 256 tokens. The readout gives tokens, vision-tower FLOPs and LLM prefill FLOPs, from formula (234.11).
How to read Figure 5

Make the image larger: Qwen2.5-VL’s token count grows with the area (1080p gives 2,691 tokens), Gemma’s stays at 256. Switch to video: Qwen2.5-VL pays for every pair of frames, so ten minutes at 2 fps and 448 × 448 is 153,600 tokens, more than its context. This is why video LLMs sample frames, lower the resolution per frame, or compress tokens (session 183, session 206). At low resolution the comparison flips for video: Qwen2.5-VL merges two frames into one temporal patch, so a 448 × 448 minute at 2 fps is 15,360 tokens, while feeding Gemma 3 the same 120 frames as separate images costs 120 × 256 = 30,720. The FLOP numbers for the ViT window attention treat each window as exactly 64 patches; real windows at the image border are smaller.

Step 7: read the report and explain every choice

Now the report. For each design choice, ask three questions: what problem does it solve, what does it cost, and which of our numbers shows it. Here is the result for the five models. Quotes and facts are from the technical reports and configs [Certain]; the “why” column connects them to earlier sessions.

ModelChoiceProblem it solvesWhere you see it
Llama 3Dense transformer, not MoE, “to maximize training stability”Simple, well-understood training at 405B scale8B: \(N = N_{\text{act}}\); no idle weights in Figure 1
Llama 3GQA with 8 KV heads at every size, “to improve inference speed and to reduce the size of key-value caches”KV cache memory and bandwidth during decoding128 KiB per token instead of 512 KiB with 32 KV heads
Llama 3128K vocabulary (100K from tiktoken + 28K for other languages); English compression from 3.17 to 3.94 characters per tokenMore text per token means less compute per documentEmbeddings are 13% of the 8B model
Llama 3RoPE base 500,000; a mask that blocks attention across documents in one packed sequenceLonger contexts; no leakage between packed documentsrope_theta in the config (session 14)
Qwen3QK-norm added, QKV bias removed “to ensure stable training”Attention logits that grow too large and cause loss spikesThe \(2 d_h\) term in (234.1); the missing biases
Qwen3 MoE128 experts, 8 active, no shared expert; global-batch load-balancing lossMany small experts give more combinations; balancing over the global batch lets experts specialize235B total, 22B active (9.4%)
Qwen3RoPE base raised from 10,000 to 1,000,000 (ABF), then YaRN and Dual Chunk Attention for 4× longer inference context; 36T training tokens, 119 languagesLong context without training on 128K sequences40,960 in config, 131,072 with YaRN
DeepSeek-V3MLA with \(d_c = 512\), \(d_r = 64\)KV cache for 128 heads would be huge68.6 KiB per token, 71× smaller than full MHA
DeepSeek-V3256 fine-grained routed experts + 1 shared, top-8; each token sent to at most 4 nodesSpecialization; limits all-to-all traffic across nodes36.6B active of 671B; n_group: 8, topk_group: 4
DeepSeek-V3Auxiliary-loss-free balancing: a per-expert bias used only for choosing experts; sigmoid scores; a small sequence-wise balance lossBalanced experts without a large auxiliary loss that hurts qualitytopk_method: noaux_tc, scoring_func: sigmoid, a 256-entry bias per layer
DeepSeek-V3Multi-Token Prediction (depth 1); FP8 mixed-precision training; 14.8T tokens, 2.788M H800 GPU hoursA denser training signal and a free draft model for speculative decoding; cheaper trainingThe extra 11.5B MTP layer; FP8 weights with 128 × 128 block scales
Gemma 35 local layers (window 1,024) per global layer; RoPE base 10k local, 1M globalKV cache at long context18.75% of the all-global cache at 32K (Figure 2)
Gemma 3QK-norm replaces Gemma 2’s soft-capping; trained with knowledge distillation; 32K pretraining, then RoPE scaling ×8 to 128KStable logits with a cheaper operation; better small modelsrope_scaling: linear, factor 8
Gemma 3400M SigLIP at 896 × 896, 256 tokens per image, optional pan and scanFixed, low visual-token costFigure 5
Qwen2.5-VLNative resolution, window attention in the ViT (4 full layers), 2 × 2 patch merger, 2-frame temporal patches, MRoPE aligned to absolute time, dynamic FPSDetail in large images at a cost that grows only linearly in the ViT; real time stamps for video groundingFigure 5; fullatt_block_indexes, mrope_section: [16, 24, 24]

Two patterns repeat across all five models. First, every design fights the KV cache: GQA (Llama, Qwen), very few KV heads (Qwen3 MoE has 4), MLA (DeepSeek), sliding windows (Gemma). Second, every recent model adds something for training stability: QK-norm (Qwen3, Gemma 3), a dense architecture (Llama 3), FP8 with fine-grained scales (DeepSeek-V3). When you read the next report, look for these two first.

The code: a config-driven calculator

This script counts parameters and KV cache from a config dict, and first checks the formula against a real PyTorch module with the same structure. It runs on CPU in under a second.

import torch, torch.nn as nn
torch.manual_seed(0)

def count_params(c):
    """Parameters of a Qwen3-style decoder (dense or MoE) from its config.json dict."""
    d, L, V = c["hidden_size"], c["num_hidden_layers"], c["vocab_size"]
    h, hkv, dh = c["num_attention_heads"], c["num_key_value_heads"], c["head_dim"]
    attn = 2 * d * h * dh + 2 * d * hkv * dh + 2 * dh          # formula (234.1), with QK-norm
    if c.get("num_experts"):                                    # MoE block, formula (234.2)
        E, k, f = c["num_experts"], c["num_experts_per_tok"], c["moe_intermediate_size"]
        ffn_total, ffn_active = E * 3 * d * f + d * E, k * 3 * d * f + d * E
    else:                                                       # dense SwiGLU
        ffn_total = ffn_active = 3 * d * c["intermediate_size"]
    emb = V * d * (1 if c["tie_word_embeddings"] else 2)
    total = L * (attn + ffn_total + 2 * d) + emb + d           # formula (234.3)
    active = L * (attn + ffn_active + 2 * d) + emb + d
    return total, active

def kv_bytes_per_token(c, bytes_per_value=2):                  # formula (234.4)
    return 2 * c["num_hidden_layers"] * c["num_key_value_heads"] * c["head_dim"] * bytes_per_value

# 1) check the formula against a real PyTorch module with the same structure
class Block(nn.Module):
    def __init__(s, d, h, hkv, dh, ff):
        super().__init__()
        s.q, s.k = nn.Linear(d, h * dh, bias=False), nn.Linear(d, hkv * dh, bias=False)
        s.v, s.o = nn.Linear(d, hkv * dh, bias=False), nn.Linear(h * dh, d, bias=False)
        s.qn, s.kn = nn.RMSNorm(dh), nn.RMSNorm(dh)            # QK-norm, one gain per head dim
        s.gate, s.up = nn.Linear(d, ff, bias=False), nn.Linear(d, ff, bias=False)
        s.down = nn.Linear(ff, d, bias=False)
        s.n1, s.n2 = nn.RMSNorm(d), nn.RMSNorm(d)

tiny = dict(hidden_size=64, num_hidden_layers=3, vocab_size=1000, num_attention_heads=4,
            num_key_value_heads=2, head_dim=16, intermediate_size=192, tie_word_embeddings=False)
model = nn.ModuleDict(dict(emb=nn.Embedding(1000, 64), head=nn.Linear(64, 1000, bias=False),
                           norm=nn.RMSNorm(64),
                           layers=nn.ModuleList([Block(64, 4, 2, 16, 192) for _ in range(3)])))
print(sum(p.numel() for p in model.parameters()), count_params(tiny)[0])   # 276000 276000

# 2) the real Qwen3-8B and Qwen3-235B-A22B configs (fields copied from config.json)
q8 = dict(hidden_size=4096, num_hidden_layers=36, vocab_size=151936, num_attention_heads=32,
          num_key_value_heads=8, head_dim=128, intermediate_size=12288, tie_word_embeddings=False)
q235 = dict(q8, num_hidden_layers=94, num_attention_heads=64, num_key_value_heads=4,
            num_experts=128, num_experts_per_tok=8, moe_intermediate_size=1536)
for name, c in [("Qwen3-8B", q8), ("Qwen3-235B-A22B", q235)]:
    total, active = count_params(c)
    print(name, total, active, kv_bytes_per_token(c) / 1024, "KiB/token")
# Qwen3-8B 8190735360 8190735360 144.0 KiB/token
# Qwen3-235B-A22B 235093634560 22190763520 188.0 KiB/token

To check against a real checkpoint, download model.safetensors.index.json and divide metadata.total_size by 2 for a BF16 model. For FP8 checkpoints (DeepSeek-V3) this check does not work directly, because the file also holds block scales; count the tensors by name instead.

Where this is used

SituationWhat you computeDecision it drives
Choosing a model for a productTotal and active parameters, KV per token, crossover contextDense vs MoE; whether long-context requests are affordable
Capacity planning for an LLM serviceFormula (234.10) per GPU type and formatNumber of GPUs, FP8 or INT4, maximum batch (session 141)
Video-RAG or video question answeringVisual tokens per minute (234.11), KV per minuteFrames per second and resolution per frame; Gemma-style fixed budget vs native resolution
Sports key-moment detection over a full matchTokens for 90 minutes of sampled frames, the KV cache they needRetrieve candidate clips first, then send only short windows to the VLM (the next session)
Speculative decodingWhether the model ships a draft (DeepSeek-V3’s MTP layer)Use the built-in draft instead of training one (session 128)
Fine-tuning plan16 bytes per parameter for full AdamW trainingFull fine-tune, LoRA or QLoRA (session 62)
InterviewsAll of the above, on a whiteboardShows you understand models as systems, not as names

Common mistakes

  • Using active parameters for memory. A 235B-A22B model needs memory for 235B weights. Active parameters set compute, not memory.
  • Using total parameters for FLOPs. The opposite mistake: an MoE token costs \(2 N_{\text{act}}\), not \(2N\).
  • Using \(h\) instead of \(h_{kv}\) in the KV cache. For Qwen3-235B this overestimates the cache 16 times.
  • Applying the GQA formula to MLA or sliding-window models. DeepSeek-V3 has num_key_value_heads: 128 in its config, but it caches a 576-wide latent. Gemma 3’s local layers stop at 1,024 tokens. Always check the attention type.
  • Trusting max_position_embeddings as the context length. Qwen3 says 40,960 but is 32K native and 128K with YaRN; read the model card.
  • Forgetting the vision tower and the MTP module. They are in the checkpoint and in GPU memory. Your count will not match total_size until you add them.
  • Counting tied embeddings twice. Gemma 3’s 262K-entry table is stored once and used twice.
  • Ignoring visual tokens in VLM serving. One 720p frame in Qwen2.5-VL is about 1,200 tokens: a 10-image prompt is a long-context request.

Interview questions

Given a config.json, how do you compute the parameter count? How do you check it?basic
Per block: attention \(2dhd_h + 2dh_{kv}d_h\) (plus QK-norm vectors and biases if present), FFN \(3dd_{ff}\) for SwiGLU (or experts plus router for MoE), plus norm vectors. Multiply by \(L\), add \(Vd\) once or twice depending on tie_word_embeddings, plus the final norm. Check by dividing total_size in model.safetensors.index.json by the bytes per value. For Qwen3-8B both give 8,190,735,360.
How much KV cache does Llama 3.1 8B need for one 128K-token request?basic
Per token \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes (128 KiB). Times 131,072 tokens: 17.2 GB in BF16, 8.6 GB in FP8. That is about the same as the 16 GB of BF16 weights, so one long request doubles the memory needed.
What does “235B-A22B” mean, and what does each number decide?core
235B total parameters, 22B active per token (8 of 128 experts in each of 94 layers, plus attention and embeddings). The total decides memory: 470 GB in BF16. The active count decides compute per token: about \(2 \times 21.6\)B FLOPs for the weights. MoE gives the quality of a large model at the per-token compute of a small one, but not at its memory.
Explain MLA and compute how much it saves for DeepSeek-V3.core
MLA projects the hidden state down to a 512-dimensional latent and caches only that plus a 64-dimensional shared RoPE key; per-head keys and values are rebuilt by an up-projection (which can be folded into the query and output projections at decode time). Cache per token: \(61 \cdot 576 \cdot 2 = 70{,}272\) bytes. Full MHA with the same 128 heads (keys 192, values 128) would cache \(61 \cdot 128 \cdot 320 \cdot 2 = 4{,}997{,}120\) bytes, 71 times more. The decoupled RoPE key is needed because RoPE rotates by position and would break the folding of the up-projection.
Gemma 3 uses 5 local layers per global layer with a 1,024 window. What does that buy at 32K context?core
With 10 global and 52 local layers, the cache is \(8192 \cdot (10T + 52 \min(T, 1024))\) bytes. At \(T = 32{,}768\): 3.12 GB versus 16.6 GB if all layers were global, 18.75%. The cost: local layers cannot look further than 1,024 tokens, so long-range information must flow through the 10 global layers. The report keeps a 1M RoPE base on global layers and 10k on local ones.
At what context length does attention cost more FLOPs than the weights for Llama 3.1 8B?core
Weights: \(2N_{\text{act}} = 2 \times 7.505\)B = 15.0 GFLOPs per token. Attention: \(L \cdot 2h(d_{qk} + d_v)T = 32 \cdot 2 \cdot 32 \cdot 256 \cdot T\). Equal at \(T^* = 7.505 \times 10^9 / (32 \cdot 32 \cdot 256) \approx 28{,}600\). For MoE models with many heads the crossover is much earlier: about 14,000 for Qwen3-235B-A22B and DeepSeek-V3.
How many H100s do you need to serve Qwen3-235B-A22B in BF16 for 8 users with 32K contexts?core
Weights 470.19 GB, KV cache \(8 \times 32{,}768 \times 192{,}512\) bytes = 50.47 GB, total 520.65 GB. At 72 GB usable per H100, 7.23 GPUs, so 8 (one node, tensor parallel 8). With FP8 weights, 235 + 50 = 285 GB fits in 4 GPUs. Then check speed: decode reads the 22B active weights and the cache every step, so bandwidth sets tokens per second.
Why did Qwen3 add QK-norm and Gemma 3 replace soft-capping with QK-norm?advanced
Attention logits are dot products of queries and keys; if their norms grow during training, logits become huge, softmax saturates and the loss spikes. QK-norm normalizes each query and key head vector (RMSNorm with a learned gain) before the dot product, which bounds the logits. Soft-capping (\(c \tanh(x/c)\)) also bounds them, but it is an extra element-wise operation on the full score matrix and does not fit well with fused attention kernels [Likely]. QK-norm costs only \(2 d_h\) parameters per layer.
How did DeepSeek-V3 balance its experts without a big auxiliary loss?advanced
Each expert has a bias that is added to its affinity score only when choosing the top-8 experts, not when weighting their outputs. After each step, the bias goes down for overloaded experts and up for underloaded ones. So balance is enforced by routing, not by a gradient that fights the language-modeling loss. A very small sequence-wise balance loss remains to avoid extreme imbalance within one sequence. Routing is also limited to 4 nodes per token to bound communication.
A 1080p image goes into Qwen2.5-VL-7B and into Gemma 3 27B. Compare the token counts and explain the trade-off.advanced
Qwen2.5-VL rounds to 1092 × 1932, so \(39 \times 69 = 2{,}691\) tokens. Gemma 3 resizes to 896 × 896 and outputs 256 tokens (more with pan and scan). Qwen keeps fine detail (small text, distant objects) and native aspect ratio, at about 10 times the LLM cost and KV cache. Gemma has a fixed, cheap budget but loses detail on large or very wide images, which is why pan and scan exists.
DeepSeek-V3 reports 2.788M H800 GPU hours. Is that consistent with its size?advanced
Training compute \(\approx 6 \times 36.6\text{B} \times 14.8\text{T} = 3.25 \times 10^{24}\) FLOPs. Divided by \(2.788 \times 10^6 \times 3600\) GPU-seconds, that is about 324 TFLOP/s per GPU. That is plausible for FP8 training on H800s with good utilization. The GPU hours include context extension and post-training, so the pretraining rate is a bit higher. This sanity check is what interviewers want: the claim is consistent with \(6ND\).

Exercises

  1. Compute the parameters of Llama 3.1 8B from the config table. Check: 8,030,261,248.
  2. Qwen3-32B has \(d = 5120\), \(L = 64\), \(h = 64\), \(h_{kv} = 8\), \(d_h = 128\), untied. Compute its KV cache per token in BF16 and for one 32K request.
  3. Qwen3-30B-A3B has \(d = 2048\), \(L = 48\), 128 experts of size 768, top-8. What fraction of its parameters is active? (Total 30,532,122,624; active 3,353,032,704.)
  4. How many tokens does Qwen2.5-VL produce for a 448 × 448 image? For 10 minutes of 448 × 448 video at 2 fps?
  5. Can DeepSeek-V3 in FP8 be served on 8 H100s (80 GB)? On 8 H200s (141 GB)? How many tokens of KV cache fit in the remaining H200 memory?
  6. Open the config of Gemma 4 31B on Hugging Face. It has 60 layers, layer_types with 5 sliding layers per full layer, sliding_window: 1024, 16 KV heads of size 256 on sliding layers, and num_global_key_value_heads: 4 with global_head_dim: 512 on full layers, plus attention_k_eq_v: true. Write the KV-per-token formula for the global layers. What must you check before trusting it?
Answerscheck
1. Attention \(2 \cdot 4096 \cdot 4096 + 2 \cdot 4096 \cdot 1024 = 41{,}943{,}040\); FFN \(3 \cdot 4096 \cdot 14{,}336 = 176{,}160{,}768\); norms 8,192; block 218,112,000; \(32 \times 218{,}112{,}000 = 6{,}979{,}584{,}000\); embeddings \(2 \cdot 128{,}256 \cdot 4096 = 1{,}050{,}673{,}152\); final norm 4,096. Total 8,030,261,248. 2. \(2 \cdot 64 \cdot 8 \cdot 128 \cdot 2 = 262{,}144\) bytes = 256 KiB per token; at 32,768 tokens, 8.59 GB. 3. \(3.35 / 30.53 = 11.0\%\). 4. \(16 \times 16 = 256\) tokens per image; 1,200 frames in 600 pairs, \(600 \times 256 = 153{,}600\) tokens. 5. Weights about 671 GB. 8 H100s give 576 GB usable: no. 8 H200s give \(0.9 \times 1128 = 1015\) GB usable, leaving about 344 GB, which at 70,272 bytes per token is about 4.9 million tokens. 6. If \(K\) and \(V\) are separate: \(10 \cdot 2 \cdot 4 \cdot 512 \cdot b\) bytes per token for the 10 full layers (81,920 bytes in BF16). If attention_k_eq_v means the value equals the key, only one tensor is cached: 40,960 bytes [Likely]. Check the modeling code before using either; also add the sliding layers: \(50 \cdot 2 \cdot 16 \cdot 256 \cdot b \cdot \min(T, 1024)\).

Key takeaways

  • Dissect in a fixed order: config, counts, costs, report. Compute first, then read the claims and check them.
  • Formulas (234.1)–(234.3) give the exact parameter count of real models; the checkpoint’s total_size confirms it to the last parameter.
  • Memory is set by total parameters; compute by active parameters (234.7). MoE models such as Qwen3-235B-A22B (9.4% active) and DeepSeek-V3 (5.5% active) separate the two.
  • KV cache per token (234.4)–(234.6): 128 KiB (Llama 3.1 8B), 188 KiB (Qwen3-235B), 68.6 KiB (DeepSeek-V3, MLA); Gemma 3’s windows cut the cache to 18.75% at 32K.
  • Serving memory (234.10) for small models with long contexts is dominated by the KV cache; for large MoE models by the weights.
  • A VLM’s cost is set by visual tokens (234.11): Qwen2.5-VL grows with image area, Gemma 3 is fixed at 256 per crop.
  • Every modern design fights the KV cache and adds a stability fix (QK-norm, dense architecture, FP8 with fine scales).

Key formulas

\[ N = L\big(2dhd_h + 2dh_{kv}d_h + 3dd_{ff} + n_{\text{norm}}d\big) + cVd + d, \qquad N_{\text{moe}}^{\text{active}} = (k + n_s)\,3df + dE \]
\[ m_{\text{KV}} = 2Lh_{kv}d_hb, \qquad m_{\text{KV}}^{\text{MLA}} = L(d_c + d_r)b, \qquad C(T) \approx 2N_{\text{act}} + \textstyle\sum_\ell 2h(d_{qk} + d_v)T_\ell \]
\[ M_{\text{serve}} = Nb_w + BTm_{\text{KV}} + M_{\text{other}}, \qquad n_{\text{img}} = \tfrac{H'}{28}\cdot\tfrac{W'}{28} \]

References for this session

  1. DeepSeek-AI (2024). DeepSeek-V3 Technical Report
  2. Qwen Team (2025). Qwen3 Technical Report
  3. Llama Team, Meta (2024). The Llama 3 Herd of Models
  4. Gemma Team (2025). Gemma 3 Technical Report
  5. Bai et al. (2025). Qwen2.5-VL Technical Report
  6. DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
  7. Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
  8. Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
  9. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
  10. DeepSeek-AI (2024). DeepSeek-V3 model card and README_WEIGHTS (Hugging Face)
  11. Qwen Team (2025). Qwen3-8B model card (Hugging Face)