Dissect a modern open model
A modern open model is a config.json file, a pile of weights and a technical report. This session shows a fixed method to take one apart: read the config, count the parameters exactly, compute FLOPs per token, KV cache per token and the memory to serve it, then explain every design choice in the report with what you learned in this course. We do it for Llama 3.1 8B, Qwen3, DeepSeek-V3, Gemma 3 and the vision tower of Qwen2.5-VL, and every count matches the real checkpoints.
What you will learn
- Read a Hugging Face config.json and know what every important field means (layers, heads, KV heads, head size, experts, vocabulary, RoPE, window, vision tower)
- Count the parameters of dense, MoE and MLA models exactly, and check the count against the checkpoint size
- Compute FLOPs per token, including the attention term that grows with context, and find where attention overtakes the weights
- Compute KV cache per token for GQA, MLA and sliding-window models, and the memory and number of GPUs needed to serve them
- Count visual tokens and vision-tower FLOPs for a native-resolution VLM (Qwen2.5-VL) and a fixed-resolution one (Gemma 3)
- Read a technical report and explain each design choice (GQA, QK-norm, MLA, fine-grained MoE, auxiliary-loss-free balancing, local and global attention, window attention in the ViT) with the problem it solves
Why this matters
Every new open model arrives as three things: a config.json file, a folder of weights, and a technical report. If you can read those three, you can answer the questions that matter in practice: how big is it really, how fast can it run, how much memory does it need to serve 50 users with long prompts, and why did its authors build it this way.
This is the last step of the course before the capstone project. Almost every earlier session gave you one tool: counting parameters and FLOPs (session 25), GQA and MLA (session 26), Mixture of Experts (session 29), RoPE and its scaling (session 14), the KV cache (session 80), number formats (session 105), and VLM vision towers (session 182). Here we use all of them at once, on real models, and we check every count against the real checkpoint files.
We dissect five models that cover the main design families of 2024–2025 [Certain]:
- Llama 3.1 8B: the plain dense baseline (GQA, SwiGLU, RoPE).
- Qwen3-8B and Qwen3-235B-A22B: one family, dense and Mixture-of-Experts, with QK-norm.
- DeepSeek-V3: MoE with 256 small experts plus Multi-head Latent Attention (MLA), the most memory-efficient KV cache here.
- Gemma 3 27B: local (sliding-window) and global attention layers, plus a SigLIP vision encoder.
- Qwen2.5-VL-7B: a vision-language model with a native-resolution ViT, for the vision-tower part.
The same method works for any newer model (Gemma 4, the next Qwen, the next DeepSeek). The last exercise gives you a fresh config to practice on.
The method: config, count, cost, report
A dissection has a fixed order. You first collect the shapes, then turn shapes into counts, then counts into costs, and only then read the report. Reading the report first is tempting, but then you only repeat the authors’ claims. Computing first lets you check them.
- input:
config.json,model.safetensors.index.json, the technical report - Read the shapes: \(d\), \(L\), \(h\), \(h_{kv}\), \(d_h\), \(d_{ff}\), vocabulary \(V\), tied or not, experts \(E\), top-\(k\), expert size \(f\), shared experts, attention type (GQA, MLA, sliding window), RoPE base and scaling, context length, vision tower
- Count parameters per block with formulas (234.1)–(234.3); multiply by \(L\); add embeddings
- Check: the checkpoint’s
total_sizein bytes \(\div\) bytes per value must equal your count. If not, find the missing piece (a bias, a norm, a vision tower) - Split into total and active parameters (MoE uses only top-\(k\) experts per token)
- KV cache per token with (234.4)–(234.6); FLOPs per token with (234.7)
- Memory to serve with (234.9) for your target context and number of users
- Read the report. For each design choice, write: what problem it solves, what it costs, and which number above shows it
Step 1: read the config
The config.json file on Hugging Face is the source of truth for shapes. Here are the fields that matter, copied from the five real configs [Certain]. When a field is missing (Llama has no head_dim), it is \(d / h\).
| Field | Llama 3.1 8B | Qwen3-8B | Qwen3-235B-A22B | DeepSeek-V3 | Gemma 3 27B (text) |
|---|---|---|---|---|---|
hidden_size \(d\) | 4,096 | 4,096 | 4,096 | 7,168 | 5,376 |
num_hidden_layers \(L\) | 32 | 36 | 94 | 61 | 62 |
num_attention_heads \(h\) | 32 | 32 | 64 | 128 | 32 |
num_key_value_heads \(h_{kv}\) | 8 | 8 | 4 | 128, but MLA caches a 512 + 64 latent | 16 |
head_dim \(d_h\) | 128 (= 4096/32) | 128 | 128 | q, k: 128 + 64 (RoPE part); v: 128 | 128 |
intermediate_size \(d_{ff}\) | 14,336 | 12,288 | experts: 1,536 | dense: 18,432; experts: 2,048 | 21,504 |
| Experts | none | none | 128, top-8, no shared | 256 routed, top-8, plus 1 shared; first 3 layers dense | none |
vocab_size \(V\) | 128,256 | 151,936 | 151,936 | 129,280 | 262,208 |
tie_word_embeddings | false | false | false | false | true |
max_position_embeddings | 131,072 | 40,960 | 40,960 | 163,840 | 131,072 |
rope_theta and scaling | 500,000, “llama3” scaling ×8 | 1,000,000 | 1,000,000 | 10,000, YaRN ×40 | 1M global, 10k local, linear ×8 |
| Special | none | QK-norm | QK-norm | MLA, 1 MTP layer, FP8 weights | window 1,024, 5 local : 1 global, QK-norm |
A few fields need care:
max_position_embeddingsis not always the advertised context. Qwen3-8B says 40,960. Its model card explains: 32,768 tokens for the output plus 8,192 for a typical prompt; the native context is 32,768, and 131,072 needs YaRN scaling turned on by hand [Certain]. DeepSeek-V3’s 163,840 is just 4,096 × 40, the YaRN factor [Certain].- MoE fields have model-specific names. Qwen3 uses
num_experts,num_experts_per_tok,moe_intermediate_size. DeepSeek usesn_routed_experts,n_shared_experts,first_k_dense_replace. Read the modeling code once when a field is new. - Hybrid attention is a pattern field. Gemma 3 has
sliding_window: 1024andsliding_window_pattern: 6: every 6th layer is global, the rest are local. With 62 layers that is 10 global and 52 local layers.
Figure 1 turns each config into a parameter budget. Pick a model.
model.safetensors.index.json.How to read Figure 1
The bar is the full model. Dense models (Llama, Qwen3-8B) are mostly FFN, then attention, with a visible slice for embeddings because their vocabularies are large. In the MoE models, the “idle experts” part is experts that a given token does not use: it must be stored in GPU memory but costs no compute for that token. In DeepSeek-V3 more than 90% of the model is idle for any single token. For Gemma 3, the “vision tower” slice is SigLIP plus its projector: only about 1.5% of the parameters.
Step 2: count the parameters
A decoder block has an attention part and an FFN part, plus small norm vectors. Write each projection as a matrix and add up the sizes. For GQA attention with \(h\) query heads, \(h_{kv}\) key-value heads and head size \(d_h\):
For a SwiGLU FFN with inner size \(d_{ff}\) there are three matrices (gate, up, down). An MoE layer has \(E\) such experts, each with a small inner size \(f\), plus a router; a token uses only \(k\) of them (plus \(n_s\) shared experts, if any):
The whole model is \(L\) blocks plus the embedding table, the output head (the same matrix if tied) and the final norm:
- Read the symbols
- \(d\)
- hidden size, the width of the residual stream
- \(h, h_{kv}, d_h\)
- number of query heads, number of key-value heads, size of one head
- \(d_{ff}, f\)
- inner size of a dense FFN, inner size of one expert
- \(E, k, n_s\)
- number of routed experts, how many each token uses, number of shared experts (always used)
- \(n_{\text{norm}}\)
- norm vectors per block: 2 for Llama and Qwen, 4 for Gemma (norms before and after each sub-layer)
- \(V, c\)
- vocabulary size; 1 if input and output embeddings share one matrix, else 2
Attention per layer: \(W_Q\) and \(W_O\) are \(4096 \times 4096\) each, \(W_K\) and \(W_V\) are \(4096 \times 1024\) each (8 heads × 128). With the two QK-norm vectors of size 128: \(2 \cdot 16{,}777{,}216 + 2 \cdot 4{,}194{,}304 + 256 = 41{,}943{,}296\).
FFN per layer: \(3 \times 4096 \times 12{,}288 = 150{,}994{,}944\). Two RMSNorm vectors: 8,192. Block total: 192,946,432. Times 36 layers plus the final norm: 6,946,075,648. This is the “6.95B non-embedding parameters” on the model card [Certain].
Embeddings: \(151{,}936 \times 4096 = 622{,}329{,}856\), twice because they are not tied. Total: 8,190,735,360.
Check: the checkpoint index says total_size = 16,381,470,720 bytes. In BF16 (2 bytes) that is exactly 8,190,735,360 parameters. The count matches to the last digit [Certain].
The same formulas give exact matches for every model in Figure 1: Llama 3.1 8B has 8,030,261,248 parameters, Qwen3-235B-A22B has 235,093,634,560, and Gemma 3 27B has 27,432,406,640 including its vision tower [Certain]. Note what Qwen3-8B tells us: with a 152K vocabulary, the two embedding matrices are 15% of the model. That is why small models (Qwen3-0.6B to 4B) tie their embeddings [Certain, from the report’s table].
MoE: total versus active
Qwen3-235B-A22B has 94 layers, each with 128 experts of size \(3 \times 4096 \times 1536 = 18{,}874{,}368\). A token uses 8 of them. Formula (234.2) gives 235.09B total and 22.19B active, which is the “A22B” in the name [Certain]. So each token pays for 9.4% of the weights in compute, but all 100% must sit in GPU memory. This one fact explains most of MoE serving: memory is set by total parameters, compute by active parameters.
MLA: the attention that compresses itself
DeepSeek-V3 does not store keys and values per head. It projects the hidden state down to a small latent vector \(\mathbf{c}^{KV}\) of size \(d_c = 512\), and stores that, plus one shared RoPE key of size \(d_r = 64\). Keys and values for all 128 heads are rebuilt from the latent by an up-projection. Queries are also compressed (to 1,536) to save training memory [Certain]. Per layer, the matrices are:
| Matrix | Shape | Parameters |
|---|---|---|
| query down \(W^{DQ}\) | 7168 × 1536 | 11,010,048 |
| query up \(W^{UQ}\) (128 heads × (128 + 64)) | 1536 × 24,576 | 37,748,736 |
| KV down \(W^{DKV}\) plus RoPE key | 7168 × (512 + 64) | 4,128,768 |
| KV up \(W^{UK}, W^{UV}\) (128 heads × (128 + 128)) | 512 × 32,768 | 16,777,216 |
| output \(W_O\) | 16,384 × 7168 | 117,440,512 |
| two latent norms | 1536 + 512 | 2,048 |
| Total per layer | 187,107,328 |
Then add 3 dense layers (FFN 18,432), 58 MoE layers (256 routed + 1 shared expert of size 2,048, a router, and a 256-entry routing bias), and untied embeddings. Formula (234.3) gives 671,026,419,200 parameters, the 671B of the report, and 36.6B active per token without the input embedding (a lookup has no FLOPs) [Certain]. The Hugging Face checkpoint is 685B because it also stores the Multi-Token Prediction (MTP) module: 11.5B unique parameters plus a copy of the embedding and head [Certain].
Step 3: KV cache per token
During generation, every layer keeps the keys and values of all earlier tokens (the KV cache). Its size per token is often the number that decides how many users a GPU can serve. For standard attention or GQA, each layer stores \(h_{kv}\) keys and \(h_{kv}\) values of size \(d_h\):
where \(b\) is bytes per value (2 for BF16, 1 for FP8). For MLA, each layer stores only the latent and the shared RoPE key, with no factor 2 and no heads:
For sliding-window attention, a local layer never keeps more than \(w\) tokens. With \(L_g\) global and \(L_l\) local layers, the cache for a context of \(T\) tokens is:
Llama 3.1 8B: \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes = 128 KiB. Qwen3-8B: \(2 \cdot 36 \cdot 8 \cdot 128 \cdot 2\) = 144 KiB. Qwen3-235B-A22B: \(2 \cdot 94 \cdot 4 \cdot 128 \cdot 2\) = 188 KiB: a model 29 times bigger than Qwen3-8B needs only 31% more cache per token, because it has only 4 KV heads.
DeepSeek-V3: \(61 \cdot (512 + 64) \cdot 2 = 70{,}272\) bytes = 68.6 KiB. If the same 128 heads cached full keys (192) and values (128), it would be \(61 \cdot 128 \cdot 320 \cdot 2 = 4{,}997{,}120\) bytes: 71 times more. That factor is the reason MLA exists.
Gemma 3 27B: a global layer costs \(2 \cdot 16 \cdot 128 \cdot 2 = 8{,}192\) bytes per token. At \(T = 32{,}768\), formula (234.6) gives \(8192 \cdot (10 \cdot 32{,}768 + 52 \cdot 1024) = 3.12\) GB, against 16.6 GB if all 62 layers were global: 18.75%.
(1 KiB = 1,024 bytes. Totals below are in GB = \(10^9\) bytes.)
How to read Figure 2
Move the context slider from 1K to 128K. At 1K, Gemma 3 has the biggest cache (62 layers, 16 KV heads); at 128K it has one of the smallest, because 52 of its layers stop growing at 1,024 tokens. DeepSeek-V3 is a 671B model with a cache smaller than Qwen3-8B. Then raise the batch: the cache scales linearly with the number of sequences, and at 128K a batch of 8 Llama 3.1 8B requests already needs more memory than one H100 has.
Step 4: FLOPs per token
A forward pass costs about 2 FLOPs per weight that the token actually uses (one multiply, one add), plus attention, which grows with the context. For one new token at position \(T\), each layer computes scores against \(T\) keys and a weighted sum of \(T\) values:
- Read the symbols
- \(N_{\text{act}}\)
- active parameters used in matrix multiplies: everything except the input embedding (a lookup) and the idle experts
- \(d_{qk}, d_v\)
- size of a query-key head and of a value head (128 and 128 for most models; 192 and 128 for DeepSeek-V3)
- \(T_\ell\)
- how many tokens layer \(\ell\) attends to
For a uniform model (all layers global, same head sizes) the attention term equals the weight term at the crossover context:
Llama 3.1 8B: \(N_{\text{act}} = 7.505\)B (8.03B minus the 0.525B input table), so \(T^* = 7.505 \times 10^9 / (32 \cdot 32 \cdot 256) \approx 28{,}630\) tokens. Beyond that, a decoding step spends more FLOPs on attention than on all the weights. For MoE models the crossover comes much earlier, because \(N_{\text{act}}\) is small compared with their many attention heads: about 14,000 tokens for both Qwen3-235B-A22B and DeepSeek-V3. Gemma 3’s local layers push its crossover past 300,000 tokens.
How to read Figure 3
At 4K, every model is dominated by its weights. At 32K, attention is already larger than the weights for Llama, Qwen3 and DeepSeek-V3. FLOPs are only half the story in decoding: a decode step is usually limited by reading the weights and the KV cache from memory, not by arithmetic. But this figure explains why long-context prefill gets slow, and why long-context models use windows, fewer KV heads or MLA.
The same count, times 3 for the backward pass, gives the training cost \(C_{\text{train}} \approx 6\,N_{\text{act}}\,D\) for \(D\) training tokens:
DeepSeek-V3: \(6 \times 36.6 \times 10^9 \times 14.8 \times 10^{12} \approx 3.25 \times 10^{24}\) FLOPs. Llama 3 405B (dense, 15.6T tokens): \(3.79 \times 10^{25}\), close to the \(3.8 \times 10^{25}\) the Llama 3 report gives [Certain]. A 671B MoE model trained with about 12 times less compute than a 405B dense one: that is the economic argument for MoE in one line.
Step 5: memory to serve
To serve a model you need the weights (all of them, including idle experts), the KV cache of every running request, and some room for activations, CUDA workspace and fragmentation:
- Read the symbols
- \(b_w\)
- bytes per weight: 2 (BF16), 1 (FP8), about 0.5 (INT4, plus small scales)
- \(B, T\)
- number of concurrent sequences and their length (prompt plus output)
- \(u\)
- the usable fraction of GPU memory; serving engines reserve about 90% by default [Likely]; we use \(u = 0.9\)
Weights in BF16: \(235.09 \times 10^9 \times 2 = 470.19\) GB. KV cache for 8 requests of 32,768 tokens: \(8 \times 32{,}768 \times 192{,}512 = 50.47\) GB. Total 520.65 GB. One H100 gives \(0.9 \times 80 = 72\) GB, so we need \(520.65 / 72 = 7.23\), rounded up to 8 GPUs, one full 8-GPU node.
In FP8 the weights drop to 235 GB, and 4 GPUs are enough for the same load. Qwen3-8B in BF16 with 16 requests of 32K tokens: \(16.38 + 77.31 = 93.69\) GB: more than one H100, and 83% of it is KV cache. For small models with long contexts, the cache, not the weights, decides the hardware.
How to read Figure 4
Start with Qwen3-235B at BF16: 8 H100s. Switch weights to FP8: the bar shrinks by half. Now pick DeepSeek-V3: even in FP8, its 671 GB of weights do not fit in 8 H100s (576 GB usable), but they fit in 8 H200s. Finally pick Llama 3.1 8B on an L4 (24 GB) with INT4 weights: the weights are only 4 GB, and the number of users you can serve is set almost entirely by the KV cache.
- input: counts from Algorithm 1, target context \(T\), concurrent requests \(B\), GPU memory \(M_{\text{GPU}}\), formats \(b_w, b_{kv}\)
- \(W \leftarrow N \cdot b_w\) // total parameters, not active
- \(K \leftarrow B \cdot M_{\text{KV}}(T)\) at \(b_{kv}\) bytes // formula (234.4), (234.5) or (234.6)
- \(n \leftarrow \lceil (W + K) / (0.9\, M_{\text{GPU}}) \rceil\); round up to a tensor-parallel size (1, 2, 4, 8)
- if \(W \gt 0.9\, n\, M_{\text{GPU}}\) with room for almost no cache: lower \(b_w\) (FP8, INT4) or add GPUs
- if \(K\) dominates: quantize the KV cache, use prefix caching, or limit \(T\)
- check speed: \(t_{\text{step}} \ge (W_{\text{act}} + K_{\text{read}}) \,/\, \text{BW}\) // decode time per step; bytes of active weights, bytes of KV read, memory bandwidth
- return \(n\), the formats, and the maximum batch that still fits
Step 6: the vision tower
A VLM adds a vision encoder and a connector. You dissect them the same way, but the important number is different: how many tokens one image or one second of video becomes, because each visual token costs as much LLM compute and KV cache as a text token.
Qwen2.5-VL-7B [Certain, config and report]: a 32-layer ViT, hidden size 1,280, 16 heads, patches of 14 × 14 pixels, RMSNorm and SwiGLU. Most layers use window attention over 112 × 112-pixel windows (8 × 8 patches); only 4 layers (indexes 7, 15, 23, 31) use full attention. Images keep their native size, rounded to a multiple of 28. A merger concatenates each 2 × 2 group of patches and maps it with a 2-layer MLP to the LLM width of 3,584. For video, two consecutive frames form one temporal patch. So:
where \(F\) is the number of sampled frames. The vision tower has 632M parameters and the merger 45M; with the 7.62B LLM the checkpoint holds 8,292,166,656 parameters, again an exact match [Certain].
Gemma 3 27B [Certain, config and report]: a 400M SigLIP encoder (27 layers, width 1,152, patch 14) that always sees a 896 × 896 square: \(64 \times 64 = 4{,}096\) patches, pooled to 256 tokens per image. Non-square or detailed images can be cut into several crops (“pan and scan”) at inference, 256 tokens each.
Qwen2.5-VL: \(H' = 728\), \(W' = 1288\), so \(52 \times 92 = 4{,}784\) patches and \(26 \times 46 = 1{,}196\) tokens. The ViT spends about 6.6 TFLOPs on it (linear layers \(2 \times 630\text{M} \times 4{,}784\) plus attention), and the LLM prefill of 1,196 tokens costs another \(2 \times 7.07\text{B} \times 1{,}196 = 16.9\) TFLOPs. In the KV cache, one such frame takes \(1{,}196 \times 56\) KiB = 68.6 MB.
Gemma 3: the frame is squashed to 896 × 896 and becomes 256 tokens, whatever its size. A minute of video at 2 frames per second and 448 × 448 is 15,360 tokens in Qwen2.5-VL. The fixed budget is cheap and predictable; the native resolution keeps small details such as text and far-away players.
How to read Figure 5
Make the image larger: Qwen2.5-VL’s token count grows with the area (1080p gives 2,691 tokens), Gemma’s stays at 256. Switch to video: Qwen2.5-VL pays for every pair of frames, so ten minutes at 2 fps and 448 × 448 is 153,600 tokens, more than its context. This is why video LLMs sample frames, lower the resolution per frame, or compress tokens (session 183, session 206). At low resolution the comparison flips for video: Qwen2.5-VL merges two frames into one temporal patch, so a 448 × 448 minute at 2 fps is 15,360 tokens, while feeding Gemma 3 the same 120 frames as separate images costs 120 × 256 = 30,720. The FLOP numbers for the ViT window attention treat each window as exactly 64 patches; real windows at the image border are smaller.
Step 7: read the report and explain every choice
Now the report. For each design choice, ask three questions: what problem does it solve, what does it cost, and which of our numbers shows it. Here is the result for the five models. Quotes and facts are from the technical reports and configs [Certain]; the “why” column connects them to earlier sessions.
| Model | Choice | Problem it solves | Where you see it |
|---|---|---|---|
| Llama 3 | Dense transformer, not MoE, “to maximize training stability” | Simple, well-understood training at 405B scale | 8B: \(N = N_{\text{act}}\); no idle weights in Figure 1 |
| Llama 3 | GQA with 8 KV heads at every size, “to improve inference speed and to reduce the size of key-value caches” | KV cache memory and bandwidth during decoding | 128 KiB per token instead of 512 KiB with 32 KV heads |
| Llama 3 | 128K vocabulary (100K from tiktoken + 28K for other languages); English compression from 3.17 to 3.94 characters per token | More text per token means less compute per document | Embeddings are 13% of the 8B model |
| Llama 3 | RoPE base 500,000; a mask that blocks attention across documents in one packed sequence | Longer contexts; no leakage between packed documents | rope_theta in the config (session 14) |
| Qwen3 | QK-norm added, QKV bias removed “to ensure stable training” | Attention logits that grow too large and cause loss spikes | The \(2 d_h\) term in (234.1); the missing biases |
| Qwen3 MoE | 128 experts, 8 active, no shared expert; global-batch load-balancing loss | Many small experts give more combinations; balancing over the global batch lets experts specialize | 235B total, 22B active (9.4%) |
| Qwen3 | RoPE base raised from 10,000 to 1,000,000 (ABF), then YaRN and Dual Chunk Attention for 4× longer inference context; 36T training tokens, 119 languages | Long context without training on 128K sequences | 40,960 in config, 131,072 with YaRN |
| DeepSeek-V3 | MLA with \(d_c = 512\), \(d_r = 64\) | KV cache for 128 heads would be huge | 68.6 KiB per token, 71× smaller than full MHA |
| DeepSeek-V3 | 256 fine-grained routed experts + 1 shared, top-8; each token sent to at most 4 nodes | Specialization; limits all-to-all traffic across nodes | 36.6B active of 671B; n_group: 8, topk_group: 4 |
| DeepSeek-V3 | Auxiliary-loss-free balancing: a per-expert bias used only for choosing experts; sigmoid scores; a small sequence-wise balance loss | Balanced experts without a large auxiliary loss that hurts quality | topk_method: noaux_tc, scoring_func: sigmoid, a 256-entry bias per layer |
| DeepSeek-V3 | Multi-Token Prediction (depth 1); FP8 mixed-precision training; 14.8T tokens, 2.788M H800 GPU hours | A denser training signal and a free draft model for speculative decoding; cheaper training | The extra 11.5B MTP layer; FP8 weights with 128 × 128 block scales |
| Gemma 3 | 5 local layers (window 1,024) per global layer; RoPE base 10k local, 1M global | KV cache at long context | 18.75% of the all-global cache at 32K (Figure 2) |
| Gemma 3 | QK-norm replaces Gemma 2’s soft-capping; trained with knowledge distillation; 32K pretraining, then RoPE scaling ×8 to 128K | Stable logits with a cheaper operation; better small models | rope_scaling: linear, factor 8 |
| Gemma 3 | 400M SigLIP at 896 × 896, 256 tokens per image, optional pan and scan | Fixed, low visual-token cost | Figure 5 |
| Qwen2.5-VL | Native resolution, window attention in the ViT (4 full layers), 2 × 2 patch merger, 2-frame temporal patches, MRoPE aligned to absolute time, dynamic FPS | Detail in large images at a cost that grows only linearly in the ViT; real time stamps for video grounding | Figure 5; fullatt_block_indexes, mrope_section: [16, 24, 24] |
Two patterns repeat across all five models. First, every design fights the KV cache: GQA (Llama, Qwen), very few KV heads (Qwen3 MoE has 4), MLA (DeepSeek), sliding windows (Gemma). Second, every recent model adds something for training stability: QK-norm (Qwen3, Gemma 3), a dense architecture (Llama 3), FP8 with fine-grained scales (DeepSeek-V3). When you read the next report, look for these two first.
The code: a config-driven calculator
This script counts parameters and KV cache from a config dict, and first checks the formula against a real PyTorch module with the same structure. It runs on CPU in under a second.
import torch, torch.nn as nn torch.manual_seed(0) def count_params(c): """Parameters of a Qwen3-style decoder (dense or MoE) from its config.json dict.""" d, L, V = c["hidden_size"], c["num_hidden_layers"], c["vocab_size"] h, hkv, dh = c["num_attention_heads"], c["num_key_value_heads"], c["head_dim"] attn = 2 * d * h * dh + 2 * d * hkv * dh + 2 * dh # formula (234.1), with QK-norm if c.get("num_experts"): # MoE block, formula (234.2) E, k, f = c["num_experts"], c["num_experts_per_tok"], c["moe_intermediate_size"] ffn_total, ffn_active = E * 3 * d * f + d * E, k * 3 * d * f + d * E else: # dense SwiGLU ffn_total = ffn_active = 3 * d * c["intermediate_size"] emb = V * d * (1 if c["tie_word_embeddings"] else 2) total = L * (attn + ffn_total + 2 * d) + emb + d # formula (234.3) active = L * (attn + ffn_active + 2 * d) + emb + d return total, active def kv_bytes_per_token(c, bytes_per_value=2): # formula (234.4) return 2 * c["num_hidden_layers"] * c["num_key_value_heads"] * c["head_dim"] * bytes_per_value # 1) check the formula against a real PyTorch module with the same structure class Block(nn.Module): def __init__(s, d, h, hkv, dh, ff): super().__init__() s.q, s.k = nn.Linear(d, h * dh, bias=False), nn.Linear(d, hkv * dh, bias=False) s.v, s.o = nn.Linear(d, hkv * dh, bias=False), nn.Linear(h * dh, d, bias=False) s.qn, s.kn = nn.RMSNorm(dh), nn.RMSNorm(dh) # QK-norm, one gain per head dim s.gate, s.up = nn.Linear(d, ff, bias=False), nn.Linear(d, ff, bias=False) s.down = nn.Linear(ff, d, bias=False) s.n1, s.n2 = nn.RMSNorm(d), nn.RMSNorm(d) tiny = dict(hidden_size=64, num_hidden_layers=3, vocab_size=1000, num_attention_heads=4, num_key_value_heads=2, head_dim=16, intermediate_size=192, tie_word_embeddings=False) model = nn.ModuleDict(dict(emb=nn.Embedding(1000, 64), head=nn.Linear(64, 1000, bias=False), norm=nn.RMSNorm(64), layers=nn.ModuleList([Block(64, 4, 2, 16, 192) for _ in range(3)]))) print(sum(p.numel() for p in model.parameters()), count_params(tiny)[0]) # 276000 276000 # 2) the real Qwen3-8B and Qwen3-235B-A22B configs (fields copied from config.json) q8 = dict(hidden_size=4096, num_hidden_layers=36, vocab_size=151936, num_attention_heads=32, num_key_value_heads=8, head_dim=128, intermediate_size=12288, tie_word_embeddings=False) q235 = dict(q8, num_hidden_layers=94, num_attention_heads=64, num_key_value_heads=4, num_experts=128, num_experts_per_tok=8, moe_intermediate_size=1536) for name, c in [("Qwen3-8B", q8), ("Qwen3-235B-A22B", q235)]: total, active = count_params(c) print(name, total, active, kv_bytes_per_token(c) / 1024, "KiB/token") # Qwen3-8B 8190735360 8190735360 144.0 KiB/token # Qwen3-235B-A22B 235093634560 22190763520 188.0 KiB/token
To check against a real checkpoint, download model.safetensors.index.json and divide metadata.total_size by 2 for a BF16 model. For FP8 checkpoints (DeepSeek-V3) this check does not work directly, because the file also holds block scales; count the tensors by name instead.
Where this is used
| Situation | What you compute | Decision it drives |
|---|---|---|
| Choosing a model for a product | Total and active parameters, KV per token, crossover context | Dense vs MoE; whether long-context requests are affordable |
| Capacity planning for an LLM service | Formula (234.10) per GPU type and format | Number of GPUs, FP8 or INT4, maximum batch (session 141) |
| Video-RAG or video question answering | Visual tokens per minute (234.11), KV per minute | Frames per second and resolution per frame; Gemma-style fixed budget vs native resolution |
| Sports key-moment detection over a full match | Tokens for 90 minutes of sampled frames, the KV cache they need | Retrieve candidate clips first, then send only short windows to the VLM (the next session) |
| Speculative decoding | Whether the model ships a draft (DeepSeek-V3’s MTP layer) | Use the built-in draft instead of training one (session 128) |
| Fine-tuning plan | 16 bytes per parameter for full AdamW training | Full fine-tune, LoRA or QLoRA (session 62) |
| Interviews | All of the above, on a whiteboard | Shows you understand models as systems, not as names |
Common mistakes
- Using active parameters for memory. A 235B-A22B model needs memory for 235B weights. Active parameters set compute, not memory.
- Using total parameters for FLOPs. The opposite mistake: an MoE token costs \(2 N_{\text{act}}\), not \(2N\).
- Using \(h\) instead of \(h_{kv}\) in the KV cache. For Qwen3-235B this overestimates the cache 16 times.
- Applying the GQA formula to MLA or sliding-window models. DeepSeek-V3 has
num_key_value_heads: 128in its config, but it caches a 576-wide latent. Gemma 3’s local layers stop at 1,024 tokens. Always check the attention type. - Trusting
max_position_embeddingsas the context length. Qwen3 says 40,960 but is 32K native and 128K with YaRN; read the model card. - Forgetting the vision tower and the MTP module. They are in the checkpoint and in GPU memory. Your count will not match
total_sizeuntil you add them. - Counting tied embeddings twice. Gemma 3’s 262K-entry table is stored once and used twice.
- Ignoring visual tokens in VLM serving. One 720p frame in Qwen2.5-VL is about 1,200 tokens: a 10-image prompt is a long-context request.
Interview questions
Given a config.json, how do you compute the parameter count? How do you check it?basic
tie_word_embeddings, plus the final norm. Check by dividing total_size in model.safetensors.index.json by the bytes per value. For Qwen3-8B both give 8,190,735,360.How much KV cache does Llama 3.1 8B need for one 128K-token request?basic
What does “235B-A22B” mean, and what does each number decide?core
Explain MLA and compute how much it saves for DeepSeek-V3.core
Gemma 3 uses 5 local layers per global layer with a 1,024 window. What does that buy at 32K context?core
At what context length does attention cost more FLOPs than the weights for Llama 3.1 8B?core
How many H100s do you need to serve Qwen3-235B-A22B in BF16 for 8 users with 32K contexts?core
Why did Qwen3 add QK-norm and Gemma 3 replace soft-capping with QK-norm?advanced
How did DeepSeek-V3 balance its experts without a big auxiliary loss?advanced
A 1080p image goes into Qwen2.5-VL-7B and into Gemma 3 27B. Compare the token counts and explain the trade-off.advanced
DeepSeek-V3 reports 2.788M H800 GPU hours. Is that consistent with its size?advanced
Exercises
- Compute the parameters of Llama 3.1 8B from the config table. Check: 8,030,261,248.
- Qwen3-32B has \(d = 5120\), \(L = 64\), \(h = 64\), \(h_{kv} = 8\), \(d_h = 128\), untied. Compute its KV cache per token in BF16 and for one 32K request.
- Qwen3-30B-A3B has \(d = 2048\), \(L = 48\), 128 experts of size 768, top-8. What fraction of its parameters is active? (Total 30,532,122,624; active 3,353,032,704.)
- How many tokens does Qwen2.5-VL produce for a 448 × 448 image? For 10 minutes of 448 × 448 video at 2 fps?
- Can DeepSeek-V3 in FP8 be served on 8 H100s (80 GB)? On 8 H200s (141 GB)? How many tokens of KV cache fit in the remaining H200 memory?
- Open the config of Gemma 4 31B on Hugging Face. It has 60 layers,
layer_typeswith 5 sliding layers per full layer,sliding_window: 1024, 16 KV heads of size 256 on sliding layers, andnum_global_key_value_heads: 4withglobal_head_dim: 512on full layers, plusattention_k_eq_v: true. Write the KV-per-token formula for the global layers. What must you check before trusting it?
Answerscheck
attention_k_eq_v means the value equals the key, only one tensor is cached: 40,960 bytes [Likely]. Check the modeling code before using either; also add the sliding layers: \(50 \cdot 2 \cdot 16 \cdot 256 \cdot b \cdot \min(T, 1024)\).Key takeaways
- Dissect in a fixed order: config, counts, costs, report. Compute first, then read the claims and check them.
- Formulas (234.1)–(234.3) give the exact parameter count of real models; the checkpoint’s
total_sizeconfirms it to the last parameter. - Memory is set by total parameters; compute by active parameters (234.7). MoE models such as Qwen3-235B-A22B (9.4% active) and DeepSeek-V3 (5.5% active) separate the two.
- KV cache per token (234.4)–(234.6): 128 KiB (Llama 3.1 8B), 188 KiB (Qwen3-235B), 68.6 KiB (DeepSeek-V3, MLA); Gemma 3’s windows cut the cache to 18.75% at 32K.
- Serving memory (234.10) for small models with long contexts is dominated by the KV cache; for large MoE models by the weights.
- A VLM’s cost is set by visual tokens (234.11): Qwen2.5-VL grows with image area, Gemma 3 is fixed at 256 per crop.
- Every modern design fights the KV cache and adds a stability fix (QK-norm, dense architecture, FP8 with fine scales).
Key formulas
References for this session
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- Qwen Team (2025). Qwen3 Technical Report
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Gemma Team (2025). Gemma 3 Technical Report
- Bai et al. (2025). Qwen2.5-VL Technical Report
- DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
- DeepSeek-AI (2024). DeepSeek-V3 model card and README_WEIGHTS (Hugging Face)
- Qwen Team (2025). Qwen3-8B model card (Hugging Face)