Efficient FFN 3: MoE from dense models, and MoE inference
Training a Mixture-of-Experts model from scratch is expensive, so this session shows two ways to build one from a dense model you already have: MoEfication cuts a dense feed-forward network into groups of neurons, and sparse upcycling copies it into many experts and keeps training. Then we serve the result: we count total and active parameters, see why memory must hold every expert, and work out what batching and expert parallelism do to the speed of Mixtral 8x7B and DeepSeek-V3.
What you will learn
- Why a ReLU feed-forward network is already a sparse mixture of experts, and how MoEfication finds the experts (balanced k-means or a co-activation graph) and builds a router
- How sparse upcycling turns a dense checkpoint into an MoE: copied experts, a new random router, and why the first step can lose most of the FFN output
- What upcycled models cost and gain: about 50% of the dense pre-training cost in the original paper, Qwen1.5-MoE-A2.7B from Qwen-1.8B, and Branch-Train-MiX
- Total vs active parameters with exact counts for Mixtral 8x7B (46.7B / 12.9B), Qwen1.5-MoE-A2.7B (14.3B / 2.7B) and DeepSeek-V3 (671B / 37.6B), and why memory must hold all of them
- How the batch changes MoE decode: the number of experts touched grows fast, so bytes per step rise toward the total, while FLOPs stay at the active count
- Expert parallelism at serving time: two all-to-all exchanges per MoE layer, stragglers from hot experts, and DeepSeek-V3's redundant experts and EP32 / EP320 deployment
Why this matters
Mixtral 8x7B has 46.7 billion parameters but uses only 12.9 billion for each token. Its total size is exactly Mistral 7B plus seven more copies of every feed-forward network and a tiny router per layer. This session is about that kind of relation: how to make a Mixture-of-Experts (MoE) model from a dense model, and what it costs to serve one.
In session 29 you met the MoE layer: a router sends each token to \(k\) of \(E\) expert feed-forward networks (FFNs). In session 95 you saw how routing is kept balanced. Two practical questions remain, and they are the two halves of this page.
- Where do the experts come from? Training a large MoE from scratch is expensive, and most teams already own a good dense checkpoint. Two families of methods reuse it. MoEfication cuts each dense FFN into groups of neurons and adds a router, so the model gets cheaper with no new parameters. Sparse upcycling copies each FFN into many experts and keeps training, so the model gets bigger and better for a fraction of the cost. The efficient-inference survey of Zhou et al. (section 5.1.1) lists both under “expert construction”. [Certain]
- What does an MoE cost at serving time? FLOPs follow the active parameters, but memory must hold all parameters, and with a batch of requests almost every expert gets used. So an MoE is cheap in one cost and not in the others. Serving systems answer with large batches and expert parallelism: experts spread over many GPUs. DeepSeek-V3 decodes on 320 GPUs with one expert per GPU. [Certain]
We use the three costs and the roofline from session 82 throughout: a step takes at least \(\max(F/P,\ Q/B)\), where \(F\) is FLOPs, \(Q\) is bytes moved, \(P\) is peak compute and \(B\) is memory bandwidth.
A dense FFN is already a sum of small experts
Start with the FFN of one block, as in session 21. It has \(d_{ff}\) hidden neurons. Neuron \(j\) has an input vector (column \(j\) of \(W_1\)) and an output vector (row \(j\) of \(W_2\)). Write the FFN as a sum over neurons:
- Read the symbols
- \(\mathbf{x}\)
- the input vector of one token, size \(d\)
- \(\mathbf{w}^{(1)}_j,\ b_j\)
- the input weights (column \(j\) of \(W_1\)) and the bias of neuron \(j\)
- \(\mathbf{w}^{(2)}_j\)
- the output weights of neuron \(j\) (row \(j\) of \(W_2\)), size \(d\)
- \(\sigma\)
- the activation function, ReLU in the models MoEfication studied
- \(\mathbf{b}_2\)
- the output bias
Two facts follow. First, the order of the neurons does not matter: any permutation of the terms gives the same sum. Second, with ReLU, a neuron with a negative pre-activation adds exactly zero. Zhang et al. measured how often that happens: in a fine-tuned T5-Large (700 million parameters), 90% of inputs activate less than 5% of the FFN neurons. [Certain] So most of the work of a dense FFN is multiplying by zero.
Now cut the \(d_{ff}\) neurons into \(K\) groups of equal size \(d_e = d_{ff}/K\). Group \(i\) is a small FFN on its own, with its own slices of \(W_1\), \(b_1\) and \(W_2\). Because the order does not matter, the dense FFN is the sum of these small FFNs:
- Read the symbols
- \(K\)
- number of experts we cut the FFN into
- \(W_1^{i}\ (d \times d_e),\ \mathbf{b}_1^{i},\ W_2^{i}\ (d_e \times d)\)
- the columns of \(W_1\), entries of \(\mathbf{b}_1\) and rows of \(W_2\) that belong to the neurons of group \(i\)
- \(E_i\)
- expert \(i\): a small FFN with \(d_e\) hidden neurons
This is an MoE with every expert switched on. Nothing has changed yet: same parameters, same output, same FLOPs. The saving comes when we run only the few experts that hold the active neurons for this token.
MoEfication: split the neurons, then build a router
MoEfication (Zhang et al., 2021) has two phases. [Certain]
Phase 1, expert construction. Choose the groups so that neurons which are active together land in the same expert. Then, for a given token, a few experts cover almost all of its active neurons. The paper tries two ways and a random baseline:
- Parameter clustering split. Treat the \(d_{ff}\) input vectors \(\mathbf{w}^{(1)}_j\) as points and run balanced k-means (k-means where every cluster must have the same size). The idea: neurons with similar input vectors fire on similar inputs.
- Co-activation graph split. Run the model on training data and count how strongly each pair of neurons is active at the same time. Then cut this graph into \(K\) equal parts with a graph-partitioning tool, keeping strongly linked neurons together.
- Random split, the baseline: equal groups with no structure.
- Read the symbols
- \(h_n(\mathbf{x})\)
- pre-activation of neuron \(n\) for input \(\mathbf{x}\)
- \(\mathbb{1}[\cdot]\)
- 1 if both neurons are active (positive) for this input, else 0
- \(\text{co-act}(n,m)\)
- the weight of the edge between neurons \(n\) and \(m\) in the co-activation graph
Phase 2, expert selection. For each token, pick \(n\) of the \(K\) experts. The best possible choice (the paper calls it groundtruth selection) is the experts with the largest sum of positive activations, but computing it needs the whole FFN, which is what we want to avoid. So the paper trains a small MLP router that predicts these sums from \(\mathbf{x}\), and also tries a cheap similarity selection: the cosine between \(\mathbf{x}\) and the mean input vector of each expert. The MoEfied layer is
- Read the symbols
- \(S(\mathbf{x})\)
- the set of \(n\) experts chosen for this token
- \(s_i(\mathbf{x})\)
- the score of expert \(i\): the true sum of its positive activations (ideal), a router prediction, or a cosine similarity
- \(n/K\)
- the fraction of FFN parameters used for this token
Note the gate: there is none. Each chosen expert enters with weight 1, because it is a piece of the original sum. If \(n = K\), \(F_m = F\) exactly.
The results, on T5 and BERT with GLUE and question-answering tasks: MoEfication uses 10% to 30% of the FFN parameters for each input and keeps over 95% of the original performance. With 25% of the FFN parameters it reports 2× faster inference on CPU and 1.2× on GPU. Experts have \(d_e = 32\) neurons, so the T5 models get 64 to 512 experts per layer. [Certain] The GPU gain is small because many small expert matrices use the GPU less well than one big one, and the router itself costs time. [Likely]
How to read Figure 1
With the k-means split, neurons of the same topic share a column, so an input lights up only two columns and \(n = 2\) experts (25% of the FFN) already cover almost all active neurons: the mean error over 64 inputs is small. Switch to the random split: the active neurons spread over all columns, and the same \(n = 2\) leaves most of them out. Switch the router: the similarity router does not see the activations, but here it picks nearly the same experts. At \(n = 8\) the error is exactly 0 for every split, by formula (96.2).
MoEfication for modern LLMs
MoEfication relies on exact zeros from ReLU. Most current LLMs use SwiGLU with SiLU, which gives small but non-zero values almost everywhere, so dropping neurons always changes the output a little. [Likely] Later work therefore splits the FFN and then trains the result. LLaMA-MoE splits the FFNs of LLaMA-2 7B into experts, adds gates, and continues pre-training on 200B tokens, giving a model with about 3.5B active parameters. [Certain] At that point MoEfication and upcycling meet: both start from dense weights and both need extra training.
- input: \(W_1, \mathbf{b}_1, W_2, \mathbf{b}_2\); number of experts \(K\); experts used \(n\); a sample of training inputs
- \(d_e \leftarrow d_{ff} / K\)
- \(\text{lab} \leftarrow\) BalancedKMeans(columns of \(W_1\), \(K\), size \(d_e\)) // or: partition the co-activation graph (96.3)
- \(P \leftarrow\) permutation that sorts neurons by \(\text{lab}\)
- for \(i = 1 \dots K\): \(W_1^{i}, \mathbf{b}_1^{i}, W_2^{i} \leftarrow\) slice \(i\) of \(W_1P,\ \mathbf{b}_1P,\ P^\top W_2\) // formula (96.2), exact
- collect targets \(t_i(\mathbf{x}) = \sum \text{ReLU}(\mathbf{x}W_1^{i} + \mathbf{b}_1^{i})\) on the sample
- train a small MLP router \(r(\mathbf{x}) \approx (t_1, \dots, t_K)\)
- at inference: \(S \leftarrow\) top-\(n\) of \(r(\mathbf{x})\); return \(\sum_{i \in S} E_i(\mathbf{x}) + \mathbf{b}_2\) // formula (96.4)
Sparse upcycling: copy the FFN, then train
MoEfication keeps the parameter count and saves FLOPs. Sparse upcycling (Komatsuzaki et al., 2022) does the opposite: it keeps the FLOPs per token roughly fixed and adds parameters, to get a better model than the dense one. [Certain] The recipe:
- Copy every weight of the dense checkpoint: embeddings, attention, norms.
- In the chosen layers, replace the FFN by an MoE layer whose \(E\) experts are all exact copies of that FFN.
- Add a router. The dense model has none, so it starts random: normal with standard deviation 0.02.
- Continue training on the usual data.
In the paper: half of the MLP layers become MoE layers (every other layer, starting with the second), each with 32 experts. The language encoder and the vision models use Expert Choice routing with capacity factor 2; the language decoder uses top-2 token-choice routing with an auxiliary loss weight of 0.01, to avoid a mismatch between full-batch training and one-token decoding. Reusing the dense optimizer state helped the vision models, not the language models. [Certain]
What happens at step 0
Since all experts are the same function, the MoE output at the first step is just the FFN output times the total gate weight of the chosen experts:
- Read the symbols
- \(g_i(\mathbf{x})\)
- gate weight of expert \(i\), from the router’s softmax
- \(G(\mathbf{x})\)
- the total gate weight of the \(k\) chosen experts: the factor by which the FFN output is scaled at step 0
If the gates are renormalized over the top-\(k\) (top-\(k\) then softmax, as in Mixtral), \(G = 1\) and the upcycled model starts as the exact dense model. If they are not (softmax over all \(E\), then top-\(k\), as in Switch and GShard), \(G\) can be far below 1. With a near-random router the softmax is close to uniform, so \(G \approx k/E\): for top-2 of 32 that is 0.0625. Router logits are not exactly zero: if the hidden vector has RMS 1 in \(d = 768\), the logits have standard deviation \(0.02\sqrt{768} = 0.55\), and \(G\) is about 0.15 (Figure 2). [Likely, depends on the hidden scale] So at step 0 the FFN contributes only about 15% of its usual output. The attention path is unchanged, so the model does not break, but quality drops. [Likely]
The paper reports this initial quality drop and recovery with more training: more MoE layers, more experts or more capacity make the drop larger; upcycling the bottom layers hurts most. Normalizing the combine weights helped vision but slightly hurt language (SuperGLUE 79.3 to 78.8). [Certain] So renormalization is not a free win; it changes the training dynamics too.
Why do identical copies ever become different? Because the router is random, different experts receive different tokens, so they get different gradients and drift apart. Anything that breaks the symmetry helps: the Qwen team reports that adding randomness to the copied weights made the upcycled model converge faster. [Certain]
How to read Figure 2
The default is the upcycling paper’s decoder setting: 32 experts, top-2, no renormalization, \(\sigma = 0.55\). The bar shows that the FFN starts at about 15% of its dense output. Set \(\sigma = 0\) to see the exact uniform case \(G = k/E\). Raise \(\sigma\): the router becomes more peaked and \(G\) grows. Click “Top-k, then renormalize”: \(G = 1\) whatever the settings, so the upcycled model starts equal to the dense one. The readout also gives the parameter and FLOP multipliers: parameters grow with \(E\), FLOPs with \(k\).
What upcycling buys
The key result: upcycled T5 Base, Large and XL and Vision Transformer Base and Large beat their dense versions on SuperGLUE and ImageNet while using only about 50% of the original dense pre-training cost as extra compute. They also beat MoE models trained from scratch with 100% of that budget. [Certain] Concretely, upcycled T5-Large and T5-Base gained 1.5 to 2 SuperGLUE points over continued dense training with 46% and 55% extra training. [Certain] The paper also notes that, with a very large compute budget, a from-scratch MoE eventually catches up, and then training from scratch may be preferable. [Certain] Upcycling wins when your extra budget is small compared with what the dense model already cost.
Real examples:
- Qwen1.5-MoE-A2.7B was made from Qwen-1.8B by upcycling with fine-grained experts: each FFN is cut into pieces (as in MoEfication), and the pieces become experts. It has 4 shared experts that are always on and 60 routed experts of which 4 are chosen. Its config: routed experts of width 1,408, a shared FFN of width 5,632 \(= 4 \times 1{,}408\), and 24 layers like Qwen-1.8B. Totals: 14.3B parameters, 2.7B active. The team reports 75% lower training cost than Qwen1.5-7B. [Certain]
- Mixtral 8x7B has exactly the shape of Mistral 7B with 8 FFNs per layer (\(d = 4096\), 32 layers, FFN width 14,336 in both configs). [Certain] Whether Mistral AI initialized it from Mistral 7B is not stated in the paper. [Speculation]
- Branch-Train-MiX (BTX) copies one seed model, trains each copy on its own domain (code, math, world knowledge), then uses their FFNs as the experts of one MoE, averages the other weights, and fine-tunes the router. Sparse upcycling is the special case with no separate domain training. [Certain]
Compare GLaM, an MoE trained from scratch: 1.2 trillion parameters, 64 experts in every other layer, top-2, so 96.6B (8%) active per token. It used about one third of the energy of GPT-3’s training and half of its inference FLOPs. [Certain] From-scratch training gives the most freedom; upcycling gives the cheapest start.
- input: dense checkpoint; experts \(E\); top-\(k\); set of layers \(\mathcal{L}\) to convert (e.g. every other layer)
- copy all non-FFN weights (embeddings, attention, norms) // optimizer state: optional
- for each layer \(\ell\): if \(\ell \in \mathcal{L}\) then
- for \(i = 1 \dots E\): \(E_i^{\ell} \leftarrow\) copy of \(F^{\ell}_{\text{dense}}\) (+ small noise to break symmetry) // formula (96.5)
- \(W_r^{\ell} \sim \mathcal{N}(0, 0.02^2)\) // new router
- else keep \(F^{\ell}_{\text{dense}}\)
- train with the usual LM loss + load-balancing loss; the paper resumes the dense learning-rate schedule where it left off
- check quality early: the step-0 drop depends on \(G\) (96.6) and on \(|\mathcal{L}|\), \(E\), capacity
Serving cost 1: total vs active parameters
Now the MoE exists, upcycled or not, and we want to serve it. Split the parameters into those used by every token (embeddings, attention, norms, routers, shared experts, dense FFN layers) and the routed experts:
- Read the symbols
- \(N_{\text{other}}\)
- parameters every token uses (including always-on shared experts)
- \(L_{\text{moe}}\)
- number of MoE layers
- \(E,\ k\)
- routed experts per layer, and how many each token uses
- \(P_e\)
- parameters of one routed expert: \(3\,d\,d_e\) for a SwiGLU expert (gate, up and down matrices)
The three costs from session 82 split cleanly. Memory size follows \(N_{\text{total}}\): every expert must sit in some GPU’s memory, because the next token may need it. FLOPs per token follow \(N_{\text{active}}\): about \(2N_{\text{active}}\). Bytes read per decode step lie in between and depend on the batch, which is the next section.
| Model (from config.json) | Routed experts | \(N_{\text{total}}\) | \(N_{\text{active}}\) | Experts’ share of \(N_{\text{total}}\) | Weights stored |
|---|---|---|---|---|---|
| Mixtral 8x7B | 8, top-2, 32 MoE layers | 46.70B | 12.88B | 96.6% | 93.4 GB (BF16) |
| Qwen1.5-MoE-A2.7B | 60, top-4, + shared FFN, 24 layers | 14.32B | 2.69B | 87.0% | 28.6 GB (BF16) |
| DeepSeek-V3 | 256, top-8, + 1 shared, 58 MoE layers (first 3 dense) | 671.0B | 37.6B | 97.4% | 671 GB (FP8) |
All counts are computed from the public configs and match the papers: Mixtral 47B / 13B, DeepSeek-V3 671B / 37B (without its extra multi-token-prediction module), Qwen 14.3B / 2.7B. [Certain] A Mixtral expert in BF16 is 352 MB; one MoE layer’s 8 experts are 2.8 GB. Note what this means: Mixtral at batch 1 does the FLOPs of a 13B model but needs the memory of a 47B model. The Mixtral paper says so directly: memory cost follows the sparse parameter count, and MoE layers suit batched workloads. [Certain]
Serving cost 2: the batch decides how many experts you read
At batch 1, a decode step reads \(N_{\text{other}}\) plus the \(k\) chosen experts in each layer: the bytes of the active parameters. That is the MoE dream: a 47B model that decodes as fast as a 13B one. But a serving system decodes a batch of \(b\) sequences per step (see session 84), and different tokens pick different experts. If routing were uniform and independent, a given expert is missed by one token with probability \(1 - k/E\), so the expected number of distinct experts touched per layer is
and the weight bytes read in one decode step are
- Read the symbols
- \(b\)
- batch size: sequences that each make one token in this step
- \(\bar{E}(b)\)
- expected number of different experts used by the batch, in one layer
- \(s\)
- bytes per weight: 2 for BF16, 1 for FP8
- \(N_{\text{other,mm}}\)
- always-used parameters that are multiplied (the input embedding is only a lookup, so it is left out)
For Mixtral (\(E = 8, k = 2\)): \(\bar{E}(4) = 5.47\), \(\bar{E}(8) = 7.20\), \(\bar{E}(16) = 7.92\). By batch 16 nearly every expert is read in every layer, so the step reads almost all 93 GB, like a dense 47B model. For DeepSeek-V3 (\(E = 256, k = 8\)) it takes longer, but at batch 64 already 222 of 256 experts are touched per layer. Real routing is not uniform (some experts are hot), which lowers \(\bar{E}\) a bit but creates imbalance (later section). [Likely]
The FLOPs, however, stay at \(2bN_{\text{active}}\). So the arithmetic intensity of the expert matrices is set by the tokens per expert, about \(bk/E\) once all experts are touched:
- Read the symbols
- \(bk/E\)
- average number of tokens each expert processes in one step
- \(I^{*} = P/B\)
- the ridge point of the GPU: FLOPs per byte needed to be compute-bound
- \(b^{*}\)
- batch at which the experts become compute-bound: \(E/k\) times the dense model’s value
A dense BF16 model on an H100 reaches the ridge (\(I^{*} = 295\) FLOP/byte) near batch 295 (from session 82). Mixtral needs \(E/k = 4\) times more, about 1,181. DeepSeek-V3 in FP8 (ridge 591 FLOP/byte for dense FP8 on an H100) needs about 9,450 sequences in one step. This is why MoE serving is all about very large batches, gathered across many GPUs. [Certain, as arithmetic]
In Figure 3 the whole step turns compute-bound a little earlier (about 1,080 sequences for Mixtral, 1,740 for Qwen1.5-MoE and 5,400 for DeepSeek-V3), because the attention and shared weights, which every token uses, reach the ridge first and their FLOPs count too. The experts on their own still sit below the ridge until \(b^{*}\).
How to read Figure 3
At batch 1 the orange MoE curve sits on the blue “same active size” line: decode is as fast as a small dense model. Move the slider: the orange curve climbs to the grey “same total size” line within a few dozen sequences (later for DeepSeek-V3, which has 256 experts). The green line is the FLOP time converted to the same scale; where it crosses the orange curve, the step turns compute-bound. The readout gives experts touched per layer, tokens per expert, the step time and the total tokens per second.
Fit. 93.4 GB of weights on 160 GB of HBM leaves 66.6 GB for the KV cache and activations.
Batch 1. Bytes read: \(2 \times (12.88 - 0.13)\) B \(= 25.5\) GB (the input embedding of 0.13B is a lookup). Bandwidth 6.7 TB/s, so at least 3.81 ms per token, 263 tokens/s. A dense 47B model would need 13.90 ms. MoE wins by 3.6×.
Batch 8. \(\bar{E}(8) = 8(1 - 0.75^8) = 7.20\) experts per layer. Bytes: 84.1 GB, 12.55 ms per step, 637 tokens/s in total. The MoE advantage over a dense 47B model is almost gone in bytes.
Batch 32. All 8 experts read: 93.1 GB, 13.90 ms, 2,302 tokens/s. Same time as a dense 47B model, but the FLOPs (0.41 ms of compute) are 3.6× smaller.
Batch 1,024. Compute time \(2 \times 1024 \times 12.75\text{B} / 1978\) TFLOP/s \(= 13.2\) ms, close to the memory time of 13.9 ms: near the ridge, about 73,700 tokens/s. A dense 47B model would already be compute-bound here at 3.6× the FLOPs, so the MoE gives 3.6× the peak throughput.
Lesson. MoE helps latency at batch 1 (fewer bytes) and throughput at huge batches (fewer FLOPs). In between, it is a 47B model in bytes.
Serving cost 3: expert parallelism and its traffic
Large MoEs do not fit one GPU, so the experts must be spread out. Two ways:
- Tensor parallelism (session 48) cuts every expert matrix across GPUs. Every GPU does a slice of every expert. Simple, but each expert’s already small matrices get even smaller.
- Expert parallelism (EP) puts whole experts on different GPUs: with \(G\) GPUs, each holds \(E/G\) experts. Attention usually runs data-parallel (each GPU on its own requests). In each MoE layer, tokens travel to the GPUs that hold their experts and the results travel back: two all-to-all exchanges (session 43, session 49).
The bytes each token sends per MoE layer, if all \(k\) experts are on other GPUs:
- Read the symbols
- \(d\)
- hidden size: each copy of a token is one \(d\)-vector
- \(s_{\text{dispatch}},\ s_{\text{combine}}\)
- bytes per number for sending tokens out and results back
Mixtral in BF16: \(2 \times 4096 \times 4 = 32{,}768\) bytes per token per layer, 1 MiB per token over 32 layers. DeepSeek-V3 sends tokens in FP8 and results in BF16 [Certain]: \(8 \times 7168 \times 3 = 172{,}032\) bytes per token per layer, about 10 MB per token over 58 layers. That is an upper bound: DeepSeek-V3 limits each token to experts on at most 4 nodes, which cuts the slow cross-node part. [Certain about the limit; the size of the saving depends on the routing, Likely]
The other problem is imbalance. Each MoE layer waits for the slowest GPU. If some experts are popular (“hot”), their GPU gets more tokens and everyone else waits:
Load-balancing losses act during training (session 95), but live traffic differs from training data. The Mixtral paper also shows that consecutive tokens often go to the same expert in higher layers (61.6% to 67.0% for “first or second choice” at layer 15, against about 46% if random), and warns that this locality can over-subscribe some experts under EP. [Certain]
DeepSeek-V3’s answer is redundant experts. It watches expert load online and gives the hottest experts a second copy on another GPU; tokens for that expert are split between the copies. In prefill, the unit is 4 nodes with 32 GPUs, attention TP4 with DP8, MoE EP32, and 32 redundant experts (each GPU holds its 8 experts plus 1 redundant one), re-chosen about every 10 minutes. In decode, the unit is 40 nodes with 320 GPUs, MoE EP320: each GPU holds only one expert, and 64 GPUs host the redundant and shared experts. To hide the all-to-all, it overlaps the attention of one micro-batch with the dispatch, MoE and combine of another. [Certain]
How to read Figure 4
Each bar is one GPU; its segments are its experts. The dashed line is perfect balance (1,024 slots per GPU). The layer takes as long as the tallest bar (formula 96.12). At \(\alpha = 0\) every expert gets the same share and all bars are equal. At \(\alpha = 1\) the hottest expert receives about a quarter of all slots and its GPU does more than twice the average work. Turn on redundant copies: the hot experts are split between two GPUs and the peak falls close to the mean.
Other tricks at serving time
- Offloading for one user. On a single small GPU, keep most experts in CPU memory and copy the needed ones in. Because consecutive tokens often reuse experts, a cache of recently used experts on the GPU helps (Eliseev and Mazur, cited by the Mixtral paper). [Certain] See session 130.
- Quantize the experts. Experts are 87% to 97% of the bytes (table above), so expert weights are the first place to quantize; DeepSeek-V3 already ships FP8 weights. [Certain]
- Separate prefill and decode. Prefill has many tokens per expert (compute-bound); decode has few (memory-bound). DeepSeek-V3 uses different units for the two, which is prefill–decode disaggregation (session 137). [Certain]
- Grouped GEMM kernels. Run all experts of a layer in one kernel call over variable-size groups of tokens, instead of a loop of small matrix multiplies (MegaBlocks style, see session 29). [Certain]
- input: on each GPU \(g\): hidden states \(X_g\) of its own requests; expert placement \(\text{host}(e)\) (with redundant copies)
- \(\text{scores} \leftarrow\) router\((X_g)\); \(\text{idx}, w \leftarrow\) top-\(k\)(scores) // local, cheap
- for each (token, expert \(e\)): pick a copy of \(e\) (the least-loaded host) // redundant experts
- all-to-all dispatch: send each token copy to its host GPU // FP8 in DeepSeek-V3, formula (96.11)
- on each GPU: grouped GEMM over its experts with the tokens it received // \(bk/E\) tokens per expert
- all-to-all combine: send results back to the token’s home GPU
- \(y \leftarrow \sum_j w_j \cdot \text{result}_j\) (+ shared expert output) // layer time \(\propto \max_g \text{load}_g\), (96.12)
- overlap: while micro-batch A is in steps 4 to 6, run attention of micro-batch B
- log per-expert token counts; every few minutes recompute which experts get redundant copies
In code
A runnable, deterministic PyTorch script with three parts: MoEfy a small ReLU FFN and measure the error for each number of experts; upcycle a SwiGLU FFN and check the step-0 output with and without renormalization; count Mixtral from its config and apply formula (96.8).
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # ---------- Part 1: MoEfication of a ReLU FFN ---------- d, dff, k_exp = 16, 64, 8 # 8 experts of 8 neurons each de = dff // k_exp topics = F.normalize(torch.randn(4, d), dim=1) owner = torch.arange(dff) % 4 # each neuron "belongs" to one topic W1 = 2.0 * topics[owner].T + 0.3 * torch.randn(d, dff) # (d, dff) b1 = -torch.ones(dff) W2 = torch.randn(dff, d) / dff ** 0.5 b2 = torch.zeros(d) def ffn(x): # dense FFN, eq. (96.1) return F.relu(x @ W1 + b1) @ W2 + b2 def balanced_kmeans(V, k, iters=20): # V: (n, d) rows = W1 columns n = V.shape[0]; size = n // k C = V[torch.randperm(n)[:k]].clone() for _ in range(iters): dist = torch.cdist(V, C) # (n, k) order = dist.flatten().argsort() lab = torch.full((n,), -1); cnt = torch.zeros(k, dtype=torch.long) for idx in order.tolist(): i, c = divmod(idx, k) if lab[i] < 0 and cnt[c] < size: lab[i] = c; cnt[c] += 1 C = torch.stack([V[lab == c].mean(0) for c in range(k)]) return lab def split(lab): # eq. (96.2): permute, then cut perm = torch.cat([torch.nonzero(lab == c).flatten() for c in range(k_exp)]) return (W1[:, perm].reshape(d, k_exp, de), b1[perm].reshape(k_exp, de), W2[perm].reshape(k_exp, de, d)) def moefied(x, n, W1e, b1e, W2e): # eq. (96.4) with ideal selection a = F.relu(torch.einsum('bd,dke->bke', x, W1e) + b1e) # (B, k, de) score = a.sum(-1) # sum of positive values keep = torch.zeros_like(score).scatter(1, score.topk(n, dim=1).indices, 1.0) return torch.einsum('bke,ked->bd', a * keep[..., None], W2e) + b2 clustered = split(balanced_kmeans(W1.T, k_exp)) random_sp = split(torch.randperm(dff) % k_exp) x = topics[torch.arange(256) % 4] + 0.3 * torch.randn(256, d) y = ffn(x) print("active neurons per input: %.1f of %d" % ((x @ W1 + b1 > 0).float().sum(1).mean(), dff)) for n in [8, 4, 2, 1]: e_c = ((moefied(x, n, *clustered) - y).norm(dim=1) / y.norm(dim=1)).mean() e_r = ((moefied(x, n, *random_sp) - y).norm(dim=1) / y.norm(dim=1)).mean() print(f"experts {n}/8 ({n/8:.1%} of FFN): rel. error clustered {e_c:.3f}, random {e_r:.3f}") # ---------- Part 2: sparse upcycling of a SwiGLU FFN ---------- class SwiGLU(nn.Module): def __init__(s, d, f): super().__init__(); s.g = nn.Linear(d, f, bias=False) s.u = nn.Linear(d, f, bias=False); s.o = nn.Linear(f, d, bias=False) def forward(s, x): return s.o(F.silu(s.g(x)) * s.u(x)) class UpcycledMoE(nn.Module): def __init__(s, dense, E, k, renorm=True): super().__init__() d = dense.g.in_features s.experts = nn.ModuleList([SwiGLU(d, dense.g.out_features) for _ in range(E)]) for e in s.experts: e.load_state_dict(dense.state_dict()) # copy, eq. (96.5) s.router = nn.Linear(d, E, bias=False) nn.init.normal_(s.router.weight, std=0.02) # new, random s.k, s.renorm = k, renorm def forward(s, x): p = s.router(x).softmax(-1) w, idx = p.topk(s.k, dim=-1) if s.renorm: w = w / w.sum(-1, keepdim=True) y = torch.zeros_like(x) for j in range(s.k): for e in range(len(s.experts)): m = idx[:, j] == e if m.any(): y[m] += w[m, j, None] * s.experts[e](x[m]) return y dense = SwiGLU(64, 256) xs = torch.randn(512, 64) with torch.no_grad(): yd = dense(xs) for renorm in [True, False]: moe = UpcycledMoE(dense, E=8, k=2, renorm=renorm) ym = moe(xs) ratio = (ym.norm(dim=1) / yd.norm(dim=1)).mean() print(f"renorm={renorm}: max |moe-dense| = {(ym-yd).abs().max():.2e}, output/dense = {ratio:.3f}") # ---------- Part 3: serving math for Mixtral 8x7B (config.json) ---------- d, L, V, ff, E, k = 4096, 32, 32000, 14336, 8, 2 expert = 3 * d * ff other = 2 * V * d + d + L * (2 * d * d + 2 * d * 1024 + 2 * d + d * E) total, active = other + L * E * expert, other + L * k * expert print(f"total {total/1e9:.2f}B active {active/1e9:.2f}B BF16 weights {2*total/1e9:.1f} GB") for b in [1, 4, 16, 64]: touched = E * (1 - (1 - k / E) ** b) # eq. (96.8) print(f"batch {b:3d}: experts touched per layer {touched:.2f} of {E}")
Output (CPU, PyTorch 2.8):
active neurons per input: 22.4 of 64 experts 8/8 (100.0% of FFN): rel. error clustered 0.000, random 0.000 experts 4/8 (50.0% of FFN): rel. error clustered 0.039, random 0.458 experts 2/8 (25.0% of FFN): rel. error clustered 0.225, random 0.698 experts 1/8 (12.5% of FFN): rel. error clustered 0.641, random 0.837 renorm=True: max |moe-dense| = 5.96e-08, output/dense = 1.000 renorm=False: max |moe-dense| = 3.88e-01, output/dense = 0.299 total 46.70B active 12.88B BF16 weights 93.4 GB batch 1: experts touched per layer 2.00 of 8 batch 4: experts touched per layer 5.47 of 8 batch 16: experts touched per layer 7.92 of 8 batch 64: experts touched per layer 8.00 of 8
Read it: with all 8 experts the MoEfied FFN is exact (formula 96.2). With the k-means split, 4 of 8 experts give only 3.9% error, against 45.8% for a random split: grouping neurons that fire together is what makes MoEfication work. The renormalized upcycled layer equals the dense FFN to float precision; without renormalization its output starts at about 30% of the dense output (8 experts, top-2, small logits, so \(G\) is near \(2/8\)).
Where this is used
| Setting | What is done | Why it fits |
|---|---|---|
| Bigger model from an existing dense checkpoint | Sparse upcycling (Qwen1.5-MoE-A2.7B from Qwen-1.8B; T5 and ViT in the paper) | Reuses sunk pre-training cost; about 50% extra compute beats dense continuation |
| Cheaper inference of an existing ReLU model, no new parameters | MoEfication with a small router | 10% to 30% of FFN parameters per input, over 95% of quality (T5, BERT) |
| Domain experts trained by separate teams | Branch-Train-MiX: merge their FFNs into one MoE | Embarrassingly parallel training, then one model with token-level routing |
| Open-weight chat models (Mixtral, Qwen MoE, DeepSeek-V3) | Batched serving with EP or TP, FP8 or INT4 experts | Fewer FLOPs per token, so high throughput at large batch |
| Local single-GPU use of a big MoE | Expert offloading with an LRU cache of experts | Batch 1 touches only \(k\) experts per layer; tokens often reuse experts |
| Video understanding with a VLM (classification, action recognition, key-moment detection) | MoE language model behind a vision encoder; long video prefill gives many tokens per expert | Prefill of thousands of visual tokens is compute-bound, so active-parameter FLOPs are what you pay |
| Offline captioning or tagging of a video archive; Video-RAG indexing | Very large decode batches across an EP group | Large \(b\) pushes tokens per expert toward the ridge (96.10) |
Common mistakes
- Sizing GPUs by active parameters. Memory must hold every expert: Mixtral needs 93.4 GB in BF16, not 26 GB.
- Expecting batch-1 speed at batch 32. By formula (96.8), a batch of 16 touches 7.9 of Mixtral’s 8 experts per layer; bytes per step approach the total.
- Thinking upcycling is free. The step-0 output can be a small fraction of the dense one (96.6), and the model needs real extra training (about 50% of the dense budget in the paper) to pay off.
- Copying experts without breaking symmetry. Identical experts with the same tokens stay identical. The random router breaks it; small noise on the copies helps more.
- MoEfying a SwiGLU model and expecting ReLU results. SiLU gives few exact zeros, so dropping experts always costs accuracy unless you retrain.
- Counting “8x7B” as 56B. Only the FFNs are copied; it is 46.7B.
- Ignoring the slowest GPU in EP. The layer waits for the busiest GPU (96.12); average load says nothing about time.
- Forgetting the all-to-all. Each MoE layer moves about \(k\,d\,(s_d + s_c)\) bytes per token twice across GPUs; on slow links this can exceed the expert compute.
Interview questions
What is the difference between total and active parameters, and what does each one determine?basic
What is MoEfication, and why does it work on T5 but less well on LLaMA-style models?core
Describe sparse upcycling step by step.basic
Why does an upcycled model lose quality at step 0, and how can you avoid it?core
How do identical expert copies become specialized?core
Estimate the decode time per token of Mixtral 8x7B at batch 1 on 2 H100s.core
Why is MoE decode at moderate batch sizes not much faster than a dense model of the same total size?advanced
What is expert parallelism, and what are its two main costs?core
How does DeepSeek-V3 deploy its MoE for inference?advanced
Upcycling or training an MoE from scratch: how do you choose?advanced
Your Mixtral service has p99 latency spikes only under heavy load. What MoE-specific causes would you check?advanced
Exercises
- A dense FFN has \(d_{ff} = 4096\) neurons, cut into experts of \(d_e = 32\). How many experts are there, and what fraction of FFN FLOPs does a token use if the router picks 32 of them?
- Upcycle with \(E = 16\), top-2, softmax over all experts then top-\(k\), and a router whose logits are all equal. What is \(G\) at step 0? What if the gates are renormalized?
- Using formula (96.7) and the Mixtral table, how many bytes does Mixtral need in INT4 (0.5 bytes per weight, scales ignored)? Does it fit one 80 GB GPU?
- For DeepSeek-V3 (\(E = 256\), \(k = 8\)), compute \(\bar{E}(b)\) for \(b = 1\) and \(b = 32\).
- With formula (96.10), what batch makes Qwen1.5-MoE-A2.7B’s experts compute-bound on one H100 in BF16?
- Use Figure 4 at \(\alpha = 0.6\): what is the imbalance without and with redundant copies? How much faster is the layer with copies?
Answerscheck
Key takeaways
- A dense FFN is exactly a sum of neuron groups (96.2). MoEfication groups co-active neurons and adds a router (96.4): 10% to 30% of FFN parameters keep over 95% of quality for ReLU models.
- Sparse upcycling copies the FFN into \(E\) experts and adds a random router (96.5). At step 0 the FFN output is scaled by \(G\) (96.6); renormalized top-\(k\) gates give an exact start. About 50% extra compute beat dense continuation in the paper.
- Memory follows total parameters, FLOPs follow active ones (96.7): Mixtral 46.7B / 12.9B, Qwen1.5-MoE 14.3B / 2.7B, DeepSeek-V3 671B / 37.6B.
- A decode batch touches \(E(1-(1-k/E)^b)\) experts per layer (96.8), so bytes per step rise quickly toward the total; the MoE’s FLOP advantage pays off only at very large batches, about \(E/k\) times the dense ridge batch (96.10).
- Expert parallelism adds two all-to-all exchanges per MoE layer (96.11) and waits for the busiest GPU (96.12); redundant copies of hot experts, as in DeepSeek-V3, cut the straggler effect.
Key formulas
References for this session
- Zhang et al. (2021). MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
- Komatsuzaki et al. (2022). Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
- Jiang et al. (2024). Mixtral of Experts
- Du et al. (2021). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Zhang et al. (2021). MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
- Komatsuzaki et al. (2022). Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
- Jiang et al. (2024). Mixtral of Experts
- Du et al. (2021). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- Fedus et al. (2021). Switch Transformers
- Qwen Team (2024). Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters (blog)
- Sukhbaatar et al. (2024). Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
- Zhu, Qu, Dong, Ruan, Tong, He, Cheng (2024). LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
- Eliseev and Mazur (2023). Fast Inference of Mixture-of-Experts Language Models with Offloading
- Mistral AI (2023). Mixtral-8x7B-v0.1: config.json
- DeepSeek-AI (2024). DeepSeek-V3: config.json
- Dai et al. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications