LLM & VLM courseBlog
29Track 2: Attention and transformers · session 14 of 15

Mixture of Experts (MoE)

A Mixture of Experts (MoE) layer replaces the one feed-forward network of a block with many smaller networks, called experts, and a small router that sends each token to only a few of them. The model then has many parameters but uses only a small part of them for each token. This session shows how the router works, how to count total and active parameters, how to keep the experts evenly used, and how experts are spread over GPUs.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 21 and 25 (the FFN, counting parameters and FLOPs); session 4 helps (gradients)

What you will learn

  • What an MoE layer computes: a router scores the experts, picks the top-k for each token, and mixes their outputs with gate weights
  • Three gating styles: softmax then top-k (Switch, GShard), top-k then softmax (Mixtral), and sigmoid scores (DeepSeek-V3)
  • Total vs active parameters, with exact counts for Mixtral 8x7B, Qwen3-30B-A3B and DeepSeek-V3, and why MoE is cheap in FLOPs but heavy in memory
  • Why routers collapse onto a few experts, and three fixes: the load-balancing loss, the router z-loss, and auxiliary-loss-free bias balancing
  • Capacity factor, dropped tokens and padding, and why many recent models are dropless
  • Fine-grained and shared experts (DeepSeekMoE), and why more, smaller experts give more combinations
  • A first look at expert parallelism: experts on different GPUs, and the two all-to-all exchanges per MoE layer

Why this matters

Mixtral 8x7B has about 47 billion parameters, but each token uses only about 13 billion of them. DeepSeek-V3 has 671 billion parameters and uses 37 billion per token. [Certain] Both are Mixture of Experts (MoE) models. This session explains how a model can “own” many parameters but “use” only a few for each token, and what this costs.

In session 21 we saw that the FFN holds about two thirds of the weights of a dense block and does most of its FLOPs. In session 25 we saw that a dense model spends about \(2N\) FLOPs per token in the forward pass, where \(N\) is the number of parameters. So in a dense model, more parameters always means more compute per token. MoE breaks this link. It replaces the one FFN with many FFNs, called experts, and adds a small router that picks only a few experts for each token. Parameters grow with the number of experts; compute grows only with the number of experts that are used.

This is now a main design choice for large open models: Mixtral, DeepSeek-V2 and V3, Qwen3 (the “A3B” and “A22B” models), and others are MoE models. [Certain] It also brings new problems that do not exist in dense models:

  • Memory. All experts must be stored, even if each token uses only a few. Mixtral needs about 93 GB for its weights in BF16, like a dense 47B model, while it computes like a 13B model.
  • Balance. If the router sends most tokens to a few experts, the other experts never learn, and the busy ones become a bottleneck. Training needs extra rules to keep the load even.
  • Communication. Large MoE models put different experts on different GPUs, so tokens must travel between GPUs twice in every MoE layer.

After this session, the lab (session 30) asks you to build a dense GPT; an MoE layer is a natural extension you can add to it. Routing, MoE inference and expert parallelism come back in depth in session 95, session 96 and session 49.

From one FFN to many experts

A transformer block has two parts: attention, which mixes information between tokens, and the FFN, which works on each token alone. Because the FFN works on each token alone, nothing stops us from giving different tokens different FFNs. That is the whole idea of MoE.

An MoE layer has \(E\) experts \(\mathrm{FFN}_1, \dots, \mathrm{FFN}_E\). Each expert is a normal FFN (in modern LLMs, a SwiGLU FFN). A router (also called a gate or gating network) looks at the token’s hidden vector \(\mathbf{x}\) and chooses \(k\) experts. Only those \(k\) experts run. Their outputs are added up with weights \(g_i\) that the router also gives:

\[ \mathrm{MoE}(\mathbf{x}) = \sum_{i \in \mathcal{T}(\mathbf{x})} g_i(\mathbf{x})\, \mathrm{FFN}_i(\mathbf{x}), \qquad |\mathcal{T}(\mathbf{x})| = k \tag{29.1} \]
Read the symbols
\(\mathbf{x} \in \R^{d}\)
the hidden vector of one token at this layer (after the normalization, as for a normal FFN)
\(E\)
the number of experts in the layer (8 in Mixtral, 128 in Qwen3-30B-A3B, 256 in DeepSeek-V3)
\(k\)
how many experts each token uses (“top-k”): 1 in Switch, 2 in Mixtral, 8 in Qwen3 and DeepSeek-V3
\(\mathcal{T}(\mathbf{x})\)
the set of the \(k\) chosen experts for this token
\(g_i(\mathbf{x})\)
the gate weight of expert \(i\): how much its output counts; experts outside \(\mathcal{T}\) have weight 0 and are not computed

The MoE layer takes the place of the FFN, so the block still adds its output to the residual stream: \(\mathbf{h} \leftarrow \mathbf{h} + \mathrm{MoE}(\mathrm{Norm}(\mathbf{h}))\). Attention usually stays dense: every token uses the same attention weights. Some models (DeepSeek-V3 is one) keep the first few blocks dense and use MoE in the rest. [Certain]

The idea is old (Jacobs, Jordan and others in 1991). The modern form starts with Shazeer et al. (2017), who put a sparsely-gated MoE layer with up to 137 billion parameters between LSTM layers. GShard (2020) put MoE into the transformer with top-2 routing and scaled a translation model beyond 600 billion parameters. Switch Transformer (2021) used top-1 routing; its Switch-Base model has 7B parameters but the same FLOPs per sequence as the 0.2B T5-Base, and its largest model, Switch-C, has 1.6 trillion parameters with 2,048 experts. [Certain]

The router: scores, top-k and gate weights

The router is a single linear layer with one row per expert, \(W_r \in \R^{E \times d}\). It gives each expert a score (a logit) for this token. Most models then turn the scores into probabilities with a softmax:

\[ \mathbf{z} = W_r \mathbf{x} \in \R^{E}, \qquad \mathbf{p} = \softmax(\mathbf{z}), \qquad \mathcal{T}(\mathbf{x}) = \operatorname{TopK}(\mathbf{p}, k) \tag{29.2} \]

There are three common ways to turn the scores into the gate weights \(g_i\):

  • Softmax over all experts, keep the top-k as they are (Switch, GShard): \(g_i = p_i\). With top-1, the single gate is less than 1, and this is how the router gets a gradient: the output is \(p_i \cdot \mathrm{FFN}_i(\mathbf{x})\), so changing \(p_i\) changes the loss.
  • Top-k, then softmax over only those k (Mixtral, and many others): the chosen weights are renormalized so they sum to 1.
  • Sigmoid scores, then normalize the top-k (DeepSeek-V3): each expert gets its own score \(s_i = \sigma(z_i)\) between 0 and 1, without competition from the others, and the chosen ones are normalized.
\[ g_i = \frac{p_i}{\sum_{j \in \mathcal{T}} p_j} = \frac{e^{z_i}}{\sum_{j \in \mathcal{T}} e^{z_j}} \quad (i \in \mathcal{T}), \qquad g_i = 0 \ \text{ otherwise} \tag{29.3} \]

Formula (29.3) is the Mixtral rule; it is the same as a softmax over only the \(k\) chosen logits. The choice of \(\mathcal{T}\) itself (the top-k) has no gradient: it is a hard decision. The router learns only through the gate weights of the experts it picked. Shazeer et al. also added random noise to the scores during training (“noisy top-k”), so that experts that are rarely picked still get a chance. [Certain]

Worked example: one token, 8 experts, top-2

The router gives the logits \(\mathbf{z} = (2.0,\ 1.5,\ 0.3,\ -0.5,\ 0.1,\ 1.0,\ -1.0,\ 0.2)\). The softmax over all 8 is \(\mathbf{p} = (0.384,\ 0.233,\ 0.070,\ 0.032,\ 0.057,\ 0.141,\ 0.019,\ 0.063)\).

Top-2 picks experts 1 and 2. Switch-style gates keep \(0.384\) and \(0.233\) (sum 0.617). Mixtral-style gates (29.3) renormalize: \(0.384/0.617 = 0.622\) and \(0.233/0.617 = 0.378\). Check: \(e^{2}/(e^{2} + e^{1.5}) = 1/(1 + e^{-0.5}) = 0.622\).

Sigmoid-style (DeepSeek-V3): \(\sigma(2.0) = 0.881\), \(\sigma(1.5) = 0.818\), normalized to \(0.519\) and \(0.481\). The same two experts are chosen, but the mix is more even. Only 2 of the 8 experts run: a quarter of the expert FLOPs.

router score of an expert not chosenchosen expert (runs)
Figure 1. The router for one token. The bars are the scores of the 8 experts (softmax probabilities, or sigmoid scores in the DeepSeek-V3 mode). The top-k experts are chosen; the number above each chosen bar is its gate weight \(g_i\) in formula (29.1). The router logits are hand-made for six example tokens.
How to read Figure 1

Pick a token, then change \(k\). The chosen experts are always the ones with the highest scores, in all three modes. What changes between modes is only the gate weights. In the Switch mode the gates do not sum to 1. In the Mixtral mode they always sum to 1, and with \(k = 1\) the single gate is exactly 1 (so the router would get no gradient from it: this is why top-1 models keep \(g_i = p_i\)). Look at the token “the”: its scores are almost flat, so the router is unsure, and a small change in the logits could switch the chosen experts. Tokens like this are where routing is unstable during training.

Figure 1 starts from logits that we made up. Figure 2 runs the whole layer for real: a token vector \(\mathbf{x}\) with \(d = 4\), a router \(W_r\) with one row per expert, and small SwiGLU experts (hidden size 3) with fixed random weights. Each step does one line of the algorithm and puts the live numbers into formulas (29.1) to (29.3).

2
token vector \(\mathbf{x}\) (flowing)router row being usedchosen expertweighted output \(g_i\,\mathbf{o}_i\) added to \(\mathbf{y}\)
  1. input: \(\mathbf{x} \in \R^4\); router \(W_r\) \((E \times 4)\); experts \(\mathrm{FFN}_1, \dots, \mathrm{FFN}_E\)
  2. for \(i = 1, \dots, E\): \(z_i \leftarrow \mathbf{w}_i \cdot \mathbf{x}\) // (29.2)
  3. \(p \leftarrow \softmax(z)\)
  4. \(\mathcal{T} \leftarrow \operatorname{TopK}(p, k)\) // a hard choice, no gradient
  5. \(g_i \leftarrow p_i / \sum_{j \in \mathcal{T}} p_j\) for \(i \in \mathcal{T}\) // (29.3)
  6. \(\mathbf{y} \leftarrow \mathbf{0}\); for \(i \in \mathcal{T}\): \(\mathbf{o}_i \leftarrow \mathrm{FFN}_i(\mathbf{x})\) // only \(k\) experts run
  7. \(\mathbf{y} \leftarrow \mathbf{y} + g_i\, \mathbf{o}_i\) // (29.1)
  8. return \(\mathbf{y}\) // the other \(E - k\) experts never ran
Figure 2. One MoE layer, computed live for one token. Left: the token vector \(\mathbf{x}\). The router (one row \(\mathbf{w}_i\) per expert) gives the logits \(z\), the scores \(p\) and the gates \(g\) in the columns. The chosen experts (orange) run a SwiGLU FFN; the four small bars next to an expert are its output \(\mathbf{o}_i\). The weighted outputs flow into \(\Sigma\), and the result \(\mathbf{y}\) is on the right. All weights come from a fixed seed, so every run gives the same numbers.
How to read Figure 2

Press Play, or Step through it. The blue dot is the token. It first visits every router row (line 2), because all \(E\) logits are needed to choose. After the top-k (line 4) the rows of the other experts fade: their FFNs are never computed. Then the dot goes to each chosen expert (line 6), and a green dot carries \(g_i \mathbf{o}_i\) to \(\Sigma\) (line 7); watch \(\mathbf{y}\) change. Change the gating style: the chosen experts stay the same, only \(g\) and \(\mathbf{y}\) change. Token 4 is a short vector, so all its logits are close to 0 and the router is almost uniform. The last step checks the sparse result against a dense loop that runs all \(E\) experts with \(g_i = 0\) for the unchosen ones: they agree exactly.

Total vs active parameters

We now need two numbers for every MoE model. The total parameters are everything you must store. The active parameters are the ones one token actually uses in the forward pass. For a model with \(L\) MoE layers, each with \(E\) experts of \(N_{\text{exp}}\) parameters and top-\(k\) routing:

\[ N_{\text{total}} = N_{\text{other}} + L\,\big(E \cdot N_{\text{exp}} + E\,d\big) \tag{29.4} \]
\[ N_{\text{active}} = N_{\text{other}} + L\,\big(k \cdot N_{\text{exp}} + E\,d\big), \qquad \text{FLOPs per token} \approx 2\,N_{\text{active}} \tag{29.5} \]
Read the symbols
\(N_{\text{other}}\)
all the dense parameters: embeddings, attention, norms, output head, and any dense FFN layers
\(N_{\text{exp}}\)
the parameters of one expert; for a SwiGLU expert with hidden size \(h\), \(N_{\text{exp}} = 3\,d\,h\)
\(E\,d\)
the router of each layer: one row of \(d\) numbers per expert (tiny)
\(2\,N_{\text{active}}\)
the forward FLOPs per token from session 25, now counted with the parameters that are used, not the ones that are stored (the attention-score FLOPs that grow with context come on top, as before)

Model names often show both numbers: “Qwen3-30B-A3B” means about 30B total and 3B active. Mixtral’s name “8x7B” is misleading: the total is not \(8 \times 7 = 56\)B, because only the FFNs are copied 8 times; attention and embeddings are shared.

Worked example: Mixtral 8x7B, counted from its config

From the Hugging Face config: \(L = 32\) layers, \(d = 4096\), expert hidden size \(h = 14{,}336\), 32 query heads and 8 key-value heads of size 128, a 32,000-token vocabulary with separate input and output embeddings, \(E = 8\), \(k = 2\). [Certain]

One SwiGLU expert: \(3 \times 4096 \times 14{,}336 = 176.2\)M. Attention per layer: \(2 \times 4096^2 + 2 \times 4096 \times 1024 = 41.9\)M. Embeddings and head: \(2 \times 32{,}000 \times 4096 = 262\)M.

Total: \(32 \times (41.9 + 8 \times 176.2)\)M plus embeddings, norms and routers \(= 46.70\)B. Active: \(32 \times (41.9 + 2 \times 176.2)\)M plus the same \(= 12.88\)B. This matches the 47B and 13B in the Mixtral paper. The experts are 96.6% of all parameters.

So Mixtral does about \(2 \times 12.88 = 25.8\) GFLOPs per token in the forward pass (before attention scores), like a 13B dense model. But its weights take \(46.7 \times 2 = 93.4\) GB in BF16, like a 47B dense model. It does not fit on one 80 GB GPU in BF16.

The same count, done in Python for the other two models in Figure 3, gives 30.53B total and 3.35B active for Qwen3-30B-A3B (48 layers, 128 experts with \(h = 768\), top-8), and 671.0B total and 37.6B active for DeepSeek-V3 (61 layers, the first 3 dense, then 256 routed experts with \(h = 2048\), 1 shared expert, top-8). Both match the reported 30.5B / 3.3B and 671B / 37B. [Certain]

Cheap per token, heavy in memory

For training, active parameters set the FLOPs, so an MoE model reaches a given loss with less compute than a dense model of the same total size. For serving, the picture is more mixed:

  • Memory follows the total. Every expert must be in GPU memory (or be loaded when needed, which is slow).
  • Decoding one token at a time is usually limited by how fast weights are read from memory, not by FLOPs. With a batch of one, only the chosen experts are read, so it is fast. With a large batch, different tokens choose different experts, and soon almost all experts are read at every step. Then the speed advantage per step shrinks, while the FLOPs advantage stays. [Likely]
  • Fine-grained experts make small matrix multiplications that use the GPU less well than one big dense one. Special kernels (grouped GEMM, MegaBlocks) exist for this. [Certain]
embeddings, attention, norms, dense layersrouted expertsshared experts
Figure 3. Total and active parameters from formulas (29.4) and (29.5), with the real shapes of three MoE models. The presets reproduce the published totals. Change the number of experts, \(k\), shared experts, or split every expert into \(m\) smaller ones (which multiplies \(E\) and \(k\) by \(m\) and divides the expert size by \(m\)).
How to read Figure 3

The top bar is what you store; the bottom bar is what one token uses. Raise \(E\) with \(k\) fixed: the top bar grows, the bottom bar does not. That is the MoE trade. Then try the split slider: total and active stay exactly the same, but the number of possible expert combinations per token explodes. This is the idea of fine-grained experts in DeepSeekMoE, explained later. The memory line assumes 2 bytes per parameter (BF16); DeepSeek-V3 is released in FP8 (1 byte per parameter), which halves it. [Certain]

Load balancing: keeping all experts busy

A new MoE model starts with a random router. Some experts get slightly more tokens by chance. Those experts get more training, so they become slightly better, so the router (which learns to pick experts that lower the loss) sends them even more tokens. This rich-get-richer loop is called routing collapse: a few experts get almost all tokens, and the others stay untrained. The model then behaves like a small dense model with many unused parameters. In expert parallelism (below) collapse is also slow: the GPU with the popular expert works while the others wait.

The auxiliary load-balancing loss

The most common fix, from Switch Transformer, adds a small extra loss. For a batch of \(T\) tokens, measure two things for each expert \(i\):

\[ \mathcal{L}_{\text{bal}} = \alpha\, E \sum_{i=1}^{E} f_i\, P_i, \qquad f_i = \frac{1}{kT} \sum_{t=1}^{T} \mathbb{1}\big[i \in \mathcal{T}(\mathbf{x}_t)\big], \qquad P_i = \frac{1}{T} \sum_{t=1}^{T} p_i(\mathbf{x}_t) \tag{29.6} \]
Read the symbols
\(f_i\)
the share of token assignments that went to expert \(i\) (counted from the hard top-k choice, so it has no gradient)
\(P_i\)
the average router probability for expert \(i\) over the batch (this part has a gradient)
\(\alpha\)
the weight of this loss; Switch used \(10^{-2}\), the Mixtral config has 0.02, Qwen3-30B-A3B has 0.001 [Certain]
\(E\)
a scale factor so that the loss equals \(\alpha\) when the load is perfectly even

Why does this work? Both \(f\) and \(P\) sum to 1. If both are uniform (\(1/E\) each), the sum is \(E \cdot E \cdot (1/E)^2 = 1\). If one expert gets more tokens and more probability, the sum gets larger. The gradient of the loss with respect to \(P_i\) is \(\alpha E f_i\): it pushes down the router probability of each expert in proportion to how busy it already is. Busy experts lose probability; idle experts gain it.

Worked example: the balance loss for 4 experts

Even load: \(f = P = (0.25, 0.25, 0.25, 0.25)\). Then \(E \sum f_i P_i = 4 \times 4 \times 0.0625 = 1.00\).

Skewed load: \(f = (0.7, 0.1, 0.1, 0.1)\) and \(P = (0.6, 0.15, 0.15, 0.1)\). Then \(4 \times (0.42 + 0.015 + 0.015 + 0.01) = 1.84\).

The gradient with respect to \(P\) is \(4 f = (2.8, 0.4, 0.4, 0.4)\) (times \(\alpha\)): seven times stronger on the busy expert. With \(\alpha = 0.01\), the extra loss is only \(0.0184\), small next to a language-model loss of about 2, but it acts on every step.

The balance loss has a cost. Its gradient goes into the same router weights as the language-model loss, and it can pull routing away from what is best for quality. A large \(\alpha\) gives even loads but a worse model; a small one may not prevent collapse. [Certain]

The router z-loss

Routers have a second problem: their logits can grow very large. Large logits make the softmax very sharp and, in BF16, the exponentials lose precision, which can cause loss spikes. ST-MoE added the router z-loss, which keeps the logits small:

\[ \mathcal{L}_z = \beta\, \frac{1}{T} \sum_{t=1}^{T} \Big( \log \sum_{j=1}^{E} e^{z_{t,j}} \Big)^2 \tag{29.7} \]

The term inside is the log of the softmax denominator. It is large when any logit is large, and the square punishes it strongly. ST-MoE used \(\beta = 10^{-3}\). [Certain] For the logits of the worked example above, \(\log \sum_j e^{z_j} = 2.957\), so the per-token term is \(8.74\). Switch also computes the router in FP32 even when the rest of the model runs in BF16 (“selective precision”). [Certain]

Balancing without an extra loss

DeepSeek-V3 uses a different idea, auxiliary-loss-free balancing (Wang et al., 2024). Each expert gets a bias \(b_i\) that is added to its score only to choose the top-k. The gate weights still use the original score. After each step, the bias of an overloaded expert goes down a little, and the bias of an underloaded expert goes up:

\[ \mathcal{T}(\mathbf{x}) = \operatorname{TopK}\big(\{ s_i + b_i \},\ k\big), \qquad b_i \leftarrow b_i - \gamma \cdot \operatorname{sign}\big(\text{load}_i - \overline{\text{load}}\big) \tag{29.8} \]

No gradient is involved, so nothing pulls against the language-model loss. DeepSeek-V3 used \(\gamma = 0.001\) for the first 14.3T training tokens and 0 for the last 500B. It also keeps a tiny sequence-level balance loss with \(\alpha = 0.0001\), only to stop extreme imbalance inside one sequence. [Certain]

Figure 4. Live training of a toy router: 8 experts, top-1, 64 tokens per step, 200 steps (each Next press jumps 5 steps). Left: tokens per expert at the current step; the dashed line is the capacity at capacity factor 1.25. Right: the imbalance (busiest expert’s load divided by the average) over all steps. In the toy, an expert’s “skill” grows with the tokens it receives, and the router learns to prefer skilled experts, which creates the rich-get-richer loop.
How to read Figure 4

With no balancing, one expert takes almost all tokens within a few dozen steps: the imbalance goes near 8 (the maximum for 8 experts) and most tokens would be dropped at capacity factor 1.25. With the balance loss, set \(\alpha\) low (0.1) and collapse still happens; around 0.2 to 0.3 the loads stay near even; at 0.4 and above the push is too strong and the loads swing from step to step (the imbalance curve gets noisy again). Bias balancing also works if \(\gamma\) is large enough to act before the collapse. The strength values are for this toy only; real models use \(\alpha\) around 0.001 to 0.02, because their loss and logit scales are different. The toy uses a fixed random seed, so every run is the same.

Capacity factor and dropped tokens

Hardware likes fixed shapes. Early MoE systems (GShard, Switch) gave every expert a fixed buffer of slots per batch, called its expert capacity:

\[ C = \left\lfloor \mathrm{CF} \cdot \frac{k\, T}{E} \right\rfloor \tag{29.9} \]
Read the symbols
\(T\)
tokens in the batch (on this device); \(kT\) is the number of token-to-expert assignments
\(kT/E\)
the number of assignments each expert would get if the load were perfectly even
\(\mathrm{CF}\)
the capacity factor: extra room for uneven loads; Switch tested 1.0, 1.25 and 2.0 [Certain]

When more tokens choose an expert than it has slots, the extra ones are dropped: the expert does not process them. In Switch, “computation is skipped and the token representation is passed directly to the next layer through the residual connection”. [Certain] The token is not lost from the sequence; it just gets no update from this layer (with top-2, it may still get its other expert). Slots that stay empty are padding: computed for nothing. A higher CF drops fewer tokens but wastes more compute and memory, and sends more data between GPUs.

Worked example: capacity in a real batch

A device holds \(T = 4096\) tokens, with \(E = 8\) experts, \(k = 2\) and \(\mathrm{CF} = 1.25\). Even load would be \(2 \times 4096 / 8 = 1024\) assignments per expert, so \(C = \lfloor 1.25 \times 1024 \rfloor = 1280\). If one popular expert is chosen 1,600 times, 320 assignments are dropped: \(320 / 8192 = 3.9\%\) of all assignments in this layer.

token being routed nowfirst choice, keptsecond choice, keptdropped (over capacity)empty slot (padding)
  1. \(C \leftarrow \lfloor \mathrm{CF} \cdot kT / E \rfloor\); \(\text{count}_i \leftarrow 0\) for every expert // (29.9)
  2. for \(c = 1, \dots, k\): // all first choices before second choices
  3. for \(t = 0, \dots, T - 1\): \(i \leftarrow\) the \(c\)-th best expert of token \(t\)
  4. if \(\text{count}_i \lt C\): put \(t\) in the next slot of expert \(i\); \(\text{count}_i \leftarrow \text{count}_i + 1\)
  5. else: drop \((t \to i)\) // \(t\) keeps only its residual
  6. \(f_i \leftarrow (\text{tokens that chose } i) / kT\); \(P_i \leftarrow \operatorname{mean}_t\, p_i(\mathbf{x}_t)\)
  7. \(\mathcal{L}_{\text{bal}} / \alpha \leftarrow E \sum_i f_i P_i\) // (29.6); 1.0 when even
Figure 5. Sixteen tokens (numbered 0 to 15, top row) are routed one by one to 4 experts with the capacity rule of formula (29.9). Each expert row fills from the left; the vertical line is the capacity \(C\); a token that arrives at a full expert is dropped (red). First choices are placed before second choices, as in GShard. The last step computes the load-balancing loss (29.6) of this batch from the real router probabilities. The router logits are fixed random numbers plus a bias toward expert 1.
How to read Figure 5

Press Play and watch each token fly from the top row into its expert. The panel shows the token’s router probabilities, its chosen expert, and whether that expert still has a free slot (\(\text{count}_i \lt C\)). With the default settings (CF 1.25, \(k = 1\), bias 1), \(C = 5\), expert 1 is chosen by 8 of the 16 tokens, and its last 3 are dropped while experts 2, 3 and 4 have empty slots: you waste compute and lose tokens at the same time. The balance term is then 1.276 instead of 1.000. Set the bias to 0: the load is only mildly uneven, one token is dropped, and the balance term falls to 1.045. Raising CF fixes the drops but adds padding. With \(k = 2\), second choices go last and are dropped first, so a token rarely loses both of its experts.

Dropless MoE

Dropping tokens hurts quality and makes the output of a token depend on the other tokens in the batch (a token can be dropped because of its neighbours). Many recent systems are dropless: each expert processes all the tokens it receives, with variable-sized groups. MegaBlocks does this with block-sparse matrix kernels. DeepSeek-V3 says it drops no tokens in training or in inference. [Certain] Dropless training needs good balance even more, because one very busy expert now slows down the whole step instead of dropping work.

Fine-grained and shared experts

DeepSeekMoE (2024) changed the shape of the experts in two ways.

Fine-grained experts. Split each expert into \(m\) smaller experts (hidden size \(h/m\)) and choose \(mk\) of them instead of \(k\). Total and active parameters stay the same (try it in Figure 3). But the number of possible expert combinations for a token grows enormously: top-2 of 16 experts gives \(\binom{16}{2} = 120\) combinations; top-8 of 64 quarter-size experts gives \(\binom{64}{8} \approx 4.4\) billion. With more combinations, each small expert can specialize on a narrower kind of knowledge, and the router can combine them more flexibly. [Certain]

Shared experts. Some knowledge is needed by almost every token (common grammar, frequent words). If every routed expert must learn it, it is stored many times. DeepSeekMoE adds \(K_s\) shared experts that every token always uses, next to the routed ones:

\[ \mathbf{h}' = \mathbf{h} + \sum_{j=1}^{K_s} \mathrm{FFN}^{\text{shared}}_j(\mathbf{x}) + \sum_{i \in \mathcal{T}(\mathbf{x})} g_i\, \mathrm{FFN}_i(\mathbf{x}) \tag{29.10} \]

Here \(\mathbf{x} = \mathrm{Norm}(\mathbf{h})\). The shared experts are just a small dense FFN that runs for every token; the routed experts can then focus on specialized knowledge. DeepSeek-V3 has 1 shared and 256 routed experts per MoE layer, each with hidden size 2,048, and uses 8 routed experts per token: 9 experts run for each token. [Certain] Not all models use shared experts: Mixtral and Qwen3-30B-A3B have none. [Certain]

Other routing styles, in one paragraph

In expert-choice routing (Zhou et al., 2022) the experts choose tokens instead of tokens choosing experts: each expert takes its top-\(C\) tokens from the batch. The load is perfect by design, and a token may get zero or many experts. But at decoding time an expert would need to see future tokens to choose, so it fits encoders and training better than autoregressive decoding. [Likely] Routing variants are the topic of session 95.

A first look at expert parallelism

DeepSeek-V3’s experts do not fit on one GPU. Expert parallelism (EP) puts different experts on different GPUs: with 8 experts and 4 GPUs, GPU 0 holds experts 1 and 2, GPU 1 holds experts 3 and 4, and so on. Attention and the other dense parts are replicated or split in other ways. Each GPU starts with its own tokens. An MoE layer then has five steps:

Algorithm 1: one MoE layer with expert parallelism (forward)
  1. input: on each GPU \(r\): its tokens \(X_r\) (shape \(T_r \times d\)), its local experts, a copy of the router \(W_r\)
  2. \(\mathbf{p}, \mathcal{T} \leftarrow\) router on \(X_r\) // formulas (29.2), (29.3); purely local
  3. group the \(k T_r\) token copies by the GPU that owns each chosen expert
  4. dispatch: all-to-all; GPU \(r\) sends to GPU \(q\) the copies for experts on \(q\) // session 43
  5. each GPU runs its local experts on the tokens it received // grouped GEMM, one group per expert
  6. combine: all-to-all in reverse; every output goes back to the GPU that owns the token
  7. \(\mathbf{y}_t \leftarrow \sum_{i \in \mathcal{T}(t)} g_i\, \mathrm{out}_i(t)\); then \(\mathbf{h}_t \leftarrow \mathbf{h}_t + \mathbf{y}_t\) // formula (29.1)

An all-to-all is a collective where every GPU sends a different chunk to every other GPU, like a distributed transpose. Each token copy travels out once and back once, so the data moved per MoE layer is about

\[ \text{bytes per layer} \approx 2 \cdot T \cdot k \cdot d \cdot b \cdot \Big(1 - \frac{1}{G}\Big) \tag{29.11} \]
Read the symbols
\(T\)
tokens in the batch over all GPUs; \(k\) copies per token
\(d\)
hidden size; each copy is one vector of \(d\) numbers
\(b\)
bytes per number (2 for BF16, 1 for FP8)
\(1 - 1/G\)
with \(G\) GPUs and random routing, about \(1/G\) of copies stay on their own GPU and need no transfer
\(2\)
dispatch plus combine (the backward pass needs two more all-to-alls)
Worked example: DeepSeek-V3 traffic per token

With \(k = 8\), \(d = 7168\) and BF16, one token sends \(8 \times 7168 \times 2 = 114{,}688\) bytes (about 115 KB) out in the dispatch of one layer, and the same amount back in the combine. Over 58 MoE layers that is about 13.3 MB per token per forward pass, before the \((1 - 1/G)\) factor. This is why DeepSeek-V3 limits each token to experts on at most 4 nodes (so most traffic stays inside a node, on fast links), and sends the dispatch in FP8, which halves it. [Certain]

Figure 6. Expert parallelism on 4 GPUs. Each GPU holds 4 tokens and 2 of the 8 experts; each token picks 2 experts (fixed random router). Step through: route, dispatch (all-to-all), expert compute, combine (all-to-all back), weighted sum.
How to read Figure 6

In the dispatch step, lines that stay inside a GPU box are local; all others cross the network. The table in the readout counts how many token copies GPU \(r\) (row) sends to GPU \(q\) (column); the diagonal is local. Each expert’s count in the compute step is its load; the slowest GPU (the one with the most received copies) sets the time of the whole layer, which is why balance matters even more with EP.

Expert parallelism is combined with data, tensor and pipeline parallelism in real training. How these fit together, and how communication is hidden behind compute, is the subject of session 49.

In code

First the plain algorithm for one MoE layer on one device, then a runnable PyTorch version.

Algorithm 2: MoE layer with top-k routing, balance loss and capacity
  1. input: tokens \(X \in \R^{T \times d}\), router \(W_r\), experts \(\mathrm{FFN}_1..\mathrm{FFN}_E\), \(k\), capacity factor CF (or none)
  2. \(Z \leftarrow X W_r^\top\) in FP32; \(P \leftarrow \softmax(Z)\) row by row // (29.2)
  3. for each token: \(\mathcal{T}_t \leftarrow\) top-k of \(P_t\); \(g \leftarrow\) renormalized top-k probabilities // (29.3)
  4. \(f_i \leftarrow\) share of assignments to expert \(i\); \(\bar P_i \leftarrow\) mean of column \(i\) of \(P\)
  5. \(\mathcal{L}_{\text{bal}} \leftarrow E \sum_i f_i \bar P_i\); \(\mathcal{L}_z \leftarrow \text{mean}_t (\log\sum_j e^{Z_{tj}})^2\) // (29.6), (29.7)
  6. \(C \leftarrow \lfloor \mathrm{CF} \cdot kT / E \rfloor\) // (29.9)
  7. for each expert \(i\): take the tokens that chose \(i\) (only the first \(C\) if capacity is used); run \(\mathrm{FFN}_i\) on them; add \(g \cdot\) output into \(Y\) at their rows
  8. add the shared experts on all tokens, if any // (29.10)
  9. return \(Y\), and \(\alpha \mathcal{L}_{\text{bal}} + \beta \mathcal{L}_z\) to add to the training loss
import math, torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)

class Expert(nn.Module):                       # one SwiGLU FFN, as in session 21
    def __init__(self, d, h):
        super().__init__()
        self.w1 = nn.Linear(d, h, bias=False); self.w3 = nn.Linear(d, h, bias=False)
        self.w2 = nn.Linear(h, d, bias=False)
    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

class MoE(nn.Module):
    def __init__(self, d, h, E, k, n_shared=0, cf=None):
        super().__init__()
        self.E, self.k, self.cf = E, k, cf
        self.router = nn.Linear(d, E, bias=False)
        self.experts = nn.ModuleList(Expert(d, h) for _ in range(E))
        self.shared = nn.ModuleList(Expert(d, h) for _ in range(n_shared))
    def forward(self, x):                      # x: (T, d), tokens already flattened
        T, d = x.shape
        logits = self.router(x).float()        # router in FP32 (29.2)
        probs = logits.softmax(-1)
        top_p, top_i = probs.topk(self.k, dim=-1)              # (T, k)
        gates = top_p / top_p.sum(-1, keepdim=True)            # renormalize (29.3)
        # load-balancing loss (29.6) and router z-loss (29.7)
        f = F.one_hot(top_i, self.E).float().sum(1).mean(0) / self.k   # share of assignments
        P = probs.mean(0)
        aux = self.E * (f * P).sum()
        zloss = torch.logsumexp(logits, -1).pow(2).mean()
        y = torch.zeros_like(x)
        dropped = 0
        cap = None if self.cf is None else math.floor(self.cf * self.k * T / self.E)   # (29.9)
        for e in range(self.E):
            tok, slot = (top_i == e).nonzero(as_tuple=True)    # which tokens chose e
            if cap is not None and len(tok) > cap:
                dropped += len(tok) - cap
                tok, slot = tok[:cap], slot[:cap]               # first come, first served
            if len(tok) == 0:
                continue
            out = self.experts[e](x[tok])                       # run expert e only on its tokens
            y.index_add_(0, tok, gates[tok, slot, None].to(x.dtype) * out)
        for s in self.shared:                                   # shared experts see every token (29.10)
            y = y + s(x)
        return y, aux, zloss, f, dropped

d, h, E, k = 64, 128, 8, 2
moe = MoE(d, h, E, k)
x = torch.randn(256, d)
y, aux, zl, f, _ = moe(x)

# 1) the sparse loop equals a dense reference that runs all experts and masks
with torch.no_grad():
    probs = moe.router(x).softmax(-1); tp, ti = probs.topk(k, -1)
    G = torch.zeros(256, E).scatter(1, ti, tp / tp.sum(-1, keepdim=True))
    dense = sum(G[:, e:e+1] * moe.experts[e](x) for e in range(E))
print("max diff vs dense:", (y - dense).abs().max().item())
# 2) total vs active parameters (29.4, 29.5)
per_expert = sum(p.numel() for p in moe.experts[0].parameters())
total = sum(p.numel() for p in moe.parameters()); active = moe.router.weight.numel() + k * per_expert
print("total", total, "active per token", active)
print("aux", round(aux.item(), 3), "z-loss", round(zl.item(), 3), "load", [round(v, 3) for v in f.tolist()])
# 3) capacity factor 1.0: some assignments are dropped
_, _, _, _, dropped = MoE(d, h, E, k, cf=1.0)(x)
print("dropped at cf=1.0:", dropped, "of", 256 * k)

# 4) a tiny training run: with and without the balancing loss
def train(alpha, steps=300):
    torch.manual_seed(1)
    m = MoE(16, 32, 8, 2); opt = torch.optim.AdamW(m.parameters(), lr=3e-3)
    with torch.no_grad():
        m.router.weight[0] += 0.3                     # a small head start for expert 0
    W = torch.randn(16, 16)
    for _ in range(steps):
        xb = torch.randn(512, 16) + 1.0               # tokens share a common direction
        yb, aux, zl, f, _ = m(xb)
        loss = F.mse_loss(yb, torch.tanh(xb @ W)) + alpha * aux + 1e-3 * zl
        opt.zero_grad(); loss.backward(); opt.step()
    return round(f.max().item() * 8, 2)               # busiest expert's load / average load
print("max/mean load:", {a: train(a) for a in (0.0, 0.01, 0.1)})

Output (PyTorch 2.8, CPU): the sparse loop matches the dense reference exactly (max difference 0.0). The layer has 197,120 parameters but one token uses 49,664 (the router plus 2 of 8 experts). At initialization the balance term is 1.002 (close to the even-load value 1) and the z-loss is 5.031. With capacity factor 1.0, 16 of 512 assignments are dropped. In the tiny training run, the busiest expert ends at 4.0 times the average load without the balance loss (with \(k = 2\) this is the maximum: that expert is chosen by every token), 1.94 times with \(\alpha = 0.01\), and 1.14 times with \(\alpha = 0.1\). [Certain]

Where this is used

SystemMoE designWhy it matters
Switch Transformer (Google, 2021)top-1, capacity factor, balance loss \(\alpha = 10^{-2}\), router in FP32; up to 2,048 expertsshowed that MoE can be simple and train in BF16; up to 7x faster pretraining than T5 at equal compute [Certain]
Mixtral 8x7B (Mistral, 2024)8 experts, top-2, softmax over the top-2, no shared expert46.7B total, 12.9B active; a strong open model with the cost per token of a 13B model
Qwen3-30B-A3B (2025)128 small experts, top-8, no shared expert, balance loss 0.00130.5B total, 3.3B active: very cheap per token for its size
DeepSeek-V3 (2024)1 shared + 256 routed experts, top-8, sigmoid gates, bias balancing, node-limited routing, dropless671B total, 37B active; trained on 14.8T tokens in 2.788M H800 GPU hours [Certain]
Vision MoE (V-MoE, 2021)MoE layers inside a ViT; routes image patches to expertsmatched strong image classifiers with as little as half the inference compute [Certain]
MoE vision-language modelsMoE language backbone; the vision encoder usually stays denseimage and video tokens are many, so a cheap per-token backbone helps; DeepSeek-VL2 is one example [Likely]
Video understanding (classification, retrieval, key-moment detection)frame or patch tokens routed to experts in a video transformer or in the language backbone of a video LLMa long video gives tens of thousands of tokens; lower FLOPs per token make long-video models affordable, but memory still follows the total parameters [Speculation]

Common mistakes

  • Quoting total parameters as compute. Mixtral’s FLOPs per token are those of a 13B model, not 47B. But its memory is that of a 47B model. Always give both numbers.
  • Thinking “8x7B” means 56B. Only the FFNs are copied; attention and embeddings are shared, so the total is 46.7B.
  • Training without any balancing. The router collapses onto a few experts within a short time (Figure 4). Log the per-expert load from the first step.
  • Too large a balance-loss weight. Perfectly even loads with a bad router are not the goal. \(\alpha\) is usually around 0.001 to 0.02.
  • Router in BF16. Large router logits in low precision cause instability. Compute the router softmax in FP32 and consider the z-loss.
  • Renormalizing the gates with top-1. Then the gate is always 1, and the router gets no gradient from the main loss.
  • Forgetting that dropped tokens depend on the batch. With a capacity limit, the same prompt can give a different output when batched with other prompts. Use dropless kernels or a large capacity at inference.
  • Expecting experts to be topic experts. In Mixtral, the routing analysis found no clear topic specialization; choices follow syntax and token type more than subject. [Certain] Do not build a system that assumes “the biology expert”.

Interview questions

What is a Mixture of Experts layer, and which part of the transformer does it replace?basic
It replaces the FFN of a block with \(E\) expert FFNs and a router. The router (a linear layer \(W_r \in \R^{E \times d}\)) scores the experts for each token, the top-k experts run, and their outputs are added with gate weights (29.1). The FFN is chosen because it holds most of the parameters and works on each token separately, so each token can go to different experts. Attention is usually kept dense.
Total vs active parameters: define both and compute them for Mixtral 8x7B.basic
Total parameters are all stored weights; active parameters are those used for one token (shared parts plus the router plus \(k\) experts per layer), formulas (29.4) and (29.5). Mixtral: 32 layers, experts of \(3 \times 4096 \times 14{,}336 = 176.2\)M, attention 41.9M per layer, embeddings 262M. Total \(= 46.7\)B, active (2 of 8 experts) \(= 12.9\)B. Compute per token follows active (\(\approx 2 \times 12.9\) GFLOPs forward); memory follows total (93 GB in BF16).
Why do MoE models need a load-balancing mechanism?core
Routing is self-reinforcing: experts that get more tokens get more training, become better, and get chosen even more. Without a counter-force the router collapses onto a few experts, wasting the parameters of the others. In expert parallelism, an overloaded expert also makes its GPU the bottleneck, and with a capacity limit its extra tokens are dropped. Balancing (an auxiliary loss, or bias adjustment) keeps the load close to even.
Write the Switch load-balancing loss and explain why it is minimized by even routing.core
\(\mathcal{L}_{\text{bal}} = \alpha E \sum_i f_i P_i\), with \(f_i\) the share of tokens sent to expert \(i\) (from the hard top-k, no gradient) and \(P_i\) the mean router probability (differentiable). With uniform \(f = P = 1/E\) the sum is \(E \cdot E \cdot 1/E^2 = 1\). The gradient with respect to \(P_i\) is \(\alpha E f_i\), so it lowers the probability of busy experts in proportion to their load. Example: \(f = (0.7, 0.1, 0.1, 0.1)\), \(P = (0.6, 0.15, 0.15, 0.1)\) gives 1.84 instead of 1. Switch used \(\alpha = 10^{-2}\).
What is the capacity factor, and what happens to a dropped token?core
Expert capacity is \(C = \lfloor \mathrm{CF} \cdot kT/E \rfloor\) slots per expert per batch (29.9). Tokens beyond \(C\) for an expert are dropped: that expert does not process them, and the token passes to the next layer through the residual connection only (with top-2 it may still get its other expert). A higher CF reduces drops but adds padding compute, memory and communication. Example: \(T = 4096\), \(E = 8\), \(k = 2\), CF 1.25 gives \(C = 1280\). Dropless systems (MegaBlocks, DeepSeek-V3) avoid drops with variable-sized expert batches.
What is the router z-loss and why is it used?core
From ST-MoE: \(\mathcal{L}_z = \beta \cdot \text{mean}_t (\log\sum_j e^{z_{t,j}})^2\) (29.7), with \(\beta = 10^{-3}\). It penalizes large router logits. Large logits make the softmax very sharp and cause round-off errors in low precision, which leads to training instability and loss spikes. Keeping the logits small improves stability without changing which expert is preferred much. Computing the router in FP32 is a second, complementary fix.
What are shared experts and fine-grained experts (DeepSeekMoE)?core
Fine-grained: split each expert into \(m\) smaller ones and route to \(mk\) of them. Total and active parameters stay the same, but the number of combinations grows from, for example, \(\binom{16}{2} = 120\) to \(\binom{64}{8} \approx 4.4 \times 10^9\), which allows more specialized experts. Shared experts: \(K_s\) experts that every token uses (29.10), which store common knowledge once, so the routed experts do not all have to learn it. DeepSeek-V3 has 1 shared and 256 routed experts, top-8.
How does DeepSeek-V3 balance experts without an auxiliary loss?advanced
Each expert has a bias \(b_i\) that is added to its sigmoid score only when choosing the top-k; the gate weights use the original scores (29.8). After each step, overloaded experts get \(b_i \leftarrow b_i - \gamma\) and underloaded ones \(b_i + \gamma\), with \(\gamma = 0.001\) for most of training. Because no gradient is involved, balancing does not fight the language-model loss, which Wang et al. found gives better quality than an auxiliary loss. A tiny sequence-wise balance loss (\(\alpha = 10^{-4}\)) is kept only against extreme cases.
Explain expert parallelism and its communication cost.advanced
Experts are placed on different GPUs. In each MoE layer, each GPU routes its tokens, sends each token copy to the GPU owning the chosen expert (all-to-all dispatch), the experts compute, and the outputs come back (all-to-all combine), then are summed with the gates. Data moved per layer is about \(2 T k d b (1 - 1/G)\) bytes (29.11), plus two more all-to-alls in backward. For DeepSeek-V3 (\(k = 8\), \(d = 7168\)), one token sends about 115 KB per layer in BF16 for dispatch. Costs are reduced by node-limited routing, FP8 dispatch, and overlapping communication with compute. Load imbalance hurts more here because the slowest GPU sets the step time.
Is an MoE model cheaper to serve than a dense model with the same quality?advanced
Per token it needs fewer FLOPs (active parameters), but it needs memory for all experts (total parameters), so it needs more GPUs or offloading. At batch size 1, decoding reads only the chosen experts and is fast. At large batch sizes, different tokens touch most experts, so each decode step reads nearly all weights, and the advantage over a dense model of the same total size shrinks, while the FLOPs advantage remains. Expert parallelism adds all-to-all latency. MoE is usually the better choice for high-throughput serving on many GPUs; small dense models are often better on one device. [Likely]
Top-1 vs top-2 routing, and softmax vs sigmoid gates: what changes?advanced
Top-1 (Switch) halves expert compute and communication compared with top-2 and was shown to work; the gate must stay \(p_i\) (not renormalized) so the router gets a gradient. Top-2 and higher give each token a mixture and a smoother output, and a second expert if one is dropped. Softmax makes experts compete for a fixed probability mass; sigmoid (DeepSeek-V3) scores each expert independently, which works well with very many experts and with bias balancing, followed by normalization of the chosen scores.

Exercises

  1. Router logits for 4 experts are \((1.0, 0.0, 2.0, -1.0)\). With top-2, which experts are chosen, and what are the Mixtral-style gate weights?
  2. Count total and active parameters of an MoE model with \(N_{\text{other}} = 1\)B, \(L = 24\) layers, \(E = 16\) experts of 50M parameters each, top-2 (ignore routers).
  3. A batch has \(T = 2048\) tokens, \(E = 16\), \(k = 1\), CF = 1.0. What is the capacity? If one expert receives 300 tokens, how many are dropped?
  4. For 2 experts with \(f = (0.9, 0.1)\) and \(P = (0.8, 0.2)\), compute \(E \sum f_i P_i\). What is it for even routing?
  5. Split each expert of a top-2-of-8 model into 4 pieces. How many experts and how many per token? How many combinations before and after?
  6. With \(k = 8\), \(d = 4096\), FP8 dispatch, how many bytes does one token send in the dispatch of one MoE layer?
Answerscheck
1. The top-2 logits are 2.0 (expert 3) and 1.0 (expert 1). Gates: \(e^{2}/(e^{2} + e^{1}) = 1/(1 + e^{-1}) = 0.731\) for expert 3 and \(0.269\) for expert 1. 2. Total \(= 1 + 24 \times 16 \times 0.05 = 20.2\)B; active \(= 1 + 24 \times 2 \times 0.05 = 3.4\)B. 3. \(C = \lfloor 1.0 \times 2048 / 16 \rfloor = 128\); \(300 - 128 = 172\) tokens are dropped. 4. \(2 \times (0.72 + 0.02) = 1.48\); even routing gives \(2 \times (0.25 + 0.25) = 1.0\). 5. 32 experts, 8 per token; \(\binom{8}{2} = 28\) combinations before, \(\binom{32}{8} = 10{,}518{,}300\) after. 6. \(8 \times 4096 \times 1 = 32{,}768\) bytes (32 KiB), and the same again in the combine if it is also sent in FP8 (DeepSeek-V3 sends the combine in BF16).

Key takeaways

  • An MoE layer replaces the FFN with \(E\) experts and a router; each token uses only its top-k experts, mixed with gate weights (29.1)–(29.3).
  • Total parameters (29.4) set memory; active parameters (29.5) set FLOPs. Mixtral: 46.7B total, 12.9B active. DeepSeek-V3: 671B total, 37B active.
  • Routers collapse without help. The balance loss (29.6), the z-loss (29.7) and bias balancing (29.8) keep the load even and the router stable.
  • Capacity (29.9) gives fixed shapes but drops tokens and adds padding; many modern systems are dropless.
  • Fine-grained and shared experts (29.10) give more combinations and store common knowledge once.
  • Expert parallelism spreads experts over GPUs and costs two all-to-alls per layer in the forward pass (29.11).

Key formulas

\[ \mathrm{MoE}(\mathbf{x}) = \sum_{i \in \operatorname{TopK}(\mathbf{p}, k)} g_i\, \mathrm{FFN}_i(\mathbf{x}), \quad \mathbf{p} = \softmax(W_r \mathbf{x}) \]
\[ N_{\text{active}} = N_{\text{other}} + L(k N_{\text{exp}} + E d), \quad \mathcal{L}_{\text{bal}} = \alpha E \sum_i f_i P_i, \quad C = \left\lfloor \mathrm{CF}\cdot \frac{kT}{E} \right\rfloor \]

References for this session

  1. Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
  2. Fedus et al. (2021). Switch Transformers
  3. Jiang et al. (2024). Mixtral of Experts
  4. DeepSeek-AI (2024). DeepSeek-V3 Technical Report
  5. Lepikhin et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  6. Zoph et al. (2022). ST-MoE: Designing Stable and Transferable Sparse Expert Models
  7. Zhou et al. (2022). Mixture-of-Experts with Expert Choice Routing
  8. Dai et al. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
  9. Wang, Gao, Zhao, Sun, Dai (2024). Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
  10. Gale, Narayanan, Young, Zaharia (2022). MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
  11. Riquelme et al. (2021). Scaling Vision with Sparse Mixture of Experts
  12. Qwen Team (2025). Qwen3 Technical Report