Efficient FFN 2: MoE routing in depth
The router of a Mixture of Experts layer decides which experts each token visits. This session looks closely at how that decision is made, why the load on experts is never perfectly even, what uneven load costs in dropped tokens, padding and slow GPUs, and the main fixes: the balance loss, the router z-loss, balanced assignment (BASE), expert-choice routing, bias balancing with group-limited routing (DeepSeek-V3), and shared and fine-grained experts.
What you will learn
- Token-choice top-k routing exactly: buffer positions, capacity, first-choice priority and batch prioritized routing
- Why even a perfect router gives uneven loads, and how to compute the drop rate and padding for any batch size and capacity factor
- Why small batches (decoding) make capacity limits useless, and how many experts a decode step really reads
- The gradient of the Switch balance loss and of the router z-loss, written out, and what each one pushes
- Two routers that balance by construction: BASE layers (a linear assignment) and expert-choice routing, and why both have trouble with autoregressive decoding
- DeepSeek-V3 routing step by step: sigmoid scores, a bias used only for choosing, and group-limited (node-limited) top-k
- Shared and fine-grained experts from the routing point of view: combinations, router cost, and what the group limit removes
Why this matters
In session 29 we met the router: a small linear layer that sends each token to a few experts. That session showed what the router does. This session looks at how it decides, and at what goes wrong when it decides badly. Almost every hard problem of MoE models, in training and in serving, is a routing problem.
A Mixture of Experts (MoE) layer is only cheap when the work is spread evenly. Suppose 64 experts sit on 64 GPUs. If one expert gets twice the average number of tokens, its GPU works twice as long, and the other 63 GPUs wait. If the layer has a fixed buffer per expert, the extra tokens are thrown away (“dropped”), and the empty slots of the quiet experts are computed for nothing (“padding”). So the router is not only a quality question. It decides the speed of the whole model.
The efficient-inference survey (Zhou et al., 2024, Section 5.1.1) lists MoE as a main way to make the FFN cheaper per token. [Certain] But that saving is real only if routing is balanced at the batch sizes you actually serve. This session explains:
- Token-choice routing in detail: how each token finds its slot in an expert’s buffer, what the capacity factor does, and who gets dropped first.
- The cost of imbalance: exact drop rates and padding, and why a batch of 32 decode tokens behaves very differently from a training batch of 4,096 tokens.
- Four families of fixes: an extra loss (the Switch balance loss and the router z-loss), a different assignment rule (BASE layers and expert choice), a bias that only changes the choice (DeepSeek-V3), and a different expert shape (shared and fine-grained experts).
The previous session, session 94, made the dense FFN cheaper (SwiGLU and weight sharing). This one makes the FFN sparse. The next one, session 96, builds MoE models from dense ones and looks at MoE serving: batching and expert parallelism.
Token-choice routing, step by step
We start with the most common rule, token choice: every token picks its own top-\(k\) experts. Each token \(t\) has a hidden vector \(\mathbf{x}_t\). The router computes logits \(\mathbf{z}_t = W_r \mathbf{x}_t \in \R^E\), probabilities \(\mathbf{p}_t = \softmax(\mathbf{z}_t)\), and the set \(\mathcal{T}_t\) of the \(k\) experts with the highest probabilities. So far this is session 29. The new part is what happens next, when many tokens arrive at the same time.
Buffers and positions
Fast hardware wants fixed tensor shapes. GShard and Switch therefore give every expert a buffer with \(C\) slots for one batch (or one “group” of tokens). Tokens are placed into the buffer in a fixed order. The slot of token \(t\) in expert \(i\) is simply the number of earlier tokens that also chose \(i\):
- Read the symbols
- \(t' \prec t\)
- “token \(t'\) comes before token \(t\) in the priority order” (by default: the position in the batch)
- \(\mathbb{1}[\cdot]\)
- 1 if the condition is true, else 0; so the sum counts earlier tokens that chose expert \(i\)
- \(T\)
- the number of tokens routed together (a batch on one device, or a “group”)
- \(E,\ k\)
- number of experts, and experts per token
- \(kT/E\)
- the load each expert would get if routing were perfectly even
- \(\mathrm{CF}\)
- the capacity factor, the extra room above the even load (Switch: “expert capacity = tokens per batch / number of experts × capacity factor”) [Certain]
In code, (95.1) is one cumulative sum: make a one-hot matrix of shape \(T \times E\) for the choices, run cumsum down the token axis, and subtract 1. This is how GShard and Switch build their dispatch tensors. [Certain] With \(k = 2\), GShard first places all first choices of the batch, then all second choices, so a second choice can never push out somebody’s first choice. [Certain]
A token that does not fit is dropped by that expert. In Switch, “computation is skipped and the token representation is passed directly to the next layer through the residual connection”. [Certain] The token is still in the sequence, but this layer adds nothing to it (with \(k = 2\) it may still get its other expert).
- input: tokens \(X \in \R^{T \times d}\), router \(W_r\), \(k\), capacity factor CF
- \(P \leftarrow \softmax(X W_r^\top)\) row by row, in FP32
- \(\text{idx}, \text{gate} \leftarrow \operatorname{TopK}(P, k)\) for each row // shape \(T \times k\)
- \(C \leftarrow \lfloor \mathrm{CF} \cdot kT/E \rfloor\) // (95.2)
- order \(\leftarrow\) token positions \(0..T-1\) (or: sort tokens by their top gate, see below)
- fill \(\leftarrow\) zeros\((E)\)
- for \(j = 1..k\): // first choices of everybody before second choices
- \(O \leftarrow\) one-hot\((\text{idx}[\text{order}, j])\) // \(T \times E\)
- pos \(\leftarrow\) fill \(+\) cumsum\((O)\) \(- 1\), read at the chosen column // (95.1)
- keep\([\text{order}, j] \leftarrow\) pos \(\lt C\); fill \(\leftarrow\) fill \(+\) column sums of \(O\)
- scatter kept tokens into a buffer of shape \(E \times C \times d\); run each expert on its \(C\) rows
- gather the outputs back; \(\mathbf{y}_t = \sum_{j:\ \text{keep}[t,j]} \text{gate}[t,j]\ \text{out}_{\text{idx}[t,j]}(t)\)
Who is dropped first: priority
With order “by position”, tokens late in the batch are dropped first, whatever their content. V-MoE (a vision MoE) proposed Batch Prioritized Routing (BPR): sort the tokens by their highest routing weight, \(\max_i p_{t,i}\), and fill the buffers in that order. Tokens the router is sure about get their expert; unsure tokens are dropped. V-MoE used this to drop unimportant image patches on purpose and save inference compute. [Certain] Note that sorting looks at the whole batch at once, so it uses information from other tokens, including later ones.
Figure 1 runs Algorithm 1 for real on a toy layer: 8 tokens with 4-dimensional hidden vectors, a fixed \(4 \times 4\) router matrix and 4 experts. You can watch each token get its logits, its softmax and its top-\(k\) experts, then watch the buffers fill in priority order, with the slot number of (95.1) and the capacity of (95.2) written out at every step. The last step computes the balance loss (95.7) of this batch, which we study in Fix 1 below.
- input: \(X\ (8 \times 4)\), \(W_r\ (4 \times 4)\), \(k\), CF
- for each token \(t\): \(z_t \leftarrow W_r x_t\); \(p_t \leftarrow \softmax(z_t)\)
- \(\mathcal{T}_t \leftarrow\) the \(k\) largest entries of \(p_t\)
- \(C \leftarrow \lfloor \mathrm{CF} \cdot kT/E \rfloor\) // (95.2)
- order \(\leftarrow 1..T\), or sort by \(\max p_t\) (BPR)
- for \(j = 1..k\): // first choices first
- for \(t\) in order: \(i \leftarrow\) \(j\)-th choice of \(t\); \(\text{pos} \leftarrow \text{fill}[i]\) // (95.1)
- if \(\text{pos} \lt C\): keep in slot pos else drop; \(\text{fill}[i] \leftarrow \text{fill}[i] + 1\)
- \(f_i\), \(P_i\), \(\mathcal{L}_{\text{bal}}/\alpha = E \sum_i f_i P_i\) // (95.7)
How to read Figure 1
Press Step and follow the highlighted line. Steps 2 to 9 compute \(\mathbf{z}_t = W_r\mathbf{x}_t\) and \(\mathbf{p}_t\) for one token each: most tokens like expert 1, a learned skew. Then the capacity \(C\) is computed and the order is fixed. With top-2 and CF 1.25, \(C = 5\): expert 4 is wanted 6 times, so exactly one pair is dropped. With priority “position” the victim is the second choice of token 5, the latest in the batch; with BPR it is the second choice of token 2, the least sure among the tokens that still want expert 4. Switch to top-1 and CF 1.0: \(C = 2\), five tokens want expert 1, and three are dropped while experts 2, 3 and 4 each leave a slot empty. The last step shows the balance loss of the batch: \(E\sum_i f_iP_i = 1.053\) for top-2, 1.192 for top-1; it would be 1 with perfectly even loads.
Capacity at inference time
The capacity factor does not have to be the same in training and in evaluation. ST-MoE trained with CF 1.25 and evaluated with CF 2.0, and recommends “top-2 routing with 1.25 capacity factor”; it also says the capacity factor “can be changed during evaluation to adjust to new memory/compute requirements”. [Certain] Many recent models avoid the question: they are dropless (MegaBlocks kernels, DeepSeek-V3), so every token is always processed. [Certain] The next section shows why capacity is especially dangerous at small batch sizes.
The cost of uneven load
Even a perfect, well-trained router cannot give every expert exactly \(kT/E\) tokens in every batch, because which experts a batch needs depends on its content. To see the smallest possible imbalance, take the best case: each token picks \(k\) distinct experts uniformly at random. Then each token chooses expert \(i\) with probability \(k/E\), independently of the other tokens, and the load \(n_i\) of expert \(i\) has a binomial distribution:
The standard deviation grows like the square root of the mean, so the relative spread is about \(1/\sqrt{kT/E}\). With 1,024 tokens per expert the spread is about 3%; with 1 token per expert it is about 100%. From (95.3) we get the expected fraction of assignments that are dropped and the fraction of buffer slots that stay empty:
- Read the symbols
- \(\max(0, n_i - C)\)
- tokens that arrive after the buffer of expert \(i\) is full
- \(\max(0, C - n_i)\)
- empty slots of expert \(i\): computed (in a fixed-shape kernel) but useless
- \(\E[\cdot]\)
- the average over the binomial distribution of (95.3); every expert has the same distribution, so one expert is enough
At CF = 1 the two numbers are equal, because the average of \(n_i - C\) is zero: every dropped token on a busy expert is matched by an empty slot on a quiet one.
Training. A device routes \(T = 4096\) tokens to \(E = 64\) experts with \(k = 2\). The even load is \(2 \times 4096/64 = 128\). At CF 1.0, \(C = 128\) and (95.4) gives 3.47% dropped and 3.47% padding. At CF 1.25, \(C = 160\): only 0.01% dropped, but 20.0% of all slots are padding.
Decoding. Now use the DeepSeek-V3 shape (\(E = 256\), \(k = 8\)) with a decode batch of \(T = 32\) tokens. The even load is \(8 \times 32/256 = 1\) token per expert. At CF 1.25, \(C = \lfloor 1.25 \rfloor = 1\), and 36.2% of all assignments are dropped, with 36.2% padding. Even CF 2.0 (\(C = 2\)) still drops 9.78% and wastes 54.9% of the slots.
All numbers are exact binomial sums, checked with a simulation of 200 random batches in the code section (3.5% and 36.21%). The lesson: a capacity factor that is safe for training is useless when there are only a few tokens per expert. [Certain]
How to read Figure 2
Start with DeepSeek-V3 and \(T = 32\) (a decode step): the red curve stays high until CF is large, because each expert expects only about one token. Move \(T\) to 4,096 (a training micro-batch on one device): the red curve falls to almost zero just above CF = 1, and the grey padding curve rises in a straight line. Then try Mixtral: with only 8 experts, each expert gets many tokens even at small \(T\), so it is much less sensitive. The number that matters is the even load \(kT/E\), shown in the readout.
Dropless routing moves the cost into time
A dropless layer processes every token, so nothing is lost, but each expert now does \(n_i\) rows of work. With expert parallelism, all GPUs wait for the busiest one:
Here \(\eta\) is the compute efficiency: the fraction of the time that is useful work, if each expert has its own GPU. With uniform random routing, Python gives an expected busiest load of about 1.21 times the mean for \(T = 4096,\ E = 64,\ k = 2\) (\(\eta \approx 0.83\)), 1.25 times for the DeepSeek-V3 shape at \(T = 4096\) (\(\eta \approx 0.80\)), 2.1 times at \(T = 256\) (\(\eta \approx 0.47\)), and 4.6 times at \(T = 32\) (\(\eta \approx 0.22\)). [Certain] So routing noise alone, with no learned skew, costs 20% of the expert time at a big batch, and more than half at small batches. Real routers add a learned skew on top. DeepSeek-V3 serves with extra copies of the busiest experts (“redundant experts”, chosen every few minutes from live statistics) to fight exactly this. [Certain]
How many experts does a decode step read?
For decoding, the main cost is usually reading weights from memory, not FLOPs (session 82). With a batch of \(B\) tokens that route independently, an expert is read if at least one token picks it. The expected number of experts read per layer is
For the DeepSeek-V3 shape (\(E = 256\), \(k = 8\)): \(U(1) = 8\), \(U(8) = 57.4\), \(U(32) = 163.3\), \(U(64) = 222.4\), \(U(256) = 255.9\). For Mixtral (\(E = 8\), \(k = 2\)) a batch of only 8 already reads 7.2 of the 8 experts. [Certain] So the “only \(k\) of \(E\) experts” saving in weight traffic disappears quickly as the batch grows, while the FLOPs saving stays. This is one reason MoE serving uses expert parallelism over many GPUs, where each GPU holds few experts and the batch is large (session 96).
How to read Figure 3
With DeepSeek-V3 and \(B = 1\), only 8 thin bars are lit: 3.1% of the routed expert weights are read. At \(B = 32\) about 163 of 256 experts are read; at \(B = 256\) almost all. Raise the skew: fewer experts are read (good for memory traffic), but the busiest expert gets many times the average load (bad for time with expert parallelism, formula (95.5)). The two goals pull in opposite directions; balance wins in practice because big serving batches are compute-heavy.
Fix 1: the balance loss, in depth
Session 29 introduced the auxiliary load-balancing loss of Switch Transformer. For a batch of \(T\) tokens:
Switch uses \(k = 1\) and \(\alpha = 10^{-2}\); the authors swept \(\alpha\) from \(10^{-1}\) to \(10^{-5}\) and found that \(10^{-2}\) “balanced load quickly without interfering with training loss”. [Certain] The factor \(E\) makes the loss equal to \(\alpha\) at perfect balance, whatever the number of experts.
Why a product of \(f\) and \(P\)?
The load \(f_i\) is what we want to control, but it comes from a hard top-k choice, so it has no gradient. The mean probability \(P_i\) is smooth and has a gradient, but on its own it is the wrong target: a router can give every expert almost the same mean probability and still send every token to the same expert (for example, if every token gives expert 1 probability 0.26 and each of the three others about 0.247, then \(P \approx (0.26, 0.247, 0.247, 0.247)\) is almost even, but \(f = (1, 0, 0, 0)\)). The product uses \(f\) as a fixed weight and \(P\) as the handle. Working out the gradient with respect to one router logit shows exactly what it does. Using \(\partial p_{t,i}/\partial z_{t,j} = p_{t,i}(\delta_{ij} - p_{t,j})\):
- Read the symbols
- \(z_{t,j}\)
- the router logit of token \(t\) for expert \(j\)
- \(\bar f_t\)
- the load of the experts “as seen by token \(t\)”: the average of the loads \(f_i\), weighted by this token’s own probabilities
- \(f_j - \bar f_t\)
- positive if expert \(j\) is busier than the experts this token usually likes
- \(\delta_{ij}\)
- 1 if \(i = j\), else 0
Gradient descent subtracts the gradient. So the logit of expert \(j\) goes down when \(j\) is busier than the token’s weighted average, and up when it is quieter. The push is scaled by \(p_{t,j}\): the loss acts mostly on experts the token already considers, and it never moves a logit whose probability is near zero. That is why the balance loss alone cannot rescue an expert that every token ignores completely.
Four experts with loads \(f = (0.7, 0.1, 0.1, 0.1)\). One token has \(\mathbf{p}_t = (0.5, 0.2, 0.2, 0.1)\). Then \(\bar f_t = 0.35 + 0.02 + 0.02 + 0.01 = 0.40\).
The factors \(p_{t,j}(f_j - \bar f_t)\) are \(0.5 \times 0.3 = 0.15\), \(0.2 \times (-0.3) = -0.06\), \(-0.06\) and \(0.1 \times (-0.3) = -0.03\), all times \(\alpha E/T\). They sum to zero (the softmax ignores a shift of all logits). One gradient step lowers the logit of the busy expert 1 and raises the other three, most for experts 2 and 3, which this token already likes. The code section checks (95.8) against autograd.
Figure 4 puts (95.7) and (95.8) into a training loop. A toy router (16 tokens, 4 experts) starts with a strong preference for expert 1. Each iteration routes the batch, measures \(f\) and \(P\), computes the loss, and takes one gradient step on the router weights using only the balance loss. You can also switch to the bias rule of DeepSeek-V3 (Fix 4 below), which needs no gradient at all, and compare how the two reach balance.
How to read Figure 4
Start with the defaults (balance loss, top-1, step size 1, preference 1.5). At iteration 0 all 16 tokens go to expert 1: \(f = (1, 0, 0, 0)\) and \(E\sum f_iP_i = 2.219\). The update step shows the factors \(p_{t,j}(f_j - \bar f_t)\) of token 1: positive for expert 1 (its logit goes down) and negative for the others. Loads become \((3, 7, 3, 3)\) after one step and exactly \((4, 4, 4, 4)\) after three; the loss then reads 1. Raise the step size to 3: the router overshoots, the busy expert jumps from one column to another, and it takes many iterations to settle. That is why \(\alpha\) is kept small in practice. Switch to bias mode: nothing happens for the first iterations, because the bias must first grow larger than the probability gap; then the loads become even (top-1, \(\gamma = 0.02\): \((4, 4, 4, 4)\) from iteration 11 on). With a large \(\gamma\) the loads keep oscillating. The probabilities \(P_i\) never change in bias mode: only the choice changes, not the router.
Over which tokens? Sequence, micro-batch, global batch
The loss (95.7) depends on which tokens are counted in \(f\) and \(P\). This choice matters more than it looks.
- Per sequence. DeepSeek-V3 keeps a tiny sequence-wise balance loss with \(\alpha = 0.0001\), “just to avoid extreme imbalance within any single sequence”. [Certain]
- Per micro-batch. Most frameworks compute \(f\) on each device’s micro-batch, which holds only a few sequences. Qiu et al. (2025) point out that this is almost a per-sequence constraint: a code document must then spread its tokens over all experts, which blocks experts from specializing on code. Computing \(f\) over the global batch (one extra all-reduce of \(E\) numbers) improved perplexity and downstream results in their runs up to 42.8B parameters. [Certain]
- Per device. DeepSeekMoE adds a device-level balance loss: it balances the sum of loads of the experts on each GPU, not each expert, because only GPU time matters for speed. It uses a small expert-level factor and a larger device-level factor. [Certain]
DeepSeek-V3 measured the same effect: at 1B parameters the validation loss was 2.258 with a sequence-wise balance loss, 2.253 with bias balancing, and 2.253 with a batch-wise balance loss; at 3B, 2.085 versus 2.080 and 2.080. [Certain] The looser the balancing scope, the better the model, as long as the batch stays balanced.
Fix 2: the router z-loss and router precision
A second failure has nothing to do with balance: the router logits grow too large. ST-MoE found that large sparse models sometimes diverge, and traced part of it to round-off in the router’s exponentials. BF16 has only 7 mantissa bits, so its round-off is “up to 65,536x worse” than FP32, and larger numbers have larger round-off. [Certain] The fix is the router z-loss \(\mathcal{L}_z = \beta \frac{1}{T}\sum_t (\log\sum_j e^{z_{t,j}})^2\) with \(\beta = 0.001\). [Certain] Its gradient is simple:
Every logit is pushed down, in proportion to its probability and to the size of \(\mathrm{lse}_t\). The largest logits shrink the most, so the scores stay in a range where BF16 is accurate. The softmax ranking changes little: the z-loss mostly removes the overall scale and shift of the logits. ST-MoE found that it stabilized all three runs it tested without a quality loss, while update clipping also stabilized them but hurt quality. [Certain]
Logits \(\mathbf{z} = (2.0,\ 1.5,\ 0.3,\ -0.5)\) give \(\mathrm{lse} = 2.627\) and \(\mathbf{p} = (0.534, 0.324, 0.098, 0.044)\). The factors \(2\,\mathrm{lse}\, p_j\) are \((2.807,\ 1.703,\ 0.513,\ 0.230)\), all times \(\beta/T\). The biggest logit gets the biggest push down, and the sum of the factors is \(2\,\mathrm{lse} = 5.25\).
A second, cheaper fix is precision: Switch computes the router in FP32 even when the rest of the model uses BF16 (“selective precision”), and the DeepSeek-V3 reference code casts both the hidden states and the router weights to FP32 before the router matrix multiply. [Certain] The router is tiny, so this costs almost nothing: for DeepSeek-V3 the router is \(2 \times 256 \times 7168 = 3.7\)M FLOPs per token per layer, about 0.46% of the 793M FLOPs of the 9 experts it selects. [Certain]
Fix 3: change the assignment rule (BASE and expert choice)
An extra loss only encourages balance. Two methods make balance part of the assignment itself.
BASE layers: routing as a linear assignment
BASE layers (Lewis et al., 2021) treat routing in training as an optimization problem: give each expert exactly \(T/E\) tokens, and among all such assignments pick the one with the highest total token-expert score. [Certain] Each expert \(e\) has an embedding \(\mathbf{w}_e\), and token \(t\) is assigned to one expert \(a_t\):
- Read the symbols
- \(\mathbf{h}_t \cdot \mathbf{w}_e\)
- the affinity of token \(t\) for expert \(e\) (the router logit)
- \(a_t\)
- the one expert that token \(t\) is sent to (BASE uses one expert per token)
- \(T/E\)
- the exact load of every expert: perfect balance, no drops, no padding
- \(\sigma\)
- the sigmoid; it scales the expert output, so the expert embedding gets a gradient
This is a classic linear assignment problem. BASE solves it with the auction algorithm of Bertsekas, which parallelizes better on GPUs than the Hungarian algorithm. [Certain] In an auction, each unassigned token “bids” for its best expert slot, at a price that is its margin over its second-best option; the slot’s price goes up, and a token that is outbid must bid again. The paper adds two more details: tokens are first shuffled randomly across workers (tokens of one sequence are similar, and that would make specialization hard), and BASE needs no auxiliary loss and no new hyperparameters. [Certain]
- input: scores \(S \in \R^{T \times E}\), capacity \(c = T/E\), small \(\varepsilon \gt 0\)
- make \(T\) slots: copy every column of \(S\) \(c\) times // slot \(j\) belongs to expert \(\lfloor j/c \rfloor\)
- price\([j] \leftarrow 0\) for all slots; all tokens are unassigned
- while some token \(t\) is unassigned:
- \(v_j \leftarrow S'[t, j] - \text{price}[j]\) for all slots; \(j^\star \leftarrow \argmax_j v_j\); \(v_2 \leftarrow\) second-best value
- price\([j^\star] \leftarrow\) price\([j^\star] + (v_{j^\star} - v_2) + \varepsilon\) // bid
- if slot \(j^\star\) had an owner, that token becomes unassigned; \(t\) takes slot \(j^\star\)
- return \(a_t = \lfloor \text{slot}(t)/c \rfloor\) // within \(T\varepsilon\) of the best total score
At inference, BASE is just greedy top-1. The paper says: “balancing the assignment leaks information about tokens in the future context. Instead, we simply greedily assign the one best expert.” [Certain] So the balance exists only in training; at test time the model relies on having learned a reasonably balanced router, which the paper shows it does, though less evenly than models with a balance loss. [Certain]
Expert choice: experts pick tokens
Expert-choice routing (Zhou et al., 2022) flips the direction. Each expert takes the \(C\) tokens of the batch that score it highest:
- Read the symbols
- \(S^\top\)
- the score matrix turned around: one row per expert, one column per token
- \(\operatorname{TopK}(S^\top, C)\)
- for each expert (row), the \(C\) best tokens: \(I\) holds their indices, \(G\) their scores, used as gate weights
- \(c\)
- the capacity factor, here meaning “experts per token on average”; \(c = 2\) matches the compute of top-2 token choice
The load is perfect by construction: every expert gets exactly \(C\) tokens. A token may be picked by many experts, by one, or by none (then it passes through the residual only). The paper reports that most tokens got one or two experts, 23% got three or four, and about 3% more than four. [Certain] With \(c = 2\), expert choice reached the perplexity of GShard top-2 in less than half the steps, and each GShard step was 20% slower because of load imbalance (8B active, 64 experts). [Certain] For example, \(T = 4096\), \(c = 2\), \(E = 64\) gives \(C = 128\) tokens per expert.
The catch is causality. To pick its top-\(C\) tokens, an expert must see all tokens of the batch, including future tokens of the same sequence. The paper itself says it “might not immediately apply to auto-regressive text generation” and that it also does not directly apply when “the batch size becomes very small during serving or inference”. [Certain] In decoding, you have one new token per sequence per step, and its routing must not depend on tokens that do not exist yet. This is why decoder-only LLMs almost all use token choice. Expert choice fits encoders and vision models, where the whole input is known. [Likely]
Six tokens, three experts, router probabilities (rows are tokens): \(t_1 = (0.55, 0.15, 0.30)\), \(t_2 = (0.60, 0.30, 0.10)\), \(t_3 = (0.50, 0.40, 0.10)\), \(t_4 = (0.45, 0.15, 0.40)\), \(t_5 = (0.20, 0.70, 0.10)\), \(t_6 = (0.40, 0.35, 0.25)\). One expert per token on average, so the even load is 2.
Token choice, top-1, CF 1.0 (\(C = 2\)): five tokens want expert 1. It keeps \(t_1, t_2\) and drops \(t_3, t_4, t_6\) (50% of the batch). Expert 2 has \(t_5\) and one empty slot; expert 3 has two empty slots. Batch prioritized routing (order \(t_5, t_2, t_1, t_3, t_4, t_6\)) drops the same three here.
Expert choice (\(C = 2\)): expert 1 takes \(t_2, t_1\); expert 2 takes \(t_5, t_3\); expert 3 takes \(t_4, t_1\). Nothing is padded, but \(t_1\) gets two experts and \(t_6\) gets none. Sum of scores 2.95.
BASE (exactly 2 per expert): the best assignment is \(t_1, t_2 \to 1\), \(t_3, t_5 \to 2\), \(t_4, t_6 \to 3\), total 2.90. Every token gets exactly one expert. Greedy top-1 (what BASE does at test time) would score 3.20 but load expert 1 with 5 tokens. The code section checks these with the auction algorithm and with brute force.
How to read Figure 5
Press Play in token choice on the worked example: tokens 1 and 2 fill expert 1, then tokens 3, 4 and 6 arrive at a full buffer and turn red. Switch to expert choice and step: expert 1 takes tokens 2 and 1, expert 2 takes 5 and 3, expert 3 takes 4 and 1. No slot is empty, but the count column shows that token 1 got 2 experts and token 6 got none. Switch to BASE and step through the 11 bids (ε = 0.05): tokens 1, 2 and 3 compete for the two slots of expert 1, prices rise by the bid margin, the outbid token bids again, and the auction ends at the assignment \((1, 1, 2, 3, 2, 3)\) with total 2.90, the exact optimum. With ε = 0.01 the same result needs 37 bids, because two slots of one expert outbid each other in tiny steps; a larger ε finishes sooner but only guarantees a total within \(T\varepsilon\) of the best. In the random batch, raise the bias toward expert 1 and compare how much each router suffers. BASE ignores the \(k\) buttons: it always uses one expert per token.
Fix 4: bias balancing and group-limited routing (DeepSeek-V3)
DeepSeek-V3 combines several ideas into one router. Each routed expert \(i\) has a centroid \(\mathbf{e}_i\), and the score is a sigmoid, \(s_{i,t} = \sigma(\mathbf{u}_t^\top \mathbf{e}_i)\), so experts do not compete inside a softmax. [Certain] Then:
- Bias for choosing only. A per-expert bias \(b_i\) is added to the scores to choose the top-k, but the gate weights use the original scores. After each training step, \(b_i\) goes down by \(\gamma\) if expert \(i\) was overloaded and up by \(\gamma\) if it was underloaded, with \(\gamma = 0.001\) for the first 14.3T tokens and 0 for the last 500B. [Certain] No gradient is involved, so balancing does not fight the language-model loss.
- Node-limited routing. In training, the 256 routed experts of a layer sit on 64 GPUs in 8 nodes, and each token may use experts on at most \(M = 4\) nodes. The nodes are chosen by “the sum of the highest \(K_r/M\) affinity scores of the experts distributed on each node”. [Certain] With \(K_r = 8\) and \(M = 4\), that is the sum of the top-2 scores per node.
- Normalize and scale. The chosen original scores are normalized to sum to 1, then multiplied by a constant
routed_scaling_factor= 2.5 (from the released config). [Certain]
The released Hugging Face code implements the node limit as groups: n_group = 8, topk_group = 4, and the group score is the sum of the top-2 biased scores in each group of 32 experts. [Certain]
- Read the symbols
- \(g\)
- a group of experts (one node in training): 8 groups of 32 experts in DeepSeek-V3
- \(\gamma_g\)
- the group score: the sum of the two best biased scores in the group
- \(\mathcal{G}\)
- the \(M = 4\) best groups; experts in other groups are masked out for this token
- \(\mathcal{T}\)
- the \(K_r = 8\) chosen routed experts, all inside the chosen groups
- \(s_i\)
- the original sigmoid score; the bias \(b_i\) appears only in the choice, never in the gate
- \(\rho\)
- the routed scaling factor, a fixed constant
Why the limit? Each routed copy of a token travels over the network (session 49). Traffic between nodes uses InfiniBand (50 GB/s in the DeepSeek-V3 cluster), while NVLink inside a node gives 160 GB/s, about 3.2 times more. [Certain] With at most 4 nodes per token, each token is first sent over InfiniBand to each of its target nodes, then forwarded inside the node over NVLink to the GPUs that hold its experts. [Certain] The paper says each token selects 3.2 experts per node on average and that the design could scale to 13 experts (4 nodes × 3.2) at the same communication cost. [Certain]
A small version: 16 experts in 4 groups of 4, keep 2 groups, choose 4 experts, bias zero. Scores: group 1 \((0.90, 0.15, 0.10, 0.05)\), group 2 \((0.70, 0.65, 0.30, 0.20)\), group 3 \((0.85, 0.15, 0.10, 0.05)\), group 4 \((0.60, 0.50, 0.45, 0.10)\).
Group scores (top-2 sums): 1.05, 1.35, 1.00, 1.10. The two best groups are 2 and 4. The top-4 inside them are experts 5, 6, 13, 14 with scores 0.70, 0.65, 0.60, 0.50 (sum 2.45). Gates (95.13): \(2.5 \times (0.70, 0.65, 0.60, 0.50)/2.45 = (0.714, 0.663, 0.612, 0.510)\).
Plain top-4 would choose experts 1, 5, 6, 9, from three groups. The best single expert (score 0.90) is excluded, because its group has only one good expert. Group-limited routing prefers groups with several good experts. A trained router adapts to this rule, so the cost in quality is small, but it is not zero. [Likely]
How to read Figure 6
Step 1 shows the scores (plus bias). Step 2 marks the two best experts in each row and writes the group score at the right. Step 3 keeps the 4 best rows. Step 4 picks the 8 best experts inside those rows and lists their gates; cells with a thick outline in the third legend colour are experts that plain top-8 (no group limit) would have chosen instead. Press “Next token” several times: often the two choices agree, sometimes 1 or 2 experts differ. Plain top-8 sometimes touches 5 or more groups (nodes); the limited one never more than 4. Switch the bias on: the chosen set can change, but the gate of each chosen expert is still computed from its original score.
Shared and fine-grained experts, from the routing side
DeepSeekMoE (2024) changes the experts, not the router, but the change is about routing. It names two problems of classic MoE with few large experts: knowledge hybridity (each expert receives very different tokens and must store unrelated knowledge) and knowledge redundancy (many experts all learn the same common knowledge). [Certain] Its two answers:
- Fine-grained segmentation: split each of \(N\) experts into \(m\) smaller ones (hidden size divided by \(m\)) and route each token to \(mK\) of them. Compute and parameters stay the same.
- Shared expert isolation: \(K_s\) experts are used by every token, without the router; to keep compute fixed, the number of routed experts per token drops by \(K_s\).
For routing, three things change. First, the number of combinations: top-2 of 16 experts gives \(\binom{16}{2} = 120\) choices per token; top-8 of 64 quarter-size experts gives \(\binom{64}{8} = 4{,}426{,}165{,}368\) (the paper’s own example); with one shared expert and top-7 of 63, \(\binom{63}{7} = 553{,}270{,}671\). [Certain] Second, the balance gets easier to measure but harder to keep: with many small experts, the even load \(kT/E\) per expert is the same as before (both \(k\) and \(E\) grow by \(m\)), but each expert is a smaller matrix, so a busy expert is a smaller absolute cost, while the router has \(m\) times more columns. Third, the group limit removes many combinations: of the \(\binom{256}{8} \approx 4.1 \times 10^{14}\) possible sets in DeepSeek-V3, only 17.5% use experts from at most 4 of the 8 groups. [Certain]
At serving time, DeepSeek-V3 treats the shared expert as a routed expert that is always chosen, so each token selects 9 experts. The decode deployment uses 320 GPUs with one expert per GPU (EP320); 64 of those GPUs hold redundant and shared experts. [Certain]
Other routers exist that this session only names. Hash layers (Roller et al., 2021) route each token by a fixed hash of its token id, with no learned router at all; balance then depends only on the token frequencies. [Certain] Sinkhorn-style routers solve a soft version of the BASE assignment; we do not cover them here.
In code
The code below implements the four routing rules of this session on plain tensors: token choice with capacity and priority (Algorithm 1), expert choice (95.11), BASE with the auction algorithm (Algorithm 2), and DeepSeek-V3 group-limited routing (95.12)–(95.13). It then checks the worked examples, the drop rates of (95.4), and the gradients (95.8) and (95.9) against autograd.
import math, itertools, torch import torch.nn.functional as F torch.manual_seed(0) def token_choice(probs, k, cf, priority="position"): """Top-k token choice with a capacity limit (GShard/Switch style).""" T, E = probs.shape C = math.floor(cf * k * T / E) # capacity (95.2) gate, idx = probs.topk(k, dim=-1) # (T, k) order = torch.arange(T) if priority == "score": # batch prioritized routing order = gate[:, 0].argsort(descending=True, stable=True) keep = torch.zeros(T, k, dtype=torch.bool) fill = torch.zeros(E, dtype=torch.long) for j in range(k): # all first choices, then all second choices onehot = F.one_hot(idx[order, j], E) # (T, E), rows in priority order pos = ((fill + onehot.cumsum(0) - 1) * onehot).sum(-1) # slot in the expert buffer (95.1) keep[order, j] = pos < C fill += onehot.sum(0) return idx, gate, keep, C def expert_choice(probs, c): """Each expert takes its top-C tokens (Zhou et al., 2022).""" T, E = probs.shape C = int(T * c / E) # (95.11) g, tok = probs.t().topk(C, dim=-1) # (E, C) per_token = torch.bincount(tok.flatten(), minlength=T) return tok, g, per_token def base_assign(scores, eps=1e-4): """Balanced assignment (BASE layers) with the auction algorithm.""" T, E = scores.shape; cap = T // E S = scores.repeat_interleave(cap, dim=1) # one column per expert slot price = torch.zeros(T); owner = [-1] * T; slot = [-1] * T free = list(range(T)) while free: t = free.pop(0) v = S[t] - price best = v.topk(2) j = best.indices[0].item() price[j] += best.values[0] - best.values[1] + eps # raise the price of the slot if owner[j] >= 0: slot[owner[j]] = -1; free.append(owner[j]) # the old owner must bid again owner[j] = t; slot[t] = j return torch.tensor(slot) // cap def group_limited(s, bias, n_group, topk_group, k, scale): """DeepSeek-V3 routing: sigmoid scores, bias for choice only, best groups first.""" T, E = s.shape choice = s + bias # bias is used only to choose grp = choice.view(T, n_group, -1).topk(2, dim=-1).values.sum(-1) # (T, n_group) keep_g = grp.topk(topk_group, dim=-1).indices mask = torch.zeros(T, n_group, dtype=torch.bool).scatter(1, keep_g, True) mask = mask.repeat_interleave(E // n_group, dim=1) idx = choice.masked_fill(~mask, float("-inf")).topk(k, dim=-1).indices w = s.gather(1, idx) # gates use the original scores return idx, scale * w / w.sum(-1, keepdim=True) # 1) the hand example: 6 tokens, 3 experts P = torch.tensor([[.55,.15,.30],[.60,.30,.10],[.50,.40,.10],[.45,.15,.40],[.20,.70,.10],[.40,.35,.25]]) idx, gate, keep, C = token_choice(P, k=1, cf=1.0) print("token choice: C =", C, "kept", keep[:, 0].int().tolist()) tok, g, per = expert_choice(P, c=1) print("expert choice: tokens per expert", (tok + 1).tolist(), "experts per token", per.tolist()) a = base_assign(P) best = max((sum(P[t, e].item() for t, e in enumerate(x)), x) for x in itertools.product(range(3), repeat=6) if all(x.count(e) == 2 for e in range(3))) print("BASE auction", (a + 1).tolist(), "score", round(P[torch.arange(6), a].sum().item(), 2), "| brute force", [e + 1 for e in best[1]], round(best[0], 2)) # 2) drops under perfectly uniform routing vs the binomial formula (95.4) def drop_rate(T, E, k, cf, trials=200): d = 0 for _ in range(trials): _, _, keep, _ = token_choice(torch.rand(T, E), k, cf) d += (~keep).sum().item() return d / (trials * T * k) print("drop, T=4096 E=64 k=2 cf=1.0:", round(100 * drop_rate(4096, 64, 2, 1.0), 2), "%") print("drop, T=32 E=256 k=8 cf=1.25:", round(100 * drop_rate(32, 256, 8, 1.25), 2), "%") # 3) the gradient of the balance loss (95.8) and of the z-loss (95.9) T, E, k = 16, 8, 2 z = torch.randn(T, E, requires_grad=True) p = z.softmax(-1) f = F.one_hot(p.topk(k).indices, E).sum(1).float().mean(0) / k (E * (f * p.mean(0)).sum()).backward() formula = (E / T) * p * (f - (p * f).sum(-1, keepdim=True)) print("balance-loss grad, autograd vs (95.8):", (z.grad - formula).abs().max().item() < 1e-6) z.grad = None lse = torch.logsumexp(z, -1) lse.pow(2).mean().backward() print("z-loss grad, autograd vs (95.9):", (z.grad - (2 / T) * lse[:, None] * z.softmax(-1)).abs().max().item() < 1e-6) # 4) group-limited routing on the worked example (16 experts, 4 groups) s = torch.tensor([[.90,.15,.10,.05, .70,.65,.30,.20, .85,.15,.10,.05, .60,.50,.45,.10]]) idx, w = group_limited(s, torch.zeros(16), n_group=4, topk_group=2, k=4, scale=2.5) print("group-limited experts", sorted((idx + 1).tolist()[0]), "gates", [round(v, 3) for v in w[0].tolist()]) print("plain top-4 experts", sorted((s.topk(4).indices + 1).tolist()[0]))
Output (PyTorch 2.8, CPU): token choice keeps \([1, 1, 0, 0, 1, 0]\) with \(C = 2\); expert choice gives tokens \([[2, 1], [5, 3], [4, 1]]\) per expert and \([2, 1, 1, 1, 1, 0]\) experts per token; the auction finds \([1, 1, 2, 3, 2, 3]\) with score 2.9, the same as brute force. The simulated drop rates are 3.5% and 36.21%, matching the binomial values 3.47% and 36.21% of (95.4). Both gradient checks print True. Group-limited routing picks experts \([5, 6, 13, 14]\) with gates \([0.714, 0.663, 0.612, 0.51]\); plain top-4 picks \([1, 5, 6, 9]\). [Certain]
Where this is used
| System | Routing rule | Balance mechanism |
|---|---|---|
| Switch Transformer (2021) | token choice, top-1, capacity factor, router in FP32 | balance loss (95.7), \(\alpha = 10^{-2}\); drops “typically under 1%” [Certain] |
| GShard (2020), ST-MoE (2022) | token choice, top-2; first choices placed before second choices; ST-MoE: train CF 1.25, eval CF 2.0 | balance loss plus router z-loss (\(\beta = 0.001\)) in ST-MoE [Certain] |
| BASE layers (2021) | balanced linear assignment in training, greedy top-1 at test time | by construction (auction algorithm); no auxiliary loss [Certain] |
| Expert choice (2022) | experts pick their top-\(C\) tokens; variable experts per token | by construction; suits encoders, not autoregressive decoding [Certain] |
| V-MoE (2021), vision | token choice with Batch Prioritized Routing; drops low-score image patches on purpose | balance losses; capacity below 1 to save inference compute [Certain] |
| DeepSeekMoE 16B (2024) | 2 shared + 64 fine-grained routed experts, top-6 | expert-level balance loss 0.001; dropless [Certain] |
| DeepSeek-V3 (2024) | sigmoid scores, group (node) limited top-8 of 256, 1 shared expert, gates × 2.5 | bias balancing \(\gamma = 0.001\), sequence-wise loss \(10^{-4}\), redundant experts when serving; dropless [Certain] |
| Video models (classification, retrieval, key-moment detection) | frame or patch tokens routed to experts; the whole clip is known in advance, so expert choice or BPR-style dropping of background patches is possible | long videos give many tokens per expert, so capacity drops are rare; skewed content (one scene) can still overload experts [Speculation] |
| Video-RAG and long-video LLMs | token choice in the language backbone of a VLM (decoding must stay causal) | large prefill batches balance well; decode steps behave as in Figure 3 [Likely] |
Common mistakes
- Using a training capacity factor at inference. With few tokens per expert, CF 1.25 can drop a third of the assignments (Figure 2). Use dropless kernels or a much larger capacity when serving.
- Forgetting that capacity makes outputs batch-dependent. Whether your token is dropped depends on the other requests in the batch. The same prompt can give a different answer under a different load.
- Computing the balance loss on a tiny micro-batch. This forces balance inside each sequence and blocks specialization. Synchronize \(f\) over the global batch, or use a batch-level method.
- Thinking the balance loss can revive a dead expert. Its gradient (95.8) is scaled by \(p_{t,j}\); an expert with near-zero probability gets almost no push. Bias balancing or good initialization is needed.
- Using expert choice in a decoder for generation. It needs the whole batch, including future tokens, to choose. It leaks information in training and does not work step by step in decoding.
- Expecting BASE to balance at inference. BASE balances only in training; at test time it is greedy top-1.
- Putting the DeepSeek-V3 bias into the gate. The bias is only for choosing; the gate uses the original score. Mixing them changes the model output.
- Running the router in BF16 without a z-loss. Large logits plus low precision lead to unstable routing and loss spikes. Keep the router in FP32.
Interview questions
How does a token find its slot in an expert’s buffer, and what happens if the buffer is full?basic
What is the capacity factor trade-off, in numbers?basic
Why is a capacity limit much worse during decoding than in training?core
Derive the gradient of the Switch balance loss with respect to a router logit. What does it push?core
What does the router z-loss do, and why is it needed?core
Explain BASE layers. Why do they need no balance loss, and what do they do at inference?core
Explain expert-choice routing, its advantages and its main limitation.core
Describe the DeepSeek-V3 router in full.advanced
Why limit each token to experts on at most 4 nodes? What does it cost?advanced
With a decode batch of 32, how many DeepSeek-V3 experts does one MoE layer read, and why does it matter?advanced
What are shared and fine-grained experts, and how do they change routing?advanced
Exercises
- A device routes \(T = 1024\) tokens to \(E = 32\) experts with \(k = 2\) and CF 1.5. What is the capacity \(C\)? How many slots does the layer have in total, and what fraction of them is padding if no token is dropped?
- For the DeepSeek-V3 shape (\(E = 256\), \(k = 8\)), what is the even load per expert for a decode batch of \(T = 64\)? What capacity does CF 1.25 give?
- Using (95.6), how many of the 8 Mixtral experts does a batch of 4 tokens read on average? And how many of the 128 Qwen3-30B-A3B experts (top-8) does a batch of 16 read?
- Three experts have loads \(f = (0.5, 0.3, 0.2)\). A token has \(\mathbf{p} = (0.6, 0.3, 0.1)\). Compute \(\bar f_t\) and the three factors \(p_j(f_j - \bar f_t)\) of (95.8). Which logits go up?
- Expert choice with \(T = 2048\), \(E = 16\), \(c = 2\): what is \(C\)? How many expert slots are computed in total, and how does that compare with top-2 token choice at CF 1.0?
- Apply group-limited routing to 8 experts in 4 groups of 2, keep 2 groups, choose 2 experts, scores \((0.9, 0.1,\ 0.6, 0.55,\ 0.7, 0.2,\ 0.3, 0.3)\), bias 0, scale 1. Which experts are chosen, with which gates? Which would plain top-2 choose?
Answerscheck
Key takeaways
- Token-choice routing places tokens into fixed expert buffers with a cumulative sum (95.1); tokens past the capacity (95.2) are dropped, and first choices go before second choices.
- Even perfect random routing gives binomial loads (95.3). Drops and padding (95.4) depend on the even load \(kT/E\): a capacity factor that is safe in training is useless in decoding.
- Dropless routing turns imbalance into time: the busiest expert sets the layer time (95.5). A decode batch quickly reads most experts (95.6).
- The balance loss pushes each logit by \(p_{t,j}(f_j - \bar f_t)\) (95.8); its scope (sequence, micro-batch, global batch) changes model quality. The z-loss (95.9) keeps logits small for BF16.
- BASE (95.10) and expert choice (95.11) balance by construction, but both look at the whole batch, so neither balances during autoregressive decoding.
- DeepSeek-V3 uses sigmoid scores, a bias used only for choosing, group-limited top-8 (95.12) and scaled gates (95.13). Shared and fine-grained experts (95.14) change the experts so routing has more, smaller choices.
Key formulas
References for this session
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Fedus et al. (2021). Switch Transformers
- Zoph et al. (2022). ST-MoE: Designing Stable and Transferable Sparse Expert Models
- Lewis et al. (2021). BASE Layers: Simplifying Training of Large, Sparse Models
- Zhou et al. (2022). Mixture-of-Experts with Expert Choice Routing
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- Lepikhin et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Dai et al. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- Wang, Gao, Zhao, Sun, Dai (2024). Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
- Riquelme et al. (2021). Scaling Vision with Sparse Mixture of Experts
- Roller, Sukhbaatar, Szlam, Weston (2021). Hash Layers For Large Sparse Models
- Qiu et al. (2025). Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
- Gale, Narayanan, Young, Zaharia (2022). MegaBlocks: Efficient Sparse Training with Mixture-of-Experts