Tensor parallelism and sequence parallelism
Tensor parallelism splits the matrix multiplications inside every layer across a few GPUs, so that they share the work and the memory of one layer. This session shows the Megatron recipe (column-parallel then row-parallel layers, attention split by heads, two all-reduces per layer in the forward pass), how sequence parallelism removes the activation memory that tensor parallelism leaves behind, and why all of this lives on fast NVLink inside one server.
What you will learn
- How a matrix multiplication splits by columns (no communication) or by rows (partial sums and one all-reduce)
- The Megatron MLP and attention: column-parallel then row-parallel, heads split across GPUs, and the f and g operators
- Why one transformer layer needs two all-reduces in the forward pass and two in the backward pass
- Vocabulary-parallel embeddings and a parallel cross-entropy that never gathers the logits
- How to estimate the communication cost, and why the compute/communication ratio is 3hβ/((t−1)F)
- Sequence parallelism: splitting LayerNorm, dropout and residual regions by tokens, at no extra communication
- Why tensor parallelism stays inside one node, and how real runs such as Llama 3 405B place it
Why this matters
FSDP and ZeRO cut the memory of the weights by storing each weight on only one GPU. But when a layer runs, every GPU still gathers the full layer and does the full matrix multiplication for its own batch. Tensor parallelism does something different: it cuts the matrix multiplication itself into pieces, so that each GPU does only a part of the math of every layer.
Why would you want that? Three reasons.
- One layer can be too big. Llama 3 70B has 80 layers with \(h = 8192\). Its weights alone are about 141 GB in BF16. FSDP can shard them, but each GPU must still hold one full gathered layer plus all the activations of its micro-batch. With long sequences, the activations are the bigger problem.
- The batch per GPU cannot grow forever. Data parallelism (DDP, FSDP) needs at least one sequence per GPU. With 16,384 GPUs and 16M tokens per step, there are not enough sequences to give each GPU its own. Tensor parallelism lets 8 GPUs share one sequence.
- Serving. To generate text from a 70B model fast, you put it on 8 GPUs and let all 8 work on every token. This is exactly tensor parallelism. In vLLM it is the option
tensor_parallel_size. [Certain]
Real systems use it everywhere. Megatron-LM introduced the method in 2019 and trained an 8.3B model on 512 GPUs at 15.1 PetaFLOPs, with 8-way tensor parallelism inside each server. [Certain] Llama 3 405B was trained with tensor parallelism of size 8, inside each server, combined with pipeline, context and data parallelism. [Certain]
This session builds on session 43 (all-reduce, reduce-scatter, all-gather and their cost) and on the FSDP sessions (session 46, session 47). The next session, session 49, adds the other ways to split a model, and shows how all of them are combined.
Splitting one matrix multiplication
Everything in this session comes from one fact: a matrix multiplication can be cut into independent pieces in two ways. Take a linear layer \(Y = XA\), with input \(X \in \R^{n \times d}\) (\(n\) tokens, \(d\) features) and weight \(A \in \R^{d \times m}\). We have \(t\) GPUs. In this session \(t\) always means the tensor-parallel size: the number of GPUs that share one layer.
Column split (column-parallel linear)
Cut \(A\) into \(t\) blocks of columns, \(A = [A_1, A_2, \dots, A_t]\), and give block \(A_i\) to GPU \(i\). Each GPU has the full input \(X\). Then each GPU computes its own block of output columns:
No GPU needs anything from another GPU. The result is split by columns: GPU \(i\) holds \(Y_i\), one slice of the output features. If you want the full \(Y\) on every GPU, you need an all-gather. But often you do not want it, as we will see.
Row split (row-parallel linear)
Now cut \(A\) into \(t\) blocks of rows, \(A = [A_1; A_2; \dots; A_t]\) (stacked on top of each other), and cut the input \(X\) the same way, by columns: \(X = [X_1, X_2, \dots, X_t]\). Then
- Read the symbols
- \(X_i\)
- the input columns (features) that belong to GPU \(i\), shape \(n \times d/t\)
- \(A_i\)
- the weight rows that belong to GPU \(i\), shape \(d/t \times m\)
- \(X_i A_i\)
- a partial sum: full output shape \(n \times m\), but only a part of the total
Each GPU computes a full-size output, but it is only a partial sum. To get the true \(Y\), the GPUs must add their partial sums: that is one all-reduce. Figure 1 shows both splits on a tiny example with integers.
How to read Figure 1
In the column split, every GPU holds all of \(X\) (gray) and a few columns of \(A\). Its output is a few finished columns of \(Y\). Nobody talks to anybody. In the row split, each GPU holds a few columns of \(X\) and the matching rows of \(A\). Its output has the full shape of \(Y\), but the numbers are wrong on their own: you must add the outputs of all GPUs (an all-reduce) to get \(Y\). Check that the sum of the partial results equals the true \(Y\) printed in the readout.
\(X = \begin{bmatrix} 1 & 2 & 0 & -1 \\ 2 & 0 & 1 & 1 \end{bmatrix}\) and \(A\) is the \(4 \times 4\) matrix in the figure. The true result is \(Y = XA = \begin{bmatrix} -1 & 3 & 3 & -1 \\ 5 & 0 & 5 & 4 \end{bmatrix}\).
Row split with \(t = 2\): GPU 1 uses the first two columns of \(X\) and the first two rows of \(A\) and gets \(\begin{bmatrix} 1 & 2 & 4 & -1 \\ 2 & 0 & 4 & 2 \end{bmatrix}\). GPU 2 gets \(\begin{bmatrix} -2 & 1 & -1 & 0 \\ 3 & 0 & 1 & 2 \end{bmatrix}\). Neither is \(Y\), but their sum is: \(1 + (-2) = -1\), \(2 + 1 = 3\), and so on.
The Megatron MLP: column first, then row
The MLP of a transformer layer (see session 21) is two linear layers with a nonlinearity between them:
Here \(h\) is the hidden size and \(f\) the MLP width (often \(4h\)). Megatron-LM splits it like this: [Certain]
- \(A\) by columns (column-parallel). Every GPU has the full \(X\) and computes \(XA_i\): its own \(f/t\) MLP features.
- GeLU on each GPU, alone. GeLU works on each number separately, so applying it to a slice of columns gives exactly the same slice of \(\mathrm{GeLU}(XA)\).
- \(B\) by rows (row-parallel). The input of a row-parallel layer must be split by columns, and that is exactly what step 2 produced. No communication is needed between the two layers.
- One all-reduce adds the partial sums: \(Z = \sum_i \mathrm{GeLU}(XA_i)\, B_i\).
So the whole MLP needs one all-reduce in the forward pass. The trick is the pairing: a column-parallel layer leaves its output split in exactly the form that a row-parallel layer wants as input.
Figure 2 runs this recipe with real numbers. It is the forward pass of Algorithm 1 (below) on a toy MLP: \(n = 4\) tokens, hidden size \(h = 4\), MLP width \(f = 8\). Every number is computed in your browser. The last part of the animation is the all-reduce itself, done as a ring: first a reduce-scatter (each GPU sends one chunk of its partial sum to the next GPU, which adds it to its own), then an all-gather (the finished chunks travel around the ring once more). These are the two halves from session 43, and they will matter again for sequence parallelism.
- \(X \leftarrow f(X)\) // identity: every GPU has all of \(X\)
- \(Y_i \leftarrow X A_i\) // column-parallel, (48.1)
- \(H_i \leftarrow \mathrm{GeLU}(Y_i)\) // element-wise, local
- \(P_i \leftarrow H_i B_i\) // row-parallel: partial sum, (48.2)
- for \(r = 1 \dots t-1\): send chunk \(i - r\) to GPU \(i+1\), which adds it // reduce-scatter
- for \(r = 1 \dots t-1\): send a finished chunk to GPU \(i+1\), which copies it // all-gather
- \(Z = \sum_i \mathrm{GeLU}(XA_i)\, B_i\) on every GPU // (48.3); \(g\) = the two loops
How to read Figure 2
Press Play, or Step by hand. Step 1: every GPU gets the same \(X\) (no communication). Step 2: each GPU multiplies \(X\) by only its own columns of \(A\), so it holds a thin slice \(Y_i\). Step 3: GeLU on the slice, still alone. Step 4: each GPU multiplies by its own rows of \(B\). Now every panel has the full \(4 \times 4\) shape, but red: each is only a partial sum. Then the ring all-reduce. In each reduce-scatter step every GPU sends one chunk (one token row when \(t = 4\)) to the next GPU, which adds it; the row tags go up by one. After \(t - 1\) steps, GPU \(i\) has one finished (green) chunk. The all-gather steps then pass the finished chunks around until every GPU has all of \(Z\). The panel on the right of the code writes the formula of the current line with the live numbers. The last step compares \(Z\) with the single-GPU result \(\mathrm{GeLU}(XA)B\): the difference is only float rounding. Switch \(t\): with 2 GPUs each chunk is 2 rows and the ring has 1 + 1 steps. The counter “sent per GPU” ends at \(2(t-1)/t \cdot n h\) numbers, the ring all-reduce cost used in formula (48.7).
Why not split \(A\) by rows?
If \(A\) were split by rows, each GPU would get a partial sum \(X_i A_i\) of the pre-activation. But GeLU is not linear: \(\mathrm{GeLU}(a + b) \neq \mathrm{GeLU}(a) + \mathrm{GeLU}(b)\). So the GPUs would have to all-reduce before the GeLU, and again after \(B\): two synchronization points instead of one. The Megatron paper gives this exact reason. [Certain]
Suppose the true pre-activation of one feature is \(0\), made of two partial sums \(1\) and \(-1\). The correct value is \(\mathrm{GeLU}(0) = 0\). Applying GeLU to each partial sum and adding gives \(\mathrm{GeLU}(1) + \mathrm{GeLU}(-1) = 0.841 + (-0.159) = 0.683\). Wrong by 0.683. In the PyTorch code below, the “row first” version is off by up to 1.38.
Modern LLMs use SwiGLU: \(Z = \big(\mathrm{SiLU}(XW_1) \odot XW_3\big)W_2\). The same rule applies: \(W_1\) and \(W_3\) are column-parallel (split the same way, so the element-wise product stays local), and \(W_2\) is row-parallel. [Certain]
The two conjugate operators f and g
Megatron describes the communication with two small operators placed around the parallel block: [Certain]
- f, at the entrance: identity in the forward pass (every GPU already has \(X\)), all-reduce in the backward pass.
- g, at the exit: all-reduce in the forward pass (add the partial sums), identity in the backward pass.
Why does f need an all-reduce in the backward pass? \(X\) was used by all \(t\) GPUs, each through its own \(A_i\). By the chain rule, the gradient of a value used in several places is the sum of the gradients from each place (see session 4):
- Read the symbols
- \(\partial L / \partial Y_i\)
- the gradient that arrives at GPU \(i\)’s slice of the MLP features, computed from its own \(B_i\)
- \(\odot\)
- element-wise product
- \(\mathrm{GeLU}'\)
- the derivative of GeLU, applied element by element
- \(A_i^\top\)
- the transpose of GPU \(i\)’s weight block; each term of the sum is computed on GPU \(i\) only
Each GPU can compute only its own term. The sum is an all-reduce. And g is the identity in the backward pass because the gradient \(\partial L / \partial Z\) is already full and identical on every GPU (it comes from replicated computation after the block). The two operators are mirror images: what one does in the forward pass, the other does in the backward pass.
The weight gradients need no communication at all: GPU \(i\) owns \(A_i\) and \(B_i\), and computes their gradients from data it already has.
Attention: split by heads
Multi-head attention (session 19) is even easier to split, because the heads are already independent. Each head has its own slices of \(W_Q\), \(W_K\), \(W_V\), and computes its own attention. So:
- \(W_Q\), \(W_K\), \(W_V\) are column-parallel, split along heads. GPU \(i\) gets the columns of its group of \(a/t\) heads, where \(a\) is the number of heads.
- Each GPU runs softmax attention for its own heads, alone. No communication.
- The output projection \(W_O\) is row-parallel. Its input (the concatenated head outputs) is already split by heads, which is a split by columns.
- One all-reduce after \(W_O\) adds the partial sums.
- Read the symbols
- \(\mathcal{H}_i\)
- the set of heads that live on GPU \(i\) (\(a/t\) of them)
- \([\dots]\)
- the outputs of those heads placed side by side, shape \(n \times h/t\)
- \(W_O^{(i)}\)
- the rows of \(W_O\) that belong to those heads, shape \(h/t \times h\)
So one transformer layer needs exactly two all-reduces in the forward pass (one after attention, one after the MLP) and two in the backward pass (the f operators at the entrance of each block). The Megatron paper states the same count. [Certain]
What about LayerNorm, dropout and the residual additions? In plain Megatron tensor parallelism they are not split. Every GPU runs them on the full activations, with identical inputs, so they give identical outputs. This is cheap in FLOPs, but it costs memory, and fixing that is the job of sequence parallelism (below).
Figure 3 walks through one full layer. Turn on sequence parallelism to see how the same layer changes.
How to read Figure 3
Without sequence parallelism, the layer alternates between “full copy” (blue) and “split by features” (orange), with a partial sum (red) right before each all-reduce. The blue steps (LayerNorm, dropout, residual) are done 4 times, once per GPU, on the same data, and each GPU stores the full tensor. Turn on sequence parallelism: the blue steps become green (each GPU keeps only \(s/4\) tokens), and each all-reduce becomes a reduce-scatter plus an all-gather. The number of bytes sent stays the same.
Embeddings and the loss
The input embedding table and the output layer are big: Llama 3 has a vocabulary of 128,256 tokens. Megatron splits both along the vocabulary: [Certain]
- Input embedding: GPU \(i\) stores the rows for tokens \(V_i\) (a range of ids). For a token outside its range it outputs zeros. An all-reduce then gives every GPU the correct embedding.
- Output layer: this is a column-parallel linear layer. GPU \(i\) computes the logits for its own part of the vocabulary. Gathering all logits would move \(b \cdot s \cdot v\) numbers: for 8,192 tokens and \(v = 128{,}256\) that is 1.05 billion numbers, 2.1 GB in BF16. Instead the loss is computed in parallel:
Each GPU sends only a few numbers per token: its local maximum, its local sum of exponentials, and the logit of the target token if it owns it. Megatron reports that this turns a \(b \times s \times v\) communication into \(b \times s\). [Certain] In PyTorch this is loss_parallel().
What tensor parallelism costs: communication
Each all-reduce moves an activation tensor of shape \(b \times s \times h\) (batch, sequence, hidden), in BF16 (2 bytes per number). From session 43, a ring all-reduce over \(t\) GPUs makes each GPU send \(2(t-1)/t\) times the data. So per layer, per forward pass:
Notice what it depends on: the activations (\(b, s, h\)), not the number of parameters. Compare with data parallelism, which sends the gradients once per step. Tensor parallelism sends activations four times per layer (two forward, two backward), and the next layer cannot start until each all-reduce is finished. It is on the critical path.
Now compare with the compute. A standard GPT layer has \(12h^2\) weights, so its forward pass costs about \(2 \cdot 12h^2 = 24h^2\) FLOPs per token (ignoring the attention scores). With \(t\) GPUs, each does \(24 bsh^2 / t\). Divide the compute time by the communication time, with \(F\) = FLOP/s each GPU achieves and \(\beta\) = bandwidth in bytes/s:
- Read the symbols
- \(h\)
- hidden size: bigger models have more compute per byte sent, so tensor parallelism works better for them
- \(\beta\)
- the link bandwidth per GPU (bytes per second)
- \(F\)
- the FLOP/s one GPU really reaches (not the peak)
- \(t - 1\)
- grows with the tensor-parallel size: more GPUs means less compute each, but nearly the same communication each
The batch and the sequence length cancel out. This ratio depends only on the model width, the hardware, and \(t\). Above 1, compute dominates; below 1, the GPUs spend more time waiting than computing.
Take \(h = 8192\), \(F = 400\) TFLOP/s (what Llama 3 405B training reached per H100, Table 4 of the Llama 3 paper [Certain]), \(t = 8\). Formula 48.8 with NVLink (\(\beta = 450\) GB/s per direction on an H100): \(3 \times 8192 \times 450 \times 10^9 / (7 \times 400 \times 10^{12}) = 3.95\). With InfiniBand (\(\beta = 50\) GB/s per GPU): \(0.44\). Over the network, the GPUs would wait more than twice as long as they compute.
Now the exact layer, with GQA and SwiGLU, and \(b \cdot s = 8192\) tokens. The activation tensor is \(8192 \times 8192 \times 2\) bytes = 128 MiB. Each all-reduce sends \(2 \cdot \tfrac{7}{8} \cdot 134.2\) MB = 234.9 MB per GPU: 0.52 ms on NVLink. Four all-reduces (forward and backward): 2.09 ms. The forward pass of the layer is 15.1 TFLOP in total (14.0 in linear layers, 1.1 in causal attention); per GPU and with backward = 2 × forward, that is 14.2 ms. So communication is about 15% of the compute on NVLink, and 133% on InfiniBand (18.8 ms). These numbers ignore latency and overlap, so they are a best case.
Figure 4 lets you change the model, the tensor-parallel size, the tokens and the link.
How to read Figure 4
Start at the default (70B, \(t = 8\), NVLink): communication is about 15% of compute. Switch to InfiniBand: it becomes larger than the compute. Move \(t\) to 16: a ring of 16 GPUs must leave the 8-GPU server, so the figure uses the InfiniBand speed automatically. Change the tokens: both bars grow together, so the ratio barely moves (formula 48.8), except for the attention part, which grows with \(s^2\). Switch to the 8B model: its smaller \(h\) gives a worse ratio.
Why tensor parallelism stays inside one node
Put the pieces together:
- Bandwidth. Inside an H100 server, NVLink and NVSwitch give each GPU about 450 GB/s per direction to every other GPU. Between servers, each GPU usually has one 400 Gb/s InfiniBand card: 50 GB/s. That is a factor of 9 (see session 42). Formula 48.8 is directly proportional to \(\beta\).
- Frequency. Four blocking collectives per layer, times 80 layers, times every micro-batch. Data parallelism can hide its gradient all-reduce behind the backward pass; tensor parallelism can hide very little, because the next operation needs the result. [Likely]
- Latency. In inference, one decoding step for one sequence all-reduces a vector of only \(h\) numbers (16 KB for \(h = 8192\)), 160 times per token for an 80-layer model. Small messages are dominated by the fixed latency \(\alpha\) of each collective, and NVLink latency is much lower than the network’s. [Likely]
- Smaller matrices. With large \(t\), each GPU’s matrix multiplication gets thin (\(h \times f/t\)) and runs less efficiently. [Likely]
So the rule of thumb: tensor parallelism up to the number of GPUs in one server (8 on DGX and HGX machines), and other kinds of parallelism across servers. The Llama 3 paper says it directly: the innermost parallelism “requires the highest network bandwidth and lowest latency, and hence is usually constrained to within the same server”. They place the dimensions in the order [TP, CP, PP, DP], from inside to outside. [Certain] The original Megatron-LM used DGX-2H servers with 300 GB/s between GPUs inside a server and 100 GB/s per server between servers, and formed its model-parallel groups inside one server. [Certain]
| Llama 3 405B run | GPUs | TP | CP | PP | DP | Sequence | BF16 MFU |
|---|---|---|---|---|---|---|---|
| Pretraining | 8,192 | 8 | 1 | 16 | 64 | 8,192 | 43% |
| Pretraining | 16,384 | 8 | 1 | 16 | 128 | 8,192 | 41% |
| Long context | 16,384 | 8 | 16 | 16 | 8 | 131,072 | 38% |
(Table 4 of the Llama 3 paper. [Certain]) Tensor parallelism stays at 8 in every run; the other dimensions change. CP (context parallelism) and PP (pipeline parallelism) are the topic of the next session.
The memory problem that tensor parallelism leaves
Tensor parallelism divides the weights by \(t\): with \(t = 8\), each GPU stores about 17.6 GB of Llama 3 70B’s 141 GB of BF16 weights. It also divides the activations that live inside the split regions (the Q, K, V, the attention scores, the MLP features). But it does not divide the activations of the replicated parts: the inputs of the two LayerNorms, the dropout masks, and the tensors around the residual additions. Every GPU stores all of them, in full.
Korthikanti et al. counted the activation memory of one GPT layer, in bytes, with 16-bit activations: [Certain]
- Read the symbols
- \(s, b, h, a\)
- sequence length, micro-batch size, hidden size, number of attention heads
- \(34\, sbh\)
- the activations that grow linearly with the tokens: \(11sbh\) in attention, \(19sbh\) in the MLP, \(4sbh\) for the two LayerNorm inputs
- \(5as^2b\)
- the attention scores, softmax output and its dropout mask: they grow with \(s^2\)
- \(10\, sbh\)
- the part tensor parallelism does not divide: LayerNorm inputs and the dropout and residual tensors, stored in full on every GPU
With \(t = 8\), the divided part is \(24/8 = 3\), and the undivided part is 10. More than three quarters of the linear term is not helped by tensor parallelism at all.
Sequence parallelism
The fix is simple once you see it. LayerNorm and dropout work on each token separately: LayerNorm normalizes one token’s \(h\) features, dropout drops numbers independently. So in those regions we can split the tensor along the sequence dimension: GPU \(i\) keeps only tokens \(\tfrac{(i-1)s}{t} + 1\) to \(\tfrac{i s}{t}\). This is sequence parallelism (Korthikanti et al., 2022). [Certain]
Now the regions alternate: sequence-parallel (split by tokens) around LayerNorm, dropout and residual; tensor-parallel (split by features) in attention and the MLP. We need to switch between the two layouts. The key fact from session 43:
A ring all-reduce is a reduce-scatter (each GPU ends with one summed slice) followed by an all-gather (everyone gets all slices). Sequence parallelism simply moves the LayerNorm and dropout in between the two halves:
- At the exit of attention or the MLP, the partial sums are reduce-scattered along the sequence: each GPU gets the finished sum for its \(s/t\) tokens. This is operator \(\bar{g}\): reduce-scatter in forward, all-gather in backward.
- Dropout, the residual add and the next LayerNorm run on those \(s/t\) tokens only.
- At the entrance of the next block, an all-gather rebuilds the full sequence for the column-parallel layers, which need every token. This is operator \(g\): all-gather in forward, reduce-scatter in backward.
Each all-reduce has turned into one reduce-scatter plus one all-gather. A ring all-reduce costs exactly that, so the communication volume does not change. The paper puts it as: tensor parallelism needs four all-reduces in a forward and backward pass, tensor plus sequence parallelism needs four all-gathers and four reduce-scatters, with the same bandwidth. [Certain] Step through Figure 3 with the checkbox on to see it.
Figure 6 shows this with numbers, on the same toy MLP as Figure 2, now with a LayerNorm in front and the residual add after it (no dropout, so the result is exact). Switch between plain tensor parallelism and tensor plus sequence parallelism. Both use the same ring steps, the same number of them, and send the same number of values. The only difference is where the all-gather happens: after the reduce-scatter (plain TP, the two halves of one all-reduce) or before the MLP (sequence parallelism, with LayerNorm and the residual add done on \(n/t\) tokens in between).
- \(x_i \leftarrow\) my \(n/t\) tokens of \(x\) // sequence-parallel region
- \(u_i \leftarrow \mathrm{LayerNorm}(x_i)\) // only my tokens
- for \(r = 1 \dots t-1\): pass \(u\) chunks around the ring // all-gather (operator \(g\))
- \(P_i \leftarrow \mathrm{GeLU}(u A_i)\, B_i\) // column then row: partial sum
- for \(r = 1 \dots t-1\): pass chunks and add // reduce-scatter (operator \(\bar g\))
- \(x_i \leftarrow x_i + z_i\) // residual add on my tokens
- \(x \leftarrow\) all \(n\) tokens, on every GPU // replicated
- \(u \leftarrow \mathrm{LayerNorm}(x)\) // same work on every GPU
- \(P_i \leftarrow \mathrm{GeLU}(u A_i)\, B_i\) // column then row: partial sum
- for \(r = 1 \dots t-1\): pass chunks and add // reduce-scatter
- for \(r = 1 \dots t-1\): pass finished chunks // all-gather; both loops = all-reduce
- \(x \leftarrow x + z\) // residual add, full, on every GPU
How to read Figure 6
Start with “TP + sequence parallel” and press Play. Each GPU begins with only its own tokens (green rows) and runs LayerNorm on them: the panel on the right shows the mean, the variance and the normalized row for one token. The all-gather steps fill the missing rows, because the column-parallel layer needs every token. The MLP step gives red partial sums, and the reduce-scatter steps add them chunk by chunk until each GPU has its own tokens, finished. The residual add then runs on those tokens only. Now press “Plain TP” at any step: the step counter stays, the ring steps are the same, and the “sent” counter ends at the same value. But every GPU now stores all 16 residual-stream values instead of \(16/t\), and runs LayerNorm on all 4 tokens. The final step checks every output row against a single-GPU computation.
The memory now divides fully by \(t\):
The same paper adds selective activation recomputation: do not store the \(5as^2b\) attention tensors; recompute them in the backward pass. They are big in memory but cheap to recompute. FlashAttention does the same thing by design (session 39). With both, the memory is \(34\, sbh / t\). [Certain]
Take \(s = 2048\), \(h = 12288\), \(a = 96\), \(b = 1\), 96 layers, as in the paper. Then \(5as/h = 80\): the attention scores are more than twice the rest. Per layer, per GPU:
- no parallelism (48.9): 2.67 GiB; tensor parallelism (48.10): 0.539 GiB;
- tensor + sequence parallelism (48.12): 0.334 GiB;
- tensor parallelism + selective recompute: \(sbh(10 + 3)\) = 0.305 GiB;
- tensor + sequence parallelism + selective recompute: \(sbh \cdot 34/8\) = 0.100 GiB.
Over 96 layers that is 51.8 GiB with tensor parallelism alone, and 9.6 GiB with all three, a reduction of 5.4 times. The paper reports “about 5×” and, for the 530B model on 2,240 A100 GPUs, a model FLOPs utilization of 54.2% instead of 42.1% with full recomputation. [Certain]
How to read Figure 7
At \(t = 1\) the first three bars are equal: with one GPU there is nothing to split. As \(t\) grows, the “TP” bar stops shrinking: its 10\(sbh\) floor stays. With sequence parallelism the whole bar shrinks as \(1/t\). Increase \(s\): the dark attention part grows with \(s^2\) and soon dominates, which is why selective recompute (or FlashAttention) matters so much for long sequences. The formula assumes a GPT layer with dropout; models without dropout, such as Llama, store somewhat less. [Likely]
The algorithms
- input: full activations \(X\) (same on every GPU), weight shards \(A_i\) (columns) and \(B_i\) (rows)
- forward: \(X' \leftarrow f(X)\) // identity
- \(H_i \leftarrow \mathrm{GeLU}(X' A_i)\) // \(n \times f/t\), no communication
- \(P_i \leftarrow H_i B_i\) // partial sum, full shape \(n \times h\)
- \(Z \leftarrow g(P_i) = \text{all-reduce}_i(P_i)\) // formula (48.2)
- backward: given \(G = \partial L/\partial Z\) (same on every GPU), \(g\) passes it through unchanged
- \(\nabla B_i \leftarrow H_i^\top G\); \(\;G_{H} \leftarrow G B_i^\top \odot \mathrm{GeLU}'(X A_i)\)
- \(\nabla A_i \leftarrow X^\top G_H\) // weight gradients: local, no communication
- \(\partial L/\partial X \leftarrow f^{\text{bwd}}(G_H A_i^\top) = \text{all-reduce}_i(G_H A_i^\top)\) // formula (48.4)
- input: \(X_i\): this GPU’s \(s/t\) tokens of the residual stream, shape \(s/t \times h\)
- \(U_i \leftarrow \mathrm{LayerNorm}(X_i)\) // sequence-parallel region
- \(U \leftarrow \text{all-gather}(U_i)\) along tokens // operator \(g\): all tokens, shape \(s \times h\)
- \(Q, K, V \leftarrow U W_{QKV}^{(i)}\) // column-parallel: heads \(\mathcal{H}_i\) only
- \(O \leftarrow \text{attention}(Q, K, V)\) for heads \(\mathcal{H}_i\) // local, causal mask, full sequence
- \(P \leftarrow O\, W_O^{(i)}\) // row-parallel: partial sum \(s \times h\)
- \(A_i \leftarrow \text{reduce-scatter}(P)\) along tokens // operator \(\bar g\): my \(s/t\) tokens, fully summed
- \(X_i \leftarrow X_i + \mathrm{Dropout}(A_i)\) // sequence-parallel: residual add
- repeat steps 2 to 8 for the MLP: LayerNorm, all-gather, \(W_1, W_3\) column-parallel, SwiGLU, \(W_2\) row-parallel, reduce-scatter, residual
- backward: every all-gather becomes a reduce-scatter and every reduce-scatter an all-gather; LayerNorm weight gradients are all-reduced over the \(t\) GPUs, because each GPU saw only \(s/t\) tokens
The last line hides a classic bug. In plain tensor parallelism the LayerNorm weights get the same gradient on every GPU (each saw the full sequence). With sequence parallelism, each GPU saw different tokens, so the LayerNorm gradients are different partial sums and must be all-reduced across the tensor-parallel group. Forget it and the copies of the LayerNorm weights slowly drift apart. [Likely]
PyTorch: checking the math on one machine
This code needs no GPUs. It simulates \(t = 4\) GPUs with slices of tensors and checks every claim of this session: the column-then-row MLP is exact (48.3), the row-first version is wrong, the input gradient needs an all-reduce (48.4), a reduce-scatter equals an all-reduce followed by a slice (48.11), and LayerNorm on token chunks equals LayerNorm on the full sequence.
import torch torch.manual_seed(0) t, n_tok, h, ffn = 4, 8, 8, 32 # 4 "GPUs", 8 tokens, hidden 8, MLP width 32 X = torch.randn(n_tok, h, dtype=torch.float64) A = torch.randn(h, ffn, dtype=torch.float64) / h ** 0.5 # first linear (h -> ffn) Bm = torch.randn(ffn, h, dtype=torch.float64) / ffn ** 0.5 # second linear (ffn -> h) gelu = torch.nn.functional.gelu # 1) dense reference, formula (48.3) Y_ref = gelu(X @ A) @ Bm # 2) Megatron MLP: A split by columns, B split by rows A_parts = A.chunk(t, dim=1) # each h x (ffn/t) B_parts = Bm.chunk(t, dim=0) # each (ffn/t) x h partial = [gelu(X @ A_parts[i]) @ B_parts[i] for i in range(t)] # no communication Y_tp = sum(partial) # the all-reduce (operator g) print("MLP forward max error:", (Y_tp - Y_ref).abs().max().item()) # 3) splitting A by rows first is wrong: GeLU of a sum is not the sum of GeLUs X_parts, A_rows = X.chunk(t, dim=1), A.chunk(t, dim=0) wrong = sum(gelu(X_parts[i] @ A_rows[i]) for i in range(t)) @ Bm print("row-first error:", (wrong - Y_ref).abs().max().item()) # 4) backward: dL/dX is the SUM of the per-GPU input gradients, formula (48.4) G = torch.randn(n_tok, h, dtype=torch.float64) # gradient arriving from above Xr = X.clone().requires_grad_(True); (gelu(Xr @ A) @ Bm).backward(G) grads = [] for i in range(t): Xi = X.clone().requires_grad_(True) # each GPU has its own copy of X (gelu(Xi @ A_parts[i]) @ B_parts[i]).backward(G) # g is the identity in backward grads.append(Xi.grad) print("dX from one GPU only:", (grads[0] - Xr.grad).abs().max().item()) print("dX after all-reduce: ", (sum(grads) - Xr.grad).abs().max().item()) # operator f # 5) sequence parallelism: reduce-scatter = all-reduce, then keep my slice of tokens full = sum(partial) rs = [sum(p.chunk(t, dim=0)[r] for p in partial) for r in range(t)] print("reduce-scatter vs slice:", max((rs[r] - full.chunk(t, 0)[r]).abs().max().item() for r in range(t))) ln = torch.nn.LayerNorm(h, dtype=torch.float64) print("LayerNorm on token chunks:", (torch.cat([ln(c) for c in X.chunk(t, 0)]) - ln(X)).abs().max().item())
Output (PyTorch 2.8, CPU): the forward error is about \(10^{-16}\), the row-first error is 1.38, the gradient from one GPU alone is off by 2.48, the all-reduced gradient is off by about \(10^{-15}\), and the last two checks print exactly 0.
PyTorch: real tensor and sequence parallelism with DTensor
PyTorch describes the same plan with parallelize_module: you say which layer is ColwiseParallel, which is RowwiseParallel, and which runs SequenceParallel. The library inserts the collectives. This script starts 4 processes on the CPU (the gloo backend), so it runs on a laptop; on GPUs you would use torchrun, "nccl" and "cuda". It builds a pre-norm MLP block, parallelizes it with sequence parallelism, feeds each process a quarter of the tokens, and compares the output and a weight gradient with the unsplit model.
import os, torch, torch.distributed as dist, torch.multiprocessing as mp from torch.distributed.device_mesh import init_device_mesh from torch.distributed.tensor import Shard from torch.distributed.tensor.parallel import (parallelize_module, ColwiseParallel, RowwiseParallel, SequenceParallel) class MLPBlock(torch.nn.Module): # x + W2(gelu(W1(LayerNorm(x)))) def __init__(self, h, ffn): super().__init__() self.norm = torch.nn.LayerNorm(h) self.w1 = torch.nn.Linear(h, ffn, bias=False) self.w2 = torch.nn.Linear(ffn, h, bias=False) def forward(self, x): return x + self.w2(torch.nn.functional.gelu(self.w1(self.norm(x)))) def run(rank, world): os.environ.update(MASTER_ADDR="127.0.0.1", MASTER_PORT="29511") dist.init_process_group("gloo", rank=rank, world_size=world) mesh = init_device_mesh("cpu", (world,), mesh_dim_names=("tp",)) torch.manual_seed(0) # same weights and data everywhere ref = MLPBlock(16, 64); x = torch.randn(2, 8, 16) # (batch, seq, hidden) tp = MLPBlock(16, 64); tp.load_state_dict(ref.state_dict()) parallelize_module(tp, mesh, { "norm": SequenceParallel(), # LayerNorm on my s/t tokens "w1": ColwiseParallel(input_layouts=Shard(1)), # all-gather tokens (g), split columns "w2": RowwiseParallel(output_layouts=Shard(1)), # reduce-scatter to token slices (g-bar) }) print(rank, tuple(tp.w1.weight.to_local().shape)) # (16, 16): a quarter of the 64 rows x_local = x.chunk(world, dim=1)[rank] # my slice of the sequence y_local = tp(x_local) y_ref = ref(x) err = (y_local - y_ref.chunk(world, dim=1)[rank]).abs().max() y_ref.sum().backward(); y_local.sum().backward() gerr = (tp.w1.weight.grad.full_tensor() - ref.w1.weight.grad).abs().max() if rank == 0: print("forward error", err.item(), "w1 grad error", gerr.item()) dist.destroy_process_group() if __name__ == "__main__": mp.spawn(run, args=(4,), nprocs=4)
Output (PyTorch 2.8, CPU): each rank holds a \(16 \times 16\) slice of w1 (PyTorch stores the weight as out × in, so a column-parallel layer splits its first dimension), the forward error is about \(10^{-7}\) (float32), and the gradient error is 0. The PyTorch tutorial uses the same pattern for a whole Llama block: wq, wk, wv, w1, w3 column-wise, wo and w2 row-wise with output_layouts=Shard(1), and the norms with SequenceParallel(). [Certain] It then wraps the result with FSDP2 over a second mesh dimension: TP inside a host, FSDP across hosts.
Where this is used
| Where | How tensor or sequence parallelism is used | Note |
|---|---|---|
| Megatron-LM, NeMo | column and row parallel layers, sequence parallelism, vocabulary-parallel loss | the reference implementation of this session [Certain] |
| Llama 3 405B pretraining | TP = 8 inside each server, with PP, CP and FSDP-style DP outside | Table above [Certain] |
| PyTorch torchtitan | parallelize_module plans with FSDP2 on a 2D mesh | see session 47 |
| LLM serving (vLLM, TensorRT-LLM, SGLang) | tensor_parallel_size: one model on several GPUs, all GPUs work on every token | small all-reduces; NVLink latency matters [Likely] |
| Vision-language models | the language model is tensor-parallel; the vision encoder is smaller and is often replicated or split the same way | [Likely] |
| Long video understanding (video QA, key-moment detection, Video-RAG over long clips) | thousands of visual tokens per clip: activation memory grows with \(s\), so sequence parallelism and selective recompute matter; very long clips need context parallelism | [Likely] |
| Video generation (diffusion transformers) | very long token sequences of space-time patches; tensor and sequence parallelism inside a node, plus context parallelism | [Likely] |
Common mistakes
- Splitting the first MLP layer by rows. Then GeLU sees partial sums. You need an extra all-reduce before the nonlinearity, or the result is wrong.
- Splitting \(W_1\) and \(W_3\) of SwiGLU differently. The element-wise product needs the same features on the same GPU. Both must be split by the same column ranges.
- Forgetting to change the head count. After splitting, each GPU has \(a/t\) query heads and \(n_{kv}/t\) KV heads. Reshape code that still uses the global \(a\) produces wrong shapes or silently wrong attention.
- Same dropout seed everywhere, or different seeds everywhere. Megatron uses the same seed for dropout in replicated regions (so the copies stay identical) and a different seed per GPU inside the tensor-parallel regions (so different features get independent masks). [Certain] Getting this wrong makes the copies diverge or correlates the masks.
- Not all-reducing LayerNorm gradients under sequence parallelism. Each GPU saw different tokens; the copies of the LayerNorm weights drift apart.
- Tensor parallelism across servers. Four blocking all-reduces per layer over 50 GB/s links: Figure 4 shows the GPUs waiting more than they compute.
- Thinking sequence parallelism adds communication. It does not: each all-reduce is replaced by a reduce-scatter plus an all-gather of the same total size.
- \(t\) that does not divide the heads, the KV heads or the MLP width. Every split dimension must be divisible by \(t\) (or KV heads must be copied).
Interview questions
What is the difference between a column-parallel and a row-parallel linear layer?basic
In Megatron’s MLP, why is the first layer column-parallel and the second row-parallel?core
How many collectives does one transformer layer need with tensor parallelism? Where are they?core
Explain the f and g operators. Why is f an all-reduce in the backward pass?core
How is attention split, and what limits the tensor-parallel size there?core
What problem does sequence parallelism solve, and how?core
Does sequence parallelism increase communication? Prove your answer.advanced
Why does tensor parallelism usually stay inside one node?basic
Compare tensor parallelism with FSDP. When do you use each, or both?advanced
How is the output layer and the cross-entropy loss handled with a 128K vocabulary?advanced
You serve a 70B model with TP = 8 and decoding is slower than expected. What do you look at?advanced
Exercises
- In Figure 1, use the row split with \(t = 4\). Write the partial result of GPU 2 and check that the four partial results add up to \(Y\).
- A GPT layer with \(h = 12288\), \(b \cdot s = 4096\) tokens, \(t = 8\). How many bytes does each GPU send per layer in the forward pass (formula 48.7)?
- Using formula 48.8, for which \(t\) does the forward compute/communication ratio of an \(h = 4096\) model drop below 2 on NVLink (\(\beta = 450\) GB/s, \(F = 400\) TFLOP/s)?
- For GPT-3 175B (\(s = 2048\), \(h = 12288\), \(a = 96\), \(b = 1\)), compute the activation memory per layer with \(t = 4\), with and without sequence parallelism.
- A model has 40 query heads and 8 KV heads. Which tensor-parallel sizes from \(\{2, 4, 8, 16\}\) work without copying KV heads?
- Why is the gradient of the LayerNorm weights identical on all GPUs under plain tensor parallelism, but not under sequence parallelism?
Answerscheck
Key takeaways
- A matrix multiplication splits by columns (no communication, output split by features, 48.1) or by rows (partial sums, one all-reduce, 48.2).
- Megatron pairs them: column-parallel then row-parallel, with the element-wise nonlinearity in between. Attention is split by heads the same way. Result: 2 all-reduces per layer in forward (operator g) and 2 in backward (operator f, 48.4).
- Communication is activations, \(8(t-1)bsh/t\) bytes per GPU per layer forward (48.7); compute/communication is \(3h\beta/((t-1)F)\) (48.8). That is why TP lives on NVLink, inside one server, usually with \(t \le 8\).
- Plain TP leaves \(10sbh\) of replicated activations (48.10). Sequence parallelism splits LayerNorm, dropout and residual regions along tokens, turning each all-reduce into reduce-scatter + all-gather (48.11) at the same cost, and divides all activations by \(t\) (48.12).
- Vocabulary-parallel embeddings and a parallel cross-entropy (48.6) avoid gathering huge logits. In practice: TP inside a node, FSDP or DP across nodes, plus pipeline and context parallelism for the largest runs.
Key formulas
References for this session
- Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Narayanan et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- PyTorch (docs). Large Scale Transformer model training with Tensor Parallel (TP)
- Li, Xue, Baranwal, Li and You (2021). Sequence Parallelism: Long Sequence Training from System Perspective
- Jacobs et al. (2023). DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
- Liu et al. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context
- Austin et al., Google DeepMind (2025). How to Scale Your Model