LLM & VLM courseBlog
48Track 3: Pretraining and scale · session 18 of 22

Tensor parallelism and sequence parallelism

Tensor parallelism splits the matrix multiplications inside every layer across a few GPUs, so that they share the work and the memory of one layer. This session shows the Megatron recipe (column-parallel then row-parallel layers, attention split by heads, two all-reduces per layer in the forward pass), how sequence parallelism removes the activation memory that tensor parallelism leaves behind, and why all of this lives on fast NVLink inside one server.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 43 and 46 (collective communication, FSDP); the transformer block from Track 2

What you will learn

  • How a matrix multiplication splits by columns (no communication) or by rows (partial sums and one all-reduce)
  • The Megatron MLP and attention: column-parallel then row-parallel, heads split across GPUs, and the f and g operators
  • Why one transformer layer needs two all-reduces in the forward pass and two in the backward pass
  • Vocabulary-parallel embeddings and a parallel cross-entropy that never gathers the logits
  • How to estimate the communication cost, and why the compute/communication ratio is 3hβ/((t−1)F)
  • Sequence parallelism: splitting LayerNorm, dropout and residual regions by tokens, at no extra communication
  • Why tensor parallelism stays inside one node, and how real runs such as Llama 3 405B place it

Why this matters

FSDP and ZeRO cut the memory of the weights by storing each weight on only one GPU. But when a layer runs, every GPU still gathers the full layer and does the full matrix multiplication for its own batch. Tensor parallelism does something different: it cuts the matrix multiplication itself into pieces, so that each GPU does only a part of the math of every layer.

Why would you want that? Three reasons.

  • One layer can be too big. Llama 3 70B has 80 layers with \(h = 8192\). Its weights alone are about 141 GB in BF16. FSDP can shard them, but each GPU must still hold one full gathered layer plus all the activations of its micro-batch. With long sequences, the activations are the bigger problem.
  • The batch per GPU cannot grow forever. Data parallelism (DDP, FSDP) needs at least one sequence per GPU. With 16,384 GPUs and 16M tokens per step, there are not enough sequences to give each GPU its own. Tensor parallelism lets 8 GPUs share one sequence.
  • Serving. To generate text from a 70B model fast, you put it on 8 GPUs and let all 8 work on every token. This is exactly tensor parallelism. In vLLM it is the option tensor_parallel_size. [Certain]

Real systems use it everywhere. Megatron-LM introduced the method in 2019 and trained an 8.3B model on 512 GPUs at 15.1 PetaFLOPs, with 8-way tensor parallelism inside each server. [Certain] Llama 3 405B was trained with tensor parallelism of size 8, inside each server, combined with pipeline, context and data parallelism. [Certain]

This session builds on session 43 (all-reduce, reduce-scatter, all-gather and their cost) and on the FSDP sessions (session 46, session 47). The next session, session 49, adds the other ways to split a model, and shows how all of them are combined.

Splitting one matrix multiplication

Everything in this session comes from one fact: a matrix multiplication can be cut into independent pieces in two ways. Take a linear layer \(Y = XA\), with input \(X \in \R^{n \times d}\) (\(n\) tokens, \(d\) features) and weight \(A \in \R^{d \times m}\). We have \(t\) GPUs. In this session \(t\) always means the tensor-parallel size: the number of GPUs that share one layer.

Column split (column-parallel linear)

Cut \(A\) into \(t\) blocks of columns, \(A = [A_1, A_2, \dots, A_t]\), and give block \(A_i\) to GPU \(i\). Each GPU has the full input \(X\). Then each GPU computes its own block of output columns:

\[ Y = XA = \big[\, XA_1,\ XA_2,\ \dots,\ XA_t \,\big], \qquad Y_i = XA_i \in \R^{n \times m/t} \tag{48.1} \]

No GPU needs anything from another GPU. The result is split by columns: GPU \(i\) holds \(Y_i\), one slice of the output features. If you want the full \(Y\) on every GPU, you need an all-gather. But often you do not want it, as we will see.

Row split (row-parallel linear)

Now cut \(A\) into \(t\) blocks of rows, \(A = [A_1; A_2; \dots; A_t]\) (stacked on top of each other), and cut the input \(X\) the same way, by columns: \(X = [X_1, X_2, \dots, X_t]\). Then

\[ Y = XA = X_1 A_1 + X_2 A_2 + \dots + X_t A_t = \sum_{i=1}^{t} X_i A_i \tag{48.2} \]
Read the symbols
\(X_i\)
the input columns (features) that belong to GPU \(i\), shape \(n \times d/t\)
\(A_i\)
the weight rows that belong to GPU \(i\), shape \(d/t \times m\)
\(X_i A_i\)
a partial sum: full output shape \(n \times m\), but only a part of the total

Each GPU computes a full-size output, but it is only a partial sum. To get the true \(Y\), the GPUs must add their partial sums: that is one all-reduce. Figure 1 shows both splits on a tiny example with integers.

Figure 1. The two ways to split \(Y = XA\). Each color is one GPU. Click a GPU name below the picture, or a colored cell, to see what that GPU computes.
How to read Figure 1

In the column split, every GPU holds all of \(X\) (gray) and a few columns of \(A\). Its output is a few finished columns of \(Y\). Nobody talks to anybody. In the row split, each GPU holds a few columns of \(X\) and the matching rows of \(A\). Its output has the full shape of \(Y\), but the numbers are wrong on their own: you must add the outputs of all GPUs (an all-reduce) to get \(Y\). Check that the sum of the partial results equals the true \(Y\) printed in the readout.

Worked example: the numbers of Figure 1

\(X = \begin{bmatrix} 1 & 2 & 0 & -1 \\ 2 & 0 & 1 & 1 \end{bmatrix}\) and \(A\) is the \(4 \times 4\) matrix in the figure. The true result is \(Y = XA = \begin{bmatrix} -1 & 3 & 3 & -1 \\ 5 & 0 & 5 & 4 \end{bmatrix}\).

Row split with \(t = 2\): GPU 1 uses the first two columns of \(X\) and the first two rows of \(A\) and gets \(\begin{bmatrix} 1 & 2 & 4 & -1 \\ 2 & 0 & 4 & 2 \end{bmatrix}\). GPU 2 gets \(\begin{bmatrix} -2 & 1 & -1 & 0 \\ 3 & 0 & 1 & 2 \end{bmatrix}\). Neither is \(Y\), but their sum is: \(1 + (-2) = -1\), \(2 + 1 = 3\), and so on.

The Megatron MLP: column first, then row

The MLP of a transformer layer (see session 21) is two linear layers with a nonlinearity between them:

\[ Z = \mathrm{GeLU}(XA)\,B, \qquad A \in \R^{h \times f},\ \ B \in \R^{f \times h} \tag{48.3} \]

Here \(h\) is the hidden size and \(f\) the MLP width (often \(4h\)). Megatron-LM splits it like this: [Certain]

  1. \(A\) by columns (column-parallel). Every GPU has the full \(X\) and computes \(XA_i\): its own \(f/t\) MLP features.
  2. GeLU on each GPU, alone. GeLU works on each number separately, so applying it to a slice of columns gives exactly the same slice of \(\mathrm{GeLU}(XA)\).
  3. \(B\) by rows (row-parallel). The input of a row-parallel layer must be split by columns, and that is exactly what step 2 produced. No communication is needed between the two layers.
  4. One all-reduce adds the partial sums: \(Z = \sum_i \mathrm{GeLU}(XA_i)\, B_i\).

So the whole MLP needs one all-reduce in the forward pass. The trick is the pairing: a column-parallel layer leaves its output split in exactly the form that a row-parallel layer wants as input.

Figure 2 runs this recipe with real numbers. It is the forward pass of Algorithm 1 (below) on a toy MLP: \(n = 4\) tokens, hidden size \(h = 4\), MLP width \(f = 8\). Every number is computed in your browser. The last part of the animation is the all-reduce itself, done as a ring: first a reduce-scatter (each GPU sends one chunk of its partial sum to the next GPU, which adds it to its own), then an all-gather (the finished chunks travel around the ring once more). These are the two halves from session 43, and they will matter again for sequence parallelism.

  1. \(X \leftarrow f(X)\) // identity: every GPU has all of \(X\)
  2. \(Y_i \leftarrow X A_i\) // column-parallel, (48.1)
  3. \(H_i \leftarrow \mathrm{GeLU}(Y_i)\) // element-wise, local
  4. \(P_i \leftarrow H_i B_i\) // row-parallel: partial sum, (48.2)
  5. for \(r = 1 \dots t-1\): send chunk \(i - r\) to GPU \(i+1\), which adds it // reduce-scatter
  6. for \(r = 1 \dots t-1\): send a finished chunk to GPU \(i+1\), which copies it // all-gather
  7. \(Z = \sum_i \mathrm{GeLU}(XA_i)\, B_i\) on every GPU // (48.3); \(g\) = the two loops
GPU 1GPU 2GPU 3GPU 4partial sumfinished (all partials added)
Figure 2. The Megatron MLP \(Z = \mathrm{GeLU}(XA)\,B\) on \(t\) GPUs, step by step. Left: the full matrices, with the columns of \(A\) and the rows of \(B\) colored by the GPU that owns them. Right: one panel per GPU with the tensor it holds at this step. The arrows are the ring links; a row tag such as 2/4 counts how many partial sums that row already contains (on narrow screens the shade of red shows it).
How to read Figure 2

Press Play, or Step by hand. Step 1: every GPU gets the same \(X\) (no communication). Step 2: each GPU multiplies \(X\) by only its own columns of \(A\), so it holds a thin slice \(Y_i\). Step 3: GeLU on the slice, still alone. Step 4: each GPU multiplies by its own rows of \(B\). Now every panel has the full \(4 \times 4\) shape, but red: each is only a partial sum. Then the ring all-reduce. In each reduce-scatter step every GPU sends one chunk (one token row when \(t = 4\)) to the next GPU, which adds it; the row tags go up by one. After \(t - 1\) steps, GPU \(i\) has one finished (green) chunk. The all-gather steps then pass the finished chunks around until every GPU has all of \(Z\). The panel on the right of the code writes the formula of the current line with the live numbers. The last step compares \(Z\) with the single-GPU result \(\mathrm{GeLU}(XA)B\): the difference is only float rounding. Switch \(t\): with 2 GPUs each chunk is 2 rows and the ring has 1 + 1 steps. The counter “sent per GPU” ends at \(2(t-1)/t \cdot n h\) numbers, the ring all-reduce cost used in formula (48.7).

Why not split \(A\) by rows?

If \(A\) were split by rows, each GPU would get a partial sum \(X_i A_i\) of the pre-activation. But GeLU is not linear: \(\mathrm{GeLU}(a + b) \neq \mathrm{GeLU}(a) + \mathrm{GeLU}(b)\). So the GPUs would have to all-reduce before the GeLU, and again after \(B\): two synchronization points instead of one. The Megatron paper gives this exact reason. [Certain]

Worked example: GeLU does not split over a sum

Suppose the true pre-activation of one feature is \(0\), made of two partial sums \(1\) and \(-1\). The correct value is \(\mathrm{GeLU}(0) = 0\). Applying GeLU to each partial sum and adding gives \(\mathrm{GeLU}(1) + \mathrm{GeLU}(-1) = 0.841 + (-0.159) = 0.683\). Wrong by 0.683. In the PyTorch code below, the “row first” version is off by up to 1.38.

Modern LLMs use SwiGLU: \(Z = \big(\mathrm{SiLU}(XW_1) \odot XW_3\big)W_2\). The same rule applies: \(W_1\) and \(W_3\) are column-parallel (split the same way, so the element-wise product stays local), and \(W_2\) is row-parallel. [Certain]

The two conjugate operators f and g

Megatron describes the communication with two small operators placed around the parallel block: [Certain]

  • f, at the entrance: identity in the forward pass (every GPU already has \(X\)), all-reduce in the backward pass.
  • g, at the exit: all-reduce in the forward pass (add the partial sums), identity in the backward pass.

Why does f need an all-reduce in the backward pass? \(X\) was used by all \(t\) GPUs, each through its own \(A_i\). By the chain rule, the gradient of a value used in several places is the sum of the gradients from each place (see session 4):

\[ \frac{\partial L}{\partial X} = \sum_{i=1}^{t} \frac{\partial L}{\partial X}\bigg|_{\text{through } A_i} = \sum_{i=1}^{t} \Big( \frac{\partial L}{\partial Y_i} \odot \mathrm{GeLU}'(XA_i) \Big) A_i^\top \tag{48.4} \]
Read the symbols
\(\partial L / \partial Y_i\)
the gradient that arrives at GPU \(i\)’s slice of the MLP features, computed from its own \(B_i\)
\(\odot\)
element-wise product
\(\mathrm{GeLU}'\)
the derivative of GeLU, applied element by element
\(A_i^\top\)
the transpose of GPU \(i\)’s weight block; each term of the sum is computed on GPU \(i\) only

Each GPU can compute only its own term. The sum is an all-reduce. And g is the identity in the backward pass because the gradient \(\partial L / \partial Z\) is already full and identical on every GPU (it comes from replicated computation after the block). The two operators are mirror images: what one does in the forward pass, the other does in the backward pass.

The weight gradients need no communication at all: GPU \(i\) owns \(A_i\) and \(B_i\), and computes their gradients from data it already has.

Attention: split by heads

Multi-head attention (session 19) is even easier to split, because the heads are already independent. Each head has its own slices of \(W_Q\), \(W_K\), \(W_V\), and computes its own attention. So:

  1. \(W_Q\), \(W_K\), \(W_V\) are column-parallel, split along heads. GPU \(i\) gets the columns of its group of \(a/t\) heads, where \(a\) is the number of heads.
  2. Each GPU runs softmax attention for its own heads, alone. No communication.
  3. The output projection \(W_O\) is row-parallel. Its input (the concatenated head outputs) is already split by heads, which is a split by columns.
  4. One all-reduce after \(W_O\) adds the partial sums.
\[ \mathrm{MHA}(X) = \sum_{i=1}^{t} \Big[\, \mathrm{head}_{j}(X) \ \text{ for } j \in \mathcal{H}_i \,\Big]\, W_O^{(i)} \tag{48.5} \]
Read the symbols
\(\mathcal{H}_i\)
the set of heads that live on GPU \(i\) (\(a/t\) of them)
\([\dots]\)
the outputs of those heads placed side by side, shape \(n \times h/t\)
\(W_O^{(i)}\)
the rows of \(W_O\) that belong to those heads, shape \(h/t \times h\)

So one transformer layer needs exactly two all-reduces in the forward pass (one after attention, one after the MLP) and two in the backward pass (the f operators at the entrance of each block). The Megatron paper states the same count. [Certain]

What about LayerNorm, dropout and the residual additions? In plain Megatron tensor parallelism they are not split. Every GPU runs them on the full activations, with identical inputs, so they give identical outputs. This is cheap in FLOPs, but it costs memory, and fixing that is the job of sequence parallelism (below).

Figure 3 walks through one full layer. Turn on sequence parallelism to see how the same layer changes.

full copy (replicated)split along hidden features (tensor-parallel)split along tokens (sequence-parallel)partial sum (must be reduced)
Figure 3. One transformer layer on \(t = 4\) GPUs. Each small box is the activation tensor of one GPU: tokens go down (\(s\) rows), hidden features go across (\(h\) columns). The colored part is what that GPU holds. The counter at the top right counts collectives in the forward pass.
How to read Figure 3

Without sequence parallelism, the layer alternates between “full copy” (blue) and “split by features” (orange), with a partial sum (red) right before each all-reduce. The blue steps (LayerNorm, dropout, residual) are done 4 times, once per GPU, on the same data, and each GPU stores the full tensor. Turn on sequence parallelism: the blue steps become green (each GPU keeps only \(s/4\) tokens), and each all-reduce becomes a reduce-scatter plus an all-gather. The number of bytes sent stays the same.

Embeddings and the loss

The input embedding table and the output layer are big: Llama 3 has a vocabulary of 128,256 tokens. Megatron splits both along the vocabulary: [Certain]

  • Input embedding: GPU \(i\) stores the rows for tokens \(V_i\) (a range of ids). For a token outside its range it outputs zeros. An all-reduce then gives every GPU the correct embedding.
  • Output layer: this is a column-parallel linear layer. GPU \(i\) computes the logits for its own part of the vocabulary. Gathering all logits would move \(b \cdot s \cdot v\) numbers: for 8,192 tokens and \(v = 128{,}256\) that is 1.05 billion numbers, 2.1 GB in BF16. Instead the loss is computed in parallel:
\[ \log \sum_{v=1}^{V} e^{z_v} = m + \log \sum_{i=1}^{t} \underbrace{\sum_{v \in V_i} e^{z_v - m}}_{\text{local on GPU } i}, \qquad m = \max_i \max_{v \in V_i} z_v \tag{48.6} \]

Each GPU sends only a few numbers per token: its local maximum, its local sum of exponentials, and the logit of the target token if it owns it. Megatron reports that this turns a \(b \times s \times v\) communication into \(b \times s\). [Certain] In PyTorch this is loss_parallel().

What tensor parallelism costs: communication

Each all-reduce moves an activation tensor of shape \(b \times s \times h\) (batch, sequence, hidden), in BF16 (2 bytes per number). From session 43, a ring all-reduce over \(t\) GPUs makes each GPU send \(2(t-1)/t\) times the data. So per layer, per forward pass:

\[ \text{bytes sent per GPU} = \underbrace{2}_{\text{all-reduces}} \times \frac{2(t-1)}{t} \times \underbrace{2\, b s h}_{\text{BF16 tensor}} = \frac{8(t-1)}{t}\, b s h \tag{48.7} \]

Notice what it depends on: the activations (\(b, s, h\)), not the number of parameters. Compare with data parallelism, which sends the gradients once per step. Tensor parallelism sends activations four times per layer (two forward, two backward), and the next layer cannot start until each all-reduce is finished. It is on the critical path.

Now compare with the compute. A standard GPT layer has \(12h^2\) weights, so its forward pass costs about \(2 \cdot 12h^2 = 24h^2\) FLOPs per token (ignoring the attention scores). With \(t\) GPUs, each does \(24 bsh^2 / t\). Divide the compute time by the communication time, with \(F\) = FLOP/s each GPU achieves and \(\beta\) = bandwidth in bytes/s:

\[ \frac{T_{\text{compute}}}{T_{\text{comm}}} = \frac{24\, b s h^2 / (t F)}{8 (t-1)\, b s h / (t \beta)} = \frac{3\, h\, \beta}{(t - 1)\, F} \tag{48.8} \]
Read the symbols
\(h\)
hidden size: bigger models have more compute per byte sent, so tensor parallelism works better for them
\(\beta\)
the link bandwidth per GPU (bytes per second)
\(F\)
the FLOP/s one GPU really reaches (not the peak)
\(t - 1\)
grows with the tensor-parallel size: more GPUs means less compute each, but nearly the same communication each

The batch and the sequence length cancel out. This ratio depends only on the model width, the hardware, and \(t\). Above 1, compute dominates; below 1, the GPUs spend more time waiting than computing.

Worked example: Llama 3 70B layer, \(t = 8\)

Take \(h = 8192\), \(F = 400\) TFLOP/s (what Llama 3 405B training reached per H100, Table 4 of the Llama 3 paper [Certain]), \(t = 8\). Formula 48.8 with NVLink (\(\beta = 450\) GB/s per direction on an H100): \(3 \times 8192 \times 450 \times 10^9 / (7 \times 400 \times 10^{12}) = 3.95\). With InfiniBand (\(\beta = 50\) GB/s per GPU): \(0.44\). Over the network, the GPUs would wait more than twice as long as they compute.

Now the exact layer, with GQA and SwiGLU, and \(b \cdot s = 8192\) tokens. The activation tensor is \(8192 \times 8192 \times 2\) bytes = 128 MiB. Each all-reduce sends \(2 \cdot \tfrac{7}{8} \cdot 134.2\) MB = 234.9 MB per GPU: 0.52 ms on NVLink. Four all-reduces (forward and backward): 2.09 ms. The forward pass of the layer is 15.1 TFLOP in total (14.0 in linear layers, 1.1 in causal attention); per GPU and with backward = 2 × forward, that is 14.2 ms. So communication is about 15% of the compute on NVLink, and 133% on InfiniBand (18.8 ms). These numbers ignore latency and overlap, so they are a best case.

Figure 4 lets you change the model, the tensor-parallel size, the tokens and the link.

Figure 4. Time per layer for one training step (forward and backward) on one GPU, using the real shapes of Llama 3 (GQA, SwiGLU). Compute assumes 400 TFLOP/s per GPU; communication is four ring all-reduces at the chosen bandwidth, with no overlap and no latency.
How to read Figure 4

Start at the default (70B, \(t = 8\), NVLink): communication is about 15% of compute. Switch to InfiniBand: it becomes larger than the compute. Move \(t\) to 16: a ring of 16 GPUs must leave the 8-GPU server, so the figure uses the InfiniBand speed automatically. Change the tokens: both bars grow together, so the ratio barely moves (formula 48.8), except for the attention part, which grows with \(s^2\). Switch to the 8B model: its smaller \(h\) gives a worse ratio.

Why tensor parallelism stays inside one node

Put the pieces together:

  • Bandwidth. Inside an H100 server, NVLink and NVSwitch give each GPU about 450 GB/s per direction to every other GPU. Between servers, each GPU usually has one 400 Gb/s InfiniBand card: 50 GB/s. That is a factor of 9 (see session 42). Formula 48.8 is directly proportional to \(\beta\).
  • Frequency. Four blocking collectives per layer, times 80 layers, times every micro-batch. Data parallelism can hide its gradient all-reduce behind the backward pass; tensor parallelism can hide very little, because the next operation needs the result. [Likely]
  • Latency. In inference, one decoding step for one sequence all-reduces a vector of only \(h\) numbers (16 KB for \(h = 8192\)), 160 times per token for an 80-layer model. Small messages are dominated by the fixed latency \(\alpha\) of each collective, and NVLink latency is much lower than the network’s. [Likely]
  • Smaller matrices. With large \(t\), each GPU’s matrix multiplication gets thin (\(h \times f/t\)) and runs less efficiently. [Likely]

So the rule of thumb: tensor parallelism up to the number of GPUs in one server (8 on DGX and HGX machines), and other kinds of parallelism across servers. The Llama 3 paper says it directly: the innermost parallelism “requires the highest network bandwidth and lowest latency, and hence is usually constrained to within the same server”. They place the dimensions in the order [TP, CP, PP, DP], from inside to outside. [Certain] The original Megatron-LM used DGX-2H servers with 300 GB/s between GPUs inside a server and 100 GB/s per server between servers, and formed its model-parallel groups inside one server. [Certain]

Figure 5. How a 2D layout uses the hardware. Each server holds one tensor-parallel group of 8 GPUs, connected by NVLink (thick orange lines through the NVSwitch). GPU \(k\) of server 1 and GPU \(k\) of server 2 hold the same shard and form a data-parallel (or FSDP) group over InfiniBand (dashed blue lines).
Llama 3 405B runGPUsTPCPPPDPSequenceBF16 MFU
Pretraining8,1928116648,19243%
Pretraining16,38481161288,19241%
Long context16,384816168131,07238%

(Table 4 of the Llama 3 paper. [Certain]) Tensor parallelism stays at 8 in every run; the other dimensions change. CP (context parallelism) and PP (pipeline parallelism) are the topic of the next session.

The memory problem that tensor parallelism leaves

Tensor parallelism divides the weights by \(t\): with \(t = 8\), each GPU stores about 17.6 GB of Llama 3 70B’s 141 GB of BF16 weights. It also divides the activations that live inside the split regions (the Q, K, V, the attention scores, the MLP features). But it does not divide the activations of the replicated parts: the inputs of the two LayerNorms, the dropout masks, and the tensors around the residual additions. Every GPU stores all of them, in full.

Korthikanti et al. counted the activation memory of one GPT layer, in bytes, with 16-bit activations: [Certain]

\[ M_{\text{none}} = s b h \left( 34 + \frac{5 a s}{h} \right) \tag{48.9} \]
\[ M_{\text{TP}} = s b h \left( 10 + \frac{24}{t} + \frac{5 a s}{h t} \right) \tag{48.10} \]
Read the symbols
\(s, b, h, a\)
sequence length, micro-batch size, hidden size, number of attention heads
\(34\, sbh\)
the activations that grow linearly with the tokens: \(11sbh\) in attention, \(19sbh\) in the MLP, \(4sbh\) for the two LayerNorm inputs
\(5as^2b\)
the attention scores, softmax output and its dropout mask: they grow with \(s^2\)
\(10\, sbh\)
the part tensor parallelism does not divide: LayerNorm inputs and the dropout and residual tensors, stored in full on every GPU

With \(t = 8\), the divided part is \(24/8 = 3\), and the undivided part is 10. More than three quarters of the linear term is not helped by tensor parallelism at all.

Sequence parallelism

The fix is simple once you see it. LayerNorm and dropout work on each token separately: LayerNorm normalizes one token’s \(h\) features, dropout drops numbers independently. So in those regions we can split the tensor along the sequence dimension: GPU \(i\) keeps only tokens \(\tfrac{(i-1)s}{t} + 1\) to \(\tfrac{i s}{t}\). This is sequence parallelism (Korthikanti et al., 2022). [Certain]

Now the regions alternate: sequence-parallel (split by tokens) around LayerNorm, dropout and residual; tensor-parallel (split by features) in attention and the MLP. We need to switch between the two layouts. The key fact from session 43:

\[ \text{all-reduce} = \text{all-gather} \circ \text{reduce-scatter} \tag{48.11} \]

A ring all-reduce is a reduce-scatter (each GPU ends with one summed slice) followed by an all-gather (everyone gets all slices). Sequence parallelism simply moves the LayerNorm and dropout in between the two halves:

  • At the exit of attention or the MLP, the partial sums are reduce-scattered along the sequence: each GPU gets the finished sum for its \(s/t\) tokens. This is operator \(\bar{g}\): reduce-scatter in forward, all-gather in backward.
  • Dropout, the residual add and the next LayerNorm run on those \(s/t\) tokens only.
  • At the entrance of the next block, an all-gather rebuilds the full sequence for the column-parallel layers, which need every token. This is operator \(g\): all-gather in forward, reduce-scatter in backward.

Each all-reduce has turned into one reduce-scatter plus one all-gather. A ring all-reduce costs exactly that, so the communication volume does not change. The paper puts it as: tensor parallelism needs four all-reduces in a forward and backward pass, tensor plus sequence parallelism needs four all-gathers and four reduce-scatters, with the same bandwidth. [Certain] Step through Figure 3 with the checkbox on to see it.

Figure 6 shows this with numbers, on the same toy MLP as Figure 2, now with a LayerNorm in front and the residual add after it (no dropout, so the result is exact). Switch between plain tensor parallelism and tensor plus sequence parallelism. Both use the same ring steps, the same number of them, and send the same number of values. The only difference is where the all-gather happens: after the reduce-scatter (plain TP, the two halves of one all-reduce) or before the MLP (sequence parallelism, with LayerNorm and the residual add done on \(n/t\) tokens in between).

  1. \(x_i \leftarrow\) my \(n/t\) tokens of \(x\) // sequence-parallel region
  2. \(u_i \leftarrow \mathrm{LayerNorm}(x_i)\) // only my tokens
  3. for \(r = 1 \dots t-1\): pass \(u\) chunks around the ring // all-gather (operator \(g\))
  4. \(P_i \leftarrow \mathrm{GeLU}(u A_i)\, B_i\) // column then row: partial sum
  5. for \(r = 1 \dots t-1\): pass chunks and add // reduce-scatter (operator \(\bar g\))
  6. \(x_i \leftarrow x_i + z_i\) // residual add on my tokens
full copy (all tokens)only some tokens (sequence split)partial sumfinished sum
Figure 6. LayerNorm → MLP → residual on \(n = 4\) tokens and \(t\) GPUs, with every number computed in your browser. Each panel is one GPU’s \(4 \times 4\) activation tensor (tokens down, hidden features across); dashed cells are token rows that this GPU does not hold. The two counters at the top count the values each GPU has sent and the residual-stream values it stores.
How to read Figure 6

Start with “TP + sequence parallel” and press Play. Each GPU begins with only its own tokens (green rows) and runs LayerNorm on them: the panel on the right shows the mean, the variance and the normalized row for one token. The all-gather steps fill the missing rows, because the column-parallel layer needs every token. The MLP step gives red partial sums, and the reduce-scatter steps add them chunk by chunk until each GPU has its own tokens, finished. The residual add then runs on those tokens only. Now press “Plain TP” at any step: the step counter stays, the ring steps are the same, and the “sent” counter ends at the same value. But every GPU now stores all 16 residual-stream values instead of \(16/t\), and runs LayerNorm on all 4 tokens. The final step checks every output row against a single-GPU computation.

The memory now divides fully by \(t\):

\[ M_{\text{TP+SP}} = \frac{s b h}{t} \left( 34 + \frac{5 a s}{h} \right) \tag{48.12} \]

The same paper adds selective activation recomputation: do not store the \(5as^2b\) attention tensors; recompute them in the backward pass. They are big in memory but cheap to recompute. FlashAttention does the same thing by design (session 39). With both, the memory is \(34\, sbh / t\). [Certain]

Worked example: GPT-3 175B, \(t = 8\)

Take \(s = 2048\), \(h = 12288\), \(a = 96\), \(b = 1\), 96 layers, as in the paper. Then \(5as/h = 80\): the attention scores are more than twice the rest. Per layer, per GPU:

  • no parallelism (48.9): 2.67 GiB; tensor parallelism (48.10): 0.539 GiB;
  • tensor + sequence parallelism (48.12): 0.334 GiB;
  • tensor parallelism + selective recompute: \(sbh(10 + 3)\) = 0.305 GiB;
  • tensor + sequence parallelism + selective recompute: \(sbh \cdot 34/8\) = 0.100 GiB.

Over 96 layers that is 51.8 GiB with tensor parallelism alone, and 9.6 GiB with all three, a reduction of 5.4 times. The paper reports “about 5×” and, for the 530B model on 2,240 A100 GPUs, a model FLOPs utilization of 54.2% instead of 42.1% with full recomputation. [Certain]

Figure 7. Activation memory of one layer on one GPU (micro-batch \(b = 1\), 16-bit), from formulas 48.9, 48.10 and 48.12. The darker part of each bar is the \(5as^2b\) attention part; the lighter part is the linear \(sbh\) part.
How to read Figure 7

At \(t = 1\) the first three bars are equal: with one GPU there is nothing to split. As \(t\) grows, the “TP” bar stops shrinking: its 10\(sbh\) floor stays. With sequence parallelism the whole bar shrinks as \(1/t\). Increase \(s\): the dark attention part grows with \(s^2\) and soon dominates, which is why selective recompute (or FlashAttention) matters so much for long sequences. The formula assumes a GPT layer with dropout; models without dropout, such as Llama, store somewhat less. [Likely]

The algorithms

Algorithm 1: tensor-parallel MLP on GPU \(i\) (Megatron)
  1. input: full activations \(X\) (same on every GPU), weight shards \(A_i\) (columns) and \(B_i\) (rows)
  2. forward: \(X' \leftarrow f(X)\) // identity
  3. \(H_i \leftarrow \mathrm{GeLU}(X' A_i)\) // \(n \times f/t\), no communication
  4. \(P_i \leftarrow H_i B_i\) // partial sum, full shape \(n \times h\)
  5. \(Z \leftarrow g(P_i) = \text{all-reduce}_i(P_i)\) // formula (48.2)
  6. backward: given \(G = \partial L/\partial Z\) (same on every GPU), \(g\) passes it through unchanged
  7. \(\nabla B_i \leftarrow H_i^\top G\); \(\;G_{H} \leftarrow G B_i^\top \odot \mathrm{GeLU}'(X A_i)\)
  8. \(\nabla A_i \leftarrow X^\top G_H\) // weight gradients: local, no communication
  9. \(\partial L/\partial X \leftarrow f^{\text{bwd}}(G_H A_i^\top) = \text{all-reduce}_i(G_H A_i^\top)\) // formula (48.4)
Algorithm 2: one transformer layer with tensor and sequence parallelism, on GPU \(i\)
  1. input: \(X_i\): this GPU’s \(s/t\) tokens of the residual stream, shape \(s/t \times h\)
  2. \(U_i \leftarrow \mathrm{LayerNorm}(X_i)\) // sequence-parallel region
  3. \(U \leftarrow \text{all-gather}(U_i)\) along tokens // operator \(g\): all tokens, shape \(s \times h\)
  4. \(Q, K, V \leftarrow U W_{QKV}^{(i)}\) // column-parallel: heads \(\mathcal{H}_i\) only
  5. \(O \leftarrow \text{attention}(Q, K, V)\) for heads \(\mathcal{H}_i\) // local, causal mask, full sequence
  6. \(P \leftarrow O\, W_O^{(i)}\) // row-parallel: partial sum \(s \times h\)
  7. \(A_i \leftarrow \text{reduce-scatter}(P)\) along tokens // operator \(\bar g\): my \(s/t\) tokens, fully summed
  8. \(X_i \leftarrow X_i + \mathrm{Dropout}(A_i)\) // sequence-parallel: residual add
  9. repeat steps 2 to 8 for the MLP: LayerNorm, all-gather, \(W_1, W_3\) column-parallel, SwiGLU, \(W_2\) row-parallel, reduce-scatter, residual
  10. backward: every all-gather becomes a reduce-scatter and every reduce-scatter an all-gather; LayerNorm weight gradients are all-reduced over the \(t\) GPUs, because each GPU saw only \(s/t\) tokens

The last line hides a classic bug. In plain tensor parallelism the LayerNorm weights get the same gradient on every GPU (each saw the full sequence). With sequence parallelism, each GPU saw different tokens, so the LayerNorm gradients are different partial sums and must be all-reduced across the tensor-parallel group. Forget it and the copies of the LayerNorm weights slowly drift apart. [Likely]

PyTorch: checking the math on one machine

This code needs no GPUs. It simulates \(t = 4\) GPUs with slices of tensors and checks every claim of this session: the column-then-row MLP is exact (48.3), the row-first version is wrong, the input gradient needs an all-reduce (48.4), a reduce-scatter equals an all-reduce followed by a slice (48.11), and LayerNorm on token chunks equals LayerNorm on the full sequence.

import torch
torch.manual_seed(0)
t, n_tok, h, ffn = 4, 8, 8, 32                  # 4 "GPUs", 8 tokens, hidden 8, MLP width 32
X  = torch.randn(n_tok, h, dtype=torch.float64)
A  = torch.randn(h, ffn, dtype=torch.float64) / h ** 0.5      # first linear  (h -> ffn)
Bm = torch.randn(ffn, h, dtype=torch.float64) / ffn ** 0.5    # second linear (ffn -> h)
gelu = torch.nn.functional.gelu

# 1) dense reference, formula (48.3)
Y_ref = gelu(X @ A) @ Bm

# 2) Megatron MLP: A split by columns, B split by rows
A_parts = A.chunk(t, dim=1)                      # each h x (ffn/t)
B_parts = Bm.chunk(t, dim=0)                     # each (ffn/t) x h
partial = [gelu(X @ A_parts[i]) @ B_parts[i] for i in range(t)]   # no communication
Y_tp = sum(partial)                              # the all-reduce (operator g)
print("MLP forward max error:", (Y_tp - Y_ref).abs().max().item())

# 3) splitting A by rows first is wrong: GeLU of a sum is not the sum of GeLUs
X_parts, A_rows = X.chunk(t, dim=1), A.chunk(t, dim=0)
wrong = sum(gelu(X_parts[i] @ A_rows[i]) for i in range(t)) @ Bm
print("row-first error:", (wrong - Y_ref).abs().max().item())

# 4) backward: dL/dX is the SUM of the per-GPU input gradients, formula (48.4)
G = torch.randn(n_tok, h, dtype=torch.float64)                 # gradient arriving from above
Xr = X.clone().requires_grad_(True); (gelu(Xr @ A) @ Bm).backward(G)
grads = []
for i in range(t):
    Xi = X.clone().requires_grad_(True)                         # each GPU has its own copy of X
    (gelu(Xi @ A_parts[i]) @ B_parts[i]).backward(G)            # g is the identity in backward
    grads.append(Xi.grad)
print("dX from one GPU only:", (grads[0] - Xr.grad).abs().max().item())
print("dX after all-reduce: ", (sum(grads) - Xr.grad).abs().max().item())   # operator f

# 5) sequence parallelism: reduce-scatter = all-reduce, then keep my slice of tokens
full = sum(partial)
rs = [sum(p.chunk(t, dim=0)[r] for p in partial) for r in range(t)]
print("reduce-scatter vs slice:", max((rs[r] - full.chunk(t, 0)[r]).abs().max().item() for r in range(t)))
ln = torch.nn.LayerNorm(h, dtype=torch.float64)
print("LayerNorm on token chunks:", (torch.cat([ln(c) for c in X.chunk(t, 0)]) - ln(X)).abs().max().item())

Output (PyTorch 2.8, CPU): the forward error is about \(10^{-16}\), the row-first error is 1.38, the gradient from one GPU alone is off by 2.48, the all-reduced gradient is off by about \(10^{-15}\), and the last two checks print exactly 0.

PyTorch: real tensor and sequence parallelism with DTensor

PyTorch describes the same plan with parallelize_module: you say which layer is ColwiseParallel, which is RowwiseParallel, and which runs SequenceParallel. The library inserts the collectives. This script starts 4 processes on the CPU (the gloo backend), so it runs on a laptop; on GPUs you would use torchrun, "nccl" and "cuda". It builds a pre-norm MLP block, parallelizes it with sequence parallelism, feeds each process a quarter of the tokens, and compares the output and a weight gradient with the unsplit model.

import os, torch, torch.distributed as dist, torch.multiprocessing as mp
from torch.distributed.device_mesh import init_device_mesh
from torch.distributed.tensor import Shard
from torch.distributed.tensor.parallel import (parallelize_module, ColwiseParallel,
                                               RowwiseParallel, SequenceParallel)

class MLPBlock(torch.nn.Module):                 # x + W2(gelu(W1(LayerNorm(x))))
    def __init__(self, h, ffn):
        super().__init__()
        self.norm = torch.nn.LayerNorm(h)
        self.w1 = torch.nn.Linear(h, ffn, bias=False)
        self.w2 = torch.nn.Linear(ffn, h, bias=False)
    def forward(self, x):
        return x + self.w2(torch.nn.functional.gelu(self.w1(self.norm(x))))

def run(rank, world):
    os.environ.update(MASTER_ADDR="127.0.0.1", MASTER_PORT="29511")
    dist.init_process_group("gloo", rank=rank, world_size=world)
    mesh = init_device_mesh("cpu", (world,), mesh_dim_names=("tp",))
    torch.manual_seed(0)                                     # same weights and data everywhere
    ref = MLPBlock(16, 64); x = torch.randn(2, 8, 16)        # (batch, seq, hidden)
    tp = MLPBlock(16, 64); tp.load_state_dict(ref.state_dict())
    parallelize_module(tp, mesh, {
        "norm": SequenceParallel(),                          # LayerNorm on my s/t tokens
        "w1": ColwiseParallel(input_layouts=Shard(1)),       # all-gather tokens (g), split columns
        "w2": RowwiseParallel(output_layouts=Shard(1)),      # reduce-scatter to token slices (g-bar)
    })
    print(rank, tuple(tp.w1.weight.to_local().shape))        # (16, 16): a quarter of the 64 rows
    x_local = x.chunk(world, dim=1)[rank]                    # my slice of the sequence
    y_local = tp(x_local)
    y_ref = ref(x)
    err = (y_local - y_ref.chunk(world, dim=1)[rank]).abs().max()
    y_ref.sum().backward(); y_local.sum().backward()
    gerr = (tp.w1.weight.grad.full_tensor() - ref.w1.weight.grad).abs().max()
    if rank == 0: print("forward error", err.item(), "w1 grad error", gerr.item())
    dist.destroy_process_group()

if __name__ == "__main__":
    mp.spawn(run, args=(4,), nprocs=4)

Output (PyTorch 2.8, CPU): each rank holds a \(16 \times 16\) slice of w1 (PyTorch stores the weight as out × in, so a column-parallel layer splits its first dimension), the forward error is about \(10^{-7}\) (float32), and the gradient error is 0. The PyTorch tutorial uses the same pattern for a whole Llama block: wq, wk, wv, w1, w3 column-wise, wo and w2 row-wise with output_layouts=Shard(1), and the norms with SequenceParallel(). [Certain] It then wraps the result with FSDP2 over a second mesh dimension: TP inside a host, FSDP across hosts.

Where this is used

WhereHow tensor or sequence parallelism is usedNote
Megatron-LM, NeMocolumn and row parallel layers, sequence parallelism, vocabulary-parallel lossthe reference implementation of this session [Certain]
Llama 3 405B pretrainingTP = 8 inside each server, with PP, CP and FSDP-style DP outsideTable above [Certain]
PyTorch torchtitanparallelize_module plans with FSDP2 on a 2D meshsee session 47
LLM serving (vLLM, TensorRT-LLM, SGLang)tensor_parallel_size: one model on several GPUs, all GPUs work on every tokensmall all-reduces; NVLink latency matters [Likely]
Vision-language modelsthe language model is tensor-parallel; the vision encoder is smaller and is often replicated or split the same way[Likely]
Long video understanding (video QA, key-moment detection, Video-RAG over long clips)thousands of visual tokens per clip: activation memory grows with \(s\), so sequence parallelism and selective recompute matter; very long clips need context parallelism[Likely]
Video generation (diffusion transformers)very long token sequences of space-time patches; tensor and sequence parallelism inside a node, plus context parallelism[Likely]

Common mistakes

  • Splitting the first MLP layer by rows. Then GeLU sees partial sums. You need an extra all-reduce before the nonlinearity, or the result is wrong.
  • Splitting \(W_1\) and \(W_3\) of SwiGLU differently. The element-wise product needs the same features on the same GPU. Both must be split by the same column ranges.
  • Forgetting to change the head count. After splitting, each GPU has \(a/t\) query heads and \(n_{kv}/t\) KV heads. Reshape code that still uses the global \(a\) produces wrong shapes or silently wrong attention.
  • Same dropout seed everywhere, or different seeds everywhere. Megatron uses the same seed for dropout in replicated regions (so the copies stay identical) and a different seed per GPU inside the tensor-parallel regions (so different features get independent masks). [Certain] Getting this wrong makes the copies diverge or correlates the masks.
  • Not all-reducing LayerNorm gradients under sequence parallelism. Each GPU saw different tokens; the copies of the LayerNorm weights drift apart.
  • Tensor parallelism across servers. Four blocking all-reduces per layer over 50 GB/s links: Figure 4 shows the GPUs waiting more than they compute.
  • Thinking sequence parallelism adds communication. It does not: each all-reduce is replaced by a reduce-scatter plus an all-gather of the same total size.
  • \(t\) that does not divide the heads, the KV heads or the MLP width. Every split dimension must be divisible by \(t\) (or KV heads must be copied).

Interview questions

What is the difference between a column-parallel and a row-parallel linear layer?basic
For \(Y = XA\): column-parallel splits \(A\) into column blocks; each GPU has the full \(X\) and computes a slice of the output features, with no communication (formula 48.1). Row-parallel splits \(A\) into row blocks and needs \(X\) split by columns; each GPU computes a full-size partial sum \(X_iA_i\), and an all-reduce adds them (48.2). A column-parallel layer’s output is exactly the input a row-parallel layer needs, which is why they are paired.
In Megatron’s MLP, why is the first layer column-parallel and the second row-parallel?core
The nonlinearity between them works element by element, so it can run on column slices without communication. If the first layer were row-parallel, each GPU would hold partial sums of the pre-activation, and \(\mathrm{GeLU}(a+b) \neq \mathrm{GeLU}(a) + \mathrm{GeLU}(b)\), so an all-reduce would be needed before GeLU and another after the second layer. Column-then-row gives one all-reduce per MLP in forward.
How many collectives does one transformer layer need with tensor parallelism? Where are they?core
Two all-reduces in forward: after the attention output projection and after the second MLP layer (operator g). Two in backward: at the entrance of attention and of the MLP, to sum the input gradients (operator f). With sequence parallelism, each becomes a reduce-scatter plus an all-gather: 2 all-gathers and 2 reduce-scatters in forward, the same in backward.
Explain the f and g operators. Why is f an all-reduce in the backward pass?core
f is identity forward, all-reduce backward; g is all-reduce forward, identity backward. The input \(X\) of a block is used by every GPU through its own weight shard. The chain rule sums gradients over all uses, and each GPU can compute only its own term \(G_H A_i^\top\), so the sum is an all-reduce (formula 48.4). g’s backward is the identity because the gradient arriving at the block output is already full and identical on all GPUs.
How is attention split, and what limits the tensor-parallel size there?core
By heads: \(W_Q, W_K, W_V\) column-parallel (each GPU gets the columns of \(a/t\) heads), attention computed locally per head, \(W_O\) row-parallel followed by one all-reduce. \(t\) must divide the number of heads; with GQA it should divide the number of KV heads too, or the KV heads must be copied. Llama 3 70B has 64 query heads and 8 KV heads, so \(t = 8\) gives 8 query heads and 1 KV head per GPU.
What problem does sequence parallelism solve, and how?core
Plain tensor parallelism replicates LayerNorm, dropout and residual computation, so each GPU stores those activations in full: the \(10sbh\) term of formula 48.10 that does not shrink with \(t\). These operations are per token, so they can be split along the sequence. The all-reduce after each block is replaced by a reduce-scatter (each GPU gets its tokens, summed), and the all-gather is moved to the entrance of the next block. Memory becomes \(sbh(34 + 5as/h)/t\) (48.12), with no extra communication.
Does sequence parallelism increase communication? Prove your answer.advanced
No. A ring all-reduce of \(D\) bytes over \(t\) GPUs sends \(2(t-1)D/t\) per GPU: \((t-1)D/t\) for its reduce-scatter half and \((t-1)D/t\) for its all-gather half. Sequence parallelism performs exactly one reduce-scatter and one all-gather of the same tensor where tensor parallelism did one all-reduce. Same bytes, same number of steps. The only extra cost is that the two halves are now separate kernels, with LayerNorm and dropout between them.
Why does tensor parallelism usually stay inside one node?basic
It all-reduces activations four times per layer, on the critical path, so it needs the fastest links. Formula 48.8 gives compute/communication \(= 3h\beta/((t-1)F)\): with \(h = 8192\), \(F = 400\) TFLOP/s and \(t = 8\), that is 3.95 on NVLink (450 GB/s) but 0.44 on InfiniBand (50 GB/s per GPU). Inside an 8-GPU server, NVLink is about 9 times faster. Llama 3 used TP = 8 inside servers and put the other dimensions outside.
Compare tensor parallelism with FSDP. When do you use each, or both?advanced
FSDP shards storage (weights, gradients, optimizer states) but each GPU computes full layers on its own data; its communication is weights and gradients, about \(3\Psi\) per step, and it can be overlapped with compute, so it tolerates slower links. TP splits the computation of each layer; it communicates activations four times per layer, cannot be hidden well, and needs NVLink. TP also divides activation memory and lets several GPUs work on one sequence, which FSDP cannot. Large runs use both: TP inside a node, FSDP across nodes (a 2D mesh).
How is the output layer and the cross-entropy loss handled with a 128K vocabulary?advanced
The output projection is column-parallel along the vocabulary, so each GPU holds logits for \(v/t\) tokens. Gathering all logits would move \(bsv\) numbers (for 8,192 tokens and \(v = 128{,}256\): 1.05 billion values, 2.1 GB in BF16). Instead, each GPU computes its local max and local sum of \(e^{z - m}\), these are all-reduced (a max and a sum), and the target logit is taken from the GPU that owns it (formula 48.6). Communication drops to a few numbers per token.
You serve a 70B model with TP = 8 and decoding is slower than expected. What do you look at?advanced
In decoding each step processes very few tokens, so each all-reduce is tiny (16 KB per sequence for \(h = 8192\)) and there are 160 per token for 80 layers: latency, not bandwidth, dominates. Check that the GPUs are connected by NVLink (not PCIe), that the all-reduce uses a low-latency algorithm, that communication and compute can overlap, and whether batching more sequences would amortize the latency. Fewer GPUs with a smaller TP (if the model fits, for example with FP8 weights) can also be faster per token. [Likely]

Exercises

  1. In Figure 1, use the row split with \(t = 4\). Write the partial result of GPU 2 and check that the four partial results add up to \(Y\).
  2. A GPT layer with \(h = 12288\), \(b \cdot s = 4096\) tokens, \(t = 8\). How many bytes does each GPU send per layer in the forward pass (formula 48.7)?
  3. Using formula 48.8, for which \(t\) does the forward compute/communication ratio of an \(h = 4096\) model drop below 2 on NVLink (\(\beta = 450\) GB/s, \(F = 400\) TFLOP/s)?
  4. For GPT-3 175B (\(s = 2048\), \(h = 12288\), \(a = 96\), \(b = 1\)), compute the activation memory per layer with \(t = 4\), with and without sequence parallelism.
  5. A model has 40 query heads and 8 KV heads. Which tensor-parallel sizes from \(\{2, 4, 8, 16\}\) work without copying KV heads?
  6. Why is the gradient of the LayerNorm weights identical on all GPUs under plain tensor parallelism, but not under sequence parallelism?
Answerscheck
1. GPU 2 uses column 2 of \(X\), \((2, 0)^\top\), and row 2 of \(A\), \((0, 1, 1, -1)\): \(\begin{bmatrix} 0 & 2 & 2 & -2 \\ 0 & 0 & 0 & 0 \end{bmatrix}\). The four partials are \(\begin{bmatrix}1&0&2&1\\2&0&4&2\end{bmatrix}\), this one, \(\begin{bmatrix}0&0&0&0\\1&1&0&2\end{bmatrix}\), \(\begin{bmatrix}-2&1&-1&0\\2&-1&1&0\end{bmatrix}\); their sum is \(\begin{bmatrix}-1&3&3&-1\\5&0&5&4\end{bmatrix} = Y\). 2. \(8 \cdot \tfrac{7}{8} \cdot 4096 \cdot 12288 = 352{,}321{,}536\) bytes, about 352 MB. 3. Ratio \(= 3 \cdot 4096 \cdot 450\times10^9 / ((t-1) \cdot 400\times10^{12}) = 13.8/(t-1)\). It is below 2 when \(t - 1 \gt 6.9\), so from \(t = 8\) (ratio 1.97). 4. \(sbh = 2048 \cdot 12288 = 25{,}165{,}824\); \(5as/h = 80\). Without SP: \(sbh(10 + 6 + 20) = 36\,sbh\) = 906 MB (0.844 GiB). With SP: \(sbh(34 + 80)/4 = 28.5\,sbh\) = 717 MB (0.668 GiB). 5. \(t\) must divide 40 and 8: \(t = 2, 4, 8\) work (at \(t = 8\): 5 query heads and 1 KV head per GPU). \(t = 16\) fails twice: 40/16 is not an integer, and there are only 8 KV heads. 6. Under plain TP, every GPU runs LayerNorm on the full sequence with the same input and the same incoming gradient, so the weight gradients are equal. Under SP, each GPU runs LayerNorm on its own \(s/t\) tokens, so each has a partial sum of the gradient; they must be all-reduced.

Key takeaways

  • A matrix multiplication splits by columns (no communication, output split by features, 48.1) or by rows (partial sums, one all-reduce, 48.2).
  • Megatron pairs them: column-parallel then row-parallel, with the element-wise nonlinearity in between. Attention is split by heads the same way. Result: 2 all-reduces per layer in forward (operator g) and 2 in backward (operator f, 48.4).
  • Communication is activations, \(8(t-1)bsh/t\) bytes per GPU per layer forward (48.7); compute/communication is \(3h\beta/((t-1)F)\) (48.8). That is why TP lives on NVLink, inside one server, usually with \(t \le 8\).
  • Plain TP leaves \(10sbh\) of replicated activations (48.10). Sequence parallelism splits LayerNorm, dropout and residual regions along tokens, turning each all-reduce into reduce-scatter + all-gather (48.11) at the same cost, and divides all activations by \(t\) (48.12).
  • Vocabulary-parallel embeddings and a parallel cross-entropy (48.6) avoid gathering huge logits. In practice: TP inside a node, FSDP or DP across nodes, plus pipeline and context parallelism for the largest runs.

Key formulas

\[ Y = [XA_1, \dots, XA_t], \qquad Y = \sum_{i} X_i A_i, \qquad Z = \sum_i \mathrm{GeLU}(XA_i)\,B_i \]
\[ \frac{T_{\text{compute}}}{T_{\text{comm}}} = \frac{3h\beta}{(t-1)F}, \qquad M_{\text{TP}} = sbh\Big(10 + \frac{24}{t} + \frac{5as}{ht}\Big), \qquad M_{\text{TP+SP}} = \frac{sbh}{t}\Big(34 + \frac{5as}{h}\Big) \]

References for this session

  1. Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  2. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  3. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  4. Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  5. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  6. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  7. Narayanan et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
  8. Llama Team, Meta (2024). The Llama 3 Herd of Models
  9. PyTorch (docs). Large Scale Transformer model training with Tensor Parallel (TP)
  10. Li, Xue, Baranwal, Li and You (2021). Sequence Parallelism: Long Sequence Training from System Perspective
  11. Jacobs et al. (2023). DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
  12. Liu et al. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context
  13. Austin et al., Google DeepMind (2025). How to Scale Your Model