H7Hard★ High-Frequency · Distributed ParallelismPart H · Optimizers & Distributed Systems

Megatron Tensor Parallelism (Column & Row Linear)

⚡Flashcard Drill➔

Shard the first MLP linear layer column-wise (zero inter-layer comm) and the second row-wise, synchronizing via a single All-Reduce.

⏱️ Time: O(B * D * H / N) 计算每卡 + 1 次 All-Reduce
💾 Space: O(D * H / N) 权重分片显存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def tp_mlp_forward(x: np.ndarray, W1_shards: list, W2_shards: list) -> np.ndarray:
"""
Megatron-style Tensor Parallel 2-layer MLP across N ranks:
1) First Linear splits columns: W1_shards[r] has shape (D, H // N)
Each rank computes local activation independently (zero communication).
2) Second Linear splits rows: W2_shards[r] has shape (H // N, D)
Each rank computes partial output, followed by a single sum All-Reduce.
"""
partial_outputs = []
for W1_r, W2_r in zip(W1_shards, W2_shards):
# Column-Parallel: X @ W1_r -> (B, H // N), followed by ReLU activation
h_local =
# Row-Parallel: h_local @ W2_r -> (B, D)
partial_outputs.append(h_local @ W2_r)
# Single All-Reduce (sum across ranks)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How many All-Reduce collectives occur per Transformer layer in standard Megatron-LM Tensor Parallelism?
Next: ZeRO-1/2/3 Model-State Memory Calculator →
💡

Core Mental Anchor / Mnemonic

Column split first avoids communication; row split second merges partial outputs with one All-Reduce.

📐 Mathematical Derivation & Core Formula

Y=GeLU(X[A1,A2])[B1B2]=∑r=1NGeLU(XAr)BrY = \mathrm{GeLU}(X [A_1, A_2]) \begin{bmatrix} B_1 \\ B_2 \end{bmatrix} = \sum_{r=1}^{N} \mathrm{GeLU}(X A_r) B_r
### Column-Parallel Followed by Row-Parallel Sharding
Sharding W1=[A1,…,AN]W_1 = [A_1, \dots, A_N] by columns lets each rank compute Hr=σ(XAr)H_r = \sigma(X A_r) locally without communication because elementwise activations act independently per column.
Sharding W2=[B1;… ;BN]W_2 = [B_1; \dots; B_N] by rows lets each rank evaluate Yr=HrBrY_r = H_r B_r and combine exact totals Y=∑r=1NYrY = \sum_{r=1}^N Y_r using one All-Reduce per block.

🔄 Tensor Dimensions & Shape Flow

X:(B,D) -> ColLinear W1_r:(D,H/N) -> local activation -> RowLinear W2_r:(H/N,D) -> AllReduce(sum) -> Y:(B,D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Keep first-layer split along columns (axis=1) and second-layer split along rows (axis=0), matching hidden divisibility by N.
  • Do not place full-vector reductions such as LayerNorm between un-synchronized shards without Sequence Parallelism.
  • Confine Tensor Parallelism inside high-bandwidth intra-node NVLink domains (TP=4 or 8) due to per-layer All-Reduce latency.

💻 Reference Code Implementation

import numpy as np

def tp_mlp_forward(x: np.ndarray, W1_shards: list, W2_shards: list) -> np.ndarray:
    """
    Megatron-style Tensor Parallel 2-layer MLP across N ranks:
    1) First Linear splits columns: W1_shards[r] has shape (D, H // N)
       Each rank computes local activation independently (zero communication).
    2) Second Linear splits rows: W2_shards[r] has shape (H // N, D)
       Each rank computes partial output, followed by a single sum All-Reduce.
    """
    partial_outputs = []
    for W1_r, W2_r in zip(W1_shards, W2_shards):
        # Column-Parallel: X @ W1_r -> (B, H // N), followed by ReLU activation
        h_local = np.maximum(0.0, x @ W1_r)
        # Row-Parallel: h_local @ W2_r -> (B, D)
        partial_outputs.append(h_local @ W2_r)
    # Single All-Reduce (sum across ranks)
    return np.sum(partial_outputs, axis=0)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
np.random.seed(42)
B, D, H, N = 4, 8, 16, 4
x = np.random.randn(B, D)
W1 = np.random.randn(D, H)
W2 = np.random.randn(H, D)
ref = np.maximum(0.0, x @ W1) @ W2
W1_shards = np.split(W1, N, axis=1)
W2_shards = np.split(W2, N, axis=0)
tp_out = tp_mlp_forward(x, W1_shards, W2_shards)
assert np.allclose(tp_out, ref, atol=1e-10), "Tensor-parallel MLP must match unsharded MLP"
print("✓ Megatron Tensor-Parallel MLP assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How many All-Reduce collectives occur per Transformer layer in standard Megatron-LM Tensor Parallelism?
2 All-Reduce operations in forward (after attention W_O and after MLP W_down) and 2 in backward (from the conjugate gradient operator on split inputs), totaling 4 per Transformer block.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Ring All-Reduce Distributed CommunicationBack to Part H & All 105Next: ZeRO-1/2/3 Model-State Memory Calculator →