H7Hard★ High-Frequency · Distributed ParallelismPart H · Optimizers & Distributed Systems
Megatron Tensor Parallelism (Column & Row Linear)
Shard the first MLP linear layer column-wise (zero inter-layer comm) and the second row-wise, synchronizing via a single All-Reduce.
⏱️ Time:
O(B * D * H / N) 计算每卡 + 1 次 All-Reduce💾 Space:
O(D * H / N) 权重分片显存🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def tp_mlp_forward(x: np.ndarray, W1_shards: list, W2_shards: list) -> np.ndarray:
"""
Megatron-style Tensor Parallel 2-layer MLP across N ranks:
1) First Linear splits columns: W1_shards[r] has shape (D, H // N)
Each rank computes local activation independently (zero communication).
2) Second Linear splits rows: W2_shards[r] has shape (H // N, D)
Each rank computes partial output, followed by a single sum All-Reduce.
"""
partial_outputs = []
for W1_r, W2_r in zip(W1_shards, W2_shards):
# Column-Parallel: X @ W1_r -> (B, H // N), followed by ReLU activation
h_local =
# Row-Parallel: h_local @ W2_r -> (B, D)
partial_outputs.append(h_local @ W2_r)
# Single All-Reduce (sum across ranks)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How many All-Reduce collectives occur per Transformer layer in standard Megatron-LM Tensor Parallelism?
💡
Core Mental Anchor / Mnemonic
Column split first avoids communication; row split second merges partial outputs with one All-Reduce.
📐 Mathematical Derivation & Core Formula
### Column-Parallel Followed by Row-Parallel Sharding
Sharding by columns lets each rank compute locally without communication because elementwise activations act independently per column.
Sharding by rows lets each rank evaluate and combine exact totals using one All-Reduce per block.
Sharding by columns lets each rank compute locally without communication because elementwise activations act independently per column.
Sharding by rows lets each rank evaluate and combine exact totals using one All-Reduce per block.
🔄 Tensor Dimensions & Shape Flow
X:(B,D) -> ColLinear W1_r:(D,H/N) -> local activation -> RowLinear W2_r:(H/N,D) -> AllReduce(sum) -> Y:(B,D)
🛡️ Numerical Stability & Pitfalls Checklist
- Keep first-layer split along columns (
axis=1) and second-layer split along rows (axis=0), matching hidden divisibility by N. - Do not place full-vector reductions such as LayerNorm between un-synchronized shards without Sequence Parallelism.
- Confine Tensor Parallelism inside high-bandwidth intra-node NVLink domains (TP=4 or 8) due to per-layer All-Reduce latency.
💻 Reference Code Implementation
import numpy as np
def tp_mlp_forward(x: np.ndarray, W1_shards: list, W2_shards: list) -> np.ndarray:
"""
Megatron-style Tensor Parallel 2-layer MLP across N ranks:
1) First Linear splits columns: W1_shards[r] has shape (D, H // N)
Each rank computes local activation independently (zero communication).
2) Second Linear splits rows: W2_shards[r] has shape (H // N, D)
Each rank computes partial output, followed by a single sum All-Reduce.
"""
partial_outputs = []
for W1_r, W2_r in zip(W1_shards, W2_shards):
# Column-Parallel: X @ W1_r -> (B, H // N), followed by ReLU activation
h_local = np.maximum(0.0, x @ W1_r)
# Row-Parallel: h_local @ W2_r -> (B, D)
partial_outputs.append(h_local @ W2_r)
# Single All-Reduce (sum across ranks)
return np.sum(partial_outputs, axis=0)
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
np.random.seed(42)
B, D, H, N = 4, 8, 16, 4
x = np.random.randn(B, D)
W1 = np.random.randn(D, H)
W2 = np.random.randn(H, D)
ref = np.maximum(0.0, x @ W1) @ W2
W1_shards = np.split(W1, N, axis=1)
W2_shards = np.split(W2, N, axis=0)
tp_out = tp_mlp_forward(x, W1_shards, W2_shards)
assert np.allclose(tp_out, ref, atol=1e-10), "Tensor-parallel MLP must match unsharded MLP"
print("✓ Megatron Tensor-Parallel MLP assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How many All-Reduce collectives occur per Transformer layer in standard Megatron-LM Tensor Parallelism?
2 All-Reduce operations in forward (after attention
W_O and after MLP W_down) and 2 in backward (from the conjugate gradient operator on split inputs), totaling 4 per Transformer block.⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.