M3-084M3: Deep Learning FoundationsDistributed Training BasicsHard
⚡Flashcard Drill➔Mastery:
3D Parallelism and Sequence Parallelism (SP) in Large Foundation Models
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
3D Parallelism combines Data (DP), Tensor (TP), and Pipeline (PP) parallelism; Sequence Parallelism shards activation-heavy operations (LayerNorm, Dropout) along the sequence dimension.
📌 Key Takeaways
- •3D Parallelism composition: Total GPUs ; maps dimensions to cluster physical network hierarchy
- •Sequence Parallelism (Megatron-SP): splits sequence tokens across TP ranks in non-matmul layers, reducing activation memory by factor
- •Context Parallelism (Ring Attention / DeepSpeed Ulysses): shards sequence dimension globally across ranks using Ring Attention to train 1M+ token contexts
📐 Mathematical Derivations
Mathematical Formulations:
① The 3D Mapping Principle:
- TP (Tensor Parallelism): Shards GEMMs. Communication activations. Map strictly inside the same node (NVLink, ).
- PP (Pipeline Parallelism): Shards layers. Peer-to-peer communication of boundary activations. Map across nodes over InfiniBand.
- DP (Data Parallelism / FSDP): Shards batches. Map across remaining nodes.
② Sequence Parallelism in Megatron-LM (Korthikanti et al., 2022):
Standard Tensor Parallelism only shards projections and MLP projections; LayerNorm, Dropout, and residual additions remain duplicated across all TP ranks, redundantly storing activations of shape .
- Megatron-SP Innovation: Notice that LayerNorm and Dropout are element-wise or token-independent along sequence length . Split the sequence dimension across workers: length becomes .
- Replace the forward All-Reduce with a Reduce-Scatter (reducing partial matmul sums into sequence shards) and replace the backward All-Reduce with an All-Gather. Total communication volume is mathematically unchanged, but activation memory is slashed by factor !
① The 3D Mapping Principle:
- TP (Tensor Parallelism): Shards GEMMs. Communication activations. Map strictly inside the same node (NVLink, ).
- PP (Pipeline Parallelism): Shards layers. Peer-to-peer communication of boundary activations. Map across nodes over InfiniBand.
- DP (Data Parallelism / FSDP): Shards batches. Map across remaining nodes.
② Sequence Parallelism in Megatron-LM (Korthikanti et al., 2022):
Standard Tensor Parallelism only shards projections and MLP projections; LayerNorm, Dropout, and residual additions remain duplicated across all TP ranks, redundantly storing activations of shape .
- Megatron-SP Innovation: Notice that LayerNorm and Dropout are element-wise or token-independent along sequence length . Split the sequence dimension across workers: length becomes .
- Replace the forward All-Reduce with a Reduce-Scatter (reducing partial matmul sums into sequence shards) and replace the backward All-Reduce with an All-Gather. Total communication volume is mathematically unchanged, but activation memory is slashed by factor !
🏭 Production Trade-offs
Long-Context Evolution: Megatron-SP is tied to TP degree (typically ). For ultra-long contexts ( tokens), use Context Parallelism (DeepSpeed Ulysses / Ring Attention), which shards the sequence globally across arbitrary numbers of nodes.
⚠️ Common Interview Pitfalls
- ✕Confusing Megatron Sequence Parallelism (which shards LayerNorm/Dropout within TP) with Context Parallelism (which shards entire attention across independent nodes)
- ✕Attempting to scale Tensor Parallelism beyond 8 GPUs across slow InfiniBand switches, causing massive latency bottlenecks
🎯 Interviewer Follow-up Inquiries
- ?How does Megatron Sequence Parallelism replace All-Reduce with Reduce-Scatter and All-Gather without increasing communication volume?
- ?How does DeepSpeed Ulysses use All-to-All communication to parallelize attention along the sequence dimension?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚