M3-084M3: Deep Learning FoundationsDistributed Training BasicsHard
⚡Flashcard Drill➔Mastery:

3D Parallelism and Sequence Parallelism (SP) in Large Foundation Models

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-085 →
📐 Mathematical Definition
total GPUs=DP×TP×PP;SP: split along sequence dim\text{total GPUs}=\text{DP}\times\text{TP}\times\text{PP};\qquad \text{SP}:\ \text{split along sequence dim}
⚡ One-Sentence Core Answer
3D Parallelism combines Data (DP), Tensor (TP), and Pipeline (PP) parallelism; Sequence Parallelism shards activation-heavy operations (LayerNorm, Dropout) along the sequence dimension.

📌 Key Takeaways

  • •
    3D Parallelism composition: Total GPUs =DP×TP×PP= \text{DP} \times \text{TP} \times \text{PP}; maps dimensions to cluster physical network hierarchy
  • •
    Sequence Parallelism (Megatron-SP): splits sequence tokens across TP ranks in non-matmul layers, reducing activation memory by factor TP\text{TP}
  • •
    Context Parallelism (Ring Attention / DeepSpeed Ulysses): shards sequence dimension globally across ranks using Ring Attention to train 1M+ token contexts

📐 Mathematical Derivations

Mathematical Formulations:
① The 3D Mapping Principle:
- TP (Tensor Parallelism): Shards GEMMs. Communication ∝\propto activations. Map strictly inside the same node (NVLink, 900GB/s900\text{GB/s}).
- PP (Pipeline Parallelism): Shards layers. Peer-to-peer communication of boundary activations. Map across nodes over InfiniBand.
- DP (Data Parallelism / FSDP): Shards batches. Map across remaining nodes.
② Sequence Parallelism in Megatron-LM (Korthikanti et al., 2022):
Standard Tensor Parallelism only shards Q,K,VQ, K, V projections and MLP projections; LayerNorm, Dropout, and residual additions remain duplicated across all TP ranks, redundantly storing activations of shape (B,S,D)(B, S, D).
- Megatron-SP Innovation: Notice that LayerNorm and Dropout are element-wise or token-independent along sequence length SS. Split the sequence dimension SS across TPTP workers: length becomes S/TPS / \text{TP}.
- Replace the forward All-Reduce with a Reduce-Scatter (reducing partial matmul sums into sequence shards) and replace the backward All-Reduce with an All-Gather. Total communication volume is mathematically unchanged, but activation memory is slashed by factor TP\text{TP}!

🏭 Production Trade-offs

Long-Context Evolution: Megatron-SP is tied to TP degree (typically ≤8\le 8). For ultra-long contexts (128K−1M128\text{K}-1\text{M} tokens), use Context Parallelism (DeepSpeed Ulysses / Ring Attention), which shards the sequence globally across arbitrary numbers of nodes.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing Megatron Sequence Parallelism (which shards LayerNorm/Dropout within TP) with Context Parallelism (which shards entire attention across independent nodes)
  • ✕
    Attempting to scale Tensor Parallelism beyond 8 GPUs across slow InfiniBand switches, causing massive latency bottlenecks
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Megatron Sequence Parallelism replace All-Reduce with Reduce-Scatter and All-Gather without increasing communication volume?
  • ?
    How does DeepSpeed Ulysses use All-to-All communication to parallelize attention along the sequence dimension?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-083: FSDP vs DeepSpeed ZeRO: Architectural Relationship, Mechanics, and Differences📋 Back to BankNext →M3-085: Systematic Troubleshooting Workflow When Training Loss Fails to Decrease