M3-078M3: Deep Learning FoundationsDistributed Training BasicsEasy
⚡Flashcard Drill➔Mastery:

Comparing Distributed Parallelism: Data (DP), Tensor (TP), and Pipeline (PP)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-079 →
📐 Mathematical Definition
DP: replicate θ, split D;TP: split W;PP: split layers\text{DP}:\ \text{replicate}\ \theta,\ \text{split}\ \mathcal{D};\quad \text{TP}:\ \text{split}\ W;\quad \text{PP}:\ \text{split layers}
⚡ One-Sentence Core Answer
Data Parallelism splits batches across workers (All-Reduce); Tensor Parallelism shards weight matrices intra-node (All-Gather/Reduce-Scatter); Pipeline Parallelism partitions layers sequentially inter-node (P2P).

📌 Key Takeaways

  • •
    Data Parallelism (DDP/FSDP): replicates or shards model parameters; scales with batch size; communication ∝\propto model parameters
  • •
    Tensor Parallelism (Megatron-LM): shards individual linear/attention matrices across GPUs; requires ultra-high bandwidth (NVLink); communication ∝\propto activations
  • •
    Pipeline Parallelism (GPipe): partitions layers across nodes; point-to-point communication; suffers from idle pipeline bubble overhead

📐 Mathematical Derivations

Taxonomy and Communication Patterns:
① Data Parallelism (DDP / FSDP):
Batch dimension BB is split across NN GPUs (B/NB/N per GPU). Each worker executes independent forward/backward passes. Gradients are synchronized via All-Reduce (2×∣params∣2 \times |\text{params}| bytes transferred). Scalable across nodes; bottlenecked by batch size lower bounds.
② Tensor Parallelism (TP / Megatron-LM):
For MLP: shard W1W_1 column-wise (d×4dTPd \times \frac{4d}{TP}) and W2W_2 row-wise (4dTP×d\frac{4d}{TP} \times d). Output: Y=GeLU(XW1)W2Y = \text{GeLU}(X W_1) W_2. Requires exactly 2 All-Reduce operations per Transformer layer. Communication is high-frequency and latency-sensitive, strictly requiring intra-node NVLink (900GB/s900\text{GB/s}).
③ Pipeline Parallelism (PP / GPipe, 1F1B):
Partitions LL layers across PP pipeline stages. Stage kk receives activations from stage k−1k-1 and sends outputs to stage k+1k+1. Communication is lightweight peer-to-peer (P2P), suitable across slow inter-node InfiniBand networks. Introduces pipeline bubble fraction: Fbubble=P−1M+P−1F_{\text{bubble}} = \frac{P - 1}{M + P - 1}, where MM is the number of micro-batches.

🏭 Production Trade-offs

3D Parallelism Composition Rule: Map Tensor Parallelism intra-node across 8 GPUs via NVLink; map Pipeline Parallelism across nodes via InfiniBand; wrap outer layer in Data Parallelism (FSDP / ZeRO) to scale across thousands of GPUs.
⚠️ Common Interview Pitfalls
  • ✕
    Running Tensor Parallelism across nodes over standard Ethernet/InfiniBand, causing extreme communication bottlenecks
  • ✕
    Setting the number of pipeline micro-batches MM too small, causing the pipeline bubble to consume >50%>50\% of compute time
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the 1F1B (One-Forward-One-Backward) schedule reduce pipeline memory footprint compared to GPipe?
  • ?
    Why does Megatron-LM pair column-parallel linear projection with row-parallel projection in attention and MLPs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-077: Why LayerNorm and Softmax Must Retain FP32 Precision in Mixed-Precision Training📋 Back to BankNext →M3-079: ZeRO (Zero Redundancy Optimizer): The Three Sharding Stages Explained