M3-078M3: Deep Learning FoundationsDistributed Training BasicsEasy
⚡Flashcard Drill➔Mastery:
Comparing Distributed Parallelism: Data (DP), Tensor (TP), and Pipeline (PP)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Data Parallelism splits batches across workers (All-Reduce); Tensor Parallelism shards weight matrices intra-node (All-Gather/Reduce-Scatter); Pipeline Parallelism partitions layers sequentially inter-node (P2P).
📌 Key Takeaways
- •Data Parallelism (DDP/FSDP): replicates or shards model parameters; scales with batch size; communication model parameters
- •Tensor Parallelism (Megatron-LM): shards individual linear/attention matrices across GPUs; requires ultra-high bandwidth (NVLink); communication activations
- •Pipeline Parallelism (GPipe): partitions layers across nodes; point-to-point communication; suffers from idle pipeline bubble overhead
📐 Mathematical Derivations
Taxonomy and Communication Patterns:
① Data Parallelism (DDP / FSDP):
Batch dimension is split across GPUs ( per GPU). Each worker executes independent forward/backward passes. Gradients are synchronized via All-Reduce ( bytes transferred). Scalable across nodes; bottlenecked by batch size lower bounds.
② Tensor Parallelism (TP / Megatron-LM):
For MLP: shard column-wise () and row-wise (). Output: . Requires exactly 2 All-Reduce operations per Transformer layer. Communication is high-frequency and latency-sensitive, strictly requiring intra-node NVLink ().
③ Pipeline Parallelism (PP / GPipe, 1F1B):
Partitions layers across pipeline stages. Stage receives activations from stage and sends outputs to stage . Communication is lightweight peer-to-peer (P2P), suitable across slow inter-node InfiniBand networks. Introduces pipeline bubble fraction: , where is the number of micro-batches.
① Data Parallelism (DDP / FSDP):
Batch dimension is split across GPUs ( per GPU). Each worker executes independent forward/backward passes. Gradients are synchronized via All-Reduce ( bytes transferred). Scalable across nodes; bottlenecked by batch size lower bounds.
② Tensor Parallelism (TP / Megatron-LM):
For MLP: shard column-wise () and row-wise (). Output: . Requires exactly 2 All-Reduce operations per Transformer layer. Communication is high-frequency and latency-sensitive, strictly requiring intra-node NVLink ().
③ Pipeline Parallelism (PP / GPipe, 1F1B):
Partitions layers across pipeline stages. Stage receives activations from stage and sends outputs to stage . Communication is lightweight peer-to-peer (P2P), suitable across slow inter-node InfiniBand networks. Introduces pipeline bubble fraction: , where is the number of micro-batches.
🏭 Production Trade-offs
3D Parallelism Composition Rule: Map Tensor Parallelism intra-node across 8 GPUs via NVLink; map Pipeline Parallelism across nodes via InfiniBand; wrap outer layer in Data Parallelism (FSDP / ZeRO) to scale across thousands of GPUs.
⚠️ Common Interview Pitfalls
- ✕Running Tensor Parallelism across nodes over standard Ethernet/InfiniBand, causing extreme communication bottlenecks
- ✕Setting the number of pipeline micro-batches too small, causing the pipeline bubble to consume of compute time
🎯 Interviewer Follow-up Inquiries
- ?How does the 1F1B (One-Forward-One-Backward) schedule reduce pipeline memory footprint compared to GPipe?
- ?Why does Megatron-LM pair column-parallel linear projection with row-parallel projection in attention and MLPs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚