M3-080M3: Deep Learning FoundationsDistributed Training BasicsMedium
⚡Flashcard Drill➔Mastery:
Collective Communication Primitives: All-Reduce, All-Gather, and Reduce-Scatter
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
All-Reduce computes sum across all ranks and distributes the full result to all; Reduce-Scatter computes sum and scatters shards; All-Gather collects distributed shards onto every rank.
📌 Key Takeaways
- •All-Reduce: combines Reduce + Broadcast; transforms input buffers of size into identical summed buffers of size
- •Reduce-Scatter: sums buffers across ranks and partitions the result so rank receives only shard (size )
- •All-Gather: the exact inverse of Reduce-Scatter; collects local shards of size to reconstruct full buffer of size on all ranks
📐 Mathematical Derivations
Mathematical Identity and Ring Algorithms (Patarasuk & Yuan, 2009):
Let be the number of ranks, and be tensor size in bytes.
① The Core Equivalence:
.
② Ring-Based Communication Complexity:
Arrange GPUs in a logical ring. Each rank sends and receives bytes per step over steps.
- Reduce-Scatter: Each rank starts with bytes. After transfers, rank holds the reduced sum of slice . Communication volume per rank: bytes.
- All-Gather: Each rank starts with a shard of size . Over ring steps, shards circulate until every rank has the full bytes. Communication volume per rank: bytes.
- Ring All-Reduce: Executes Reduce-Scatter followed by All-Gather. Total data transmitted per rank: bytes, completely independent of the number of ranks !
Let be the number of ranks, and be tensor size in bytes.
① The Core Equivalence:
.
② Ring-Based Communication Complexity:
Arrange GPUs in a logical ring. Each rank sends and receives bytes per step over steps.
- Reduce-Scatter: Each rank starts with bytes. After transfers, rank holds the reduced sum of slice . Communication volume per rank: bytes.
- All-Gather: Each rank starts with a shard of size . Over ring steps, shards circulate until every rank has the full bytes. Communication volume per rank: bytes.
- Ring All-Reduce: Executes Reduce-Scatter followed by All-Gather. Total data transmitted per rank: bytes, completely independent of the number of ranks !
🏭 Production Trade-offs
Role in ZeRO / FSDP Architecture: ZeRO-3 decomposes standard DDP's All-Reduce into its constituent halves: Reduce-Scatter during backward pass (sharding gradients across ranks) and All-Gather during forward and backward passes (reconstructing parameter shards dynamically).
⚠️ Common Interview Pitfalls
- ✕Assuming All-Reduce communication time scales linearly with cluster node count ; under Ring-AllReduce, per-GPU transfer volume is independent of
- ✕Confusing Reduce-Scatter (which performs algebraic summation) with Scatter (which merely partitions unsummed data)
🎯 Interviewer Follow-up Inquiries
- ?Why is the communication volume per GPU under Ring-AllReduce asymptotically , independent of cluster size ?
- ?How does Tree-AllReduce achieve lower latency than Ring-AllReduce for small message sizes?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚