Collective Comm & NVLink Topology

Every synchronization in multi-GPU training reduces to collective primitives. Let each rank hold a data shard of $S$ bytes and there be N ranks: ① AllGather (each rank broadcasts its shard to the other N−1 ranks; at the end every rank holds all $NS$ bytes): each rank must receive N−1 shards, volume $(N-1)S$ — every rank's data is read once by each of the other N−1 ranks; ② ReduceScatter (reduce, then keep only one's own part of the result): $\frac{N-1}{N}S$ per rank; ③ AllReduce (reduce then distribute to all) = ReduceScatter + AllGather: the ring implementation costs $2\frac{N-1}{N}S \to 2S$ per rank (for large N each rank transmits only about twice its own data; bandwidth independent of N); ④ AllToAll (every rank sends to and receives from every other) — exactly what MoE expert-parallel token routing does: with uniform load, $\frac{N-1}{N}S$ per rank. Hardware bandwidth: NVLink 4.0 (H100) 900GB/s per GPU, NVLink 5.0 (B200/GB200) 1.8TB/s per GPU; InfiniBand 800Gbps (XDR) = 100GB/s, 400Gbps (NDR) = 50GB/s — NVLink 5.0 is about 18× a single 800Gbps IB link. Rail-optimized topology: the i-th GPU NIC of every node plugs into the same leaf switch (the i-th “rail”), so cross-node traffic only happens between same-index GPUs, preventing AllReduce/AllGather flows from hopping across rails and causing hotspots/congestion; together with NVSwitch full-mesh (intra-node) and leaf-spine/3D-Torus (inter-node), this forms NCCL's top-level topology design.

the core module of training-infra interviews (the prerequisite of distributed-parallel): derive Ring-AllReduce's volume, how AllGather vs AllReduce differ, why MoE uses AllToAll, how NVLink compares to IB in orders of magnitude, what rail-optimization is.

drops N-way synchronization cost from naive $O(NS)$ (every rank sends its full data to N−1 others) to $O(S)$ — bandwidth stops growing with the number of GPUs, so parallelism can scale to hundreds or thousands of cards; NVLink's huge bandwidth carries TP's frequent small messages while IB carries cross-node gradient sync, and topology design keeps both flows congestion-free.

Test Your Knowledge: Practice Questions for "Collective Comm & NVLink Topology"

Single choice pitfall questions with instant feedback and mistake tracking.