Collective Comm & NVLink Topology

Every synchronization in multi-GPU training reduces to collective primitives. Let each rank hold a data shard of $S$ bytes and there be N ranks: ① AllGather (each rank broadcasts its shard to the other N−1 ranks; at the end every rank holds all $NS$ bytes): each rank must receive N−1 shards, volume $(N-1)S$ — every rank's data is read once by each of the other N−1 ranks; ② ReduceScatter (reduce, then keep only one's own part of the result): $\frac{N-1}{N}S$ per rank; ③ AllReduce (reduce then distribute to all) = ReduceScatter + AllGather: the ring implementation costs $2\frac{N-1}{N}S \to 2S$ per rank (for large N each rank transmits only about twice its own data; bandwidth independent of N); ④ AllToAll (every rank sends to and receives from every other) — exactly what MoE expert-parallel token routing does: with uniform load, $\frac{N-1}{N}S$ per rank. Hardware bandwidth: NVLink 4.0 (H100) 900GB/s per GPU, NVLink 5.0 (B200/GB200) 1.8TB/s per GPU; InfiniBand 800Gbps (XDR) = 100GB/s, 400Gbps (NDR) = 50GB/s — NVLink 5.0 is about 18× a single 800Gbps IB link. Rail-optimized topology: the i-th GPU NIC of every node plugs into the same leaf switch (the i-th “rail”), so cross-node traffic only happens between same-index GPUs, preventing AllReduce/AllGather flows from hopping across rails and causing hotspots/congestion; together with NVSwitch full-mesh (intra-node) and leaf-spine/3D-Torus (inter-node), this forms NCCL's top-level topology design.

the core module of training-infra interviews (the prerequisite of distributed-parallel): derive Ring-AllReduce's volume, how AllGather vs AllReduce differ, why MoE uses AllToAll, how NVLink compares to IB in orders of magnitude, what rail-optimization is.

drops N-way synchronization cost from naive $O(NS)$ (every rank sends its full data to N−1 others) to $O(S)$ — bandwidth stops growing with the number of GPUs, so parallelism can scale to hundreds or thousands of cards; NVLink's huge bandwidth carries TP's frequent small messages while IB carries cross-node gradient sync, and topology design keeps both flows congestion-free.

Derive AllGather's volume: shard $S$ per rank, N ranks, each ends with $NS$ and must receive $(N-1)S$ new bytes — explain why it grows linearly with N while Ring-AllReduce does not.

Decompose AllReduce into ReduceScatter + AllGather: what is each phase's volume ($\frac{N-1}{N}S$ and $(N-1)S$) and step count (N−1); verify the total $2\frac{N-1}{N}S$ by addition.

Why MoE expert parallelism is AllToAll, not AllGather: tokens route by expert ID, each rank sends to and receives from all ranks; compare volume $\frac{N-1}{N}S$ under uniform load with AllReduce.

Recite NVLink/IB numbers: NVLink 4.0 = 900GB/s, NVLink 5.0 = 1.8TB/s, IB 800Gbps = 100GB/s, 400Gbps = 50GB/s; compute how many × NVLink 5.0 is over IB 800Gbps (≈18×) and why TP must use NVLink while DP can use IB.

Describe rail-optimized topology: the i-th GPU of every node plugs into one shared leaf switch forming a “rail”; what problem it solves (same-index communication, no cross-rail hotspots) and its relation to NVSwitch and leaf-spine.