4D Parallelism DP/TP/PP/SP

Four parallelism dimensions that shard parameters, gradients, activations and sequences across GPUs. ① DP (data parallel): every rank holds a full model copy, only data is sharded; one full gradient AllReduce per step, the ring version costs $2\frac{N-1}{N}S \approx 2S$ per rank ($S$ = per-rank gradient size, growing linearly with parameter count — a 70B model has about 140GB of gradients per rank, so DP sync is expensive for large models); communication happens only between steps and can be hidden by bucket-async overlap. ② TP (tensor parallel): intra-layer matrices are split column/row-wise (see megatron-3d-bubble); each transformer layer runs 2 AllReduces, each message only about $b \times s \times h$ activations (independent of parameter count, but frequent and latency-sensitive) — it needs intra-node NVLink (900GB/s+) to be viable. ③ PP (pipeline parallel): layers are split into stages; communication only at stage boundaries, one activation and one gradient message (~$b \times s \times h$) per microbatch — low volume, low frequency, latency adds linearly, so it fits cross-node IB; the cost is the pipeline bubble (bubble ratio $\frac{p-1}{m+p-1}$, see megatron-3d-bubble). ④ SP (sequence parallel): the sequence dimension is split into N chunks across N ranks, fixing KV/activation memory growing linearly with seq length; two implementations: Megatron-SP (combined with TP, also shards LayerNorm/Dropout activations to cut TP's activation traffic) and Ring-Attention (KV blocks circulate on a ring, O(1) memory per rank). 3D combination DP × TP × PP: GPT-3 175B used 1024 A100s = DP 64 × TP 8 × PP 2; the allocation rule: fill TP inside a node first (it consumes NVLink bandwidth), PP across nodes (least traffic), DP outermost for gradient sync — communication cost decreases from the inside out while scalability increases from the inside out.

a training-infra interview staple: how much each of DP/TP/PP communicates, why TP cannot span nodes, how SP relates to TP, and how to configure 3D parallelism for a given GPU count.

the single GPU runs out of memory (weights + optimizer + activations) and saturates neither compute nor bandwidth — TP shards weights, PP shards layers, DP shards data, SP shards the sequence; stacked together, arbitrarily large models train on arbitrarily many GPUs with near-linear efficiency, and every per-layer message lands on the right interconnect (intra-node NVLink vs inter-node IB).

Derive DP's communication: the ring AllReduce per step is $2\frac{N-1}{N}S \approx 2S$ ($S$ = per-rank gradient), independent of N; explain why DP is still expensive for large models — S grows linearly with parameters (70B gradients ≈ 140GB).

TP communication profile: 2 AllReduces per layer, each ~$b \times s \times h$ and independent of parameter count; argue why TP must stay intra-node (NVLink 900GB/s, low latency) and what happens across nodes.

PP communication: why its volume is small (one activation + one gradient per microbatch between adjacent stages, ~$b \times s \times h$) and fits across nodes; what is its cost (bubble $\frac{p-1}{m+p-1}$, see its card).

SP: what is sharded (the sequence dimension), what it fixes (KV/activation memory linear in seq), and the communication cost and fit of Megatron-SP vs Ring-Attention.

3D configuration problem: given total GPUs and model size, allocate TP intra-node → PP inter-node → DP outermost; recite GPT-3 175B's 1024 GPUs = DP 64 × TP 8 × PP 2 and justify each dimension.