4D Parallelism DP/TP/PP/SP

Four parallelism dimensions that shard parameters, gradients, activations and sequences across GPUs. ① DP (data parallel): every rank holds a full model copy, only data is sharded; one full gradient AllReduce per step, the ring version costs $2\frac{N-1}{N}S \approx 2S$ per rank ($S$ = per-rank gradient size, growing linearly with parameter count — a 70B model has about 140GB of gradients per rank, so DP sync is expensive for large models); communication happens only between steps and can be hidden by bucket-async overlap. ② TP (tensor parallel): intra-layer matrices are split column/row-wise (see megatron-3d-bubble); each transformer layer runs 2 AllReduces, each message only about $b \times s \times h$ activations (independent of parameter count, but frequent and latency-sensitive) — it needs intra-node NVLink (900GB/s+) to be viable. ③ PP (pipeline parallel): layers are split into stages; communication only at stage boundaries, one activation and one gradient message (~$b \times s \times h$) per microbatch — low volume, low frequency, latency adds linearly, so it fits cross-node IB; the cost is the pipeline bubble (bubble ratio $\frac{p-1}{m+p-1}$, see megatron-3d-bubble). ④ SP (sequence parallel): the sequence dimension is split into N chunks across N ranks, fixing KV/activation memory growing linearly with seq length; two implementations: Megatron-SP (combined with TP, also shards LayerNorm/Dropout activations to cut TP's activation traffic) and Ring-Attention (KV blocks circulate on a ring, O(1) memory per rank). 3D combination DP × TP × PP: GPT-3 175B used 1024 A100s = DP 64 × TP 8 × PP 2; the allocation rule: fill TP inside a node first (it consumes NVLink bandwidth), PP across nodes (least traffic), DP outermost for gradient sync — communication cost decreases from the inside out while scalability increases from the inside out.

a training-infra interview staple: how much each of DP/TP/PP communicates, why TP cannot span nodes, how SP relates to TP, and how to configure 3D parallelism for a given GPU count.

the single GPU runs out of memory (weights + optimizer + activations) and saturates neither compute nor bandwidth — TP shards weights, PP shards layers, DP shards data, SP shards the sequence; stacked together, arbitrarily large models train on arbitrarily many GPUs with near-linear efficiency, and every per-layer message lands on the right interconnect (intra-node NVLink vs inter-node IB).

Test Your Knowledge: Practice Questions for "4D Parallelism DP/TP/PP/SP"

Single choice pitfall questions with instant feedback and mistake tracking.