5.1
5.1 Large-Scale Training Systems
3D Parallelism (TP/PP/DP), Context Parallelism (CP), PyTorch FSDP, DeepSpeed ZeRO memory sharding, long-horizon fault tolerance, and async checkpoint recovery.
3D Parallelism (TP/PP/DP), FSDP & ZeRO Memory Sharding
Megatron-LM TP/PP communication scheduling, Context/Sequence Parallelism (CP/SP) for long context, PyTorch FSDP, DeepSpeed ZeRO-1/2/3 parameter/optimizer sharding, and activation checkpointing memory trade-offs.
🏢 Companies
NVIDIA、Microsoft (DeepSpeed)、Meta (FairScale)、DeepSeek、Google DeepMind、OpenAI
🛠️ Tech Stack
Tensor ParallelismPipeline ParallelismZeRO-3FSDPContext ParallelismMegatron-LMActivation Checkpointing
💼 Roles & Salary
Training Engineer、Distributed Training Engineer、ML Systems Engineer
💰 $245K - $510K / year (Distributed Training) | ¥750K - ¥1.85M / year
📚 Prerequisites: Distributed Parallelism Theory (3D/FSDP) • CUDA Memory Math & ZeRO-1/2/3 Derivations • NCCL Collectives & Topology Modeling • PyTorch Distributed & ProcessGroup Internals
Training Fault Tolerance, Checkpointing & Observability
10K+ GPU async checkpoint persistence and rapid recovery, straggler and silent hang detection, FP8/BF16 mixed-precision numerical stability, automated loss spike remediation, and W&B telemetry.
🏢 Companies
NVIDIA、DeepSeek、Meta、Together AI、CentML、xAI
🛠️ Tech Stack
Fault ToleranceAsync CheckpointingStraggler DetectionLoss Spike RemediationFP8 StabilityTraining TelemetryW&B
💼 Roles & Salary
Distributed Training Engineer、Performance Engineer、Research Engineer
💰 $235K - $480K / year (Training Reliability) | ¥700K - ¥1.75M / year
📚 Prerequisites: Hardware Diagnostic & Straggler Detection • Async Checkpointing I/O Pipelines • FP8/BF16 Numerical Underflow/Overflow • Distributed Training Telemetry & Monitoring