Checkpointing & Recovery

A checkpoint is a complete snapshot of training state so that thousand-GPU runs can resume from the latest step after failures (node crash / OOM / power loss) instead of starting over. It must save: ① model weights; ② optimizer state — Adam's first/second moments $m_t$, $v_t$ and the FP32 master weight copy (weights alone cannot resume losslessly; lost momentum and LR progress make the resumed loss unstable); ③ progress — epoch/step counters and the LR-scheduler step; ④ randomness — each rank's RNG states (data shuffle, not just init); ⑤ data cursors — DataLoader shard boundaries. Sharded checkpoints: one huge monolithic file is slow to write and fragile (a 70B FP16 file is 140 GB, minutes to persist); HF safetensors splits weights into shards plus an index.json — parallel writes, mmap zero-copy reads, and no pickle, hence no arbitrary-code-execution on load; under ZeRO-3/FSDP parameters are already sharded, so each rank persists only its own $16\Psi/N_d$ shard and reads it back on resume (or All-Gathers once to export a consolidated file). Async checkpointing: a synchronous save stop-the-worlds training (it stalls for the disk write); async saves first copy a consistent snapshot from GPU to CPU memory (with copy-on-write/versioning so later updates don't corrupt it), then a background thread/process writes it out while training continues. Resuming: reconstruct optimizer state, reset epoch/step and RNG, restore shuffle order and data sharding — ZeRO-3 requires every rank to read back its own shard. Training stability: pair saving with stability — on loss/gradient spikes, roll back to the last-good checkpoint, lower the LR, or skip the bad batch; save frequency trades IO overhead against failure loss, commonly using time/step thresholds plus per-epoch candidate checkpoints.

interview favorites are what a checkpoint must contain and why weights alone are not enough, what HF sharded/safetensors fixes, how async saves take a consistent snapshot, and resume training with spike rollback.

on thousand-GPU runs the probability of at least one failure is non-trivial (MTBF shrinks with scale) and no checkpoint means days of compute wasted; sharded + async checkpointing pushes save cost from minutes of downtime to background overlap (nearly zero throughput loss) and cuts resume from hours of warmup to minutes; combined with last-good rollback and spike detection, training becomes recoverable against both hardware faults and numeric divergence.

List everything a checkpoint must save (weights, Adam $m_t$/$v_t$, step/epoch and LR schedule, RNG states, DataLoader cursors) and explain why saving only weights makes the resumed loss unstable (lost optimizer state and LR progress).

What sharded checkpointing fixes: a single 140 GB FP16 70B file takes minutes to persist and is fragile; how HF safetensors sharding (index.json + parallel writes + mmap zero-copy reads) solves it and why it is safer than pickle (.bin).

Async checkpointing: why synchronous saves must stop-the-world (weights keep changing); how async saves obtain a consistent snapshot (GPU→CPU copy + copy-on-write/versioning + background persistence) and the trade-offs.

How to save/resume under ZeRO-3/FSDP sharding: each rank persists its own $16\Psi/N_d$ shard and reads it back on resume (or gathers a consolidated file); why RNG and shuffle order must be restored as well.

Training stability: how to detect and handle loss/gradient spikes (roll back to a last-good checkpoint, lower LR, skip the bad batch), how to trade off save frequency (IO cost vs failure loss), with a per-epoch candidate-checkpoint strategy.