M3-091M3: Deep Learning FoundationsTraining Diagnostics & DebuggingHard
⚡Flashcard Drill➔Mastery:
Reproducibility Debugging in Deep Learning: RNG Seeds, Deterministic Algorithms, and Pipelines
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Full reproducibility requires fixing random seeds across all libraries, enabling deterministic GPU algorithms, and isolating multi-processing data loader worker states.
📌 Key Takeaways
- •Global seed configuration: must seed Python
random, NumPy, PyTorch CPU, and PyTorch CUDA simultaneously - •Deterministic algorithms: enable
torch.use_deterministic_algorithms(True)to replace non-deterministic GPU atomic operations - •DataLoader workers: configure
worker_init_fnto prevent all parallel workers from drawing identical duplicate random sequences
📐 Mathematical Derivations
Reproducibility Checklist and Implementation:
① Universal Seed Setup:
```python
import torch, random, numpy as np, os
def seed_everything(seed=42):
random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
# Enforce deterministic algorithms
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
``
① Universal Seed Setup:
```python
import torch, random, numpy as np, os
def seed_everything(seed=42):
random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
# Enforce deterministic algorithms
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
``
<br/>② <strong>Non-Deterministic CUDA Kernels</strong>:<br/>Operations like torch.baddbmm, index_add_, atomic additions in scatter/gather, and backward passes of 2D convolutions utilize non-deterministic floating-point accumulation order in GPU hardware. Floating-point addition is non-associative: $(a + b) + c \ne a + (b + c)$. Forcing determinism replaces fast atomic adds with deterministic reduction trees, incurring a 5–20% throughput penalty.<br/>③ <strong>DataLoader Worker Isolation</strong>:<br/>If worker_init_fn` is omitted, PyTorch DataLoader workers fork the exact same base PRNG state, producing identical random augmentations across parallel threads.🏭 Production Trade-offs
Development vs Production: Enforce strict determinism during regression testing and bug isolation. In large-scale foundation pretraining, disable strict deterministic flags (
torch.backends.cudnn.benchmark = True) to maximize training throughput.⚠️ Common Interview Pitfalls
- ✕Seeding only
torch.manual_seed()while leaving NumPy and Pythonrandomunseeded, allowing data augmentation to introduce non-determinism - ✕Assuming reproducibility is guaranteed across different GPU architectures (e.g., A100 vs H100); different hardware instructions alter floating-point reduction order
🎯 Interviewer Follow-up Inquiries
- ?Why does floating-point non-associativity cause GPU atomic additions to produce non-deterministic results?
- ?How does
worker_init_fnin PyTorch DataLoaders guarantee distinct random sequences across worker threads?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚