M10-013M10: Engineering Deep-Dive & Technical BQAI Infra Deep-Dive & Postmortem BQHard
⚡Flashcard Drill➔Mastery:
AI Infra Deep-Dive: Isolating Silent Distributed NCCL Hangs & VRAM OOM Spikes
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Prove hands-on systems mastery by systematically diagnosing distributed silent hangs, collective deadlocks, or sudden memory OOM spikes across GPU clusters.
📌 Key Takeaways
- •Diagnosing Silent Hangs: inspect DCGM GPU-util=0% + high VRAM -> dump py-spy stacks across ranks to spot mismatched collective branches -> inspect NCCL Flight Recorder
- •Hunting Stragglers: run quick intra/inter-node allreduce diagnostics to isolate degraded NVLink/PCIe switches or thermal-throttled GPUs and cordon immediately
- •Capturing Subtle OOMs: enable
torch.cuda.memory._record_memory_historyto catch retained graph refs or dynamic-shape CUDA caching allocator fragmentation
📐 Mathematical Derivations
Every distributed hang reduces cleanly to either mismatched per-rank control flow or one degraded hardware link throttling the collective ring.
🏭 Production Trade-offs
Run pre-flight NCCL bandwidth probes cleanly before launching massive training jobs so faulty optical transceivers never burn cluster hours.
⚠️ Common Interview Pitfalls
- ✕Blindly restarting hung distributed jobs without capturing py-spy / NCCL Flight Recorder traces first
- ✕Appending un-detached PyTorch loss tensors into metric logs, retaining entire autograd graphs indefinitely
🎯 Interviewer Follow-up Inquiries
- ?How do InfiniBand hardware counters diagnose PFC storms or degraded optical links?
- ?Why does PyTorch CUDA OOM trigger with 15GB cached free blocks, and how does
expandable_segments:Truehelp?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚