M10-013M10: Engineering Deep-Dive & Technical BQAI Infra Deep-Dive & Postmortem BQHard
⚡Flashcard Drill➔Mastery:

AI Infra Deep-Dive: Isolating Silent Distributed NCCL Hangs & VRAM OOM Spikes

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M10-014 →
📐 Mathematical Definition
Hang / OOM / P99 Spike→Nsight + Py-Spy + NCCL LogDeterministic Isolation\text{Hang / OOM / P99 Spike} \xrightarrow{\text{Nsight + Py-Spy + NCCL Log}} \text{Deterministic Isolation}
⚡ One-Sentence Core Answer
Prove hands-on systems mastery by systematically diagnosing distributed silent hangs, collective deadlocks, or sudden memory OOM spikes across GPU clusters.

📌 Key Takeaways

  • •
    Diagnosing Silent Hangs: inspect DCGM GPU-util=0% + high VRAM -> dump py-spy stacks across ranks to spot mismatched collective branches -> inspect NCCL Flight Recorder
  • •
    Hunting Stragglers: run quick intra/inter-node allreduce diagnostics to isolate degraded NVLink/PCIe switches or thermal-throttled GPUs and cordon immediately
  • •
    Capturing Subtle OOMs: enable torch.cuda.memory._record_memory_history to catch retained graph refs or dynamic-shape CUDA caching allocator fragmentation

📐 Mathematical Derivations

Every distributed hang reduces cleanly to either mismatched per-rank control flow or one degraded hardware link throttling the collective ring.

🏭 Production Trade-offs

Run pre-flight NCCL bandwidth probes cleanly before launching massive training jobs so faulty optical transceivers never burn cluster hours.
⚠️ Common Interview Pitfalls
  • ✕
    Blindly restarting hung distributed jobs without capturing py-spy / NCCL Flight Recorder traces first
  • ✕
    Appending un-detached PyTorch loss tensors into metric logs, retaining entire autograd graphs indefinitely
🎯 Interviewer Follow-up Inquiries
  • ?
    How do InfiniBand hardware counters diagnose PFC storms or degraded optical links?
  • ?
    Why does PyTorch CUDA OOM trigger with 15GB cached free blocks, and how does expandable_segments:True help?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM10-012: AIE Resilience BQ: Leading Vendor-Agnostic LLM Gateways & Model Migrations📋 Back to BankNext →M10-014: AI Infra Optimization BQ: End-to-End Roofline Kernel & Serving Throughput Tuning