M10-014M10: Engineering Deep-Dive & Technical BQAI Infra Deep-Dive & Postmortem BQHard
⚡Flashcard Drill➔Mastery:

AI Infra Optimization BQ: End-to-End Roofline Kernel & Serving Throughput Tuning

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M10-015 →
📐 Mathematical Definition
Cost/Million Tok↓  ⟺  Prefill Compute Bound+Decode Batch Saturation\text{Cost/Million Tok} \downarrow \iff \text{Prefill Compute Bound} + \text{Decode Batch Saturation}
⚡ One-Sentence Core Answer
Walk through concrete hardware profiling (Nsight Systems) exposing CPU launch overhead or HBM memory ceilings, resolved via CUDA Graphs, operator fusion, and FP8 quantization.

📌 Key Takeaways

  • •
    Profile First: open Nsight Systems traces to see whether GPU bubbles stem from CPU kernel-launch latency or HBM bandwidth saturation
  • •
    Fix 1 (Kill CPU Launch Bubbles): capture decode step execution via CUDA Graphs cleanly to eliminate ~40% host dispatch overhead
  • •
    Fix 2 (Fuse HBM Round-Trips): fuse RMSNorm/Residual/RoPE in Triton + ship FP8/AWQ weights to halve memory traffic
  • •
    Empirical Lift: demonstrate concrete pre/post tok/s, SM occupancy, and P99 TPOT metrics

📐 Mathematical Derivations

Respect Amdahl's Law rigorously: optimize host tokenization, IPC copies, or dispatch bubbles first whenever profiling shows idle GPU streams.

🏭 Production Trade-offs

Bucket decode batch sizes to discrete powers-of-two (1, 2, 4, ..., 64) for pre-captured CUDA Graphs cleanly.
⚠️ Common Interview Pitfalls
  • ✕
    Tuning random kernels blindly without capturing an Nsight trace upfront
  • ✕
    Breaking CUDA Graph capture invariants via implicit .cpu() syncs or dynamic host allocations
🎯 Interviewer Follow-up Inquiries
  • ?
    How does FlashInfer / vLLM reconcile dynamic Paged KV tables cleanly with static CUDA Graph capture?
  • ?
    How do you diagnose register spilling cleanly when tuning Triton tile block sizes?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM10-013: AI Infra Deep-Dive: Isolating Silent Distributed NCCL Hangs & VRAM OOM Spikes📋 Back to BankNext →M10-015: AI Infra Fault Tolerance BQ: Designing Zero-Waste Checkpointing & Elastic Recovery