M9-011M9: Senior ML & AI System DesignLLM Serving, RAG & Agent SysDesignHard
⚡Flashcard Drill➔Mastery:
Design a 100K-QPS LLM Gateway (PD Disaggregation, Prefix Routing & Semantic Cache)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Eliminate prefill-vs-decode latency spikes via RDMA-connected disaggregated worker pools, consistent prefix-hash routing, semantic caches, and cascading routers.
📌 Key Takeaways
- •L1 Interference Fix: separating compute-bound Prefill from memory-bound Decode stabilizes both TTFT and TPOT
- •L2 RDMA KV Transfer: NIXL/Mooncake transfers prefill KV blocks seamlessly to high-batch Decode workers
- •L2 Prefix-Hash Routing: sticky consistent hashing elevates Radix prefix hit rates above 75%
- •L3 FinOps Cascade: tenant-scoped semantic cache -> 8B fast path -> Frontier escalation + KV admission gates
📐 Mathematical Derivations
Decoupling prefill and decode sizing maximizes useful throughput per dollar while sealing P99 decode smoothness from prompt-spike stalls.
🏭 Production Trade-offs
Combine prefix consistent hashing with bounded-load overflow above 85% KV occupancy to maintain cache locality without hot-spotting.
⚠️ Common Interview Pitfalls
- ✕Plain Round-Robin balancing wipes out per-node Radix prefix cache hits
- ✕Omit tenant ACLs in semantic cache keys causes catastrophic cross-tenant leakage
🎯 Interviewer Follow-up Inquiries
- ?How should Prefill-to-Decode node ratios autoscale dynamically as prompt/output lengths shift?
- ?How do you prevent RDMA KV transfers from saturating links on 128K prompts?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚