M9-011M9: Senior ML & AI System DesignLLM Serving, RAG & Agent SysDesignHard
⚡Flashcard Drill➔Mastery:

Design a 100K-QPS LLM Gateway (PD Disaggregation, Prefix Routing & Semantic Cache)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M9-012 →
📐 Mathematical Definition
TTFT(Lin)⊥TPOT(Bdecode)∧Consistent Prefix Hash\text{TTFT}(L_{\text{in}}) \perp \text{TPOT}(B_{\text{decode}})\quad \land \quad\text{Consistent Prefix Hash}
⚡ One-Sentence Core Answer
Eliminate prefill-vs-decode latency spikes via RDMA-connected disaggregated worker pools, consistent prefix-hash routing, semantic caches, and cascading routers.

📌 Key Takeaways

  • •
    L1 Interference Fix: separating compute-bound Prefill from memory-bound Decode stabilizes both TTFT and TPOT
  • •
    L2 RDMA KV Transfer: NIXL/Mooncake transfers prefill KV blocks seamlessly to high-batch Decode workers
  • •
    L2 Prefix-Hash Routing: sticky consistent hashing elevates Radix prefix hit rates above 75%
  • •
    L3 FinOps Cascade: tenant-scoped semantic cache -> 8B fast path -> Frontier escalation + KV admission gates

📐 Mathematical Derivations

Decoupling prefill and decode sizing maximizes useful throughput per dollar while sealing P99 decode smoothness from prompt-spike stalls.

🏭 Production Trade-offs

Combine prefix consistent hashing with bounded-load overflow above 85% KV occupancy to maintain cache locality without hot-spotting.
⚠️ Common Interview Pitfalls
  • ✕
    Plain Round-Robin balancing wipes out per-node Radix prefix cache hits
  • ✕
    Omit tenant ACLs in semantic cache keys causes catastrophic cross-tenant leakage
🎯 Interviewer Follow-up Inquiries
  • ?
    How should Prefill-to-Decode node ratios autoscale dynamically as prompt/output lengths shift?
  • ?
    How do you prevent RDMA KV transfers from saturating links on 128K prompts?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM9-010: Design a Multimodal Document & Image Search Engine (SigLIP + ColPali MaxSim)📋 Back to BankNext →M9-012: Design a Zero-Leak Multi-Tenant RBAC Enterprise Hybrid RAG Hub