M4-062M4: Sequences & TransformersKV Cache & Inference OptimizationsHard
⚡Flashcard Drill➔Mastery:
Methods to Reduce KV Cache Memory Footprint
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
KV cache memory is minimized across four complementary dimensions: architecture design (MQA/GQA/MLA), numerical quantization (FP8/INT4), token pruning/eviction (H2O/StreamingLLM), and memory management (PagedAttention).
📌 Key Takeaways
- •Architecture level: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce KV heads; Multi-head Latent Attention (MLA in DeepSeek) compresses KV into low-rank latent vectors
- •Quantization level: FP8, INT8, or INT4 KV cache quantization reduces per-token memory by to with minimal degradation in perplexity
- •Token pruning & eviction: StreamingLLM (attention sinks + local window) and H2O (Heavy Hitter Oracle) drop unimportant historical tokens
- •Paging level: PagedAttention (vLLM) eliminates internal and external memory fragmentation
📐 Mathematical Derivations
Mathematical Mechanism: 1. Memory Baseline: For sequence length , layers , query heads , KV heads , head dimension , and precision bytes : 2. GQA / MQA Compression: In standard MHA, . GQA sets ( groups), achieving a -fold reduction in KV cache size. MQA sets , achieving an -fold ( to ) reduction. 3. DeepSeek MLA (Multi-head Latent Attention): MLA projects Key and Value into a compressed low-rank latent space: During decoding, only the compressed vector and a decoupled RoPE key are cached, reducing KV cache per token to of GQA. 4. Quantization: Transitioning from FP16 () to FP8 () or INT4 () scales memory linearly by or . Dynamic per-channel or per-block scaling factors preserve dynamic range: . 5. Token Pruning (H2O / StreamingLLM): Keeps only sink tokens and sliding window tokens, bounding cache memory to rather than .
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Quality vs. Memory Trade-off: MQA incurs slight capacity loss on complex reasoning tasks; GQA strikes the optimal balance and has become the industry standard for LLaMA-2/3, Mistral, and Qwen. ② Quantization Sensitivities: Keys are vulnerable to outlier channels (which skew scaling factors), while Values are more uniform. Asymmetric quantization (e.g., K at INT8, V at FP4) or per-channel scaling is frequently adopted. ③ MLA vs GQA: MLA achieves superior compression over GQA by combining low-rank projection with RoPE decoupling, allowing DeepSeek-V2/V3 to serve massive context lengths with modest VRAM. ④ Pruning vs Needle-in-a-Haystack: Eviction methods (H2O, Scissorhands) perform well on generic text generation but degrade severely on long-context retrieval and code analysis if critical middle tokens are discarded. ⑤ Interview Strategy: Categorize solutions systematically into Architecture (GQA/MLA), Quantization (FP8/INT4), Eviction (StreamingLLM), and Virtual Memory (PagedAttention), providing precise sizing formulas.
⚠️ Common Interview Pitfalls
- ✕Confusing weight quantization with KV cache quantization (KV cache quantization requires dynamic per-token or per-tensor scales during runtime)
- ✕Assuming MQA and GQA can be applied post-training without architectural pre-training or fine-tuning
- ✕Overlooking that KV pruning breaks long-context associative recall tasks
🎯 Interviewer Follow-up Inquiries
- ?How does DeepSeek's Multi-head Latent Attention (MLA) decouple RoPE from the compressed KV cache?
- ?What causes accuracy degradation when quantizing KV cache to INT4, and how can it be mitigated?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚