M4-057M4: Sequences & TransformersKV Cache & Inference OptimizationsEasy
⚡Flashcard Drill➔Mastery:
KV Cache Principles and Why It Cannot Be Used During Training
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The KV Cache caches Key and Value representations of historical tokens during autoregressive inference to eliminate redundant recomputation, whereas training processes the entire sequence simultaneously in parallel via causal masking without sequential dependencies.
📌 Key Takeaways
- •Inference generation is sequential token-by-token: without KV Cache, computing token requires recalculating representations for all previous tokens, leading to complexity
- •With KV Cache, historical and tensors are stored in GPU memory, reducing per-step computation to matrix-vector multiplications ( cumulative)
- •During training, causal masking enables fully parallel computation of all positions at once, making step-by-step caching unnecessary and inapplicable
📐 Mathematical Derivations
Mathematical Mechanism: The Bottleneck of Autoregressive Generation: At step , the attention output is . If historical keys and values are not cached, all previous projections and () must be recomputed from scratch across all layers. The total computation over generated tokens becomes for attention per layer and for projection weights, yielding cumulative latency quadratic in generated length. With KV Cache: At step , only , , and are computed ( projection cost). and are appended to the existing cache: and . The attention operation computes ( operations), reducing overall generation complexity from (or ) to . Why Not Used in Training: Training utilizes teacher forcing where the ground-truth sequence is completely known. Thanks to causal attention masking (a lower-triangular mask setting upper-triangular logits to ), the full matrices and all attention scores are computed in parallel matrix multiplications via Tensor Cores. There are no sequential generation steps, and full activation tensors are required for backpropagation.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Memory vs. Computation Trade-off: KV Cache trades GPU VRAM for latency. For an FP16 model with sequence length , layers , heads , and head dimension , the memory required per sequence is bytes. For a 70B model with , KV cache can reach several gigabytes per request, becoming the primary bottleneck for serving concurrency. ② Memory Bandwidth Bound: During the decoding phase, generating each token requires reading the entire KV cache from High Bandwidth Memory (HBM) into SRAM for a single query vector (), resulting in an arithmetic intensity (FLOPs/byte) and making decoding severely memory-bandwidth bound. ③ Dynamic Memory Allocation: Traditional static allocation pre-allocates maximum sequence length, wasting up to 60-80% of VRAM due to internal and external fragmentation (solved by PagedAttention in vLLM). ④ Architectural Optimizations: MQA (Multi-Query Attention) and GQA (Grouped-Query Attention) directly reduce the number of KV heads, slashing KV cache footprint by to . ⑤ Interview Strategy: Clarify the contrast between parallel GEMM training and memory-bound autoregressive decoding, derive the KV cache size formula, and explain why inference transitions from compute-bound prefill to bandwidth-bound decoding.
⚠️ Common Interview Pitfalls
- ✕Confusing the training computational paradigm with autoregressive sequential decoding
- ✕Forgetting to multiply by 2 for both Key and Value tensors when calculating cache memory size
- ✕Overlooking that decoding is memory bandwidth-bound rather than compute-bound
🎯 Interviewer Follow-up Inquiries
- ?How do you calculate the exact KV Cache size per token for a 70B model?
- ?Why does PagedAttention resolve memory fragmentation in KV Cache management?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚