KV Cache Optimization

KV Cache optimization compresses the K/V tensors cached per layer during autoregressive inference to break the memory wall of long contexts and large batches. Memory formula: $2 \times 2 \times L \times H \times d \times s \times b$ bytes (K and V per token × 2 bytes for FP16 × layers × heads × head dim × sequence length × batch size). Four main lines: ① structural compression — GQA shares one KV head group across multiple Q heads (e.g. 64 → 8, memory /8); MLA (DeepSeek-V2) compresses each layer's K/V into a low-rank latent vector $c_t^{KV} = W^{DKV} h_t$, and at inference the $W^{UK}$ projection is absorbed into $W^Q$ and the output projection, caching only the latent — vs MHA's 86GB, KV memory drops to 5.1GB (−93.3%); ② KV quantization — storing K/V in INT8/FP8 halves memory with little accuracy loss; ③ StreamingLLM — keeping only the "attention sink" (first 4 tokens) plus a sliding window, so long-session memory drops from $O(s)$ to $O(\text{window})$; ④ H2O sparsity — retaining only heavy-hitter tokens by cumulative attention score, with dynamic replacement.

long-context (32K/128K) online inference, multi-turn chatbots, and the decode side of PD separation; interview favorites: the KV cache memory formula, GQA vs MHA, how MLA saves memory, and how to survive memory blowup on long conversations.

with large batches and long sequences, KV cache grows far faster than weights (each new token adds $2 \times 2 \times L \times H \times d$ bytes), directly capping concurrency and context length. GQA/MLA cut an order of magnitude structurally (MLA: 86GB → 5.1GB), KV quantization halves it again, and StreamingLLM/H2O make memory depend on the window instead of full length — combined, these let the same GPU serve several times more concurrent connections and longer contexts.

Test Your Knowledge: Practice Questions for "KV Cache Optimization"

Single choice pitfall questions with instant feedback and mistake tracking.