KV Cache Optimization

KV Cache optimization compresses the K/V tensors cached per layer during autoregressive inference to break the memory wall of long contexts and large batches. Memory formula: $2 \times 2 \times L \times H \times d \times s \times b$ bytes (K and V per token × 2 bytes for FP16 × layers × heads × head dim × sequence length × batch size). Four main lines: ① structural compression — GQA shares one KV head group across multiple Q heads (e.g. 64 → 8, memory /8); MLA (DeepSeek-V2) compresses each layer's K/V into a low-rank latent vector $c_t^{KV} = W^{DKV} h_t$, and at inference the $W^{UK}$ projection is absorbed into $W^Q$ and the output projection, caching only the latent — vs MHA's 86GB, KV memory drops to 5.1GB (−93.3%); ② KV quantization — storing K/V in INT8/FP8 halves memory with little accuracy loss; ③ StreamingLLM — keeping only the "attention sink" (first 4 tokens) plus a sliding window, so long-session memory drops from $O(s)$ to $O(\text{window})$; ④ H2O sparsity — retaining only heavy-hitter tokens by cumulative attention score, with dynamic replacement.

long-context (32K/128K) online inference, multi-turn chatbots, and the decode side of PD separation; interview favorites: the KV cache memory formula, GQA vs MHA, how MLA saves memory, and how to survive memory blowup on long conversations.

with large batches and long sequences, KV cache grows far faster than weights (each new token adds $2 \times 2 \times L \times H \times d$ bytes), directly capping concurrency and context length. GQA/MLA cut an order of magnitude structurally (MLA: 86GB → 5.1GB), KV quantization halves it again, and StreamingLLM/H2O make memory depend on the window instead of full length — combined, these let the same GPU serve several times more concurrent connections and longer contexts.

Write the KV cache memory formula $2 \times 2 \times L \times H \times d \times s \times b$ and explain each factor; roughly how many GB does a 70B (e.g. 80 layers, 64 heads, head dim 128) use at 8K context with batch 1?

GQA vs MHA/MQA: why sharing KV heads (e.g. 64 → 8) barely hurts accuracy (attention head redundancy) and how much memory it saves?

MLA's low-rank compression: how does $c_t^{KV} = W^{DKV} h_t$ compress per-layer KV into a latent; why can the $W^Q$/$W^U$ projections be fused at inference; the 86GB → 5.1GB numbers?

KV quantization (INT8/FP8): why are K/V easier to quantize than activations (stable value ranges); the cost of halving memory and how it composes with GQA/MLA?

Why must StreamingLLM keep the attention sink (softmax normalization stability, first 4 tokens); how does H2O keep heavy-hitter tokens by cumulative attention score, and how does it differ from sliding windows?