M4-060M4: Sequences & TransformersKV Cache & Inference OptimizationsMedium
⚡Flashcard Drill➔Mastery:
Continuous Batching vs. Static Batching in LLM Serving
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Continuous batching operates at the iteration level to dynamically insert arriving requests and evict completed requests at every decoding step, eliminating the massive bubble overhead of static sequence-level batching.
📌 Key Takeaways
- •Static batching: requests are grouped at the sequence level and must wait for the longest sequence to finish, causing massive GPU idle bubbles and high tail latency
- •Continuous (iteration-level) batching: the batch is reconstituted at every single token step; finished sequences exit immediately and new prompts enter without waiting
- •Enables to higher serving throughput and drastically reduces average Time-to-First-Token (TTFT)
📐 Mathematical Derivations
Mathematical Mechanism: Static Batching Bubble Problem: Consider a batch of requests with prompt lengths and generation lengths . The static batch execution time is determined by . Requests that finish early must output padding tokens until the slowest request finishes. The GPU execution efficiency is: In conversational workloads with high length variance (e.g., lengths ranging from 20 to 2048 tokens), can drop below 20%, wasting >80% of GPU compute and memory bandwidth on padding. Continuous Batching Mechanism: Execution is scheduled at the granularity of a single decoding iteration . At step , the batch tensor contains active tokens from ongoing sequences. When sequence outputs an EOS token, its state is immediately evicted, releasing its KV cache. A pending request from the queue can immediately execute its prefill phase (or first decode step) in step . The effective utilization approaches , bounded only by KV cache VRAM limits and scheduling overhead.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① PagedAttention Dependency: Continuous batching causes dynamic, non-contiguous memory allocations as sequences grow and terminate at different rates. Without virtual memory paging (PagedAttention), dynamic memory allocation suffers severe external fragmentation. ② Preemption and Eviction: When VRAM is exhausted due to growing KV caches in the active batch, the engine must either preempt (swap to CPU memory) or recompute (drop KV cache and re-run prefill later) low-priority requests. ③ Mixed Batch Scheduling: Co-scheduling prefill tokens (GEMM) and decode tokens (GEMV) in the same iteration requires chunked prefill to prevent prefills from starving ongoing decode latencies. ④ System Architecture: Pioneered by Orca and popularized by vLLM, TensorRT-LLM, and TGI, continuous batching is the universal de facto standard in production LLM inference engines. ⑤ Interview Strategy: Contrast sequence-level scheduling with step-level scheduling, draw the execution timeline diagram showing how bubbles are eliminated, and explain how it ties into KV cache management.
⚠️ Common Interview Pitfalls
- ✕Believing static batching padding does not consume memory or compute (padding tokens still undergo matrix multiplications)
- ✕Overlooking that continuous batching requires dynamic memory management like PagedAttention to prevent VRAM fragmentation
- ✕Failing to explain what happens when memory runs out during continuous generation (preemption/recomputation)
🎯 Interviewer Follow-up Inquiries
- ?How does an engine handle VRAM exhaustion when all active requests in a continuous batch expand their KV caches?
- ?What is the difference between iteration-level scheduling in Orca and vLLM?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚