M4-060M4: Sequences & TransformersKV Cache & Inference OptimizationsMedium
⚡Flashcard Drill➔Mastery:

Continuous Batching vs. Static Batching in LLM Serving

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-061 →
📐 Mathematical Definition
static: wait for longest;continuous: refill slots every iteration\text{static}:\ \text{wait for longest};\qquad \text{continuous}:\ \text{refill slots every iteration}
⚡ One-Sentence Core Answer
Continuous batching operates at the iteration level to dynamically insert arriving requests and evict completed requests at every decoding step, eliminating the massive bubble overhead of static sequence-level batching.

📌 Key Takeaways

  • •
    Static batching: requests are grouped at the sequence level and must wait for the longest sequence to finish, causing massive GPU idle bubbles and high tail latency
  • •
    Continuous (iteration-level) batching: the batch is reconstituted at every single token step; finished sequences exit immediately and new prompts enter without waiting
  • •
    Enables 3×3\times to 10×10\times higher serving throughput and drastically reduces average Time-to-First-Token (TTFT)

📐 Mathematical Derivations

Mathematical Mechanism: Static Batching Bubble Problem: Consider a batch of BB requests with prompt lengths LipromptL_{i}^{\text{prompt}} and generation lengths LigenL_{i}^{\text{gen}}. The static batch execution time is determined by max⁡i(Liprompt+Ligen)\max_i (L_i^{\text{prompt}} + L_i^{\text{gen}}). Requests that finish early must output padding tokens until the slowest request finishes. The GPU execution efficiency η\eta is: η=∑i=1B(Liprompt+Ligen)B×max⁡i(Liprompt+Ligen)\eta = \frac{\sum_{i=1}^B (L_i^{\text{prompt}} + L_i^{\text{gen}})}{B \times \max_i (L_i^{\text{prompt}} + L_i^{\text{gen}})} In conversational workloads with high length variance (e.g., lengths ranging from 20 to 2048 tokens), η\eta can drop below 20%, wasting >80% of GPU compute and memory bandwidth on padding. Continuous Batching Mechanism: Execution is scheduled at the granularity of a single decoding iteration tt. At step tt, the batch tensor Xt∈RBt×1×dX_t \in \mathbb{R}^{B_t \times 1 \times d} contains active tokens from BtB_t ongoing sequences. When sequence kk outputs an EOS token, its state is immediately evicted, releasing its KV cache. A pending request from the queue can immediately execute its prefill phase (or first decode step) in step t+1t+1. The effective utilization approaches η≈1.0\eta \approx 1.0, bounded only by KV cache VRAM limits and scheduling overhead.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① PagedAttention Dependency: Continuous batching causes dynamic, non-contiguous memory allocations as sequences grow and terminate at different rates. Without virtual memory paging (PagedAttention), dynamic memory allocation suffers severe external fragmentation. ② Preemption and Eviction: When VRAM is exhausted due to growing KV caches in the active batch, the engine must either preempt (swap to CPU memory) or recompute (drop KV cache and re-run prefill later) low-priority requests. ③ Mixed Batch Scheduling: Co-scheduling prefill tokens (GEMM) and decode tokens (GEMV) in the same iteration requires chunked prefill to prevent prefills from starving ongoing decode latencies. ④ System Architecture: Pioneered by Orca and popularized by vLLM, TensorRT-LLM, and TGI, continuous batching is the universal de facto standard in production LLM inference engines. ⑤ Interview Strategy: Contrast sequence-level scheduling with step-level scheduling, draw the execution timeline diagram showing how bubbles are eliminated, and explain how it ties into KV cache management.
⚠️ Common Interview Pitfalls
  • ✕
    Believing static batching padding does not consume memory or compute (padding tokens still undergo matrix multiplications)
  • ✕
    Overlooking that continuous batching requires dynamic memory management like PagedAttention to prevent VRAM fragmentation
  • ✕
    Failing to explain what happens when memory runs out during continuous generation (preemption/recomputation)
🎯 Interviewer Follow-up Inquiries
  • ?
    How does an engine handle VRAM exhaustion when all active requests in a continuous batch expand their KV caches?
  • ?
    What is the difference between iteration-level scheduling in Orca and vLLM?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-059: Losslessness and Theoretical Guarantee of Speculative Decoding📋 Back to BankNext →M4-061: Prefix Caching and RadixAttention Mechanisms