Continuous Batching

Continuous Batching (iteration-level scheduling, introduced by Orca) does not wait for an entire batch to finish before serving the next one; at the end of every decoding iteration (one token step) the batch membership is reshuffled: requests that finished (naturally or by max-length truncation) are evicted, waiting requests that completed prefill are inserted, and each in-flight request keeps its own KV-cache progress. Contrast static (request-level) batching, which pins a batch until all requests complete — the slowest request (e.g. 40 tokens) blocks the fastest, whose slot idles the rest of the time → low GPU utilization. With continuous batching every step tries to keep the batch full, raising throughput roughly 2-3× (Orca reports about 2.5×; the win is biggest for mixed long/short workloads, since short requests finish fast, free slots, and new ones fill in). Key implementation points: ① a new request must prefill first (compute the whole prompt's KV in one pass) before joining decode iterations; how prefill and decode are co-scheduled decides TTFT; ② preemption: under memory pressure the engine can preempt some requests' decode slots (common policies: new-prefill admission or priority-based eviction); evicted requests' KV is swapped to CPU or discarded to be restarted/recomputed later; ③ sequences of unequal length in the batch need padding or chunked attention to avoid wasted compute; ④ it composes naturally with PagedAttention, whose paged KV allows slots to be allocated/freed block-by-block as requests come and go, fragmentation-free.

the default scheduler in every mainstream engine (vLLM/SGLang/TensorRT-LLM); biggest gains for online chat/API traffic (random arrival, mixed lengths); interview staples: "static vs continuous batching", "preemption and swap", "why GPU utilization goes up".

static batching wastes GPU cycles from two sources — the slowest request in a batch blocks everyone (bubble), and waits between batch arrivals. Continuous batching refines the scheduling granularity from request to iteration, so every step has finished requests freeing slots and new requests filling them, driving compute density toward 100%; on the same hardware this is roughly 2-3× more throughput, with more controllable TTFT since new requests no longer wait for a whole batch to drain.

Test Your Knowledge: Practice Questions for "Continuous Batching"

Single choice pitfall questions with instant feedback and mistake tracking.