I6Medium★ LLM Serving 必考 · 吞吐核心Part I · 推理生成、解码与量化
Continuous Batching 连续批处理调度器
按 Iteration(单步)维度释放已结束请求并在下一解码步立即插入新请求,消除静态批处理的木桶空转等待。
⏱️ 时间复杂度:
O(|Batch|) 每解码步调度💾 空间复杂度:
O(|Waiting| + |Running|)🎯 免登录实战自测闭环(先做再看答案)
按「机制回忆 → 关键行填空 → 边界反例 → 追问与错题记录」顺序自测,进度自动保存在本地。
Step 1 · 机制口述与数值稳定性自检
Step 2 · 核心代码关键行补全
class ContinuousBatchScheduler:
"""Iteration-level LLM scheduler admitting new waiting requests as soon as slots & KV budget open."""
def __init__(self, max_batch_size: int, kv_token_budget: int):
self.max_batch_size =
self.kv_token_budget = kv_token_budget
self.waiting = [] # list of dicts: {id, prompt_len, remaining_gen}
self.running = [] # list of dicts: {id, used_kv, remaining_gen}
self.finished_ids = []
def submit(self, req_id: str, prompt_len: int, gen_len: int) -> None:
self.waiting.append({"id": req_id, "prompt_len": prompt_len, "remaining_gen": gen_len})
def step(self) -> list:
# 1. Admit waiting requests within concurrency & KV budget
used_kv_total =
while self.waiting and len(self.running) < self.max_batch_size:
nxt = self.waiting[0]
if used_kv_total + nxt["prompt_len"] + 1 > self.kv_token_budget:
break
self.waiting.pop(0)
self.running.append({"id": nxt["id"], "used_kv": nxt["prompt_len"], "remaining_gen": nxt["remaining_gen"]})
used_kv_total += nxt["prompt_len"]
# 2. Run one iteration step across all active requests
stepped_ids = []
still_running = []
for req in self.running:
req["used_kv"] += 1
req["remaining_gen"] -= 1
stepped_ids.append(req["id"])
if req["remaining_gen"] <= 0:
self.finished_ids.append(req["id"])
else:
still_running.append(req)
self.running = still_running
return stepped_ids
Step 3 · 反例与边界断言核对
Step 4 · 面试追问盲答与错题记录
Q1: 什么是 Chunked Prefill(分块预填充)?它为什么对 Continuous Batching 很关键?
💡
实战一句话记忆口诀
静态整批等最慢,连续按步随到随插,槽空立刻填新客,预算卡住防爆显存
📐 核心数学公式与机理推导
### Iteration-Level Scheduling 与木桶效应消除
传统静态批处理(Static Batching)必须等待同批所有请求全部输出
Orca / vLLM 采用 Iteration-Level Scheduling(连续批处理):
- 在每一个解码 step 结束后立即检查已完成请求并回收其 KV 显存;
- 在下一个 step 开始前,只要并发槽位与 KV Token 预算允许,立刻从等待队列拉入新请求执行 Prefill;
- 现代服务(如 Sarathi / vLLM Chunked Prefill)进一步把长 Prefill 切块与 Decode 混批,防止大 Prompt 卡顿正在解码用户的延迟(TTFT vs ITL 平衡)。
传统静态批处理(Static Batching)必须等待同批所有请求全部输出
[EOS] 后才能接入下一批,短请求完成后对应的 GPU 算力与批槽位全部空转浪费。Orca / vLLM 采用 Iteration-Level Scheduling(连续批处理):
- 在每一个解码 step 结束后立即检查已完成请求并回收其 KV 显存;
- 在下一个 step 开始前,只要并发槽位与 KV Token 预算允许,立刻从等待队列拉入新请求执行 Prefill;
- 现代服务(如 Sarathi / vLLM Chunked Prefill)进一步把长 Prefill 切块与 Decode 混批,防止大 Prompt 卡顿正在解码用户的延迟(TTFT vs ITL 平衡)。
🔄 张量维度与形状流转说明
waiting queue -> 检查 max_batch 与 kv_budget -> 插入 running -> 全批单步 decode -> 回收 finished -> 下一步立即补位
🛡️ 数值稳定性自检与边界要点
- 准入判断时除了看条数上限
max_batch_size,还必须核对活跃请求的 KV 总预算,防止中途显存 OOM 触发大规模抢占换出 - 当长请求连续占位而突发短请求需要优先进入或发生 KV 满载时,需支持抢占回退(Preemption / Swap / Recompute)
💻 完整参考代码实现
class ContinuousBatchScheduler:
"""Iteration-level LLM scheduler admitting new waiting requests as soon as slots & KV budget open."""
def __init__(self, max_batch_size: int, kv_token_budget: int):
self.max_batch_size = max_batch_size
self.kv_token_budget = kv_token_budget
self.waiting = [] # list of dicts: {id, prompt_len, remaining_gen}
self.running = [] # list of dicts: {id, used_kv, remaining_gen}
self.finished_ids = []
def submit(self, req_id: str, prompt_len: int, gen_len: int) -> None:
self.waiting.append({"id": req_id, "prompt_len": prompt_len, "remaining_gen": gen_len})
def step(self) -> list:
# 1. Admit waiting requests within concurrency & KV budget
used_kv_total = sum(r["used_kv"] for r in self.running)
while self.waiting and len(self.running) < self.max_batch_size:
nxt = self.waiting[0]
if used_kv_total + nxt["prompt_len"] + 1 > self.kv_token_budget:
break
self.waiting.pop(0)
self.running.append({"id": nxt["id"], "used_kv": nxt["prompt_len"], "remaining_gen": nxt["remaining_gen"]})
used_kv_total += nxt["prompt_len"]
# 2. Run one iteration step across all active requests
stepped_ids = []
still_running = []
for req in self.running:
req["used_kv"] += 1
req["remaining_gen"] -= 1
stepped_ids.append(req["id"])
if req["remaining_gen"] <= 0:
self.finished_ids.append(req["id"])
else:
still_running.append(req)
self.running = still_running
return stepped_ids
🧪 本地测试与正确性断言
可直接复制到本地 Python / Jupyter 验证:
sched = ContinuousBatchScheduler(max_batch_size=2, kv_token_budget=30)
sched.submit("r1", prompt_len=4, gen_len=1)
sched.submit("r2", prompt_len=4, gen_len=3)
sched.submit("r3", prompt_len=4, gen_len=2)
s1 = sched.step() # runs r1, r2; r1 finishes!
s2 = sched.step() # r3 immediately enters batch alongside r2 without waiting for r2 to finish!
assert s1 == ["r1", "r2"], f"Step 1 mismatch: {s1}"
assert s2 == ["r2", "r3"], f"Step 2 must admit r3 immediately: {s2}"
print("✓ Continuous Batching scheduler assertion passed")🎯 核心工程追问与标准应答
Q1:什么是 Chunked Prefill(分块预填充)?它为什么对 Continuous Batching 很关键?
当一条超长 Prompt(如 16k tokens)在同一步执行完整 Prefill 时,单次计算会独占 GPU 数百毫秒,拖累同批正在 Decode 用户的 Token 间延迟(ITL/TPOT)。Chunked Prefill 将长 Prefill 切成固定 token 预算的小块(如 512 tokens),和正在进行的 Decode 请求合并调度,显著降低尾延迟抖动。
⚡
想检验本题的肌肉记忆?进入 3 阶闪卡主动回忆盲测
隐藏答案遮罩,自测:审题锚点 ➔ 数理公式机理 ➔ 工业标杆代码与连环追问。3 分钟攻破认知盲区!