N5Medium★ AI 网关必考 · FinOps 限流Part N · 数据特征管道与生产系统算子
多租户 LLM TPM 令牌桶预扣减与流式核销限流器
解决流式 LLM 输出事先未知长度的限流难题:请求发出前按 worst-case 原子预占配额,流结束按实际 usage 多退少补。
⏱️ 时间复杂度:
O(1) per op💾 空间复杂度:
O(Tenants)🎯 免登录实战自测闭环(先做再看答案)
按「机制回忆 → 关键行填空 → 边界反例 → 追问与错题记录」顺序自测,进度自动保存在本地。
Step 1 · 机制口述与数值稳定性自检
Step 2 · 核心代码关键行补全
class TenantTokenBucket:
"""
Multi-tenant LLM gateway TPM limiter supporting atomic pre-reservation + post-stream settlement.
"""
def __init__(self, capacity: float, refill_per_sec: float):
self.capacity = float(capacity)
self.refill_per_sec = float(refill_per_sec)
self.buckets = {} # tenant_id -> [tokens_available, last_ts]
def _refill(self, tenant_id: str, now_s: float) -> list:
if tenant_id not in self.buckets:
self.buckets[tenant_id] = [self.capacity, now_s]
return self.buckets[tenant_id]
state = self.buckets[tenant_id]
elapsed =
state[0] = min(self.capacity, state[0] + elapsed * self.refill_per_sec)
state[1] = now_s
return state
def pre_reserve(self, tenant_id: str, reserve_tokens: float, now_s: float) -> bool:
state = self._refill(tenant_id, now_s)
if state[0] < reserve_tokens:
return False
state[0] -= reserve_tokens
return True
def settle(self, tenant_id: str, reserved_tokens: float, actual_tokens: float) -> float:
state = self.buckets[tenant_id]
refund = reserved_tokens - actual_tokens
state[0] =
return state[0]
Step 3 · 反例与边界断言核对
💡
实战一句话记忆口诀
按需预扣上限值,流完退补真实数,时钟不倒流隔离每租户
📐 核心数学公式与机理推导
### 为什么普通 API 限流(RPM)会拖垮大模型网关?
一个带 60k 上下文工具检索的 Agent 请求消耗的 GPU 成本可以是普通对话的 500 倍。因此生产网关必须按 Token(TPM)做隔离:
1. 流式事前不知真实 `completion_tokens`:分发前先按
2. 流结束后结算退款:根据服务端返回的真实
3. 分布式落地时用 Redis Lua 保证 Read-Refill-Deduct 原子性。
一个带 60k 上下文工具检索的 Agent 请求消耗的 GPU 成本可以是普通对话的 500 倍。因此生产网关必须按 Token(TPM)做隔离:
1. 流式事前不知真实 `completion_tokens`:分发前先按
reserve = prompt_tokens + max_tokens 原子扣减;2. 流结束后结算退款:根据服务端返回的真实
usage.total_tokens 退回差额 reserve - actual,既封死并发瞬间超发漏洞,又不误伤正常租户配额;3. 分布式落地时用 Redis Lua 保证 Read-Refill-Deduct 原子性。
🔄 张量维度与形状流转说明
tenant_id -> refill(now - last_ts) -> atomic pre_reserve(max_tokens) -> stream end -> settle(reserved - actual)
🛡️ 数值稳定性自检与边界要点
- 必须先做
elapsed = max(0.0, now_s - last_ts)保护,避免跨机器轻微时钟回拨扣成负数 - 并发请求发出前必须用 Pre-Reserve + Post-Settle 而非事后记账,否则 50 个并发大上下文请求可瞬时穿透日配额
- 客户端中途断开 SSE 连接时必须在服务端
finally块核销已生成 Token,防止预占额度永久泄漏或绕过计费
💻 完整参考代码实现
class TenantTokenBucket:
"""
Multi-tenant LLM gateway TPM limiter supporting atomic pre-reservation + post-stream settlement.
"""
def __init__(self, capacity: float, refill_per_sec: float):
self.capacity = float(capacity)
self.refill_per_sec = float(refill_per_sec)
self.buckets = {} # tenant_id -> [tokens_available, last_ts]
def _refill(self, tenant_id: str, now_s: float) -> list:
if tenant_id not in self.buckets:
self.buckets[tenant_id] = [self.capacity, now_s]
return self.buckets[tenant_id]
state = self.buckets[tenant_id]
elapsed = max(0.0, now_s - state[1])
state[0] = min(self.capacity, state[0] + elapsed * self.refill_per_sec)
state[1] = now_s
return state
def pre_reserve(self, tenant_id: str, reserve_tokens: float, now_s: float) -> bool:
state = self._refill(tenant_id, now_s)
if state[0] < reserve_tokens:
return False
state[0] -= reserve_tokens
return True
def settle(self, tenant_id: str, reserved_tokens: float, actual_tokens: float) -> float:
state = self.buckets[tenant_id]
refund = reserved_tokens - actual_tokens
state[0] = min(self.capacity, max(0.0, state[0] + refund))
return state[0]
🧪 本地测试与正确性断言
可直接复制到本地 Python / Jupyter 验证:
limiter = TenantTokenBucket(capacity=1000.0, refill_per_sec=100.0)
# Pre-reserve worst-case 800 tokens for Tenant A
assert limiter.pre_reserve("tA", 800.0, now_s=0.0) is True
# Burst second request for 400 tokens must reject when only 200 remain
assert limiter.pre_reserve("tA", 400.0, now_s=0.0) is False
# Settle first request actually used 250 tokens -> refund 550 -> balance becomes 750
bal = limiter.settle("tA", reserved_tokens=800.0, actual_tokens=250.0)
assert abs(bal - 750.0) < 1e-6
# Tenant B maintains isolated bucket
assert limiter.pre_reserve("tB", 900.0, now_s=0.0) is True
print("✓ Tenant token-bucket pre-reserve & settle assertion passed")🎯 核心工程追问与标准应答
Q1:在多节点 FastAPI 网关集群下,如何避免这段限流状态出现并发竞争超发?
将状态迁入 Redis,使用单条 Redis Lua Script 在单线程原子上下文内执行
GET -> Refill -> Compare & DECR -> EXPIRE,并用 Redis TIME 或单调网关时间戳对齐时钟。⚡
想检验本题的肌肉记忆?进入 3 阶闪卡主动回忆盲测
隐藏答案遮罩,自测:审题锚点 ➔ 数理公式机理 ➔ 工业标杆代码与连环追问。3 分钟攻破认知盲区!