N5Medium★ AI 网关必考 · FinOps 限流Part N · 数据特征管道与生产系统算子

多租户 LLM TPM 令牌桶预扣减与流式核销限流器

⚡闪卡盲测➔

解决流式 LLM 输出事先未知长度的限流难题:请求发出前按 worst-case 原子预占配额,流结束按实际 usage 多退少补。

⏱️ 时间复杂度: O(1) per op
💾 空间复杂度: O(Tenants)
🎯 免登录实战自测闭环(先做再看答案)
按「机制回忆 → 关键行填空 → 边界反例 → 追问与错题记录」顺序自测,进度自动保存在本地。
Step 1 · 机制口述与数值稳定性自检
Step 2 · 核心代码关键行补全
class TenantTokenBucket:
"""
Multi-tenant LLM gateway TPM limiter supporting atomic pre-reservation + post-stream settlement.
"""
def __init__(self, capacity: float, refill_per_sec: float):
self.capacity = float(capacity)
self.refill_per_sec = float(refill_per_sec)
self.buckets = {} # tenant_id -> [tokens_available, last_ts]
def _refill(self, tenant_id: str, now_s: float) -> list:
if tenant_id not in self.buckets:
self.buckets[tenant_id] = [self.capacity, now_s]
return self.buckets[tenant_id]
state = self.buckets[tenant_id]
elapsed =
state[0] = min(self.capacity, state[0] + elapsed * self.refill_per_sec)
state[1] = now_s
return state
def pre_reserve(self, tenant_id: str, reserve_tokens: float, now_s: float) -> bool:
state = self._refill(tenant_id, now_s)
if state[0] < reserve_tokens:
return False
state[0] -= reserve_tokens
return True
def settle(self, tenant_id: str, reserved_tokens: float, actual_tokens: float) -> float:
state = self.buckets[tenant_id]
refund = reserved_tokens - actual_tokens
state[0] =
return state[0]
Step 3 · 反例与边界断言核对
Step 4 · 面试追问盲答与错题记录
Q1: 在多节点 FastAPI 网关集群下,如何避免这段限流状态出现并发竞争超发?
下一题: 分布式通信 Ring-AllReduce 两阶段带宽最优仿真 →
💡

实战一句话记忆口诀

按需预扣上限值,流完退补真实数,时钟不倒流隔离每租户

📐 核心数学公式与机理推导

B(t)=min⁡(Cmax⁡,  B(t0)+(t−t0)⋅r),reconcile=Treserved−TactualB(t) = \min\big(C_{\max},\; B(t_0) + (t - t_0)\cdot r\big), \quad \text{reconcile} = T_{\text{reserved}} - T_{\text{actual}}
### 为什么普通 API 限流(RPM)会拖垮大模型网关?
一个带 60k 上下文工具检索的 Agent 请求消耗的 GPU 成本可以是普通对话的 500 倍。因此生产网关必须按 Token(TPM)做隔离:
1. 流式事前不知真实 `completion_tokens`:分发前先按 reserve = prompt_tokens + max_tokens 原子扣减;
2. 流结束后结算退款:根据服务端返回的真实 usage.total_tokens 退回差额 reserve - actual,既封死并发瞬间超发漏洞,又不误伤正常租户配额;
3. 分布式落地时用 Redis Lua 保证 Read-Refill-Deduct 原子性。

🔄 张量维度与形状流转说明

tenant_id -> refill(now - last_ts) -> atomic pre_reserve(max_tokens) -> stream end -> settle(reserved - actual)

🛡️ 数值稳定性自检与边界要点

  • 必须先做 elapsed = max(0.0, now_s - last_ts) 保护,避免跨机器轻微时钟回拨扣成负数
  • 并发请求发出前必须用 Pre-Reserve + Post-Settle 而非事后记账,否则 50 个并发大上下文请求可瞬时穿透日配额
  • 客户端中途断开 SSE 连接时必须在服务端 finally 块核销已生成 Token,防止预占额度永久泄漏或绕过计费

💻 完整参考代码实现

class TenantTokenBucket:
    """
    Multi-tenant LLM gateway TPM limiter supporting atomic pre-reservation + post-stream settlement.
    """
    def __init__(self, capacity: float, refill_per_sec: float):
        self.capacity = float(capacity)
        self.refill_per_sec = float(refill_per_sec)
        self.buckets = {}  # tenant_id -> [tokens_available, last_ts]

    def _refill(self, tenant_id: str, now_s: float) -> list:
        if tenant_id not in self.buckets:
            self.buckets[tenant_id] = [self.capacity, now_s]
            return self.buckets[tenant_id]
        state = self.buckets[tenant_id]
        elapsed = max(0.0, now_s - state[1])
        state[0] = min(self.capacity, state[0] + elapsed * self.refill_per_sec)
        state[1] = now_s
        return state

    def pre_reserve(self, tenant_id: str, reserve_tokens: float, now_s: float) -> bool:
        state = self._refill(tenant_id, now_s)
        if state[0] < reserve_tokens:
            return False
        state[0] -= reserve_tokens
        return True

    def settle(self, tenant_id: str, reserved_tokens: float, actual_tokens: float) -> float:
        state = self.buckets[tenant_id]
        refund = reserved_tokens - actual_tokens
        state[0] = min(self.capacity, max(0.0, state[0] + refund))
        return state[0]

🧪 本地测试与正确性断言

可直接复制到本地 Python / Jupyter 验证:

limiter = TenantTokenBucket(capacity=1000.0, refill_per_sec=100.0)
# Pre-reserve worst-case 800 tokens for Tenant A
assert limiter.pre_reserve("tA", 800.0, now_s=0.0) is True
# Burst second request for 400 tokens must reject when only 200 remain
assert limiter.pre_reserve("tA", 400.0, now_s=0.0) is False
# Settle first request actually used 250 tokens -> refund 550 -> balance becomes 750
bal = limiter.settle("tA", reserved_tokens=800.0, actual_tokens=250.0)
assert abs(bal - 750.0) < 1e-6
# Tenant B maintains isolated bucket
assert limiter.pre_reserve("tB", 900.0, now_s=0.0) is True
print("✓ Tenant token-bucket pre-reserve & settle assertion passed")

🎯 核心工程追问与标准应答

Q1:在多节点 FastAPI 网关集群下,如何避免这段限流状态出现并发竞争超发?
将状态迁入 Redis,使用单条 Redis Lua Script 在单线程原子上下文内执行 GET -> Refill -> Compare & DECR -> EXPIRE,并用 Redis TIME 或单调网关时间戳对齐时钟。
⚡
想检验本题的肌肉记忆?进入 3 阶闪卡主动回忆盲测
隐藏答案遮罩,自测:审题锚点 ➔ 数理公式机理 ➔ 工业标杆代码与连环追问。3 分钟攻破认知盲区!
🎯 立即在闪卡中盲测此题➔
← 上一题: Prefill / Decode 分离式推理调度仿真 (DistServe / Mooncake)返回 Part N 与全部 105 题下一题: 分布式通信 Ring-AllReduce 两阶段带宽最优仿真 →