N5Medium★ AI Gateway Essential · FinOpsPart N · Data Pipelines & Production SysOps

Multi-Tenant Token Bucket with Pre-Reserve & Settlement

⚡Flashcard Drill➔

Enforce strict LLM TPM fairness across tenants despite unknown stream output lengths via atomic pre-reservation and post-stream settlement.

⏱️ Time: O(1) per op
💾 Space: O(Tenants)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
class TenantTokenBucket:
"""
Multi-tenant LLM gateway TPM limiter supporting atomic pre-reservation + post-stream settlement.
"""
def __init__(self, capacity: float, refill_per_sec: float):
self.capacity = float(capacity)
self.refill_per_sec = float(refill_per_sec)
self.buckets = {} # tenant_id -> [tokens_available, last_ts]
def _refill(self, tenant_id: str, now_s: float) -> list:
if tenant_id not in self.buckets:
self.buckets[tenant_id] = [self.capacity, now_s]
return self.buckets[tenant_id]
state = self.buckets[tenant_id]
elapsed =
state[0] = min(self.capacity, state[0] + elapsed * self.refill_per_sec)
state[1] = now_s
return state
def pre_reserve(self, tenant_id: str, reserve_tokens: float, now_s: float) -> bool:
state = self._refill(tenant_id, now_s)
if state[0] < reserve_tokens:
return False
state[0] -= reserve_tokens
return True
def settle(self, tenant_id: str, reserved_tokens: float, actual_tokens: float) -> float:
state = self.buckets[tenant_id]
refund = reserved_tokens - actual_tokens
state[0] =
return state[0]
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What happens if an organization wants USD-weighted buckets across heterogeneous 8B vs Frontier models?
Next: Ring-AllReduce (Reduce-Scatter + All-Gather) →
💡

Core Mental Anchor / Mnemonic

Pre-reserve worst-case budget atomically; settle real tokens post-stream in `finally`.

📐 Mathematical Derivation & Core Formula

B(t)=min⁡(Cmax⁡,  B(t0)+(t−t0)⋅r),reconcile=Treserved−TactualB(t) = \min\big(C_{\max},\; B(t_0) + (t - t_0)\cdot r\big), \quad \text{reconcile} = T_{\text{reserved}} - T_{\text{actual}}
### Solving Unknown Output-Length Rate Limiting
HTTP RPM fails when prompts span 50 vs 100k tokens. Atomically reserve prompt + max_tokens before dispatch, then refund reserved - actual via finally stream cleanup.

🔄 Tensor Dimensions & Shape Flow

tenant bucket -> continuous refill -> pre_reserve -> stream complete -> settle delta

🛡️ Numerical Stability & Pitfalls Checklist

  • Clamp elapsed >= 0 to defend against clock jitter across distributed gateway nodes.
  • Always trigger settle inside a stream finally block even when clients abort connections early.

💻 Reference Code Implementation

class TenantTokenBucket:
    """
    Multi-tenant LLM gateway TPM limiter supporting atomic pre-reservation + post-stream settlement.
    """
    def __init__(self, capacity: float, refill_per_sec: float):
        self.capacity = float(capacity)
        self.refill_per_sec = float(refill_per_sec)
        self.buckets = {}  # tenant_id -> [tokens_available, last_ts]

    def _refill(self, tenant_id: str, now_s: float) -> list:
        if tenant_id not in self.buckets:
            self.buckets[tenant_id] = [self.capacity, now_s]
            return self.buckets[tenant_id]
        state = self.buckets[tenant_id]
        elapsed = max(0.0, now_s - state[1])
        state[0] = min(self.capacity, state[0] + elapsed * self.refill_per_sec)
        state[1] = now_s
        return state

    def pre_reserve(self, tenant_id: str, reserve_tokens: float, now_s: float) -> bool:
        state = self._refill(tenant_id, now_s)
        if state[0] < reserve_tokens:
            return False
        state[0] -= reserve_tokens
        return True

    def settle(self, tenant_id: str, reserved_tokens: float, actual_tokens: float) -> float:
        state = self.buckets[tenant_id]
        refund = reserved_tokens - actual_tokens
        state[0] = min(self.capacity, max(0.0, state[0] + refund))
        return state[0]

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

limiter = TenantTokenBucket(capacity=1000.0, refill_per_sec=100.0)
# Pre-reserve worst-case 800 tokens for Tenant A
assert limiter.pre_reserve("tA", 800.0, now_s=0.0) is True
# Burst second request for 400 tokens must reject when only 200 remain
assert limiter.pre_reserve("tA", 400.0, now_s=0.0) is False
# Settle first request actually used 250 tokens -> refund 550 -> balance becomes 750
bal = limiter.settle("tA", reserved_tokens=800.0, actual_tokens=250.0)
assert abs(bal - 750.0) < 1e-6
# Tenant B maintains isolated bucket
assert limiter.pre_reserve("tB", 900.0, now_s=0.0) is True
print("✓ Tenant token-bucket pre-reserve & settle assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What happens if an organization wants USD-weighted buckets across heterogeneous 8B vs Frontier models?
Convert tokens into micro-dollar cost units (tokens * price_per_token) before calling the exact same pre-reserve + settle Lua bucket.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Prefill / Decode Disaggregated Serving SimulatorBack to Part N & All 105Next: Ring-AllReduce (Reduce-Scatter + All-Gather) →