Stage 6 · Per-Tenant Token Quota Rate-Limiter & Graceful Circuit-Breaker Pattern
Stage 6 FinOps protection pattern enforcing sliding-window RPM/TPM + monthly USD spend buckets per `tenant_id` in Redis—smoothly downgrading models at 80% soft caps and breaking circuits gracefully at 100% hard caps without noisy-neighbor spillover.
Best For
Fixed-seat B2B SaaS AI products preventing power-user loops or noisy neighbors from bankrupting blended unit gross margins.
Human-AI Boundary
Account admins receive proactive telemetry alerts at 80% spend and can self-serve quota top-ups rather than hitting mysterious silent errors.
Anti-Patterns to Avoid
- Enforcing only HTTP request-count limits (RPM) when a single 128k-token Agent request costs 500x more than a short chat turn
- Hard-crashing all requests with HTTP 500 on soft limit instead of gracefully switching to small-model / semantic-cache mode
Latency/Cost Profile & Eval Metrics
Atomic Redis Lua check + post-stream token reconciliation < 3ms gateway overhead · Zero upstream provider overrun beyond configured per-workspace USD hard limit
- Noisy-Neighbor Isolation & Hard-Cap Enforcement: 0 tenant overruns > 102% budget; < 0.1% impact on healthy workspaces during overload (Verify atomic pre-reservation under 500 concurrent requests per tenant)
💰 Business ROI Model & Evidence Level
production_practiceEliminates runaway $10k+ single-tenant billing spikes from misconfigured loops, safeguarding >=60% platform-wide gross margin floors.
🏗️ End-to-End Architecture Pipeline
Gateway runs atomic Lua token+USD reservation before upstream dispatch and settles exact `usage` refunds upon stream finish.
At 80% cap, force semantic cache + smaller tier fallback; at 100% cap, short-circuit cleanly with structured 429 retry metadata.
🚨 Production Failure Modes & Signatures
- Streaming client disconnects mid-response bypassing post-hook usage accounting if not tracked via `finally` stream cleanup
- Race condition allowing 50 parallel burst requests to bypass non-atomic Read-Then-Write quota checks
🛡️ Compliance & Governance Hard Constraints
- Keep separate upstream API key pools or concurrency semaphores across Free/Trial vs Enterprise SLA tiers
- Never leak global or other-tenant quota counters inside HTTP 429 headers or error payloads
Bidirectional Taxonomy Links
Internal Tools & Primary Sources
- Stripe / Kong AI Gateway — Token Rate Limiting Architecture (engineering_blog, 2026-11-17)