High-Concurrency Serving

High-concurrency LLM serving = capacity planning and traffic governance that sustain a target QPS on finite GPU compute while honoring latency budgets (TTFT, TBT/TPOT, end-to-end P50/P95). Core mechanisms: ① Capacity estimation and latency budgets: per-GPU decode throughput is finite (~1-2K tokens/s, HBM-bandwidth-bound); derive max per-instance concurrency from peak throughput × average output length × latency budget. Example: one GPU at 2000 tokens/s, average output 500 tokens, TBT budget 20 ms (50 token/s per request) → each request consumes 50 token/s, so max concurrency ≈ $2000/50 = 40$; multiply by the number of replicas for cluster capacity. Beyond this, the latency budget breaks (queuing time grows non-linearly with utilization). ② Rate limiting: the token-bucket algorithm — bucket capacity $C$ = allowed burst (requests admitted at once), refill rate $r$ = long-term average QPS cap; each request consumes one token; an empty bucket means reject/queue. ③ Queuing and backpressure: each instance keeps a bounded request queue; when full, reject (503) or client backoff-retry; backpressure signals propagate from slow replicas upstream to gateways/clients, throttling them to prevent retry storms → full queues → cascading failure; timeouts and circuit breaking avoid infinite waits. ④ Multi-replica routing: gateways route by least-connections / shortest queue (more robust to slow replicas than round-robin); use consistent hashing when prefix-cache affinity matters (the same session/prefix lands on the same replica, raising prompt-cache hit rates); health checks drain failed nodes. ⑤ Graceful degradation: priority queues (paid/high-priority admission), output truncation on timeout, fallback to smaller models, rejecting low-value traffic — keeping P95 budgets intact through spikes.

production LLM API gateways and capacity management; interview favorites: "capacity plan given QPS and latency budgets", "how to set token-bucket params", "what to do when the queue is full", "how to route across replicas".

LLM inference consumes non-linear resources (memory + compute + KV cache) under bandwidth-bound throughput; ungoverned, peak traffic makes queues grow unbounded, P95 explodes and cascading restarts follow. Rate limiting/queuing/backpressure shape traffic into capacity; multi-replica routing with prefix affinity lifts cache hits and utilization; graceful degradation keeps priority users within budget under overload — numerically, queuing-theory (M/M/c approximation) shows wait times degrade exponentially past ~80% utilization, so capacity plans typically target 60-70% peak utilization with headroom.

Test Your Knowledge: Practice Questions for "High-Concurrency Serving"

Single choice pitfall questions with instant feedback and mistake tracking.