High-Concurrency Serving

High-concurrency LLM serving = capacity planning and traffic governance that sustain a target QPS on finite GPU compute while honoring latency budgets (TTFT, TBT/TPOT, end-to-end P50/P95). Core mechanisms: ① Capacity estimation and latency budgets: per-GPU decode throughput is finite (~1-2K tokens/s, HBM-bandwidth-bound); derive max per-instance concurrency from peak throughput × average output length × latency budget. Example: one GPU at 2000 tokens/s, average output 500 tokens, TBT budget 20 ms (50 token/s per request) → each request consumes 50 token/s, so max concurrency ≈ $2000/50 = 40$; multiply by the number of replicas for cluster capacity. Beyond this, the latency budget breaks (queuing time grows non-linearly with utilization). ② Rate limiting: the token-bucket algorithm — bucket capacity $C$ = allowed burst (requests admitted at once), refill rate $r$ = long-term average QPS cap; each request consumes one token; an empty bucket means reject/queue. ③ Queuing and backpressure: each instance keeps a bounded request queue; when full, reject (503) or client backoff-retry; backpressure signals propagate from slow replicas upstream to gateways/clients, throttling them to prevent retry storms → full queues → cascading failure; timeouts and circuit breaking avoid infinite waits. ④ Multi-replica routing: gateways route by least-connections / shortest queue (more robust to slow replicas than round-robin); use consistent hashing when prefix-cache affinity matters (the same session/prefix lands on the same replica, raising prompt-cache hit rates); health checks drain failed nodes. ⑤ Graceful degradation: priority queues (paid/high-priority admission), output truncation on timeout, fallback to smaller models, rejecting low-value traffic — keeping P95 budgets intact through spikes.

production LLM API gateways and capacity management; interview favorites: "capacity plan given QPS and latency budgets", "how to set token-bucket params", "what to do when the queue is full", "how to route across replicas".

LLM inference consumes non-linear resources (memory + compute + KV cache) under bandwidth-bound throughput; ungoverned, peak traffic makes queues grow unbounded, P95 explodes and cascading restarts follow. Rate limiting/queuing/backpressure shape traffic into capacity; multi-replica routing with prefix affinity lifts cache hits and utilization; graceful degradation keeps priority users within budget under overload — numerically, queuing-theory (M/M/c approximation) shows wait times degrade exponentially past ~80% utilization, so capacity plans typically target 60-70% peak utilization with headroom.

Hand-compute capacity: GPU at 2000 tokens/s, average output 500 tokens, TBT budget 20 ms → max concurrency $2000/50 = 40$; replicas needed for a target QPS; why latency degrades past ~80% utilization (M/M/c).

Token-bucket rate limiting: how to set capacity $C$ (burst) and refill rate $r$ (average QPS); handling bursts vs steady traffic; reject vs queue on overflow and the reasoning.

Queuing and backpressure: why the queue must be bounded; options when full (503 reject, backoff retry, degrade); how backpressure prevents retry storms and cascades, and the role of timeout circuit breaking.

Multi-replica routing: round-robin vs least-connections vs consistent hashing; why prefix-cache affinity calls for consistent hashing (pinning a session to a replica), and how failed replicas are drained.

Latency budgets and graceful degradation: how to split TTFT/TBT/P95 budgets; what priority queues, output truncation and small-model fallback sacrifice vs protect under overload.