Autoscaling & Cost

Autoscaling = adding/removing GPU replicas based on live load — the valve between cost and latency. Three core signals: ① QPS — the most intuitive but the most lagging; it cannot capture per-request load differences (a 1-token and a 1K-token request are both 1 QPS); ② queue length / queuing latency — the most direct overload signal; backlog means replicas are saturated, making it a better trigger than QPS; ③ GPU utilization — the healthy band is usually 70-90%: below 70% means idle GPUs burning money, above 90% means queuing latency grows non-linearly and jitter worsens. In practice, serving uses K8s HPA (custom metrics; cooldown prevents oscillation) or the Ray autoscaler; training clusters scale on queued-job backlog. Cost side: ① Spot instances — 60-90% cheaper than on-demand but can be reclaimed (about 2-minute warning), suited to stateless, interruptible, retryable workloads (batch inference/training); critical online services run spot + on-demand hybrid pools; ② reserved instances / committed use discounts; ③ utilization monitoring plus off-peak batch processing to fill idle time.

any workload with traffic variation (day peaks/night troughs, viral-event bursts); queue-based scaling for training clusters; interview favorites: which metrics to scale on, what GPU utilization target to pick, and how to save money with spot instances.

fixed provisioning is either overloaded at peaks (queuing, timeouts, dropped requests) or idle at troughs (GPU utilization under 50% for much of the day — wasted spend). Scaling replicas to follow load on queue length / utilization, then layering spot and reserved commitments on top, can cut cost by more than half at the same SLA — the core trade-off is scaling latency (cold start / image pull) vs overload risk.

Test Your Knowledge: Practice Questions for "Autoscaling & Cost"

Single choice pitfall questions with instant feedback and mistake tracking.