Autoscaling & Cost

Autoscaling = adding/removing GPU replicas based on live load — the valve between cost and latency. Three core signals: ① QPS — the most intuitive but the most lagging; it cannot capture per-request load differences (a 1-token and a 1K-token request are both 1 QPS); ② queue length / queuing latency — the most direct overload signal; backlog means replicas are saturated, making it a better trigger than QPS; ③ GPU utilization — the healthy band is usually 70-90%: below 70% means idle GPUs burning money, above 90% means queuing latency grows non-linearly and jitter worsens. In practice, serving uses K8s HPA (custom metrics; cooldown prevents oscillation) or the Ray autoscaler; training clusters scale on queued-job backlog. Cost side: ① Spot instances — 60-90% cheaper than on-demand but can be reclaimed (about 2-minute warning), suited to stateless, interruptible, retryable workloads (batch inference/training); critical online services run spot + on-demand hybrid pools; ② reserved instances / committed use discounts; ③ utilization monitoring plus off-peak batch processing to fill idle time.

any workload with traffic variation (day peaks/night troughs, viral-event bursts); queue-based scaling for training clusters; interview favorites: which metrics to scale on, what GPU utilization target to pick, and how to save money with spot instances.

fixed provisioning is either overloaded at peaks (queuing, timeouts, dropped requests) or idle at troughs (GPU utilization under 50% for much of the day — wasted spend). Scaling replicas to follow load on queue length / utilization, then layering spot and reserved commitments on top, can cut cost by more than half at the same SLA — the core trade-off is scaling latency (cold start / image pull) vs overload risk.

Choosing scaling signals: pros and cons of QPS, queue length / queuing latency and GPU utilization; why is queue backlog a better scale-up trigger than QPS (uneven request load)?

Why target GPU utilization at 70-90%: below 70% wastes idle capacity, above 90% queuing latency grows non-linearly; how does the target couple to the latency budget?

Autoscaling mechanics: K8s HPA custom metrics and cooldown to prevent oscillation; how do you mitigate scale-up cold-start latency (image pull / model load) with warm pools / preloaded models?

Spot instances: why are they 60-90% cheaper, how does reclamation work (about 2-minute warning), which workloads fit, and how do online services keep SLA with spot + on-demand hybrid pools?

The cost-optimization playbook: how spot, reserved instances, utilization monitoring and off-peak batch processing stack; where does a full cost-optimization pass start?