🏗️ System DesignID:
queue-depth-gpu-autoscalingQueue-Depth-Driven GPU Autoscaling
队列深度驱动的 GPU 弹性扩缩容
🎯Core Definition
Queue-Depth-Driven GPU Autoscaling (Kubernetes KEDA, Karpenter, Prometheus Custom Metrics) governs dynamic compute elasticity across heavy generative workloads, harmonizing strict queue wait SLAs with aggressive multi-million-dollar cloud GPU cost optimizations; standard CPU/GPU-utilization HPAs fail because a GPU running a single task reports 100% load, blind to 5,000 pending backlog requests; KEDA continuously monitors message queue backlog depth () and target queue latency () to derive required GPU replicas: ( is single-GPU throughput), leveraging pre-warmed container pools and Spot GPU instances for elastic cluster scaling.
💡Use Cases
AI image/video generation bursts during viral trends, automatic overnight scale-down, and hybrid Spot/On-Demand fleet cost governance.
⚡Key Problems Solved
Statically provisioning peak GPU capacity leaves 70%+ of hardware idle during off-peak hours (burning millions); utilization-based scaling triggers severe latency collapses; queue-depth metric scaling scales directly with user demand.
🎯5 High-Frequency Exam Points
1
Derive the mathematical formula for target replica counts based on queue backlog depth, worker capacity, and target latency?
2
How to accelerate multi-gigabyte GPU container cold starts from minutes to seconds using Stargz lazy-pulling and pre-warmed daemon nodes?
3
Design a hybrid Spot and On-Demand GPU fleet architecture that exploits 70% Spot discounts while hedging against node reclamation?
4
How does Kubernetes Pod Disruption Budget and preStop hook drain active long-running diffusion tasks before terminating GPU instances?
5
Explain the design of stabilization cooldown windows preventing metric flapping and thrashing during transient load dips?
📖 In-depth Guide:📄 multimodal-generative-system-design →
Updated 2026-08-14🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "Queue-Depth-Driven GPU Autoscaling"
Single choice pitfall questions with instant feedback and mistake tracking.