Cluster Scheduling Ray/K8s

Cluster scheduling answers 'which task goes to which GPU and how to avoid wasting resources'. The K8s default scheduler (kube-scheduler) is a two-phase pipeline: ① Filtering: drop nodes that violate hard constraints — resource requests (CPU/memory/extended resources like GPUs), NodeSelector and node affinity, taints/tolerations, port and volume conflicts; ② Scoring: rank the feasible nodes with policies (least-requested spreads load, most-requested packs tightly, custom priorities) and bind the Pod to the best one. GPU scheduling: the device plugin advertises GPUs as an indivisible extended resource $nvidia.com/gpu$; the scheduler allocates whole cards, never fractional ones — memory and SMs cannot be safely split, so allocation is by card, not by MB; for finer granularity use MIG (Multi-Instance GPU, hardware slicing on A100/H100, up to 7 instances per card, e.g. $1g.5gb$, $7g.80gb$), where each instance owns fixed SMs and memory partitions with hardware isolation and no performance jitter — unlike vGPU time-slicing. Ray is an application-level scheduler built on top of K8s: Driver (entry script) → Head (global control plane GCS + Raylet) → one Raylet per node (local scheduler + Object Store, an in-memory shared store giving zero-copy data transfer between tasks on the same node) → Workers (execute Tasks/Actors). Autoscaling: the Ray Autoscaler adds/removes nodes based on the pending-task queue and reclaims idle ones; K8s scales via HPA or custom metrics (QPS, queue length, GPU utilization); on scale-down or spot preemption, in-flight work is drained gracefully, backed by replicas and checkpoint-resume.

interview favorites are how the K8s scheduler picks a node, why GPUs cannot be scheduled fractionally and what MIG is, drawing the Ray architecture, and the layering of Ray on K8s.

multi-tenant clusters mix priorities, heterogeneous GPUs and memory fragmentation, making manual placement wasteful — filter-then-score guarantees feasible and optimal placement; whole-card allocation + MIG eliminates fragmentation and stops small tasks from idling a full card (up to 7 MIG instances per A100); Ray's local scheduling and object store keep data off the network, and autoscaling reclaims idle GPUs, raising cluster GPU utilization significantly.

Describe the K8s scheduler's two phases: the hard constraints checked in filtering (resources, NodeSelector, affinity, taints/tolerations, ports/volumes), the scoring policies (least-requested vs most-requested) and their trade-offs, and why filtering must come first.

Why GPU is an extended resource allocated in whole cards (memory and SMs cannot be safely partitioned), how the device plugin advertises $nvidia.com/gpu$; can a 30 GB job fit a card with only 20 GB free, and why.

What is MIG: hardware slicing of A100/H100 (up to 7 instances, e.g. $1g.5gb$) with dedicated SM and memory partitions per instance; the key difference from vGPU time-slicing (isolation, no jitter), and when to choose it.

Sketch the Ray architecture: Driver → Head (GCS global control plane + Raylet) → per-node Raylet (local scheduler + Object Store) → Workers; why the object store gives zero-copy sharing between same-node tasks while only cross-node traffic hits the network.

Autoscaling: the Ray Autoscaler adds/removes nodes by pending task counts, K8s via HPA/custom metrics; how in-flight work is handled gracefully on scale-down and spot preemption (drain, replicas, checkpoint-resume).