⚡Flashcard Drill➔
HubBS3 · Model & AgentS6 · Security & FinOpsVerified: 2026-11-17

Role Lens · AI Infrastructure & LLM Inference Systems Engineer

Systems engineering role focused on GPU cluster efficiency and low-latency LLM serving—spanning Paged/RadixAttention, TP/PP/ZeRO parallelism, AWQ/FP8 quantization, and autoscaling inference gateways.

Best For

Backend systems engineers, C++ devs, and SREs pivoting into GPU serving, distributed training infra, and high-throughput LLM gateways.

Human-AI Boundary

Infra engineers verify GPU memory headrooms and prefix-routing affinity bounds under synthetic chaos soak tests prior to production rollout.

Anti-Patterns to Avoid

  • Treating distributed AI inference like stateless web microservices without calculating KV Cache HBM footprints or Compute-vs-Memory-Bandwidth rooflines
  • Deploying 70B+ models on naive static batching instead of continuous iteration-level batching and prefix caching

Latency/Cost Profile & Eval Metrics

Target TTFT < 200ms P95 and Memory Utilization > 90% under continuous batching · GPU-hours per 1M output tokens across Prefill/Decode phases

  • Serving Throughput & Hardware Utilization Benchmark: >= 90% KV cache block efficiency; 0 OOM restarts under 2x burst soak tests (Simulate bursty prompt lengths in /tools/science-coding Part H & Part I)

💰 Business ROI Model & Evidence Level

curated_competency_matrix

🎯 Role Competency Weight Matrix (Hub B)

Inference Engines, Paged/RadixAttention & Scheduling40%
Iteration-level scheduling, PD disaggregation, speculative decoding
Distributed Parallelism & Quantization Kernels35%
ZeRO sharding math, Tensor Parallelism, outlier-aware quantization
Distributed Systems Reliability, Quotas & FinOps25%
Prefix-aware LB, Redis token rate-limiting, tail-latency mitigation

🎤 Interview Defense & System Design Deep-Dive Q&A

  1. Why is LLM autoregressive Decode memory-bandwidth bound while Prefill is compute-bound, and how does PD disaggregation resolve interference between them?
  2. Walk through how much GPU HBM is required per rank for a 70B FP16 model under ZeRO-1 vs ZeRO-2 vs ZeRO-3 across 8x 80GB H100s.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔

Bidirectional Taxonomy Links

[HubA] Company Intelligence Hub · Meta AI/MLE, Ranking & Infra Loop[HubA] Company Intelligence Hub · NVIDIA Inference Systems, CUDA/TRT-LLM & Infra Loop[L0 Card] vLLM PagedAttention & Continuous Batching[L0 Card] KV Cache & PagedAttention[L0 Card] Prompt Caching & RadixAttention[L0] vLLM PagedAttention & Continuous Batching Serving Engine[L0] SGLang RadixAttention Automatic Prefix Caching & Constrained Serving[L0] Confidence-Based Cascading Model Router & Pareto FinOps Primitive[L1] Stage 6 · Per-Tenant Token Quota Rate-Limiter & Graceful Circuit-Breaker Pattern[L1] Stage 3 · Structured Output Constrained FSM Decoding, Reasoning Protection & Retry Mode[L2] 10M-Doc Multi-Tenant Enterprise RAG & Compound Agent System Design Blueprint[L3] Verifiable-Citation SEC 10-K Research Copilot (Reference Portfolio Project)[HubA] Company Intelligence Hub · OpenAI AI Stack & Hiring Loop[HubA] Company Intelligence Hub · Anthropic AI Stack & Hiring Loop[HubA] Company Intelligence Hub · Databricks AI Stack & Hiring Loop[HubA] Company Intelligence Hub · Google DeepMind, Gemini & Cloud AI Loop

Internal Tools & Primary Sources