HubBS3 · Model & AgentS6 · Security & FinOpsVerified: 2026-11-17
Role Lens · AI Infrastructure & LLM Inference Systems Engineer
Systems engineering role focused on GPU cluster efficiency and low-latency LLM serving—spanning Paged/RadixAttention, TP/PP/ZeRO parallelism, AWQ/FP8 quantization, and autoscaling inference gateways.
Best For
Backend systems engineers, C++ devs, and SREs pivoting into GPU serving, distributed training infra, and high-throughput LLM gateways.
Human-AI Boundary
Infra engineers verify GPU memory headrooms and prefix-routing affinity bounds under synthetic chaos soak tests prior to production rollout.
Anti-Patterns to Avoid
- Treating distributed AI inference like stateless web microservices without calculating KV Cache HBM footprints or Compute-vs-Memory-Bandwidth rooflines
- Deploying 70B+ models on naive static batching instead of continuous iteration-level batching and prefix caching
Latency/Cost Profile & Eval Metrics
Target TTFT < 200ms P95 and Memory Utilization > 90% under continuous batching · GPU-hours per 1M output tokens across Prefill/Decode phases
- Serving Throughput & Hardware Utilization Benchmark: >= 90% KV cache block efficiency; 0 OOM restarts under 2x burst soak tests (Simulate bursty prompt lengths in /tools/science-coding Part H & Part I)
💰 Business ROI Model & Evidence Level
curated_competency_matrix🎯 Role Competency Weight Matrix (Hub B)
Inference Engines, Paged/RadixAttention & Scheduling40%
Iteration-level scheduling, PD disaggregation, speculative decoding
Distributed Parallelism & Quantization Kernels35%
ZeRO sharding math, Tensor Parallelism, outlier-aware quantization
Distributed Systems Reliability, Quotas & FinOps25%
Prefix-aware LB, Redis token rate-limiting, tail-latency mitigation
🎤 Interview Defense & System Design Deep-Dive Q&A
- Why is LLM autoregressive Decode memory-bandwidth bound while Prefill is compute-bound, and how does PD disaggregation resolve interference between them?
- Walk through how much GPU HBM is required per rank for a 70B FP16 model under ZeRO-1 vs ZeRO-2 vs ZeRO-3 across 8x 80GB H100s.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
Bidirectional Taxonomy Links
[HubA] Company Intelligence Hub · Meta AI/MLE, Ranking & Infra Loop[HubA] Company Intelligence Hub · NVIDIA Inference Systems, CUDA/TRT-LLM & Infra Loop[L0 Card] vLLM PagedAttention & Continuous Batching[L0 Card] KV Cache & PagedAttention[L0 Card] Prompt Caching & RadixAttention[L0] vLLM PagedAttention & Continuous Batching Serving Engine[L0] SGLang RadixAttention Automatic Prefix Caching & Constrained Serving[L0] Confidence-Based Cascading Model Router & Pareto FinOps Primitive[L1] Stage 6 · Per-Tenant Token Quota Rate-Limiter & Graceful Circuit-Breaker Pattern[L1] Stage 3 · Structured Output Constrained FSM Decoding, Reasoning Protection & Retry Mode[L2] 10M-Doc Multi-Tenant Enterprise RAG & Compound Agent System Design Blueprint[L3] Verifiable-Citation SEC 10-K Research Copilot (Reference Portfolio Project)[HubA] Company Intelligence Hub · OpenAI AI Stack & Hiring Loop[HubA] Company Intelligence Hub · Anthropic AI Stack & Hiring Loop[HubA] Company Intelligence Hub · Databricks AI Stack & Hiring Loop[HubA] Company Intelligence Hub · Google DeepMind, Gemini & Cloud AI Loop
Internal Tools & Primary Sources
- TalentMe AI Infra Curriculum Matrix (industry_report, 2026-11-17)