🖥️ AI InfrastructureID: monitoring-observability

Monitoring & Observability

Q: What do TTFT and TPOT measure and how do they interact? How is $T = TTFT + TPOT \times (N-1)$ used?

【Monitoring & Observability 核心考点 1】What do TTFT and TPOT measure and how do they interact? How is $T = TTFT + TPOT \times (N-1)$ used?。技术解析：Definition: LLM serving monitoring has three layers: ① system-level metrics — latency (P50/P95/P99), throughput (QPS, tokens/sec), error rate (4xx/5xx and timeouts), GPU utilization, KV-cache hit rate and queue depth; ② token-level metrics — TTFT (time to first token, from reques... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/monitoring-observability

Q: Why track P99 latency instead of the average? Where do tail requests come from in LLM serving?

【Monitoring & Observability 核心考点 2】Why track P99 latency instead of the average? Where do tail requests come from in LLM serving?。技术解析：Definition: LLM serving monitoring has three layers: ① system-level metrics — latency (P50/P95/P99), throughput (QPS, tokens/sec), error rate (4xx/5xx and timeouts), GPU utilization, KV-cache hit rate and queue depth; ② token-level metrics — TTFT (time to first token, from reques... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/monitoring-observability

Q: Under a QPS spike, how do traces separate input-side, serving-side and model-side bottlenecks?

【Monitoring & Observability 核心考点 3】Under a QPS spike, how do traces separate input-side, serving-side and model-side bottlenecks?。技术解析：Definition: LLM serving monitoring has three layers: ① system-level metrics — latency (P50/P95/P99), throughput (QPS, tokens/sec), error rate (4xx/5xx and timeouts), GPU utilization, KV-cache hit rate and queue depth; ② token-level metrics — TTFT (time to first token, from reques... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/monitoring-observability

Q: How to set SLOs and alerts? How to design thresholds and severity tiers for error rate and latency?

【Monitoring & Observability 核心考点 4】How to set SLOs and alerts? How to design thresholds and severity tiers for error rate and latency?。技术解析：Definition: LLM serving monitoring has three layers: ① system-level metrics — latency (P50/P95/P99), throughput (QPS, tokens/sec), error rate (4xx/5xx and timeouts), GPU utilization, KV-cache hit rate and queue depth; ② token-level metrics — TTFT (time to first token, from reques... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/monitoring-observability

Q: How do serving-side metrics — KV-cache hit rate, GPU utilization, queue depth — drive scaling decisions?

【Monitoring & Observability 核心考点 5】How do serving-side metrics — KV-cache hit rate, GPU utilization, queue depth — drive scaling decisions?。技术解析：Definition: LLM serving monitoring has three layers: ① system-level metrics — latency (P50/P95/P99), throughput (QPS, tokens/sec), error rate (4xx/5xx and timeouts), GPU utilization, KV-cache hit rate and queue depth; ② token-level metrics — TTFT (time to first token, from reques... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/monitoring-observability

监控与可观测性

🎯Core Definition

LLM serving monitoring has three layers: ① system-level metrics — latency (P50/P95/P99), throughput (QPS, tokens/sec), error rate (4xx/5xx and timeouts), GPU utilization, KV-cache hit rate and queue depth; ② token-level metrics — TTFT (time to first token, from request arrival to the first returned token) and TPOT (time per output token), with total latency

T = TTFT + TPOT \times (N-1)

where

TPOT = \frac{T_{\text{decode}}}{N-1}

; ③ logs & traces — recording requests/responses, parameters and tool-call chains as distributed traces, so one multi-turn conversation is one queryable trace; ④ alerting & RCA — tiered alerts tied to an SLO budget, tracing anomalies back to the input side (very long prompts), serving side (queuing, OOM, autoscaling) or model side (degraded outputs).

💡Use Cases

day-to-day stability and incident response for production LLM services; interviews ask how to measure and optimize TTFT/TPOT, why P99 matters more than average latency, and how to localize a bottleneck under a QPS spike.

⚡Key Problems Solved

a black box only says “it got slower”, observability says where: a high TTFT points at prefill or queuing, a high TPOT at decode-side bandwidth/compute limits; averages hide long tails that P99 exposes (huge prompts, imbalanced batches); traces stitch multi-replica, multi-service, multi-tool-call chains together, cutting root-cause time from hours to minutes and supplying the data behind rate limiting, autoscaling and SLO decisions.

🎯5 High-Frequency Exam Points

What do TTFT and TPOT measure and how do they interact? How is

T = TTFT + TPOT \times (N-1)

used?

Why track P99 latency instead of the average? Where do tail requests come from in LLM serving?

Under a QPS spike, how do traces separate input-side, serving-side and model-side bottlenecks?

How to set SLOs and alerts? How to design thresholds and severity tiers for error rate and latency?

How do serving-side metrics — KV-cache hit rate, GPU utilization, queue depth — drive scaling decisions?

📖 In-depth Guide:📄 high-concurrency-ai-system →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "Monitoring & Observability"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardModel Testing & Gates Next Card →Security & Privacy

🔗 More AI Infrastructure Knowledge Cards

Activation Memory Estimation Agent Runtime (cross-module)Autoscaling & Cost Checkpointing & Recovery