Monitoring & Observability

LLM serving monitoring has three layers: ① system-level metrics — latency (P50/P95/P99), throughput (QPS, tokens/sec), error rate (4xx/5xx and timeouts), GPU utilization, KV-cache hit rate and queue depth; ② token-level metrics — TTFT (time to first token, from request arrival to the first returned token) and TPOT (time per output token), with total latency $T = TTFT + TPOT \times (N-1)$ where $TPOT = \frac{T_{\text{decode}}}{N-1}$; ③ logs & traces — recording requests/responses, parameters and tool-call chains as distributed traces, so one multi-turn conversation is one queryable trace; ④ alerting & RCA — tiered alerts tied to an SLO budget, tracing anomalies back to the input side (very long prompts), serving side (queuing, OOM, autoscaling) or model side (degraded outputs).

day-to-day stability and incident response for production LLM services; interviews ask how to measure and optimize TTFT/TPOT, why P99 matters more than average latency, and how to localize a bottleneck under a QPS spike.

a black box only says “it got slower”, observability says where: a high TTFT points at prefill or queuing, a high TPOT at decode-side bandwidth/compute limits; averages hide long tails that P99 exposes (huge prompts, imbalanced batches); traces stitch multi-replica, multi-service, multi-tool-call chains together, cutting root-cause time from hours to minutes and supplying the data behind rate limiting, autoscaling and SLO decisions.

What do TTFT and TPOT measure and how do they interact? How is $T = TTFT + TPOT \times (N-1)$ used?

Why track P99 latency instead of the average? Where do tail requests come from in LLM serving?

Under a QPS spike, how do traces separate input-side, serving-side and model-side bottlenecks?

How to set SLOs and alerts? How to design thresholds and severity tiers for error rate and latency?

How do serving-side metrics — KV-cache hit rate, GPU utilization, queue depth — drive scaling decisions?