9.4
9.4 AI Observability, Quality & Incident Response
OpenTelemetry GenAI semantic conventions, end-to-end span distributed tracing, embedding drift (MMD/Wasserstein), online quality probes, and error budget burn rate alerts.
End-to-End Distributed Tracing & OpenTelemetry GenAI
Standardized OpenTelemetry GenAI semantic conventions, nested agent/tool/RAG span cascaded tracing, TTFT and TPOT latency breakdowns, real-time payload inspection, and interactive visualization.
🏢 Companies
Langfuse、Arize AI (Phoenix)、Datadog、Dynatrace、Braintrust
🛠️ Tech Stack
Distributed TracingOpenTelemetry GenAISpan CascadeTTFT / TPOTLangfuseArize PhoenixDatadog
💼 Roles & Salary
Observability Engineer、LLMOps Engineer、AI Platform Engineer
💰 $195K - $410K / year (Distributed Tracing & GenAI APM) | ¥520K - ¥1.22M / year
📚 Prerequisites: OpenTelemetry GenAI Semantic Conventions • Distributed Trace Collectors & APM Dashboards • TTFT / TPOT Latency Decomposition Analysis • Unstructured Payload Sanitization & Sampling Policies
Quality Drift Monitoring, Embedding Drift & Incident Response
High-dimensional embedding drift detection via MMD & PSI, online quality and feedback probes, SLO error budget burn rate alerting, and cross-layer security incident response runbooks.
🏢 Companies
Arize AI、Langfuse、WhyLabs、Datadog、Dynatrace
🛠️ Tech Stack
Embedding DriftMMDPSIQuality ProbesError Budget Burn RateIncident ResponseWhyLabsArize AI
💼 Roles & Salary
Observability Engineer、SRE、Evaluation Engineer
💰 $190K - $400K / year (AI Quality Drift & SRE Reliability) | ¥500K - ¥1.18M / year
📚 Prerequisites: High-dimensional Drift Math (MMD / Wasserstein / PSI) • Online Quality Scoring Probes & Feedback Flywheels • SRE Multi-Window Error Budget Burn Rate Alerting • AI Security Incident Response & Graceful Rollback Runbooks