L2 Scenario · On-Call SRE Telemetry Triage & Root-Cause Diagnostics Agent
Production incident-response workflow deduplicating cascading microservice alerts, correlating recent deploys, error spikes, and telemetry spans via quarantined read-only subagents inside 15 seconds.
Best For
SRE and Cloud Platform teams compressing alert storm fatigue and initial incident MTTR.
Human-AI Boundary
Agent operates strictly read-only over telemetry APIs; mitigation rollbacks stay gated behind on-call engineer approval.
Anti-Patterns to Avoid
- Granting write/restart Kubernetes privileges automatically to diagnostic LLM loops during cascading partial network partitions
- Feeding unsummarized megabyte JSON stacktraces straight into coordinator prompts
Latency/Cost Profile & Eval Metrics
First evidence dossier ready in <= 15s post PagerDuty webhook · Fixed max 8 diagnostic steps + parallel Subagent log compaction
- Top-2 Candidate Service Identification Recall: >= 90% accuracy across replayed historical SEV postmortem timelines (Measure against root-cause service documented in human postmortems)
💰 Business ROI Model & Evidence Level
production_practiceTurns 30 minutes of fragmented dashboard log-hunting into a 15-second unified evidence timeline—cutting customer-facing downtime duration significantly.
🏗️ End-to-End Architecture Pipeline
Group cascading alerts using dependency topology and fetch deploy commits/config flips over the preceding 2-hour window.
Dispatch bounded workers per suspect service to produce timestamped telemetry citations and draft rollback commands.
🛡️ Compliance & Governance Hard Constraints
- Read-only scoped IAM service account for diagnostic telemetry connectors
- Mask auth tokens or customer PII accidentally logged inside stacktraces via Presidio scrubber
📦 Project Deliverables & Portfolio Artifacts
- Read-Only Telemetry Connector Suite (Prometheus + Jaeger + Git Diff)
- Historical Postmortem Incident Replay Evaluation Benchmark
Bidirectional Taxonomy Links
Internal Tools & Primary Sources
- OpenTelemetry GenAI Observability & Distributed Tracing (official_docs, 2026-11-17)