⚡Flashcard Drill➔
L2S2 · Data & RetrievalS3 · Model & AgentS5 · Eval & LLMOpsVerified: 2026-11-17

L2 Scenario · On-Call SRE Telemetry Triage & Root-Cause Diagnostics Agent

Production incident-response workflow deduplicating cascading microservice alerts, correlating recent deploys, error spikes, and telemetry spans via quarantined read-only subagents inside 15 seconds.

Best For

SRE and Cloud Platform teams compressing alert storm fatigue and initial incident MTTR.

Human-AI Boundary

Agent operates strictly read-only over telemetry APIs; mitigation rollbacks stay gated behind on-call engineer approval.

Anti-Patterns to Avoid

  • Granting write/restart Kubernetes privileges automatically to diagnostic LLM loops during cascading partial network partitions
  • Feeding unsummarized megabyte JSON stacktraces straight into coordinator prompts

Latency/Cost Profile & Eval Metrics

First evidence dossier ready in <= 15s post PagerDuty webhook · Fixed max 8 diagnostic steps + parallel Subagent log compaction

  • Top-2 Candidate Service Identification Recall: >= 90% accuracy across replayed historical SEV postmortem timelines (Measure against root-cause service documented in human postmortems)

💰 Business ROI Model & Evidence Level

production_practice

Turns 30 minutes of fragmented dashboard log-hunting into a 15-second unified evidence timeline—cutting customer-facing downtime duration significantly.

🏗️ End-to-End Architecture Pipeline

Step 1: Service-Graph Deduplication & Deploy Diff Correlation

Group cascading alerts using dependency topology and fetch deploy commits/config flips over the preceding 2-hour window.

Step 2: Bounded Worker Telemetry Triage & Cited Briefing Card

Dispatch bounded workers per suspect service to produce timestamped telemetry citations and draft rollback commands.

🛡️ Compliance & Governance Hard Constraints

  • Read-only scoped IAM service account for diagnostic telemetry connectors
  • Mask auth tokens or customer PII accidentally logged inside stacktraces via Presidio scrubber

📦 Project Deliverables & Portfolio Artifacts

  • Read-Only Telemetry Connector Suite (Prometheus + Jaeger + Git Diff)
  • Historical Postmortem Incident Replay Evaluation Benchmark
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔

Bidirectional Taxonomy Links

Internal Tools & Primary Sources