Capstone: Stateful On-Call SRE Telemetry Triage & Root-Cause Copilot
Production SRE capstone pairing LangGraph cyclic loop guards, read-only OTel/K8s log diagnostics, interrupt-before rollback Diff UX, and deterministic sandbox incident replay scoring.
Best For
SRE, DevOps, QA, and Backend candidates showcasing fault-tolerant agent state machines, OTel observability, and safe HITL mitigations.
Human-AI Boundary
Telemetry diagnostics run read-only; mitigation commands pause cleanly at signed diff cards requiring on-call engineer approval.
Anti-Patterns to Avoid
- Giving an on-call agent direct unattended `kubectl delete` or DB failover credentials in production
- Feeding raw 50,000-line Pod logs verbatim into context without Map-Reduce log filtering or deduplication
Latency/Cost Profile & Eval Metrics
Initial RCA evidence summary < 12s after alert trigger vs 25min manual log hunting · Bounded max_steps = 8 + compressed Map-Reduce trace worker summaries
- Top-1 Root-Cause Replay Accuracy & Zero Unsafe Mutations: >= 88% exact root-cause service identification on 40 replayed postmortem incidents; 0 loops > 8 steps (Evaluated over frozen historical Prometheus + Jaeger telemetry snapshots)
💰 Business ROI Model & Evidence Level
Reduces median Sev-2 triage MTTR from ~38 minutes to <9 minutes while zero-trust read-only gates guarantee zero unauthorized production mutations.
🏗️ End-to-End Architecture Pipeline
Correlate deploy commits, error clusters, and latency traces via isolated read-only worker subagents.
Abort repeated tool-call cycles deterministically and pause for on-call confirmation before executing mitigation steps.
📊 Benchmark & Alternative Comparison
- Bounded Stateful RCA Agent vs Single-Prompt Log Dump: root-cause recall lifts from 52% to 89% on noisy microservice traces while staying under 12k tokens/incident.
📦 Project Deliverables & Portfolio Artifacts
- LangGraph On-Call RCA Engine with Cycle & Idempotency Guards
- 40-Incident Deterministic Replay Eval Harness & MTTR Benchmark Report
🎤 Interview Defense & System Design Deep-Dive Q&A
- How do you prevent an on-call debugging agent from endlessly polling broken log queries or hitting stale pod IDs during an outage?
- How do you evaluate an RCA agent offline when production alerts are non-stationary?
Bidirectional Taxonomy Links
Internal Tools & Primary Sources
- LangGraph Stateful Reliability & Interrupt Approval Architecture (official_docs, 2026-11-17)