← Back to Registry Hub/Site Search/L1 · stage5_eval_llmops
⚡Flashcard Drill➔
L1S5 · Eval & LLMOpsS6 · Security & FinOpsVerified: 2026-11-17

Stage 5 · Canary Prompt & Model-Upgrade Shadow Traffic A/B Gate Pattern

Stage 5 progressive delivery pattern pairing deterministic offline shadow replay with live 1–5% traffic splits—evaluating candidate model/prompt bundles on paired significance tests, P95 TTFT, schema faults, and blended unit cost before full rollout.

Best For

High-traffic services rolling out foundation model migrations, fine-tuned weights, retrieval re-tunings, or revised agent prompts.

Human-AI Boundary

Automated monitors hard-rollback on schema errors, safety violations, or P95 breaches; humans approve promotions on marginal quality-cost trade-offs.

Anti-Patterns to Avoid

  • Flipping 100% of production traffic from Model A to Model B overnight purely based on vendor leaderboard scores
  • Computing A/B metrics over different unaligned query cohorts or too few samples (<200 paired requests)

Latency/Cost Profile & Eval Metrics

Shadow path runs async off critical user path; live Canary isolates impact to <= 5% user sessions · 2x inference cost on 5% sampled shadow traffic window

  • Paired Judge Win-Rate & Cost-Latency Guardrail: Delta win-rate >= -1.0% (95% CI) AND P95 TTFT / cost-per-task within +10% SLA (Auto-rollback canary weights if schema error rate > 0.2%)

💰 Business ROI Model & Evidence Level

verified_benchmark

Catches silent quality drops, schema drift, and tail latency spikes on <3% budget overhead—preventing high-visibility full-fleet regressions.

🏗️ End-to-End Architecture Pipeline

Step 1: Immutable Bundle Versioning & Sticky Cohort Split

Freeze model, prompt, temperature, and tool manifests into immutable `bundle_sha` tags routed deterministically per session.

Step 2: Online Telemetry Paired Scoring & Auto-Rollback

Monitor parse failures, HITL accept ratios, latency, and paired judge diffs; auto-revert traffic weight on invariant violation.

📊 Benchmark & Alternative Comparison

  • Sticky Session Canary vs Unpinned Random Request A/B: eliminates multi-turn state corruption while yielding cleaner paired significance intervals.

🚨 Production Failure Modes & Signatures

  • Vendor silent checkpoint update breaking deterministic prompt adherence even when `model_id` string stayed identical
  • Metric peeking every 30 minutes causing high false-positive rollbacks before statistical power stabilizes

📦 Project Deliverables & Portfolio Artifacts

  • Immutable Prompt+Model Release Bundle Manifest & Sticky Router Config
  • Paired Significance & Auto-Rollback Alerting Dashboard Spec
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔

Bidirectional Taxonomy Links

Internal Tools & Primary Sources