M2-115M2: Classical Machine LearningClass Imbalance LearningHard
⚡Flashcard Drill➔Mastery:

Post-Deployment Monitoring and Evaluation Under Severe Class Imbalance

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-116 →
📐 Mathematical Definition
monitor: πt, PR-AUCt, Recall@fixed Precisiont\text{monitor}:\ \pi_t,\ \text{PR-AUC}_t,\ \text{Recall@fixed Precision}_t
⚡ One-Sentence Core Answer
Monitor positive prior drift, segmented precision/recall, calibration quality, and business metrics; severe imbalance makes global aggregated metrics deceptive.

📌 Key Takeaways

  • •
    Positive prior drift: shifts in π=P(y=1)\pi = P(y=1) invalidate fixed decision thresholds τ\tau
  • •
    Segmented monitoring: track Recall and Precision across critical business slices to prevent localized failure
  • •
    Dynamic threshold adjustment: adapt thresholds via quantile-based targeting or rolling cost optimization

📐 Mathematical Derivations

Impact of Prior Drift on Thresholds: Recall that Bayes optimal decision threshold satisfies τ∗=cFPcFP+cFN\tau^* = \frac{c_{\text{FP}}}{c_{\text{FP}} + c_{\text{FN}}}. The posterior odds relate to likelihood ratio via Bayes rule: P(Y=1∣X)P(Y=0∣X)=P(X∣Y=1)P(X∣Y=0)⋅π1−π\frac{P(Y=1|X)}{P(Y=0|X)} = \frac{P(X|Y=1)}{P(X|Y=0)} \cdot \frac{\pi}{1 - \pi}.
If the true positive class prevalence π\pi drops from 5%5\% to 1%1\% online, the posterior probability of positive predictions collapses: P(Y=1∣X)P(Y=1|X) drops proportionally. A model operating with a fixed threshold τ=0.5\tau = 0.5 will suffer an immediate collapse in predicted positive volume and operational recall. Tracking π\pi and recalibrating thresholds dynamically via score quantiles is mandatory.

🏭 Production Trade-offs

Production Monitoring Dashboard Essentials: ① Rolling PR-AUC and Brier score computed on verified delayed labels; ② Distribution of prediction scores (Population Stability Index, PSI); ③ Prediction rate (fraction of positive predictions per hour); ④ Subgroup performance (e.g., breakdown by user tier or geographic region).
⚠️ Common Interview Pitfalls
  • ✕
    Monitoring overall Accuracy or ROC-AUC, which remain completely flat at 0.99 even when positive recall collapses to zero
  • ✕
    Hardcoding fixed probability thresholds without alerting on positive prevalence π\pi shifts
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Population Stability Index (PSI) detect covariate shift in imbalanced prediction scores?
  • ?
    How can a business dynamically adapt its classification threshold when fraud rates double during holiday sales?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-114: Imbalanced Ensembling: EasyEnsemble and BalanceCascade Explained📋 Back to BankNext →M2-116: MICE (Multiple Imputation by Chained Equations): Iterative Process and Key Details