M2-090M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningHard
⚡Flashcard Drill➔Mastery:
Why Offline and Online Evaluations Diverge: Causes and Mitigation Strategies
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Divergence stems from distribution drift, proxy metric misalignment, feedback loops, and latency; mitigate via off-policy evaluation, correlation tracking, and canary A/B tests.
📌 Key Takeaways
- •Data distribution divergence: offline data is generated by historical logging policy (counterfactual gap)
- •Proxy metric mismatch: offline NDCG/AUC gains may fail to translate into online CTR/GMV/retention
- •Feedback loops: online models influence user behavior dynamically, creating self-reinforcing loops absent in static logs
📐 Mathematical Derivations
Core Divergence Mechanisms:
① Off-Policy Counterfactual Gap: Offline logs reflect actions chosen by historical logging policy . The expected online reward of candidate policy is , but logs only contain rewards for . Standard offline estimation suffers from policy mismatch. Correct using Inverse Propensity Scoring (IPS): , or Doubly Robust (DR) estimation to reduce propensity variance.
② Feedback Loops & Cannibalization: Recommending high-CTR clickbait increases short-term clicks but drives long-term customer attrition and returns.
③ Train-Serving Skew: Asynchronous feature pipelines, online feature calculation timeouts, and missing realtime signals introduce discrepancies between offline training matrices and online feature store values.
① Off-Policy Counterfactual Gap: Offline logs reflect actions chosen by historical logging policy . The expected online reward of candidate policy is , but logs only contain rewards for . Standard offline estimation suffers from policy mismatch. Correct using Inverse Propensity Scoring (IPS): , or Doubly Robust (DR) estimation to reduce propensity variance.
② Feedback Loops & Cannibalization: Recommending high-CTR clickbait increases short-term clicks but drives long-term customer attrition and returns.
③ Train-Serving Skew: Asynchronous feature pipelines, online feature calculation timeouts, and missing realtime signals introduce discrepancies between offline training matrices and online feature store values.
🏭 Production Trade-offs
Actionable bridges: ① Offline-Online Correlation Tracking: Periodically plot scatter plots of offline metric gain vs online A/B lift across past experiments; discard offline metrics that exhibit weak Spearman rank correlation with online business KPIs. ② Canary & Shadow Deployments: Run shadow traffic (replicating production requests through new model without serving responses) to verify latency, feature alignment, and prediction drift before user-facing A/B tests. ③ Guardrail Metrics: Couple primary optimization targets (clicks) with guardrail metrics (uninstalls, refund rates, latency p99).
⚠️ Common Interview Pitfalls
- ✕Assuming an offline AUC improvement of +0.01 guarantees an online revenue increase in production A/B tests
- ✕Failing to log action propensity scores in production, making unbiased counterfactual evaluation mathematically impossible
🎯 Interviewer Follow-up Inquiries
- ?How does Doubly Robust (DR) estimation combine model-based regression and Inverse Propensity Scoring (IPS) to bound variance?
- ?What causes train-serving skew in real-time streaming feature pipelines, and how can feature stores prevent it?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚