M2-090M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningHard
⚡Flashcard Drill➔Mastery:

Why Offline and Online Evaluations Diverge: Causes and Mitigation Strategies

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-091 →
📐 Mathematical Definition
offline proxy ≠ online metric\text{offline}\ \text{proxy}\ \ne\ \text{online}\ \text{metric}
⚡ One-Sentence Core Answer
Divergence stems from distribution drift, proxy metric misalignment, feedback loops, and latency; mitigate via off-policy evaluation, correlation tracking, and canary A/B tests.

📌 Key Takeaways

  • •
    Data distribution divergence: offline data is generated by historical logging policy (counterfactual gap)
  • •
    Proxy metric mismatch: offline NDCG/AUC gains may fail to translate into online CTR/GMV/retention
  • •
    Feedback loops: online models influence user behavior dynamically, creating self-reinforcing loops absent in static logs

📐 Mathematical Derivations

Core Divergence Mechanisms:
① Off-Policy Counterfactual Gap: Offline logs reflect actions chosen by historical logging policy π0\pi_0. The expected online reward of candidate policy π\pi is Ea∼π[R(s,a)]\mathbb{E}_{a \sim \pi}[R(s, a)], but logs only contain rewards for a∼π0a \sim \pi_0. Standard offline estimation suffers from policy mismatch. Correct using Inverse Propensity Scoring (IPS): V^IPS(π)=1N∑i=1Nπ(ai∣si)π0(ai∣si)ri\hat{V}_{\text{IPS}}(\pi) = \frac{1}{N} \sum_{i=1}^N \frac{\pi(a_i | s_i)}{\pi_0(a_i | s_i)} r_i, or Doubly Robust (DR) estimation to reduce propensity variance.
② Feedback Loops & Cannibalization: Recommending high-CTR clickbait increases short-term clicks but drives long-term customer attrition and returns.
③ Train-Serving Skew: Asynchronous feature pipelines, online feature calculation timeouts, and missing realtime signals introduce discrepancies between offline training matrices and online feature store values.

🏭 Production Trade-offs

Actionable bridges: ① Offline-Online Correlation Tracking: Periodically plot scatter plots of offline metric gain vs online A/B lift across past experiments; discard offline metrics that exhibit weak Spearman rank correlation with online business KPIs. ② Canary & Shadow Deployments: Run shadow traffic (replicating production requests through new model without serving responses) to verify latency, feature alignment, and prediction drift before user-facing A/B tests. ③ Guardrail Metrics: Couple primary optimization targets (clicks) with guardrail metrics (uninstalls, refund rates, latency p99).
⚠️ Common Interview Pitfalls
  • ✕
    Assuming an offline AUC improvement of +0.01 guarantees an online revenue increase in production A/B tests
  • ✕
    Failing to log action propensity scores π0(a∣s)\pi_0(a|s) in production, making unbiased counterfactual evaluation mathematically impossible
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Doubly Robust (DR) estimation combine model-based regression and Inverse Propensity Scoring (IPS) to bound variance?
  • ?
    What causes train-serving skew in real-time streaming feature pipelines, and how can feature stores prevent it?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-089: Handling Randomness and Seed Variance in Hyperparameter Tuning📋 Back to BankNext →M2-091: Evaluating Feature Importance in Neural Networks vs Tree Models