M2-090M2: Classical Machine LearningEvaluation Metrics & Hyperparameter TuningHard
⚡Flashcard Drill➔Mastery:
Why Offline and Online Evaluations Diverge: Causes and Mitigation Strategies
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Divergence stems from distribution drift, proxy metric misalignment, feedback loops, and latency; mitigate via off-policy evaluation, correlation tracking, and canary A/B tests.
📌 Key Takeaways
- •Data distribution divergence: offline data is generated by historical logging policy (counterfactual gap)
- •Proxy metric mismatch: offline NDCG/AUC gains may fail to translate into online CTR/GMV/retention
- •Feedback loops: online models influence user behavior dynamically, creating self-reinforcing loops absent in static logs
📐 Mathematical Derivations
Core Divergence Mechanisms:
① Off-Policy Counterfactual Gap: Offline logs reflect actions chosen by historical logging policy . The expected online reward of candidate policy is , but logs only contain rewards for . Standard offline estimation suffers from policy mismatch. Correct using Inverse Propensity Scoring (IPS): , or Doubly Robust (DR) estimation to reduce propensity variance.
② Feedback Loops & Cannibalization: Recommending high-CTR clickbait increases short-term clicks but drives long-term customer attrition and returns.
③ Train-Serving Skew: Asynchronous feature pipelines, online feature calculation timeouts, and missing realtime signals introduce discrepancies between offline training matrices and online feature store values.
① Off-Policy Counterfactual Gap: Offline logs reflect actions chosen by historical logging policy . The expected online reward of candidate policy is , but logs only contain rewards for . Standard offline estimation suffers from policy mismatch. Correct using Inverse Propensity Scoring (IPS): , or Doubly Robust (DR) estimation to reduce propensity variance.
② Feedback Loops & Cannibalization: Recommending high-CTR clickbait increases short-term clicks but drives long-term customer attrition and returns.
③ Train-Serving Skew: Asynchronous feature pipelines, online feature calculation timeouts, and missing realtime signals introduce discrepancies between offline training matrices and online feature store values.
🏭 Production Trade-offs
Actionable bridges: ① Offline-Online Correlation Tracking: Periodically plot scatter plots of offline metric gain vs online A/B lift across past experiments; discard offline metrics that exhibit weak Spearman rank correlation with online business KPIs. ② Canary & Shadow Deployments: Run shadow traffic (replicating production requests through new model without serving responses) to verify latency, feature alignment, and prediction drift before user-facing A/B tests. ③ Guardrail Metrics: Couple primary optimization targets (clicks) with guardrail metrics (uninstalls, refund rates, latency p99).
⚠️ Common Interview Pitfalls
- ✕Assuming an offline AUC improvement of +0.01 guarantees an online revenue increase in production A/B tests
- ✕Failing to log action propensity scores in production, making unbiased counterfactual evaluation mathematically impossible
🎯 Interviewer Follow-up Inquiries
- ?How does Doubly Robust (DR) estimation combine model-based regression and Inverse Propensity Scoring (IPS) to bound variance?
- ?What causes train-serving skew in real-time streaming feature pipelines, and how can feature stores prevent it?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚
Associated Knowledge Base Guides & Mindmaps
中文速览与面试要点
线上评估与离线评估为什么会不一致?如何缩小差距
离线数据分布、指标代理、反馈回路、延迟效应都与线上不同;需回放评估、反事实评估与在线小流量验证。
核心要点
- 不一致的五个来源:① 数据分布差异 ——离线数据由旧策略产生(行为策略),新策略偏好的样本可能缺乏覆盖;且线上分布随时间漂移(季节、竞品、用户群变化)。② 指标代理差异 ——离线用 Recall@K/NDCG/AUC,线上用 CTR/CVR/留存/GMV;两者可能不单调对应(如离线 NDCG 提升但线上点击下降,因为排序变化影响了多样性)。③ 反馈回路 ——线上系统的输出会改变用户行为与后续数据(用户点击后系统更倾向推荐同类,形成自我强化),离线数据不含这种动态。④ 延迟与长周期效应 ——转化、留存、退货等指标有延迟,短期离线评估无法捕捉;且短期提升可能损害长期(如激进的推荐提高短期点击但损害留存)。⑤ 系统与实现差异 ——训练-服务偏移(特征计算不一致)、延迟约束(离线可用大模型,线上受限)、缓存与降级策略。
- 缩小差距的方法:① 反事实评估(OPE) ——用 IPS/Doubly Robust 等方法从历史日志估计新策略的效果,关键是记录 倾向性(propensity) (每次曝光的概率);优点是无需上线;局限是需要覆盖假设、方差大、对新策略的分布外动作不可靠。② 回放评估(Replay) ——用历史日志模拟新策略(只保留新策略会选择的动作),估计无偏但 有效样本量随策略差异增大而骤降 (新策略选的动作在日志中罕见)。③ 离线上线相关性分析 ——用历史 A/B 结果验证离线指标的代理质量(画散点图看相关性/排序一致性);若相关性弱,说明离线指标不是好代理,需换指标。④ 小流量在线验证 ——用小流量 A/B 或影子流量验证离线结论(影子流量只观测不返回,无用户风险);这是最终的判据。⑤ 短期指标 + 长期护栏 ——同时监控短期(CTR)与长期(留存、退货、满意度)指标,防短期优化损害长期。⑥ 训练-服务一致性 ——用同一套特征计算逻辑、定期做离线回放对比线上日志,检测 skew。⑦ 因果推断方法 ——对无法做 A/B 的场景(如全量改版),用 DID/合成控制/断点回归估计效应。
- 用 OPE(IPS/DR)做离线策略评估
- 离线上线做相关性分析,校准代理指标
- 用小流量 A/B 验证离线结论
常见陷阱
- 认为离线指标提升线上必然提升
- 不做离线上线相关性分析就直接上线