← Back to ML Engineer Mind Map
💻 ML EngineerID: mle-offline-online-eval-gap

Offline-Online Eval Gap & A/B Testing

离线线上指标鸿沟与 A/B 验证
🎯Core Definition
The Offline-Online Metric Gap Root-Cause & A/B Validation Architecture addresses the perennial industry dilemma where offline AUC improvements fail to translate into online business gains (revenue, GMV, retention); primary root causes include: 1) Train-Serving Skew & feature leakage; 2) Objective Disconnect (AUC optimizes rank ordering, but ad revenue requires calibrated absolute probabilities, and feeds demand diversity); 3) Position Bias (users click top items regardless of intrinsic quality); 4) Closed Feedback Loops (models only observe feedback on past recommendations); resolution frameworks include double-blind A/B testing, Team Draft Interleaving (100x sensitivity boost), GAUC group evaluations, and Counterfactual Offline Evaluation (Inverse Propensity Scoring).
💡Use Cases
Pre-launch model candidate gatekeeping, diagnosing unexpected A/B metric drops, and aligning ML loss with business KPIs.
⚡Key Problems Solved
Eliminates wasted engineering cycles chasing offline metric gains that fail to move business metrics; provides a battle-tested diagnostic playbook and causal validation framework.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of Offline-Online Eval Gap & A/B Testing.
Use Cases
In production engineering, what are the primary use cases for Offline-Online Eval Gap & A/B Testing and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
List the systematic 5-point root-cause diagnostic checklist when offline AUC gains fail to yield online business improvements?
2
Explain why GAUC (Group AUC) must replace Global AUC to eliminate cross-user activity disparities in recommendation evaluation?
3
Explain how Inverse Propensity Scoring (IPS) unbiases offline evaluation against historical logging policy selection bias?
4
Why does Team Draft Interleaving achieve statistical significance with 1% of the sample size and 1/10 the duration of standard A/B tests?
5
How to design multi-week A/B experimentation windows to isolate transient Novelty Effects and capture true long-term retention?
Updated 2026-08-14
🎯
Verify Mastery: Practice Exam Questions for "Offline-Online Eval Gap & A/B Testing"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardReal-Time Risk & Delayed Feedback

🔗 More ML Engineer Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/MLE/card/mle-offline-online-eval-gap
中文速览与面试要点

离线线上指标鸿沟与 A/B 验证 (Offline-Online Eval Gap & A/B Testing)

定义: 离线与线上指标鸿沟排查与 A/B 实验验证体系 (Offline-Online Metric Gap Root-Cause & A/B Validation Architecture) 是解决工业级机器学习模型“离线 AUC 显著提升但线上真实商业 KPI(转化率、GMV、留存率)不升反降”这一最令人头疼的核心工程难题的方法论;离线线上鸿沟的核心根因:1) 离在线特征不一致 (Train-Serving Skew / 特征穿越);2) 离线指标与商业目标不一致(如 AUC 只关心相对序,但广告计费需要精准的绝对值,推荐更依赖多目标与多样性);3) 位置偏置与曝光未点击偏差 (Position Bias);4) 数据反馈循环与幸存者偏差 (Feedback Loop);解决体系:严格实施双盲 A/B 测试、交替实验 (Interleaving 灵敏度提升 100 倍)、GAUC 分组评估、保序回归概率校准与因果推断反事实评估 (Counterfactual Offline Evaluation)。 使用场景: 推荐与搜索模型上线前的准入评审、A/B 实验效果不符合离线预期的排障分析、模型商业化决策。 解决的问题: 算法团队花费数月优化离线指标却无法转化为线上真金白银的商业增长;本体系提供了系统化的鸿沟排查清单与严密的在线因果验证标准。

核心要点

  • 系统化列出导致“离线 AUC 涨但线上业务指标跌”的五大根因排查清单与针对性解决方案?
  • 为什么在推荐系统中必须使用 GAUC (Group AUC: 按用户分组加权 AUC) 替代全局 AUC 以消除跨用户活跃度偏置?
  • 反事实离线评估 (Counterfactual Offline Evaluation) 中的逆倾向得分加权 (Inverse Propensity Scoring, IPS: R_ IPS = 1N Y_i P_ new(A_i|X_i)P_ old(A_i|X_i)) 原理?
  • Interleaving (交替实验) 相比标准 A/B 测试在快速筛选淘汰弱模型中的样本量与时间效率优势?
  • 新颖性效应 (Novelty Effect) 与慢炖效应 (Slow Cooker Effect) 在长期 A/B 实验中的观测周期设定与指标平稳性判断?