💻 ML EngineerID:
mle-offline-online-eval-gapOffline-Online Eval Gap & A/B Testing
离线线上指标鸿沟与 A/B 验证
🎯Core Definition
The Offline-Online Metric Gap Root-Cause & A/B Validation Architecture addresses the perennial industry dilemma where offline AUC improvements fail to translate into online business gains (revenue, GMV, retention); primary root causes include: 1) Train-Serving Skew & feature leakage; 2) Objective Disconnect (AUC optimizes rank ordering, but ad revenue requires calibrated absolute probabilities, and feeds demand diversity); 3) Position Bias (users click top items regardless of intrinsic quality); 4) Closed Feedback Loops (models only observe feedback on past recommendations); resolution frameworks include double-blind A/B testing, Team Draft Interleaving (100x sensitivity boost), GAUC group evaluations, and Counterfactual Offline Evaluation (Inverse Propensity Scoring).
💡Use Cases
Pre-launch model candidate gatekeeping, diagnosing unexpected A/B metric drops, and aligning ML loss with business KPIs.
⚡Key Problems Solved
Eliminates wasted engineering cycles chasing offline metric gains that fail to move business metrics; provides a battle-tested diagnostic playbook and causal validation framework.
🎯5 High-Frequency Exam Points
1
List the systematic 5-point root-cause diagnostic checklist when offline AUC gains fail to yield online business improvements?
2
Explain why GAUC (Group AUC) must replace Global AUC to eliminate cross-user activity disparities in recommendation evaluation?
3
Explain how Inverse Propensity Scoring (IPS) unbiases offline evaluation against historical logging policy selection bias?
4
Why does Team Draft Interleaving achieve statistical significance with 1% of the sample size and 1/10 the duration of standard A/B tests?
5
How to design multi-week A/B experimentation windows to isolate transient Novelty Effects and capture true long-term retention?
🔗Foundational Prerequisite Cards (Click to Review)
📖 In-depth Guide:📄 mle-system-design-guide →
Updated 2026-08-14🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "Offline-Online Eval Gap & A/B Testing"
Single choice pitfall questions with instant feedback and mistake tracking.