M7-001Medium评估体系 · Evaluation stack🔥 大厂高频出现 14 次
离线指标(AUC/NDCG)提升了,但线上指标没动,可能是什么原因?
Offline metrics (AUC, NDCG) improved but online metrics did not move. What could explain it?
💡 考场高分一句话答案:
五个常见原因:离线评估的样本有偏(position bias)、指标与业务目标不一致、训练-服务偏斜、线上有探索与冷启动、反馈回路改变。
Five usual causes: biased offline samples (position bias), metric-objective mismatch, train-serving skew, the absence of exploration online, and feedback loops that changed the data distribution.
📌 核心考察要点 (Key Points)
- Position bias:离线用已曝光样本评估,模型学到的是“位置好所以被点”,线上换了排序就失效
- 指标错配:NDCG 优化整体排序,业务只看 top-3;或 AUC 高但 top-k 差
- 训练-服务偏斜:特征在线上缺失/延迟/口径不同
- 探索缺失:线上要探索新内容,离线评估的全是 exploitation
- Position bias: offline evaluation uses logged exposures, so the model learns 'high position gets clicked' and fails once ranking changes
- Metric mismatch: NDCG optimises the whole ranking while the business cares about the top three
- Train-serving skew: features missing, delayed or defined differently in production
- No exploration: offline evaluation is pure exploitation, while production must explore
⚖️ 业务权衡与取舍 (Trade-offs & Calibration)
最隐蔽的是反馈回路:模型推什么用户就点什么,离线数据因此越来越偏向模型已有的偏好,形成自我强化。这会同时让离线指标虚高、线上多样性下降。解法是保留随机探索流量,并用 inverse propensity scoring 校正离线评估。
⚠️ 常见高频面试陷阱与避坑指南
- 认为“离线涨线上不涨”是运气问题
- 直接在离线数据上调参直到线上有效(过拟合到线上)
🔍 大厂面试官连环追问 (Follow-ups)
- 如何用 IPS 校正 position bias?
- 如何检测训练-服务偏斜?
🎯 关联大厂考题实战训练 (6 题)
嫌字多看太累?进入苏格拉底式对话推演实验室
零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.