M2-006Medium上线评估 · Launch evaluation
实验里 OEC 显著提升 2%,但全量上线后整体指标完全没动。为什么?
An experiment shows a significant +2% on the OEC, but after full launch the overall metric does not move at all. Why?
💡 考场高分一句话答案:
常见原因:实验人群与全量人群不同、效应被稀释、存在 novelty 效应消退、或上线实现与实验实现不一致。
Common causes: the experiment population differs from the launch population, the effect dilutes, novelty fades, or the launch implementation differs from the experimental one.
📌 核心考察要点 (Key Points)
- 人群差异:实验可能只覆盖了部分平台/地区,或参与者是更活跃的用户(selection)
- 稀释:全量后新增了大量不敏感用户,效应按比例缩小
- 时间效应:实验期短,novelty 消退后效应归零
- 实现差异:实验用的是特制后端,全量走的是另一条链路;或上线时改了参数
- Population: the experiment may have covered only some platforms or regions, or skewed towards more engaged users
- Dilution: full launch adds many insensitive users, shrinking the effect proportionally
- Time: the experiment was short, and the effect decayed once novelty passed
- Implementation: the experiment ran a bespoke backend while production runs another path, or parameters changed at launch
⚖️ 业务权衡与取舍 (Trade-offs & Calibration)
这是 Google 内部非常重视的问题,叫 “实验到生产的效应衰减(treatment effect decay)”。系统性的解法是:(1) 实验期足够长以越过 novelty 峰值;(2) 用 triggered analysis 对齐人群;(3) 上线后保留小比例 holdout 做持续对照;(4) 记录“实验实现 vs 生产实现”的差异清单。
⚠️ 常见高频面试陷阱与避坑指南
- 认为“实验有效就一定上线有效”,不做上线后验证
- 用实验期效应直接做财务预测,不乘衰减系数
🔍 大厂面试官连环追问 (Follow-ups)
- 如何设计一个能提前发现衰减的实验?
- 如果上线后效应只有实验的一半,你会怎么向管理层解释?
嫌字多看太累?进入苏格拉底式对话推演实验室
零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.