M1-016Medium指标口径 · Metric definitions
什么是 metric dilution(指标稀释)?怎么避免?
What is metric dilution and how do you avoid it?
💡 考场高分一句话答案:
把未被改动影响的用户算进分母,导致真实效应被按比例缩小;解法是 triggered analysis,只分析真正触发的用户。
Including users the change cannot affect shrinks the measured effect proportionally. The fix is triggered analysis on the actually-exposed subset.
📌 核心考察要点 (Key Points)
- 场景:改动只影响“搜索了含广告关键词”的用户,但分析时用了全部 DAU 作分母
- 后果:真实 +10% 的效应用全量分母算出来只有 +1%,可能被判为不显著而误杀
- 解法:实验时记录 trigger 标志(对照组也记录“本会触发”),分析时只取 triggered 子集
- 注意:triggered 分析的结论只对触发人群成立,不能外推到全量
- Scenario: the change only affects users who search for commercial queries, but the analysis uses all DAU as the denominator
- Consequence: a real +10% effect measures as +1% and may be declared non-significant, killing a good change
- Fix: log a trigger flag in the experiment (including for the control group) and analyse the triggered subset only
- Caveat: triggered results generalise only to the triggered population, not to the whole user base
📐 第一性原理与数理推导机理
设触发比例为 $q$,真实效应为 $\tau$,则全量口径下的观测效应为 $q\tau$。样本量需求按 $1/q^2$ 增长——这就是为什么稀释会让实验“看起来没效果”。triggered analysis 把 $q$ 恢复为 1,功效随之恢复。
⚖️ 业务权衡与取舍 (Trade-offs & Calibration)
triggered analysis 的前提是对照组也要有 trigger 标志(counterfactual logging),否则无法构造可比的子集。这需要在实验设计阶段就和工程约定好,是典型的“设计决定分析上限”。
⚠️ 常见高频面试陷阱与避坑指南
- 事后按“是否点击了新功能”划分 triggered——那是结果变量,会产生选择偏差
- triggered 结论外推到全量,高估整体收益
🔍 大厂面试官连环追问 (Follow-ups)
- triggered analysis 与 CUPED 能叠加吗?
- 如果 trigger 比例只有 2%,实验要跑多久?
嫌字多看太累?进入苏格拉底式对话推演实验室
零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.