M1-016Medium指标口径 · Metric definitions

什么是 metric dilution(指标稀释)?怎么避免?

What is metric dilution and how do you avoid it?
💡 考场高分一句话答案:

把未被改动影响的用户算进分母,导致真实效应被按比例缩小;解法是 triggered analysis,只分析真正触发的用户。

Including users the change cannot affect shrinks the measured effect proportionally. The fix is triggered analysis on the actually-exposed subset.

📌 核心考察要点 (Key Points)

  • 场景:改动只影响“搜索了含广告关键词”的用户,但分析时用了全部 DAU 作分母
  • 后果:真实 +10% 的效应用全量分母算出来只有 +1%,可能被判为不显著而误杀
  • 解法:实验时记录 trigger 标志(对照组也记录“本会触发”),分析时只取 triggered 子集
  • 注意:triggered 分析的结论只对触发人群成立,不能外推到全量
  • Scenario: the change only affects users who search for commercial queries, but the analysis uses all DAU as the denominator
  • Consequence: a real +10% effect measures as +1% and may be declared non-significant, killing a good change
  • Fix: log a trigger flag in the experiment (including for the control group) and analyse the triggered subset only
  • Caveat: triggered results generalise only to the triggered population, not to the whole user base

📐 第一性原理与数理推导机理

设触发比例为 $q$,真实效应为 $\tau$,则全量口径下的观测效应为 $q\tau$。样本量需求按 $1/q^2$ 增长——这就是为什么稀释会让实验“看起来没效果”。triggered analysis 把 $q$ 恢复为 1,功效随之恢复。

⚖️ 业务权衡与取舍 (Trade-offs & Calibration)

triggered analysis 的前提是对照组也要有 trigger 标志(counterfactual logging),否则无法构造可比的子集。这需要在实验设计阶段就和工程约定好,是典型的“设计决定分析上限”。

⚠️ 常见高频面试陷阱与避坑指南

  • 事后按“是否点击了新功能”划分 triggered——那是结果变量,会产生选择偏差
  • triggered 结论外推到全量,高估整体收益

🔍 大厂面试官连环追问 (Follow-ups)

  • triggered analysis 与 CUPED 能叠加吗?
  • 如果 trigger 比例只有 2%,实验要跑多久?

嫌字多看太累?进入苏格拉底式对话推演实验室

零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。

💬 开启对话式破冰 ➔
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔