M3-004Medium实验流程 · Experiment workflow
一个实验应该跑多久?跑 3 天和跑 14 天有什么区别?
How long should an experiment run? What is the difference between three days and fourteen?
💡 考场高分一句话答案:
至少覆盖一个完整周周期,且要越过 novelty 峰值;3 天会被周内模式与新奇效应双重污染。
At least one full weekly cycle and past the novelty peak. Three days is contaminated by both intra-week patterns and novelty.
📌 核心考察要点 (Key Points)
- 周周期性:工作日与周末的用户构成、行为强度不同,少于 7 天会引入日程偏差
- Novelty:新 UI 的效应通常在第 3–7 天达到峰值后回落,短实验会高估
- 样本累积:跑得越久样本越多,但边际收益递减;超过 4 周要警惕外部事件污染
- 规则:先按功效算出最小天数,再向上取整到整周,且不早于 novelty 消退
- Weekly periodicity: weekday and weekend users differ in composition and intensity, so under seven days introduces day-of-week bias
- Novelty: the effect of a new UI usually peaks around day three to seven and then settles, so short tests overstate it
- Accumulation: more days means more sample, but with diminishing returns; beyond four weeks watch for external contamination
- Rule: compute the minimum days from power, round up to whole weeks, and do not stop before novelty has passed
⚖️ 业务权衡与取舍 (Trade-offs & Calibration)
判断 novelty 是否消退的方法是画效应随天数的曲线:若效应在第 5 天后趋于平稳,说明 novelty 已过;若持续下滑,说明效应主要是新奇感,长期可能归零。这个图在面试里可以直接口述,非常有说服力。
⚠️ 常见高频面试陷阱与避坑指南
- 跑 3 天看到显著就停(peeking + 周内偏差 + novelty)
- 跑 8 周导致外部事件污染与用户疲劳
🔍 大厂面试官连环追问 (Follow-ups)
- 如何在不延长实验的前提下估计长期效应?
- 如果效应曲线持续上升,说明什么?
嫌字多看太累?进入苏格拉底式对话推演实验室
零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.