M3-004Medium实验流程 · Experiment workflow

一个实验应该跑多久?跑 3 天和跑 14 天有什么区别?

How long should an experiment run? What is the difference between three days and fourteen?
💡 考场高分一句话答案:

至少覆盖一个完整周周期,且要越过 novelty 峰值;3 天会被周内模式与新奇效应双重污染。

At least one full weekly cycle and past the novelty peak. Three days is contaminated by both intra-week patterns and novelty.

📌 核心考察要点 (Key Points)

  • 周周期性:工作日与周末的用户构成、行为强度不同,少于 7 天会引入日程偏差
  • Novelty:新 UI 的效应通常在第 3–7 天达到峰值后回落,短实验会高估
  • 样本累积:跑得越久样本越多,但边际收益递减;超过 4 周要警惕外部事件污染
  • 规则:先按功效算出最小天数,再向上取整到整周,且不早于 novelty 消退
  • Weekly periodicity: weekday and weekend users differ in composition and intensity, so under seven days introduces day-of-week bias
  • Novelty: the effect of a new UI usually peaks around day three to seven and then settles, so short tests overstate it
  • Accumulation: more days means more sample, but with diminishing returns; beyond four weeks watch for external contamination
  • Rule: compute the minimum days from power, round up to whole weeks, and do not stop before novelty has passed

⚖️ 业务权衡与取舍 (Trade-offs & Calibration)

判断 novelty 是否消退的方法是画效应随天数的曲线:若效应在第 5 天后趋于平稳,说明 novelty 已过;若持续下滑,说明效应主要是新奇感,长期可能归零。这个图在面试里可以直接口述,非常有说服力。

⚠️ 常见高频面试陷阱与避坑指南

  • 跑 3 天看到显著就停(peeking + 周内偏差 + novelty)
  • 跑 8 周导致外部事件污染与用户疲劳

🔍 大厂面试官连环追问 (Follow-ups)

  • 如何在不延长实验的前提下估计长期效应?
  • 如果效应曲线持续上升,说明什么?

嫌字多看太累?进入苏格拉底式对话推演实验室

零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。

💬 开启对话式破冰 ➔
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔