M1-087M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Hard
⚡Flashcard Drill➔Mastery:
Explain Methodologies for Measuring Long-Term Treatment Effects Beyond Short-Term Metrics
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Short-term experiments suffer from novelty bias, primation bias, and delayed user attrition; evaluated via long-running holdouts, post-experiment holdouts (reverse experiments), and surrogate metric modeling.
📌 Key Takeaways
- •Failure of Short-Term Tests: A feature might boost 1-week clicks due to novelty, but cause long-term churn at 3 months; aggressive notification alerts increase short-term opens but drive permanent app uninstalls.
- •Universal Holdouts: Retains a persistent 1% Control group across all launched features for 6-12 months to measure compounding platform health.
- •Reverse Holdouts: Once a feature is shipped to 95% of users, keep a 5% holdout on the old experience to measure long-term steady-state divergence.
- •Surrogate Metric Modeling (Athey et al., 2019): Uses intermediate behavioral patterns that mathematically satisfy the conditional independence surrogate property for long-term outcomes.
📐 Mathematical Derivations
Surrogate Index Framework: Let be the true long-term metric (e.g. 1-year subscriber retention), be treatment assignment, and be a vector of intermediate short-term surrogate metrics (e.g. 2-week active days, search diversity, profile completion). The Surrogacy Condition states: (treatment affects long-term metric strictly through surrogates ). Under surrogacy and unconfoundedness, . One fits a regression model on historical cohort data, and then evaluates the treatment effect on the predicted surrogate index: , providing instantaneous forecasts of 1-year impact from 2-week experiments.
🏭 Production Trade-offs
Comparison of strategies: (1) Long-running holdouts measure true steady-state reality without statistical modeling assumptions, but incur severe business opportunity costs (holding users on outdated experiences for a year). (2) Surrogate modeling enables rapid 2-week decisions, but breaks down if a new treatment influences long-term retention through a novel channel not captured in historical surrogates .
⚠️ Common Interview Pitfalls
- ✕Relying on proxy metrics that fail the statistical surrogacy condition (e.g. Optimizing short-term click volume which can harm user trust).
- ✕Ignoring user survivorship bias in long-running holdout cohorts.
🎯 Interviewer Follow-up Inquiries
- ?What is the Prentice Criterion for statistical validation of surrogate endpoints?
- ?How do Reverse Holdouts prevent cumulative platform degradation over quarterly feature releases?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚