M1-087M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Hard
⚡Flashcard Drill➔Mastery:

Explain Methodologies for Measuring Long-Term Treatment Effects Beyond Short-Term Metrics

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-088 →
📐 Mathematical Definition
holdout:persistent control group for months\text{holdout}: \text{persistent control group for months}
⚡ One-Sentence Core Answer
Short-term experiments suffer from novelty bias, primation bias, and delayed user attrition; evaluated via long-running holdouts, post-experiment holdouts (reverse experiments), and surrogate metric modeling.

📌 Key Takeaways

  • •
    Failure of Short-Term Tests: A feature might boost 1-week clicks due to novelty, but cause long-term churn at 3 months; aggressive notification alerts increase short-term opens but drive permanent app uninstalls.
  • •
    Universal Holdouts: Retains a persistent 1% Control group across all launched features for 6-12 months to measure compounding platform health.
  • •
    Reverse Holdouts: Once a feature is shipped to 95% of users, keep a 5% holdout on the old experience to measure long-term steady-state divergence.
  • •
    Surrogate Metric Modeling (Athey et al., 2019): Uses intermediate behavioral patterns that mathematically satisfy the conditional independence surrogate property for long-term outcomes.

📐 Mathematical Derivations

Surrogate Index Framework: Let YY be the true long-term metric (e.g. 1-year subscriber retention), WW be treatment assignment, and SS be a vector of intermediate short-term surrogate metrics (e.g. 2-week active days, search diversity, profile completion). The Surrogacy Condition states: W⊥Y∣SW \perp Y \mid S (treatment affects long-term metric YY strictly through surrogates SS). Under surrogacy and unconfoundedness, E[Y(w)]=ES[E[Y∣S,W=w]]=ES[E[Y∣S]]E[Y(w)] = E_S[E[Y\mid S, W=w]] = E_S[E[Y\mid S]]. One fits a regression model g(S)=E[Y∣S]g(S) = E[Y\mid S] on historical cohort data, and then evaluates the treatment effect on the predicted surrogate index: τ^long=1N∑i[g(Si(1))−g(Si(0))]\hat{\tau}_{\text{long}} = \frac{1}{N}\sum_{i} [g(S_i(1)) - g(S_i(0))], providing instantaneous forecasts of 1-year impact from 2-week experiments.

🏭 Production Trade-offs

Comparison of strategies: (1) Long-running holdouts measure true steady-state reality without statistical modeling assumptions, but incur severe business opportunity costs (holding users on outdated experiences for a year). (2) Surrogate modeling enables rapid 2-week decisions, but breaks down if a new treatment influences long-term retention through a novel channel not captured in historical surrogates SS.
⚠️ Common Interview Pitfalls
  • ✕
    Relying on proxy metrics that fail the statistical surrogacy condition (e.g. Optimizing short-term click volume which can harm user trust).
  • ✕
    Ignoring user survivorship bias in long-running holdout cohorts.
🎯 Interviewer Follow-up Inquiries
  • ?
    What is the Prentice Criterion for statistical validation of surrogate endpoints?
  • ?
    How do Reverse Holdouts prevent cumulative platform degradation over quarterly feature releases?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-086: Explain Trigger Analysis in Online Experiments and How It Enhances Statistical Power📋 Back to BankNext →M1-088: Explain Regression Discontinuity Design (RDD) and Its Core Continuity Assumption