M7-099M7: Retrieval, Ranking & RecSysOnline Metrics & GuardrailsHard
⚡Flashcard Drill➔Mastery:

Explain Sample Size Determination, Statistical Power, and Minimum Detectable Effect (MDE) in A/B Testing

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M7-100 →
📐 Mathematical Definition
n≈2(z1−α/2+z1−β)2σ2MDE2;MDE↓⇒n↑ (quadratic)n\approx\frac{2(z_{1-\alpha/2}+z_{1-\beta})^2\sigma^2}{\text{MDE}^2};\qquad \text{MDE}\downarrow\Rightarrow n\uparrow\ \text{(quadratic)}
⚡ One-Sentence Core Answer
Sample size scales proportionally with metric variance and inversely with the square of the Minimum Detectable Effect (n ~ sigma^2 / MDE^2); balancing MDE against business value prevents under-powered experiments and wasteful multi-month test durations.

📌 Key Takeaways

  • •
    The sample size scaling law: Halving the target MDE quadruples required sample size (1 / MDE^2 scaling), demanding massive traffic investments for marginal gains.
  • •
    Statistical parameters: Type I error alpha (false positive rate, typically 5%) and Statistical Power 1 - beta (probability of detecting a true effect, typically 80%).
  • •
    Continuous vs. binary metric variance: Continuous revenue metrics have high skew and variance (requiring larger N); binary CTR metrics scale with p(1-p).
  • •
    CUPED sample size reduction: Reducing metric variance sigma^2 by 50% reduces required sample size by exactly 50% for the same target MDE.

📐 Mathematical Derivations

Mathematical & Statistical Formulation: Sample Size Mechanics.

(1) Two-Sample Hypothesis Testing Framework:
Let null hypothesis be H0:μT−μC=0H_0: \mu_T - \mu_C = 0, and alternative be H1:μT−μC=δ=MDEH_1: \mu_T - \mu_C = \delta = \text{MDE}.
- Significance Level α\alpha: P(Reject H0∣H0 is true)=αP(\text{Reject } H_0 \mid H_0 \text{ is true}) = \alpha (Standard: α=0.05  ⟹  z1−α/2=1.96\alpha = 0.05 \implies z_{1-\alpha/2} = 1.96).
- Statistical Power 1−β1 - \beta: P(Reject H0∣H1 is true)=1−βP(\text{Reject } H_0 \mid H_1 \text{ is true}) = 1 - \beta (Standard: 1−β=0.80  ⟹  z1−β=0.841 - \beta = 0.80 \implies z_{1-\beta} = 0.84).

(2) Sample Size Formula per Variant ($n$):
For a continuous metric with standard deviation σ\sigma across both variants:
n≈2⋅(z1−α/2+z1−β)2⋅σ2δ2=2⋅(1.96+0.84)2⋅σ2MDE2≈15.68⋅σ2MDE2n \approx \frac{2 \cdot (z_{1-\alpha/2} + z_{1-\beta})^2 \cdot \sigma^2}{\delta^2} = \frac{2 \cdot (1.96 + 0.84)^2 \cdot \sigma^2}{\text{MDE}^2} \approx \frac{15.68 \cdot \sigma^2}{\text{MDE}^2}
For a binary proportion metric (e.g., Conversion Rate pp):
σ2=p(1−p)  ⟹  n≈15.68⋅p(1−p)MDE2\sigma^2 = p(1 - p) \implies n \approx \frac{15.68 \cdot p(1 - p)}{\text{MDE}^2}

(3) The Quadratic Penalty ($1 / \text{MDE}^2$):
Suppose baseline conversion rate p=0.05p = 0.05 (5%5\%):
- To detect a relative +10%+10\% lift (MDE=0.005\text{MDE} = 0.005):
n≈15.68×0.05×0.95(0.005)2=0.74480.000025≈29,792 users per variantn \approx \frac{15.68 \times 0.05 \times 0.95}{(0.005)^2} = \frac{0.7448}{0.000025} \approx 29,792 \text{ users per variant}Total N≈60,000 users (Feasible in 1 day).\text{Total } N \approx 60,000 \text{ users (Feasible in 1 day)}.
- To detect a relative +1%+1\% lift (MDE=0.0005\text{MDE} = 0.0005):
n≈0.7448(0.0005)2=0.74480.00000025≈2,979,200 users per variantn \approx \frac{0.7448}{(0.0005)^2} = \frac{0.7448}{0.00000025} \approx 2,979,200 \text{ users per variant}Total N≈6,000,000 users (Requires 100x more traffic!)\text{Total } N \approx 6,000,000 \text{ users (Requires 100x more traffic!)}

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① The Under-Powered Experiment Trap ($1 - \beta —running an A/B test with insufficient sample size does not mean 'it will only catch large wins'; it means true +1%+1\% algorithmic improvements fail to reach statistical significance (p>0.05p > 0.05) and are discarded, while random noise outliers that happen to breach p<0.05p < 0.05 are shipped (the 'Winner's Curse'); power calculations must be executed before launching any test. ② CUPED as a sample size multiplier—because n∝σ2n \propto \sigma^2, applying CUPED to reduce variance by 40% drops required sample size from 100k to 60k users, shortening experiment duration from 14 days to 8 days. ③ Metric variance capping and log transformation—continuous revenue distributions contain extreme whale spenders (>\10,000),causing), causing\sigma^2toexplode;cappingrevenueatthe99thpercentileortakingto explode; capping revenue at the 99th percentile or taking\ln(1 + \text{spend})cutscuts\sigma^2by3x,vastlyreducingsamplesizerequirements.④<strong>MinimumDetectableEffect(MDE)alignmentwithbusinessvalue</strong>—settingMDEtoosmallwastestestingcapacityontrivialgains;settingMDEtoolargemissesmeaningfulcumulativeoptimizations;MDEshouldbecalibratedtothesmallestliftthatjustifiesengineeringmaintenancecosts.⑤<strong>Fixed−horizontestingvs.Continuouspeeking</strong>—evaluatingby 3x, vastly reducing sample size requirements. ④ <strong>Minimum Detectable Effect (MDE) alignment with business value</strong>—setting MDE too small wastes testing capacity on trivial gains; setting MDE too large misses meaningful cumulative optimizations; MDE should be calibrated to the smallest lift that justifies engineering maintenance costs. ⑤ <strong>Fixed-horizon testing vs. Continuous peeking</strong>—evaluatingp−valuesdailyduringanexperimentinflatesfalsepositiverates;samplesizeformulasassumeevaluationoccursstrictlyatpre−calculatedhorizon-values daily during an experiment inflates false positive rates; sample size formulas assume evaluation occurs strictly at pre-calculated horizonN.⑥<strong>Interviewtakeaway</strong>—writeoutthesamplesizeformula. ⑥ <strong>Interview takeaway</strong>—write out the sample size formulan \approx \frac{16 \sigma^2}{\text{MDE}^2},explainthequadratic, explain the quadratic1/\text{MDE}^2$ scaling law with a concrete calculation, define Type I/II errors, and explain how CUPED reduces sample size requirements.
⚠️ Common Interview Pitfalls
  • ✕
    Halving the target MDE without recognizing that required traffic volume increases by a factor of 4, causing tests to run for months.
  • ✕
    Running under-powered experiments on small traffic slices, discarding genuine algorithmic improvements that fail to reach statistical significance.
  • ✕
    Calculating sample size on skewed continuous revenue metrics without outlier capping, underestimating required traffic by a factor of 10.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does the 'Winner's Curse' cause statistically significant results from under-powered experiments to drastically overestimate true effect sizes?
  • ?
    How does outlier percentile capping on skewed revenue metrics reduce required sample size without introducing unacceptable bias?
  • ?
    What mathematical adjustments convert sample size calculations from user-level randomization to session-level randomization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM7-098: Explain the Architectural and Algorithmic Requirements of an Enterprise Experimentation Platform📋 Back to BankNext →M7-100: Explain Multiple Testing Corrections (FDR, Bonferroni) and Sequential Testing (Alpha Spending) in A/B Experiments