M7-099M7: Retrieval, Ranking & RecSysOnline Metrics & GuardrailsHard
⚡Flashcard Drill➔Mastery:
Explain Sample Size Determination, Statistical Power, and Minimum Detectable Effect (MDE) in A/B Testing
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Sample size scales proportionally with metric variance and inversely with the square of the Minimum Detectable Effect (n ~ sigma^2 / MDE^2); balancing MDE against business value prevents under-powered experiments and wasteful multi-month test durations.
📌 Key Takeaways
- •The sample size scaling law: Halving the target MDE quadruples required sample size (1 / MDE^2 scaling), demanding massive traffic investments for marginal gains.
- •Statistical parameters: Type I error alpha (false positive rate, typically 5%) and Statistical Power 1 - beta (probability of detecting a true effect, typically 80%).
- •Continuous vs. binary metric variance: Continuous revenue metrics have high skew and variance (requiring larger N); binary CTR metrics scale with p(1-p).
- •CUPED sample size reduction: Reducing metric variance sigma^2 by 50% reduces required sample size by exactly 50% for the same target MDE.
📐 Mathematical Derivations
Mathematical & Statistical Formulation: Sample Size Mechanics.
(1) Two-Sample Hypothesis Testing Framework:
Let null hypothesis be , and alternative be .
- Significance Level : (Standard: ).
- Statistical Power : (Standard: ).
(2) Sample Size Formula per Variant ($n$):
For a continuous metric with standard deviation across both variants:
For a binary proportion metric (e.g., Conversion Rate ):
(3) The Quadratic Penalty ($1 / \text{MDE}^2$):
Suppose baseline conversion rate ():
- To detect a relative lift ():
- To detect a relative lift ():
(1) Two-Sample Hypothesis Testing Framework:
Let null hypothesis be , and alternative be .
- Significance Level : (Standard: ).
- Statistical Power : (Standard: ).
(2) Sample Size Formula per Variant ($n$):
For a continuous metric with standard deviation across both variants:
For a binary proportion metric (e.g., Conversion Rate ):
(3) The Quadratic Penalty ($1 / \text{MDE}^2$):
Suppose baseline conversion rate ():
- To detect a relative lift ():
- To detect a relative lift ():
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① The Under-Powered Experiment Trap ($1 - \beta —running an A/B test with insufficient sample size does not mean 'it will only catch large wins'; it means true algorithmic improvements fail to reach statistical significance () and are discarded, while random noise outliers that happen to breach are shipped (the 'Winner's Curse'); power calculations must be executed before launching any test. ② CUPED as a sample size multiplier—because , applying CUPED to reduce variance by 40% drops required sample size from 100k to 60k users, shortening experiment duration from 14 days to 8 days. ③ Metric variance capping and log transformation—continuous revenue distributions contain extreme whale spenders (>\10,000\sigma^2\ln(1 + \text{spend})\sigma^2pNn \approx \frac{16 \sigma^2}{\text{MDE}^2}1/\text{MDE}^2$ scaling law with a concrete calculation, define Type I/II errors, and explain how CUPED reduces sample size requirements.
⚠️ Common Interview Pitfalls
- ✕Halving the target MDE without recognizing that required traffic volume increases by a factor of 4, causing tests to run for months.
- ✕Running under-powered experiments on small traffic slices, discarding genuine algorithmic improvements that fail to reach statistical significance.
- ✕Calculating sample size on skewed continuous revenue metrics without outlier capping, underestimating required traffic by a factor of 10.
🎯 Interviewer Follow-up Inquiries
- ?Why does the 'Winner's Curse' cause statistically significant results from under-powered experiments to drastically overestimate true effect sizes?
- ?How does outlier percentile capping on skewed revenue metrics reduce required sample size without introducing unacceptable bias?
- ?What mathematical adjustments convert sample size calculations from user-level randomization to session-level randomization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚