M7-100M7: Retrieval, Ranking & RecSysOnline Metrics & GuardrailsHard
⚡Flashcard Drill➔Mastery:
Explain Multiple Testing Corrections (FDR, Bonferroni) and Sequential Testing (Alpha Spending) in A/B Experiments
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Evaluating multiple metrics simultaneously inflates Family-Wise Error Rates, solved by Bonferroni or False Discovery Rate (FDR) corrections; repeatedly monitoring data over time inflates Type I errors, solved by sequential testing and alpha spending functions.
📌 Key Takeaways
- •The multi-metric trap: Testing m metrics at alpha = 0.05 yields Family-Wise Error Rate FWER = 1 - (1 - alpha)^m (e.g., 64% false positive rate for m = 20).
- •The continuous peeking trap: Checking A/B test dashboards daily and stopping as soon as p < 0.05 inflates true false positive rates from 5% to over 30%.
- •Multiple testing corrections: Bonferroni enforces strict FWER control (alpha / m); Benjamini-Hochberg (BH) controls False Discovery Rate (FDR).
- •Sequential testing & Alpha spending: Distributes total error budget alpha across sequential interim analyses (O'Brien-Fleming, Pocock).
📐 Mathematical Derivations
Mathematical & Statistical Formulation: Error Inflation & Correction Mechanics.
(1) The Multiple Comparisons Pathology (Multi-Metric):
Let independent null hypotheses be tested simultaneously at significance level . The Family-Wise Error Rate (FWER) is:
For : . For : .
- Bonferroni Correction (FWER Control):
Guarantees , but is severely conservative, destroying statistical power.
- Benjamini-Hochberg Procedure (FDR Control):
Sort -values in ascending order: . Find maximum index such that:
Reject all null hypotheses . Guarantees False Discovery Rate .
(2) The Continuous Peeking Pathology (Sequential Testing):
Checking dashboard -values daily and stopping the test as soon as exploits random walk fluctuations. Even under a true null effect, the cumulative probability that a Brownian motion crosses the boundary over 30 days exceeds .
(3) Alpha Spending Functions (Lan & DeMets, 1983):
Allocates total significance budget as information fraction accumulates:
- O'Brien-Fleming Alpha Spending:
Spends virtually zero during early interim looks (requiring extreme significance to stop early), reserving the bulk of for the final horizon . Allows safe early termination for massive wins/losses without inflating Type I error.
(1) The Multiple Comparisons Pathology (Multi-Metric):
Let independent null hypotheses be tested simultaneously at significance level . The Family-Wise Error Rate (FWER) is:
For : . For : .
- Bonferroni Correction (FWER Control):
Guarantees , but is severely conservative, destroying statistical power.
- Benjamini-Hochberg Procedure (FDR Control):
Sort -values in ascending order: . Find maximum index such that:
Reject all null hypotheses . Guarantees False Discovery Rate .
(2) The Continuous Peeking Pathology (Sequential Testing):
Checking dashboard -values daily and stopping the test as soon as exploits random walk fluctuations. Even under a true null effect, the cumulative probability that a Brownian motion crosses the boundary over 30 days exceeds .
(3) Alpha Spending Functions (Lan & DeMets, 1983):
Allocates total significance budget as information fraction accumulates:
- O'Brien-Fleming Alpha Spending:
Spends virtually zero during early interim looks (requiring extreme significance to stop early), reserving the bulk of for the final horizon . Allows safe early termination for massive wins/losses without inflating Type I error.
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① Bonferroni vs. Benjamini-Hochberg (FDR)—Bonferroni is suitable when a single false positive has catastrophic consequences (e.g., medical device safety); in internet platforms where teams track 50 secondary metrics, Bonferroni eliminates all statistical power; Benjamini-Hochberg (FDR ) provides the optimal trade-off between discovery and false positive control. ② Always Valid p-values (mSPRT / Sequential Testing)—Netflix, Uber, and Optimizely deploy Mixture Sequential Probability Ratio Tests (mSPRT); mSPRT generates 'always valid -values' that remain mathematically sound under continuous peeking, allowing product managers to stop tests at any moment without penalty. ③ Decision vs. Diagnostic metric classification—platforms avoid multiple testing penalties by pre-registering a single Decision Metric (e.g., Checkout Conversion) that decides the ship/no-ship verdict (tested at full ); all other 40 metrics are classified as Diagnostic Metrics used only for post-hoc debugging. ④ Early stopping for catastrophic regressions—while early stopping for small wins causes false positives, early stopping when guardrail metrics show extreme degradation (, revenue) is strictly mandatory to prevent business harm; asymmetric alpha spending allows rapid aborts on failure. ⑤ Bayesian A/B testing as an alternative—Bayesian testing calculates the posterior probability that treatment beats control: ; Bayesian frameworks naturally handle continuous monitoring and quantify risk via expected loss . ⑥ Interview takeaway—calculate FWER inflation , contrast Bonferroni (FWER) with Benjamini-Hochberg (FDR), explain why continuous peeking breaks Type I error control, and describe O'Brien-Fleming alpha spending.
⚠️ Common Interview Pitfalls
- ✕Peeking at A/B test dashboards daily and terminating the experiment the first moment p < 0.05 is achieved, inflating false positive rates to 30%+.
- ✕Testing 30 metrics simultaneously and claiming algorithmic success because one secondary metric reached p = 0.04 without FDR correction.
- ✕Applying Bonferroni corrections across dozens of exploratory metrics, destroying statistical power and discarding genuine product improvements.
🎯 Interviewer Follow-up Inquiries
- ?How does the Mixture Sequential Probability Ratio Test (mSPRT) construct 'always valid p-values' that allow continuous dashboard monitoring?
- ?What is the mathematical distinction between Family-Wise Error Rate (FWER) and False Discovery Rate (FDR)?
- ?How does O'Brien-Fleming alpha spending distribute significance thresholds across interim looks to allow safe early stopping?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚