M7-100M7: Retrieval, Ranking & RecSysOnline Metrics & GuardrailsHard
⚡Flashcard Drill➔Mastery:

Explain Multiple Testing Corrections (FDR, Bonferroni) and Sequential Testing (Alpha Spending) in A/B Experiments

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M7-101 →
📐 Mathematical Definition
FWER=1−(1−α)m;alpha spending: α(t) sums to α\text{FWER}=1-(1-\alpha)^m;\qquad \text{alpha spending}:\ \alpha(t)\ \text{sums to }\alpha
⚡ One-Sentence Core Answer
Evaluating multiple metrics simultaneously inflates Family-Wise Error Rates, solved by Bonferroni or False Discovery Rate (FDR) corrections; repeatedly monitoring data over time inflates Type I errors, solved by sequential testing and alpha spending functions.

📌 Key Takeaways

  • •
    The multi-metric trap: Testing m metrics at alpha = 0.05 yields Family-Wise Error Rate FWER = 1 - (1 - alpha)^m (e.g., 64% false positive rate for m = 20).
  • •
    The continuous peeking trap: Checking A/B test dashboards daily and stopping as soon as p < 0.05 inflates true false positive rates from 5% to over 30%.
  • •
    Multiple testing corrections: Bonferroni enforces strict FWER control (alpha / m); Benjamini-Hochberg (BH) controls False Discovery Rate (FDR).
  • •
    Sequential testing & Alpha spending: Distributes total error budget alpha across sequential interim analyses (O'Brien-Fleming, Pocock).

📐 Mathematical Derivations

Mathematical & Statistical Formulation: Error Inflation & Correction Mechanics.

(1) The Multiple Comparisons Pathology (Multi-Metric):
Let mm independent null hypotheses be tested simultaneously at significance level α=0.05\alpha = 0.05. The Family-Wise Error Rate (FWER) is:
FWER=P(At least 1 false positive)=1−(1−α)m\text{FWER} = P(\text{At least 1 false positive}) = 1 - (1 - \alpha)^m
For m=10m = 10: 1−(0.95)10=40.1%1 - (0.95)^{10} = 40.1\%. For m=20m = 20: 1−(0.95)20=64.2%1 - (0.95)^{20} = 64.2\%.
- Bonferroni Correction (FWER Control):
Reject H0,i  ⟺  pi≤αm\text{Reject } H_{0, i} \iff p_i \le \frac{\alpha}{m}
Guarantees FWER≤α\text{FWER} \le \alpha, but is severely conservative, destroying statistical power.
- Benjamini-Hochberg Procedure (FDR Control):
Sort mm pp-values in ascending order: p(1)≤p(2)≤⋯≤p(m)p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}. Find maximum index kk such that:
k=max⁡{i∈[1,m]:p(i)≤im⋅q∗}k = \max\left\{ i \in [1, m] : p_{(i)} \le \frac{i}{m} \cdot q^* \right\}
Reject all null hypotheses H(1),…,H(k)H_{(1)}, \dots, H_{(k)}. Guarantees False Discovery Rate E[False RejectionsTotal Rejections]≤q∗\mathbb{E}\left[ \frac{\text{False Rejections}}{\text{Total Rejections}} \right] \le q^*.

(2) The Continuous Peeking Pathology (Sequential Testing):
Checking dashboard pp-values daily and stopping the test as soon as p<0.05p < 0.05 exploits random walk fluctuations. Even under a true null effect, the cumulative probability that a Brownian motion crosses the 1.96σ1.96\sigma boundary over 30 days exceeds 30%30\%.

(3) Alpha Spending Functions (Lan & DeMets, 1983):
Allocates total significance budget α\alpha as information fraction t=n/Nmax∈(0,1]t = n / N_{\text{max}} \in (0, 1] accumulates:
- O'Brien-Fleming Alpha Spending:
α(t)=2−2Φ(z1−α/2t)\alpha(t) = 2 - 2 \Phi\left( \frac{z_{1-\alpha/2}}{\sqrt{t}} \right)
Spends virtually zero α\alpha during early interim looks (requiring extreme significance p<0.0001p < 0.0001 to stop early), reserving the bulk of α\alpha for the final horizon t=1t=1. Allows safe early termination for massive wins/losses without inflating Type I error.

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① Bonferroni vs. Benjamini-Hochberg (FDR)—Bonferroni is suitable when a single false positive has catastrophic consequences (e.g., medical device safety); in internet platforms where teams track 50 secondary metrics, Bonferroni eliminates all statistical power; Benjamini-Hochberg (FDR ≤10%\le 10\%) provides the optimal trade-off between discovery and false positive control. ② Always Valid p-values (mSPRT / Sequential Testing)—Netflix, Uber, and Optimizely deploy Mixture Sequential Probability Ratio Tests (mSPRT); mSPRT generates 'always valid pp-values' that remain mathematically sound under continuous peeking, allowing product managers to stop tests at any moment without penalty. ③ Decision vs. Diagnostic metric classification—platforms avoid multiple testing penalties by pre-registering a single Decision Metric (e.g., Checkout Conversion) that decides the ship/no-ship verdict (tested at full α=0.05\alpha = 0.05); all other 40 metrics are classified as Diagnostic Metrics used only for post-hoc debugging. ④ Early stopping for catastrophic regressions—while early stopping for small wins causes false positives, early stopping when guardrail metrics show extreme degradation (p<0.001p < 0.001, −5%-5\% revenue) is strictly mandatory to prevent business harm; asymmetric alpha spending allows rapid aborts on failure. ⑤ Bayesian A/B testing as an alternative—Bayesian testing calculates the posterior probability that treatment beats control: P(μT>μC∣Data)P(\mu_T > \mu_C \mid \text{Data}); Bayesian frameworks naturally handle continuous monitoring and quantify risk via expected loss E[max⁡(0,μC−μT)]\mathbb{E}[\max(0, \mu_C - \mu_T)]. ⑥ Interview takeaway—calculate FWER inflation 1−(1−α)m1-(1-\alpha)^m, contrast Bonferroni (FWER) with Benjamini-Hochberg (FDR), explain why continuous peeking breaks Type I error control, and describe O'Brien-Fleming alpha spending.
⚠️ Common Interview Pitfalls
  • ✕
    Peeking at A/B test dashboards daily and terminating the experiment the first moment p < 0.05 is achieved, inflating false positive rates to 30%+.
  • ✕
    Testing 30 metrics simultaneously and claiming algorithmic success because one secondary metric reached p = 0.04 without FDR correction.
  • ✕
    Applying Bonferroni corrections across dozens of exploratory metrics, destroying statistical power and discarding genuine product improvements.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the Mixture Sequential Probability Ratio Test (mSPRT) construct 'always valid p-values' that allow continuous dashboard monitoring?
  • ?
    What is the mathematical distinction between Family-Wise Error Rate (FWER) and False Discovery Rate (FDR)?
  • ?
    How does O'Brien-Fleming alpha spending distribute significance thresholds across interim looks to allow safe early stopping?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM7-099: Explain Sample Size Determination, Statistical Power, and Minimum Detectable Effect (MDE) in A/B Testing📋 Back to BankNext →M7-101: Explain Layered Experimentation Frameworks and Mutually Exclusive Domains in Scaled A/B Testing