M1-044M1: Mathematics & Statistics FundamentalsHypothesis TestingMedium
⚡Flashcard Drill➔Mastery:

Define the Multiple Testing Problem and Contrast Common Corrections (Bonferroni vs. Benjamini-Hochberg FDR)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-045 →
📐 Mathematical Definition
Bonferroni: α′=α/m,BH: p(i)≤imq\text{Bonferroni}:\ \alpha'=\alpha/m,\qquad \text{BH}:\ p_{(i)}\le\frac{i}{m}q
⚡ One-Sentence Core Answer
Testing multiple hypotheses simultaneously inflates family-wise false positives exponentially: 1−(1−α)m1 - (1-\alpha)^m; controlled via conservative Bonferroni (FWER) or powerful Benjamini-Hochberg (FDR).

📌 Key Takeaways

  • •
    Family-Wise Error Rate (FWER): Probability of making at least one false positive: FWER=P(V≥1)≤1−(1−α)m≈mα\text{FWER} = P(V \ge 1) \le 1 - (1-\alpha)^m \approx m\alpha for small α\alpha.
  • •
    Bonferroni correction: Rejects if pi≤α/mp_i \le \alpha / m; guarantees strict FWER control but severely inflates Type II error (lacks power).
  • •
    False Discovery Rate (FDR): Expected proportion of false discoveries among rejections: FDR=E[VR∣R>0]\text{FDR} = E\left[\frac{V}{R} \mid R > 0\right].
  • •
    Benjamini-Hochberg (BH): Sorts p-values p(1)≤⋯≤p(m)p_{(1)} \le \dots \le p_{(m)} and finds largest kk where p(k)≤kmqp_{(k)} \le \frac{k}{m}q.

📐 Mathematical Derivations

Proof of Bonferroni via Boole's inequality (union bound): Let AiA_i be the event of falsely rejecting true null hypothesis H0,iH_{0, i}. Then FWER=P(⋃i=1m0Ai)≤∑i=1m0P(Ai)=∑i=1m0α′=m0α′≤mα′\text{FWER} = P\left(\bigcup_{i=1}^{m_0} A_i\right) \le \sum_{i=1}^{m_0} P(A_i) = \sum_{i=1}^{m_0} \alpha' = m_0 \alpha' \le m \alpha'. Setting α′=αm\alpha' = \frac{\alpha}{m} strictly guarantees FWER≤α\text{FWER} \le \alpha. The Benjamini-Hochberg procedure sorts p-values p(1)≤p(2)≤⋯≤p(m)p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}. It finds k∗=max⁡{k:p(k)≤kmq}k^* = \max\left\{k : p_{(k)} \le \frac{k}{m} q\right\} and rejects all hypotheses H(1),…,H(k∗)H_{(1)}, \dots, H_{(k^*)}. Benjamini & Hochberg (1995) proved that for independent test statistics, this procedure guarantees FDR≤m0mq≤q\text{FDR} \le \frac{m_0}{m} q \le q.

🏭 Production Trade-offs

In industrial experimentation: (1) A/B/n tests with multiple variants or safety guardrail metric checks use Bonferroni or Holm-Bonferroni to strictly avoid shipping damaging regressions. (2) Feature selection, genome-wide association, and metric exploration platforms use Benjamini-Hochberg FDR because tolerating 5% false discoveries among 100 claimed discoveries (q=0.05q=0.05) yields far superior discovery power than conservative FWER.
⚠️ Common Interview Pitfalls
  • ✕
    Looking at 20 metrics and highlighting the single one with p<0.05p < 0.05 without multiple testing correction (the Texas sharpshooter fallacy).
  • ✕
    Applying Bonferroni correction when metrics are near-perfectly correlated, resulting in massive over-correction.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the Holm-Bonferroni step-down procedure uniformly dominate standard Bonferroni in power?
  • ?
    How does the Benjamini-Yekutieli (BY) procedure extend FDR control to arbitrary positive and negative dependencies?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-043: Explain the Mathematical Relationship Between Sample Size and Statistical Power with Sizing Formulas📋 Back to BankNext →M1-045: Explain Sequential Testing and the Peeking Problem, and Detail Correct Continuous Monitoring Frameworks