📈 Data ScientistID: ds-multiple-testing-fwer-fdr-bh

Multiple Comparisons: FWER vs FDR-BH

Q: Contrast Bonferroni, Holm-Bonferroni, and Benjamini-Hochberg (BH) across ranking algorithms, statistical conservatism, and Type-II error rates?

【Multiple Comparisons: FWER vs FDR-BH 核心考点 1】Contrast Bonferroni, Holm-Bonferroni, and Benjamini-Hochberg (BH) across ranking algorithms, statistical conservatism, and Type-II error rates?。技术解析：Definition: The Multiple Comparisons in Experimentation (FWER vs FDR-BH Corrections) framework formalizes decision protocols when evaluating multi-arm variants (A/B/n testing), multi-metric dashboards, or multidimensional subgroup drilldowns; 2 governing statistical philosophies:... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DS/card/ds-multiple-testing-fwer-fdr-bh

Q: Write the complete algorithm for Benjamini-Hochberg step-up procedure determining the cutoff index $k^*$ such that $p_{(k^*)} \le \frac{k^*}{K} q$?

【Multiple Comparisons: FWER vs FDR-BH 核心考点 2】Write the complete algorithm for Benjamini-Hochberg step-up procedure determining the cutoff index $k^*$ such that $p_{(k^*)} \le \frac{k^*}{K} q$?。技术解析：Definition: The Multiple Comparisons in Experimentation (FWER vs FDR-BH Corrections) framework formalizes decision protocols when evaluating multi-arm variants (A/B/n testing), multi-metric dashboards, or multidimensional subgroup drilldowns; 2 governing statistical philosophies:... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DS/card/ds-multiple-testing-fwer-fdr-bh

Q: Map FWER vs FDR to business scenarios: mission-critical GMV safety vs marketing tag discovery vs multi-armed bandits?

【Multiple Comparisons: FWER vs FDR-BH 核心考点 3】Map FWER vs FDR to business scenarios: mission-critical GMV safety vs marketing tag discovery vs multi-armed bandits?。技术解析：Definition: The Multiple Comparisons in Experimentation (FWER vs FDR-BH Corrections) framework formalizes decision protocols when evaluating multi-arm variants (A/B/n testing), multi-metric dashboards, or multidimensional subgroup drilldowns; 2 governing statistical philosophies:... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DS/card/ds-multiple-testing-fwer-fdr-bh

Q: Explain how post-hoc subgroup cherry-picking triggers both multiple testing bias and Simpson's Paradox?

【Multiple Comparisons: FWER vs FDR-BH 核心考点 4】Explain how post-hoc subgroup cherry-picking triggers both multiple testing bias and Simpson's Paradox?。技术解析：Definition: The Multiple Comparisons in Experimentation (FWER vs FDR-BH Corrections) framework formalizes decision protocols when evaluating multi-arm variants (A/B/n testing), multi-metric dashboards, or multidimensional subgroup drilldowns; 2 governing statistical philosophies:... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DS/card/ds-multiple-testing-fwer-fdr-bh

Q: Explain how Benjamini-Yekutieli (BY) generalizes FDR control under arbitrary positive dependence and correlation across metrics?

【Multiple Comparisons: FWER vs FDR-BH 核心考点 5】Explain how Benjamini-Yekutieli (BY) generalizes FDR control under arbitrary positive dependence and correlation across metrics?。技术解析：Definition: The Multiple Comparisons in Experimentation (FWER vs FDR-BH Corrections) framework formalizes decision protocols when evaluating multi-arm variants (A/B/n testing), multi-metric dashboards, or multidimensional subgroup drilldowns; 2 governing statistical philosophies:... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DS/card/ds-multiple-testing-fwer-fdr-bh

多重比较校正：Bonferroni vs FDR

🎯Core Definition

The Multiple Comparisons in Experimentation (FWER vs FDR-BH Corrections) framework formalizes decision protocols when evaluating multi-arm variants (A/B/n testing), multi-metric dashboards, or multidimensional subgroup drilldowns; 2 governing statistical philosophies: 1) Family-Wise Error Rate (FWER): strictly controlling the probability of committing even a single Type-I error across all

K

tests (

P(V \ge 1) \le \alpha

), implemented via conservative Bonferroni (

\alpha' = \alpha / K

) or step-down Holm-Bonferroni (

\alpha_i = \alpha / (K - i + 1)

), mandatory for mission-critical core revenue decisions; 2) False Discovery Rate (FDR): controlling the expected proportion of false discoveries among all rejected nulls (

\mathbb{E}[V / R] \le q

), implemented via the Benjamini-Hochberg (BH) step-up ranking procedure (

p_{(i)} \le \frac{i}{K} q

), dramatically boosting statistical power for exploratory multi-metric dashboards.

💡Use Cases

Multi-arm bandit/A/B/n feature launches, automated metric dashboard anomaly detection, and subgroup heterogeneity audits.

⚡Key Problems Solved

Testing 50 guardrail metrics guarantees spurious false discoveries without correction; FWER/FDR corrections eliminate false-alarm noise while preserving discovery power.

🎯5 High-Frequency Exam Points

Contrast Bonferroni, Holm-Bonferroni, and Benjamini-Hochberg (BH) across ranking algorithms, statistical conservatism, and Type-II error rates?

Write the complete algorithm for Benjamini-Hochberg step-up procedure determining the cutoff index

k^*

such that

p_{(k^*)} \le \frac{k^*}{K} q

Map FWER vs FDR to business scenarios: mission-critical GMV safety vs marketing tag discovery vs multi-armed bandits?

Explain how post-hoc subgroup cherry-picking triggers both multiple testing bias and Simpson's Paradox?

Explain how Benjamini-Yekutieli (BY) generalizes FDR control under arbitrary positive dependence and correlation across metrics?

🔗Foundational Prerequisite Cards (Click to Review)

🎓[RS]Bonferroni Correction→📐[Math]Hypothesis Testing→

📖 In-depth Guide:📄 ds-statistics-and-experiment-design →

Updated 2026-08-14

🎯

Test Your Knowledge: Practice Questions for "Multiple Comparisons: FWER vs FDR-BH"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardP-hacking, Peeking & mSPRT Sequential Next Card →Bootstrap Resampling & Empirical CI

🔗 More Data Scientist Knowledge Cards

Hypothesis Testing, Type I/II & Power Sample Size Derivation & MDE Budget CUPED Variance Reduction Derivation SRM Detection & Chi-Square Diagnostics