Multiple Comparisons: FWER vs FDR-BH

The Multiple Comparisons in Experimentation (FWER vs FDR-BH Corrections) framework formalizes decision protocols when evaluating multi-arm variants (A/B/n testing), multi-metric dashboards, or multidimensional subgroup drilldowns; 2 governing statistical philosophies: 1) Family-Wise Error Rate (FWER): strictly controlling the probability of committing even a single Type-I error across all $K$ tests ($P(V \ge 1) \le \alpha$), implemented via conservative Bonferroni ($\alpha' = \alpha / K$) or step-down Holm-Bonferroni ($\alpha_i = \alpha / (K - i + 1)$), mandatory for mission-critical core revenue decisions; 2) False Discovery Rate (FDR): controlling the expected proportion of false discoveries among all rejected nulls ($\mathbb{E}[V / R] \le q$), implemented via the Benjamini-Hochberg (BH) step-up ranking procedure ($p_{(i)} \le \frac{i}{K} q$), dramatically boosting statistical power for exploratory multi-metric dashboards.

Multi-arm bandit/A/B/n feature launches, automated metric dashboard anomaly detection, and subgroup heterogeneity audits.

Testing 50 guardrail metrics guarantees spurious false discoveries without correction; FWER/FDR corrections eliminate false-alarm noise while preserving discovery power.

Contrast Bonferroni, Holm-Bonferroni, and Benjamini-Hochberg (BH) across ranking algorithms, statistical conservatism, and Type-II error rates?

Write the complete algorithm for Benjamini-Hochberg step-up procedure determining the cutoff index $k^*$ such that $p_{(k^*)} \le \frac{k^*}{K} q$?

Map FWER vs FDR to business scenarios: mission-critical GMV safety vs marketing tag discovery vs multi-armed bandits?

Explain how post-hoc subgroup cherry-picking triggers both multiple testing bias and Simpson's Paradox?

Explain how Benjamini-Yekutieli (BY) generalizes FDR control under arbitrary positive dependence and correlation across metrics?