Hypothesis Testing

Hypothesis testing makes a binary decision between the null $H_0$ and the alternative $H_1$: reject $H_0$ when the test statistic falls in the rejection region. Two error types: Type I (falsely rejecting $H_0$) has probability $\alpha$, Type II (missing a real effect) has probability $\beta$, and power $= 1-\beta$; the p-value is the probability of observing the current or more extreme result under $H_0$ — not the probability that $H_0$ is true.

significance in A/B testing, feature screening, paper claims; interviews typically probe the $\alpha$/$\beta$ trade-off, p-value misconceptions, and Bonferroni vs FDR-BH corrections under multiple testing.

running $k$ independent tests inflates false positives — the family-wise error rate approaches $1-(1-\alpha)^k \approx k\alpha$; Bonferroni lowers each threshold to $\alpha' = \alpha/k$, strictly bounding the family-wise error at $\alpha$ but losing power;FDR-BH instead controls the false discovery rate $\text{FDR} = \mathbb{E}[V/R]$ (the fraction of false rejections), which keeps far more power when $k$ is in the thousands, e.g. genomics or feature screening.

Define Type I error $\alpha$, Type II error $\beta$ and power $1-\beta$. Why does lowering $\alpha$ raise $\beta$?

Precise definition of a p-value? Why is it not the probability that $H_0$ is true?

How does the family-wise error accumulate over $k$ tests? Principle and cost of Bonferroni $\alpha'=\alpha/k$?

FWER vs FDR? The concrete steps of the Benjamini–Hochberg procedure?

How do $\alpha$, effect size and sample size affect power? How to do power analysis / sample-size planning?