Hypothesis Testing

Hypothesis testing makes a binary decision between the null $H_0$ and the alternative $H_1$: reject $H_0$ when the test statistic falls in the rejection region. Two error types: Type I (falsely rejecting $H_0$) has probability $\alpha$, Type II (missing a real effect) has probability $\beta$, and power $= 1-\beta$; the p-value is the probability of observing the current or more extreme result under $H_0$ — not the probability that $H_0$ is true.

significance in A/B testing, feature screening, paper claims; interviews typically probe the $\alpha$/$\beta$ trade-off, p-value misconceptions, and Bonferroni vs FDR-BH corrections under multiple testing.

running $k$ independent tests inflates false positives — the family-wise error rate approaches $1-(1-\alpha)^k \approx k\alpha$; Bonferroni lowers each threshold to $\alpha' = \alpha/k$, strictly bounding the family-wise error at $\alpha$ but losing power;FDR-BH instead controls the false discovery rate $\text{FDR} = \mathbb{E}[V/R]$ (the fraction of false rejections), which keeps far more power when $k$ is in the thousands, e.g. genomics or feature screening.

Test Your Knowledge: Practice Questions for "Hypothesis Testing"

Single choice pitfall questions with instant feedback and mistake tracking.