Bootstrap

Bootstrap estimates the sampling distribution of a statistic $\hat\theta = g(x_1, \dots, x_n)$ by resampling with replacement: independently $B$ times, draw $n$ observations from the original sample and compute $\hat\theta^{*b}$ ($b = 1, \dots, B$); the empirical distribution of these $B$ values approximates the distribution of $\hat\theta$. Variance and standard error estimate:

and the percentile interval takes quantiles of the resampled distribution $\bigl[\hat\theta^{*}_{(\alpha/2)},\; \hat\theta^{*}_{(1-\alpha/2)}\bigr]$. The statistical rationale: the empirical distribution $\hat F_n$ approximates the true $F$, so resampling is "re-running the experiment" under $\hat F_n$.

statistics without analytic standard errors (median, correlation), confidence intervals for small samples, and uncertainty of model metrics (AUC, RMSE); interviews often ask why bootstrap works and how it relates to the CLT.

analytic standard errors rely on the CLT, large samples and differentiable statistics, whereas bootstrap works for any statistic under exchangeable i.i.d. data, costing only $O(B n)$ resampling operations; its limits: resamples contain no out-of-sample information, and dependent data (time series) requires variants such as block bootstrap.

Test Your Knowledge: Practice Questions for "Bootstrap"

Single choice pitfall questions with instant feedback and mistake tracking.