Bootstrap

Bootstrap estimates the sampling distribution of a statistic $\hat\theta = g(x_1, \dots, x_n)$ by resampling with replacement: independently $B$ times, draw $n$ observations from the original sample and compute $\hat\theta^{*b}$ ($b = 1, \dots, B$); the empirical distribution of these $B$ values approximates the distribution of $\hat\theta$. Variance and standard error estimate:

and the percentile interval takes quantiles of the resampled distribution $\bigl[\hat\theta^{*}_{(\alpha/2)},\; \hat\theta^{*}_{(1-\alpha/2)}\bigr]$. The statistical rationale: the empirical distribution $\hat F_n$ approximates the true $F$, so resampling is "re-running the experiment" under $\hat F_n$.

statistics without analytic standard errors (median, correlation), confidence intervals for small samples, and uncertainty of model metrics (AUC, RMSE); interviews often ask why bootstrap works and how it relates to the CLT.

analytic standard errors rely on the CLT, large samples and differentiable statistics, whereas bootstrap works for any statistic under exchangeable i.i.d. data, costing only $O(B n)$ resampling operations; its limits: resamples contain no out-of-sample information, and dependent data (time series) requires variants such as block bootstrap.

Full procedure and formula for estimating $\mathrm{Var}(\hat\theta)$ by bootstrap? Why is it "using the empirical distribution to approximate the true distribution"?

How is the percentile interval constructed? How does it differ from the normal approximation $\hat\theta \pm z_{\alpha/2} \widehat{\mathrm{SE}}$?

Bootstrap vs CLT: when are they equivalent, and when is bootstrap superior (no analytic SE, small samples)?

Parametric vs non-parametric bootstrap: what are the differences?

Assumptions and failure modes of bootstrap (independence, information ceiling of resamples, time series)?