Bias-Variance Decomposition

The bias-variance decomposition splits expected squared test error into three terms: $\text{EGE} = \text{Bias}^2 + \text{Var} + \sigma^2$. Assume the true relation $y = f^*(x) + \epsilon$ with $\mathbb{E}[\epsilon] = 0$, $\text{Var}(\epsilon) = \sigma^2$, and $\epsilon$ independent of the input; the model $\hat f$ is fit on a random training set $D$, and $\bar f(x) = \mathbb{E}_D[\hat f(x)]$. Full proof (for fixed $x$): rewrite $y - \hat f = \epsilon + (f^* - \hat f)$ and expand: $\mathbb{E}[(y - \hat f)^2] = \mathbb{E}[\epsilon^2] + 2\mathbb{E}[\epsilon(f^* - \hat f)] + \mathbb{E}[(f^* - \hat f)^2]$. The cross term vanishes because $\epsilon$ is independent of the training set $D$ (hence of $\hat f$) and $\mathbb{E}[\epsilon] = 0$: $2\mathbb{E}[\epsilon(f^* - \hat f)] = 2\mathbb{E}[\epsilon]\mathbb{E}[f^* - \hat f] = 0$. For the third term, write $f^* - \hat f = (f^* - \bar f) + (\bar f - \hat f)$; under $\mathbb{E}_D$ the cross term $2(f^* - \bar f)\mathbb{E}_D[\bar f - \hat f] = 2(f^* - \bar f)(\bar f - \bar f) = 0$, so $\mathbb{E}_D[(f^* - \hat f)^2] = (f^* - \bar f)^2 + \mathbb{E}_D[(\hat f - \bar f)^2] = \text{Bias}^2 + \text{Var}$. Combining: $\text{EGE} = \text{Bias}^2 + \text{Var} + \sigma^2$, where $\sigma^2$ is irreducible noise that no model can remove.

the standard framework for interview questions on why complex models overfit/underfit: more complexity lowers bias (hypothesis space covers the truth) but raises variance (more sensitive to the training set); it also explains bagging (reduces variance), boosting (reduces bias) and regularization (raises bias, lowers variance).

a quantitative diagnosis for capacity selection: low training error with high test error → variance-dominated → add data / regularization / bagging; high error on both → bias-dominated → add capacity / features. Notes: the decomposition holds per fixed $x$ and is typically averaged over the input distribution; it does not hold for classification under 0-1 loss, where decomposable alternatives like the Brier score are used.

Whiteboard-prove $\mathbb{E}[(y-\hat f)^2] = \text{Bias}^2 + \text{Var} + \sigma^2$: show the expansion and both cross-term cancellations ($\mathbb{E}[\epsilon] = 0$ and $\mathbb{E}_D[\hat f] = \bar f$).

Why is the cross term $2\mathbb{E}[\epsilon(f^* - \hat f)]$ zero? Which independence/expectation assumptions does it rely on? Why is $\sigma^2$ irreducible?

As model complexity grows, how do Bias and Var change? Which term dominates in underfitting vs overfitting?

Explain via the decomposition: why does bagging mainly reduce variance, and boosting mainly reduce bias?

Why does the decomposition fail for classification with 0-1 loss? What alternative (e.g. Brier score) can be used?