Regularization Geometry

Regularization is equivalent to adding a penalty or constraining the solution: $\min_w L(w)$ s.t. $\Vert w\Vert_p \le t$. L2 (ridge): penalty $\frac{\lambda}{2}\Vert w\Vert_2^2$, the constraint set is a ball; the loss contours (data-determined ellipses) touch the sphere at a point where all coordinates are uniformly shrunk but generally nonzero. Closed form: $w^* = (X^TX + \lambda I)^{-1}X^Ty$; via SVD $X = U\Sigma V^T$ this becomes $w^* = V(\Sigma^T\Sigma + \lambda I)^{-1}\Sigma^TU^Ty$, i.e. each eigen-direction is scaled by $\frac{\sigma_i^2}{\sigma_i^2 + \lambda}$ — directions with small singular values are shrunk most, and $X^TX + \lambda I$ is always invertible. L1 (lasso): the constraint set $\Vert w\Vert_1 \le t$ is a diamond (hypercube in higher dims) whose vertices lie exactly on the coordinate axes; the loss ellipse often touches the diamond at a vertex or edge → some coordinates are exactly zero → sparse solutions.

high-dimensional feature selection (L1), stabilizing ill-conditioned/collinear $X^TX$ inversion (L2), and interview questions on why L1 gives sparse solutions (geometric and Bayesian views).

in the Bayesian view, regularization is MAP with a prior. L2 corresponds to a Gaussian prior $w \sim \mathcal{N}(0, \tau^2 I)$ with $\lambda = \sigma^2/\tau^2$: $\hat w_{MAP} = \arg\max_w -\frac{1}{2\sigma^2}\Vert y - Xw\Vert^2 - \frac{1}{2\tau^2}\Vert w\Vert^2$ — the smooth Gaussian tail shrinks all weights but keeps them nonzero. L1 corresponds to a Laplace prior $p(w) \propto e^{-\lambda|w|}$, whose density has a sharp peak at 0 concentrating probability mass near zero → the posterior mode pushes weights to exactly 0. Regularization reduces variance (fights overfitting) and makes the solution unique, at the cost of introducing bias.

Test Your Knowledge: Practice Questions for "Regularization Geometry"

Single choice pitfall questions with instant feedback and mistake tracking.