Regularization Geometry

Regularization is equivalent to adding a penalty or constraining the solution: $\min_w L(w)$ s.t. $\Vert w\Vert_p \le t$. L2 (ridge): penalty $\frac{\lambda}{2}\Vert w\Vert_2^2$, the constraint set is a ball; the loss contours (data-determined ellipses) touch the sphere at a point where all coordinates are uniformly shrunk but generally nonzero. Closed form: $w^* = (X^TX + \lambda I)^{-1}X^Ty$; via SVD $X = U\Sigma V^T$ this becomes $w^* = V(\Sigma^T\Sigma + \lambda I)^{-1}\Sigma^TU^Ty$, i.e. each eigen-direction is scaled by $\frac{\sigma_i^2}{\sigma_i^2 + \lambda}$ — directions with small singular values are shrunk most, and $X^TX + \lambda I$ is always invertible. L1 (lasso): the constraint set $\Vert w\Vert_1 \le t$ is a diamond (hypercube in higher dims) whose vertices lie exactly on the coordinate axes; the loss ellipse often touches the diamond at a vertex or edge → some coordinates are exactly zero → sparse solutions.

high-dimensional feature selection (L1), stabilizing ill-conditioned/collinear $X^TX$ inversion (L2), and interview questions on why L1 gives sparse solutions (geometric and Bayesian views).

in the Bayesian view, regularization is MAP with a prior. L2 corresponds to a Gaussian prior $w \sim \mathcal{N}(0, \tau^2 I)$ with $\lambda = \sigma^2/\tau^2$: $\hat w_{MAP} = \arg\max_w -\frac{1}{2\sigma^2}\Vert y - Xw\Vert^2 - \frac{1}{2\tau^2}\Vert w\Vert^2$ — the smooth Gaussian tail shrinks all weights but keeps them nonzero. L1 corresponds to a Laplace prior $p(w) \propto e^{-\lambda|w|}$, whose density has a sharp peak at 0 concentrating probability mass near zero → the posterior mode pushes weights to exactly 0. Regularization reduces variance (fights overfitting) and makes the solution unique, at the cost of introducing bias.

Draw the L1 diamond and L2 ball; explain why contour tangency at diamond vertices yields sparse solutions while ball tangencies are generically off-axis.

Derive the ridge closed form $w = (X^TX+\lambda I)^{-1}X^Ty$ and use SVD to analyze the per-direction shrinkage $\sigma_i^2/(\sigma_i^2+\lambda)$.

Bayesian view: write the log-posterior under Gaussian/Laplace priors and show L2 = Gaussian-prior MAP, L1 = Laplace-prior MAP.

Why can L1 land exactly on non-differentiable corners? Write and explain the lasso subgradient condition $0 \in X^T(Xw-y) + \lambda\partial\Vert w\Vert_1$.

How does the regularization strength $\lambda$ trade off bias vs variance? Why is regularization described as trading bias for variance?