M2-004M2: Classical Machine LearningLinear RegressionMedium
⚡Flashcard Drill➔Mastery:

Explain Why Ridge Regression Mitigates Multicollinearity and Derive Its Closed-Form Solution

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-005 →
📐 Mathematical Definition
w^ridge=(X⊤X+λI)−1X⊤y\hat w_{ridge}=(X^\top X+\lambda I)^{-1}X^\top y
⚡ One-Sentence Core Answer
Ridge regression adds an L2 penalty λ∥w∥22\lambda \|w\|_2^2, modifying the normal equations to β^Ridge=(XTX+λI)−1XTy\hat{\beta}_{\text{Ridge}} = (X^T X + \lambda I)^{-1} X^T y; adding λ>0\lambda > 0 strictly shifts all eigenvalues upward, guaranteeing invertibility.

📌 Key Takeaways

  • •
    Closed-form solution: β^Ridge=(XTX+λI)−1XTy\hat{\beta}_{\text{Ridge}} = (X^T X + \lambda I)^{-1} X^T y.
  • •
    Guaranteed Invertibility: If XTXX^T X has eigenvalues σi2≥0\sigma_i^2 \ge 0, then XTX+λIX^T X + \lambda I has eigenvalues σi2+λ≥λ>0\sigma_i^2 + \lambda \ge \lambda > 0, guaranteeing positive definiteness.
  • •
    Condition Number Reduction: Drops condition number from σmax⁡2σmin⁡2\frac{\sigma_{\max}^2}{\sigma_{\min}^2} down to σmax⁡2+λσmin⁡2+λ\frac{\sigma_{\max}^2 + \lambda}{\sigma_{\min}^2 + \lambda}, stabilizing numerical inversion.

📐 Mathematical Derivations

The Ridge objective is L(β)=12∥y−Xβ∥22+λ2∥β∥22L(\beta) = \frac{1}{2}\|y - X\beta\|_2^2 + \frac{\lambda}{2}\|\beta\|_2^2. Taking the gradient: ∇βL(β)=−XT(y−Xβ)+λβ=(XTX+λI)β−XTy\nabla_\beta L(\beta) = -X^T (y - X\beta) + \lambda \beta = (X^T X + \lambda I)\beta - X^T y. Setting ∇βL=0\nabla_\beta L = 0 yields the Ridge normal equations: (XTX+λI)β=XTy(X^T X + \lambda I)\beta = X^T y. Since XTXX^T X is symmetric positive semi-definite (vTXTXv=∥Xv∥2≥0v^T X^T X v = \|Xv\|^2 \ge 0), adding λI\lambda I for any λ>0\lambda > 0 makes the matrix strictly positive definite: vT(XTX+λI)v=∥Xv∥2+λ∥v∥2>0v^T (X^T X + \lambda I)v = \|Xv\|^2 + \lambda \|v\|^2 > 0 for all v≠0v \ne 0. Thus (XTX+λI)−1(X^T X + \lambda I)^{-1} exists unconditionally, even when N<dN < d.

🏭 Production Trade-offs

Bias-Variance Tradeoff analysis: β^Ridge=(XTX+λI)−1XTXβtrue\hat{\beta}_{\text{Ridge}} = (X^T X + \lambda I)^{-1} X^T X \beta_{\text{true}}, which introduces deliberate shrinkage bias. However, the variance is Var(β^Ridge)=σ2(XTX+λI)−1XTX(XTX+λI)−1\text{Var}(\hat{\beta}_{\text{Ridge}}) = \sigma^2 (X^T X + \lambda I)^{-1} X^T X (X^T X + \lambda I)^{-1}. By the Hoerl & Kennard (1970) theorem, there always exists a λ>0\lambda > 0 such that the reduction in variance strictly exceeds the squared bias, yielding a lower Mean Squared Error than unbiased OLS.
⚠️ Common Interview Pitfalls
  • ✕
    Applying Ridge regularization without standardizing features (unstandardized features with larger scales are unfairly penalized less).
  • ✕
    Regularizing the intercept term β0\beta_0 (the intercept should never be penalized; center yy and XX instead).
🎯 Interviewer Follow-up Inquiries
  • ?
    How does SVD express Ridge regression as singular value shrinkage factors σi2σi2+λ\frac{\sigma_i^2}{\sigma_i^2 + \lambda}?
  • ?
    Why is Ridge regression equivalent to MAP estimation under an isotropic Gaussian prior?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-003: Explain Multicollinearity, Its Impact on Coefficient Stability and Significance Tests, and Diagnostic Thresholds📋 Back to BankNext →M2-005: Derive the Least Squares Solution for Linear Regression and Explain Its Orthogonal Projection Geometry