M1-029M1: Mathematics & Statistics FundamentalsConvex Optimization & KKTMedium
⚡Flashcard Drill➔Mastery:

Explain Geometrically and Algebraically Why L1 Regularization Promotes Sparsity While L2 Does Not

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-030 →
📐 Mathematical Definition
min⁡∥y−Xw∥2+λ∥w∥1vsmin⁡∥y−Xw∥2+λ∥w∥22\min\|y-Xw\|^2+\lambda\|w\|_1\quad\text{vs}\quad \min\|y-Xw\|^2+\lambda\|w\|_2^2
⚡ One-Sentence Core Answer
Geometrically, L1 contour balls have sharp non-smooth corners on coordinate axes where loss contours touch first; algebraically, L1 subgradient produces a constant shrinkage threshold, setting small weights exactly to zero.

📌 Key Takeaways

  • •
    Geometric perspective: The L1 norm ball ∥w∥1≤C\|w\|_1 \le C is a diamond/cross-polytope with vertices on coordinate axes where wi=0w_i=0.
  • •
    Algebraic perspective: Soft-thresholding operator Sλ(w)=sign(w)max⁡(∣w∣−λ,0)S_\lambda(w) = \text{sign}(w)\max(|w|-\lambda, 0) truncates weights with magnitude <λ< \lambda to exact zero.
  • •
    L2 (Ridge) shrinks weights proportionally by factor 11+λ\frac{1}{1+\lambda}, asymptotically approaching zero without ever reaching it.

📐 Mathematical Derivations

For an orthogonal design matrix, the 1D objective is min⁡w12(w−w^)2+λ∣w∣\min_w \frac{1}{2}(w - \hat{w})^2 + \lambda |w|. The subdifferential is w−w^+λ∂∣w∣=0w - \hat{w} + \lambda \partial |w| = 0, where ∂∣w∣=sign(w)\partial |w| = \text{sign}(w) if w≠0w \ne 0 and [−1,1][-1, 1] if w=0w=0. Solving gives the soft-thresholding solution: w∗=w^−λw^* = \hat{w} - \lambda if w^>λ\hat{w} > \lambda; w∗=w^+λw^* = \hat{w} + \lambda if w^<−λ\hat{w} < -\lambda; and w∗=0w^* = 0 if ∣w^∣≤λ|\hat{w}| \le \lambda. In contrast, for L2 objective min⁡w12(w−w^)2+λ2w2\min_w \frac{1}{2}(w - \hat{w})^2 + \frac{\lambda}{2}w^2, setting derivative to zero gives w∗=w^1+λw^* = \frac{\hat{w}}{1+\lambda}, which is a smooth linear scaling that equals zero only when empirical estimate w^=0\hat{w}=0.

🏭 Production Trade-offs

L1 (Lasso) performs automatic feature selection, producing highly interpretable models and drastically reducing memory during inference by pruning dead features. However, under high multicollinearity, L1 arbitrarily selects one feature from a correlated group and discards the rest. Elastic Net blends L1 and L2 penalties: λ1∥w∥1+λ2∥w∥22\lambda_1 \|w\|_1 + \lambda_2 \|w\|_2^2, combining sparsity with group selection stability.
⚠️ Common Interview Pitfalls
  • ✕
    Using standard gradient descent on L1 objectives (the gradient at w=0w=0 is undefined, causing oscillations around zero; proximal gradient must be used).
  • ✕
    Assuming L2 weight decay sets small weights to absolute zero.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the Bayesian perspective interpret L1 as a Laplace prior and L2 as a Gaussian prior?
  • ?
    Why does L0 regularization produce the ultimate sparse solution, and why is it NP-hard to optimize?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-028: Define Strong Duality and Slater's Condition, and Explain the Practical Utility of Dual Formulations📋 Back to BankNext →M1-030: Explain Subgradients and Proximal Operators, and How Proximal Gradient Descent Optimizes Non-Smooth L1 Penalties