M2-011M2: Classical Machine LearningRegularization (L1 / L2)Easy
⚡Flashcard Drill➔Mastery:

Compare the Geometric and Functional Differences Between L1, L2, and ElasticNet Regularization

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-012 →
📐 Mathematical Definition
L1:λ∥w∥1,L2:λ∥w∥22,EN:λ1∥w∥1+λ2∥w∥22L1:\lambda\|w\|_1,\quad L2:\lambda\|w\|_2^2,\quad EN:\lambda_1\|w\|_1+\lambda_2\|w\|_2^2
⚡ One-Sentence Core Answer
L1 (Lasso) promotes sparsity by shrinking coefficients to exact zero on diamond corners; L2 (Ridge) shrinks coefficients smoothly without sparsity; ElasticNet combines both, grouping correlated features while maintaining sparsity.

📌 Key Takeaways

  • •
    Constraint Ball Geometry: L1 ball ∥w∥1≤C\|w\|_1 \le C has sharp vertices on axes where wi=0w_i=0; L2 ball ∥w∥22≤C\|w\|_2^2 \le C is smooth and spherical.
  • •
    L1 (Lasso): Automatic feature selection, sparse models; under collinearity, arbitrarily selects one feature from a correlated cluster.
  • •
    L2 (Ridge): Solves multicollinearity, distributes weights evenly among correlated features; no sparsity.
  • •
    ElasticNet: L=Loss+λ1∥w∥1+λ22∥w∥22L = \text{Loss} + \lambda_1 \|w\|_1 + \frac{\lambda_2}{2}\|w\|_2^2; strictly convex, retains group selection property.

📐 Mathematical Derivations

Analysis of ElasticNet grouping effect: Let features xix_i and xjx_j be highly correlated (r=xiTxjr = x_i^T x_j). For strictly convex ElasticNet with λ2>0\lambda_2 > 0, Zou & Hastie (2005) proved that the difference in estimated coefficients satisfies: ∥β^i−β^j∥≤∥y∥2λ22(1−r)\|\hat{\beta}_i - \hat{\beta}_j\| \le \frac{\|y\|_2}{\lambda_2} \sqrt{2(1 - r)}. As correlation r→1r \to 1, ∥β^i−β^j∥→0\|\hat{\beta}_i - \hat{\beta}_j\| \to 0, forcing correlated features to receive nearly identical coefficients. In pure Lasso (λ2=0\lambda_2 = 0), this bound does not exist, causing one coefficient to take the entire weight while the other is set to zero arbitrarily.

🏭 Production Trade-offs

When to choose which: (1) L1: High-dimensional sparse feature regimes (e.g. 1 million text n-grams or gene expressions) where 99% of features are noise. (2) L2: Dense numerical signals where all features contribute small predictive value and collinearity must be controlled. (3) ElasticNet: High-dimensional data with multicollinear feature clusters (p≫np \gg n), achieving both sparsity and group stability.
⚠️ Common Interview Pitfalls
  • ✕
    Using pure Lasso when p>np > n (Lasso can select at most nn non-zero features before saturating; ElasticNet can select all pp features).
  • ✕
    Forgetting to scale/normalize features prior to applying L1/L2 penalties.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why can Lasso select at most nn features when p>np > n, and how does ElasticNet overcome this limit?
  • ?
    How does the L1/2 norm compare to L1, and why is it non-convex?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-010: Explain Complete Separation (Hauck-Donner Effect) in Logistic Regression and Standard Industrial Solutions📋 Back to BankNext →M2-012: Explain Hyperparameter Selection for Regularization Strength Lambda and Diagnostic Extremes