M2-011M2: Classical Machine LearningRegularization (L1 / L2)Easy
⚡Flashcard Drill➔Mastery:
Compare the Geometric and Functional Differences Between L1, L2, and ElasticNet Regularization
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
L1 (Lasso) promotes sparsity by shrinking coefficients to exact zero on diamond corners; L2 (Ridge) shrinks coefficients smoothly without sparsity; ElasticNet combines both, grouping correlated features while maintaining sparsity.
📌 Key Takeaways
- •Constraint Ball Geometry: L1 ball has sharp vertices on axes where ; L2 ball is smooth and spherical.
- •L1 (Lasso): Automatic feature selection, sparse models; under collinearity, arbitrarily selects one feature from a correlated cluster.
- •L2 (Ridge): Solves multicollinearity, distributes weights evenly among correlated features; no sparsity.
- •ElasticNet: ; strictly convex, retains group selection property.
📐 Mathematical Derivations
Analysis of ElasticNet grouping effect: Let features and be highly correlated (). For strictly convex ElasticNet with , Zou & Hastie (2005) proved that the difference in estimated coefficients satisfies: . As correlation , , forcing correlated features to receive nearly identical coefficients. In pure Lasso (), this bound does not exist, causing one coefficient to take the entire weight while the other is set to zero arbitrarily.
🏭 Production Trade-offs
When to choose which: (1) L1: High-dimensional sparse feature regimes (e.g. 1 million text n-grams or gene expressions) where 99% of features are noise. (2) L2: Dense numerical signals where all features contribute small predictive value and collinearity must be controlled. (3) ElasticNet: High-dimensional data with multicollinear feature clusters (), achieving both sparsity and group stability.
⚠️ Common Interview Pitfalls
- ✕Using pure Lasso when (Lasso can select at most non-zero features before saturating; ElasticNet can select all features).
- ✕Forgetting to scale/normalize features prior to applying L1/L2 penalties.
🎯 Interviewer Follow-up Inquiries
- ?Why can Lasso select at most features when , and how does ElasticNet overcome this limit?
- ?How does the L1/2 norm compare to L1, and why is it non-convex?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚