M1-029M1: Mathematics & Statistics FundamentalsConvex Optimization & KKTMedium
⚡Flashcard Drill➔Mastery:
Explain Geometrically and Algebraically Why L1 Regularization Promotes Sparsity While L2 Does Not
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Geometrically, L1 contour balls have sharp non-smooth corners on coordinate axes where loss contours touch first; algebraically, L1 subgradient produces a constant shrinkage threshold, setting small weights exactly to zero.
📌 Key Takeaways
- •Geometric perspective: The L1 norm ball is a diamond/cross-polytope with vertices on coordinate axes where .
- •Algebraic perspective: Soft-thresholding operator truncates weights with magnitude to exact zero.
- •L2 (Ridge) shrinks weights proportionally by factor , asymptotically approaching zero without ever reaching it.
📐 Mathematical Derivations
For an orthogonal design matrix, the 1D objective is . The subdifferential is , where if and if . Solving gives the soft-thresholding solution: if ; if ; and if . In contrast, for L2 objective , setting derivative to zero gives , which is a smooth linear scaling that equals zero only when empirical estimate .
🏭 Production Trade-offs
L1 (Lasso) performs automatic feature selection, producing highly interpretable models and drastically reducing memory during inference by pruning dead features. However, under high multicollinearity, L1 arbitrarily selects one feature from a correlated group and discards the rest. Elastic Net blends L1 and L2 penalties: , combining sparsity with group selection stability.
⚠️ Common Interview Pitfalls
- ✕Using standard gradient descent on L1 objectives (the gradient at is undefined, causing oscillations around zero; proximal gradient must be used).
- ✕Assuming L2 weight decay sets small weights to absolute zero.
🎯 Interviewer Follow-up Inquiries
- ?How does the Bayesian perspective interpret L1 as a Laplace prior and L2 as a Gaussian prior?
- ?Why does L0 regularization produce the ultimate sparse solution, and why is it NP-hard to optimize?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚