M2-096M2: Classical Machine LearningRegularization (L1 / L2)Medium
⚡Flashcard Drill➔Mastery:
Bayesian Interpretation of L1 and L2 Regularization
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
L2 regularization corresponds to a Gaussian prior (shrinkage); L1 regularization corresponds to an independent Laplace prior (sparsity via its non-differentiable peak at zero).
📌 Key Takeaways
- •MAP framework:
- •L2 / Ridge: Gaussian prior
- •L1 / Lasso: Laplace prior , whose sharp peak at zero forces coefficients exactly to zero
📐 Mathematical Derivations
Maximum A Posteriori (MAP) Derivation: .
① Gaussian Prior $\implies$ L2: If , . The MAP objective is: . Regularization weight .
② Laplace Prior $\implies$ L1: If , , and . The MAP objective is: .
Why Laplace induces exact sparsity: The log-prior density has a non-zero derivative everywhere including as . When data likelihood gradient at zero , the posterior mode is locked precisely at the non-differentiable corner .
① Gaussian Prior $\implies$ L2: If , . The MAP objective is: . Regularization weight .
② Laplace Prior $\implies$ L1: If , , and . The MAP objective is: .
Why Laplace induces exact sparsity: The log-prior density has a non-zero derivative everywhere including as . When data likelihood gradient at zero , the posterior mode is locked precisely at the non-differentiable corner .
🏭 Production Trade-offs
Posterior inference distinction: MAP gives a single point estimate (the posterior mode). In full Bayesian regression, the posterior mean under an L1 prior is actually non-sparse; true sparsity is a property of the mode (MAP), not the mean.
⚠️ Common Interview Pitfalls
- ✕Claiming that a full Bayesian posterior under a Laplace prior produces sparse sample paths, when only the MAP point estimate is sparse
- ✕Assuming L1 and L2 priors imply different data likelihood models, rather than purely different prior distributions on
🎯 Interviewer Follow-up Inquiries
- ?Why is the posterior mean under a Laplace prior non-sparse while the MAP estimate is sparse?
- ?What prior distribution corresponds to ElasticNet regularization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚