M2-096M2: Classical Machine LearningRegularization (L1 / L2)Medium
⚡Flashcard Drill➔Mastery:

Bayesian Interpretation of L1 and L2 Regularization

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-097 →
📐 Mathematical Definition
p(w)=N(0,τ2I)⇒L2;p(w)∝e−∣w∣/b⇒L1p(w)=\mathcal N(0,\tau^2I)\Rightarrow \text{L2};\qquad p(w)\propto e^{-|w|/b}\Rightarrow \text{L1}
⚡ One-Sentence Core Answer
L2 regularization corresponds to a Gaussian prior (shrinkage); L1 regularization corresponds to an independent Laplace prior (sparsity via its non-differentiable peak at zero).

📌 Key Takeaways

  • •
    MAP framework: arg⁡max⁡w[log⁡P(D∣w)+log⁡P(w)]\arg\max_w [\log P(D|w) + \log P(w)]
  • •
    L2 / Ridge: Gaussian prior wj∼N(0,τ2)  ⟹  −12τ2∥w∥22w_j \sim \mathcal{N}(0, \tau^2) \implies -\frac{1}{2\tau^2} \|w\|_2^2
  • •
    L1 / Lasso: Laplace prior wj∼Laplace(0,b)  ⟹  −1b∥w∥1w_j \sim \text{Laplace}(0, b) \implies -\frac{1}{b} \|w\|_1, whose sharp peak at zero forces coefficients exactly to zero

📐 Mathematical Derivations

Maximum A Posteriori (MAP) Derivation: w^MAP=arg⁡max⁡w[∑i=1nlog⁡P(yi∣xi,w)+log⁡P(w)]\hat{w}_{\text{MAP}} = \arg\max_w \left[ \sum_{i=1}^n \log P(y_i | x_i, w) + \log P(w) \right].
① Gaussian Prior $\implies$ L2: If wj∼N(0,σ02)w_j \sim \mathcal{N}(0, \sigma_0^2), log⁡P(w)=−12σ02∑jwj2−C\log P(w) = -\frac{1}{2\sigma_0^2} \sum_j w_j^2 - C. The MAP objective is: min⁡wL(w)+σ2σ02∥w∥22\min_w \mathcal{L}(w) + \frac{\sigma^2}{\sigma_0^2} \|w\|_2^2. Regularization weight λ=σ2/σ02\lambda = \sigma^2 / \sigma_0^2.
② Laplace Prior $\implies$ L1: If wj∼Laplace(0,b)w_j \sim \text{Laplace}(0, b), P(wj)=12bexp⁡(−∣wj∣b)P(w_j) = \frac{1}{2b} \exp\left(-\frac{|w_j|}{b}\right), and log⁡P(w)=−1b∥w∥1−C\log P(w) = -\frac{1}{b} \|w\|_1 - C. The MAP objective is: min⁡wL(w)+σ2b∥w∥1\min_w \mathcal{L}(w) + \frac{\sigma^2}{b} \|w\|_1.
Why Laplace induces exact sparsity: The log-prior density −1b∣wj∣-\frac{1}{b}|w_j| has a non-zero derivative ±1/b\pm 1/b everywhere including as wj→0w_j \to 0. When data likelihood gradient at zero ∣∇L∣<1/b|\nabla \mathcal{L}| < 1/b, the posterior mode is locked precisely at the non-differentiable corner wj=0w_j = 0.

🏭 Production Trade-offs

Posterior inference distinction: MAP gives a single point estimate (the posterior mode). In full Bayesian regression, the posterior mean under an L1 prior is actually non-sparse; true sparsity is a property of the mode (MAP), not the mean.
⚠️ Common Interview Pitfalls
  • ✕
    Claiming that a full Bayesian posterior under a Laplace prior produces sparse sample paths, when only the MAP point estimate is sparse
  • ✕
    Assuming L1 and L2 priors imply different data likelihood models, rather than purely different prior distributions on ww
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the posterior mean under a Laplace prior non-sparse while the MAP estimate is sparse?
  • ?
    What prior distribution corresponds to ElasticNet regularization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-095: Relationship Between Logistic Regression and the Maximum Entropy Model📋 Back to BankNext →M2-097: Why Weight Decay Must Be Decoupled in Adam (AdamW)