M2-096M2: Classical Machine LearningRegularization (L1 / L2)Medium
⚡Flashcard Drill➔Mastery:
Bayesian Interpretation of L1 and L2 Regularization
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
L2 regularization corresponds to a Gaussian prior (shrinkage); L1 regularization corresponds to an independent Laplace prior (sparsity via its non-differentiable peak at zero).
📌 Key Takeaways
- •MAP framework:
- •L2 / Ridge: Gaussian prior
- •L1 / Lasso: Laplace prior , whose sharp peak at zero forces coefficients exactly to zero
📐 Mathematical Derivations
Maximum A Posteriori (MAP) Derivation: .
① Gaussian Prior $\implies$ L2: If , . The MAP objective is: . Regularization weight .
② Laplace Prior $\implies$ L1: If , , and . The MAP objective is: .
Why Laplace induces exact sparsity: The log-prior density has a non-zero derivative everywhere including as . When data likelihood gradient at zero , the posterior mode is locked precisely at the non-differentiable corner .
① Gaussian Prior $\implies$ L2: If , . The MAP objective is: . Regularization weight .
② Laplace Prior $\implies$ L1: If , , and . The MAP objective is: .
Why Laplace induces exact sparsity: The log-prior density has a non-zero derivative everywhere including as . When data likelihood gradient at zero , the posterior mode is locked precisely at the non-differentiable corner .
🏭 Production Trade-offs
Posterior inference distinction: MAP gives a single point estimate (the posterior mode). In full Bayesian regression, the posterior mean under an L1 prior is actually non-sparse; true sparsity is a property of the mode (MAP), not the mean.
⚠️ Common Interview Pitfalls
- ✕Claiming that a full Bayesian posterior under a Laplace prior produces sparse sample paths, when only the MAP point estimate is sparse
- ✕Assuming L1 and L2 priors imply different data likelihood models, rather than purely different prior distributions on
🎯 Interviewer Follow-up Inquiries
- ?Why is the posterior mean under a Laplace prior non-sparse while the MAP estimate is sparse?
- ?What prior distribution corresponds to ElasticNet regularization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚
Associated Knowledge Base Guides & Mindmaps
中文速览与面试要点
解释 L1 与 L2 正则化的贝叶斯解释
L2 = 高斯先验下的 MAP;L1 = Laplace 先验下的 MAP;正则强度 = 先验方差的倒数。
核心要点
- 推导:MAP 估计最大化 log p(D|w)+log p(w)。① 高斯先验 p(w)=N(0,τ²I) → log p(w)=−‖w‖²/(2τ²)+const → 目标变为 log 似然 − λ‖w‖²,其中 λ=1/(2τ²),即 L2 正则(岭回归/权重衰减) ;② Laplace 先验 p(w)∝exp(−|w|/b) → log p(w)=−‖w‖₁/b → 目标变为 log 似然 − λ‖w‖₁,即 L1 正则(LASSO) 。 为什么 Laplace 导致稀疏 :Laplace 分布在 w=0 处有 尖峰(不可导的角点) ,其对数先验在该点的'梯度'有跳变;当似然项对某系数的'推力'不够强时,后验众数恰好落在角点上(w=0)。相比之下高斯先验在 0 处平滑(导数为 0),后验众数只会被'收缩'但不会恰好为零。这给出了 L1 稀疏性的 概率解释 (与几何解释——菱形可行域的顶点——相互印证)。
- 这一视角的实践价值:① 正则强度 = 先验信念强度 ——λ 大对应先验方差小(强信念'系数应接近 0'),λ 小对应弱先验趋近 MLE;这解释了为什么数据量增加时最优 λ 应减小(先验被数据稀释)。② 选择先验的依据 ——若认为'多数特征确实无关'(稀疏真值),用 Laplace(L1);若认为'所有特征都有小贡献',用高斯(L2);若不确定,用 ElasticNet(混合先验)或层次先验(Horseshoe 先验,自适应稀疏)。③ 完整贝叶斯 vs MAP ——MAP 只取后验众数(丢弃不确定性);完整贝叶斯需对后验积分(如贝叶斯线性回归的后验预测分布),能得到预测的不确定性估计,但计算更贵。④ 对超参的先验 ——λ 本身也可加先验(层次贝叶斯),用经验贝叶斯或 MCMC 自动确定,避免交叉验证的成本。⑤ 与深度学习的关系 ——weight decay 就是高斯先验(这解释了为什么它对所有权重同等收缩),而 稀疏先验 (L1、Horseshoe)在需要剪枝的场景更有用;此外 LoRA 的零初始化 可理解为'强先验:增量应为 0'。
- λ = 1/(2τ²)(高斯)或 λ = 1/b(Laplace)
- Laplace 在 0 处的尖峰 ⇒ 稀疏
常见陷阱
- 认为正则化只是'防止过拟合的技巧'(它有明确的先验解释)
- 在高斯先验下期待稀疏解