M2-098M2: Classical Machine LearningRegularization (L1 / L2)Hard
⚡Flashcard Drill➔Mastery:
Spectral Norm Regularization and Lipschitz Continuity Constraints
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Bounding the spectral norm (maximum singular value) of each layer's weight matrix bounds the network's global Lipschitz constant, stabilizing GAN training and ensuring robustness.
📌 Key Takeaways
- •Lipschitz bound: ; for feedforward networks
- •Spectral Normalization: , enforcing via power iteration
- •Application: prevents discriminator explosion in Wasserstein GANs and enhances adversarial robustness
📐 Mathematical Derivations
Mathematical Formulation: A linear transformation has Lipschitz constant equal to its spectral norm: .
For a deep neural network , assuming activation functions are 1-Lipschitz (e.g., ReLU, Leaky ReLU, GeLU), the composition satisfies: .
Spectral Normalization (Miyato et al., 2018): Normalizes weight matrices by their top singular value: . During each training step, is estimated in via a single step of the power iteration method: , , with .
For a deep neural network , assuming activation functions are 1-Lipschitz (e.g., ReLU, Leaky ReLU, GeLU), the composition satisfies: .
Spectral Normalization (Miyato et al., 2018): Normalizes weight matrices by their top singular value: . During each training step, is estimated in via a single step of the power iteration method: , , with .
🏭 Production Trade-offs
System design trade-offs: Spectral normalization provides a soft, gradient-friendly alternative to hard weight clipping in WGANs without degrading model representation capacity. It adds negligible computational overhead () compared to full SVD.
⚠️ Common Interview Pitfalls
- ✕Confusing the spectral norm (maximum singular value) with the Frobenius norm (entrywise Euclidean norm)
- ✕Using full SVD to compute spectral norms during every iteration instead of power iteration, causing severe training bottlenecks
🎯 Interviewer Follow-up Inquiries
- ?How does power iteration compute the dominant singular value in a single forward-backward pass?
- ?Why did Spectral Normalization replace gradient penalty (WGAN-GP) in large-scale generative models like BigGAN?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚
Associated Knowledge Base Guides & Mindmaps
中文速览与面试要点
解释谱范数正则化与 Lipschitz 约束的关系
谱范数 = 层的 Lipschitz 常数;约束谱范数即约束输出对输入的敏感度。
核心要点
- 数学关系:对多层网络 f(x)=W_Lσ(…σ(W_1x)…),若激活函数的 Lipschitz 常数为 1(ReLU、LeakyReLU、以及有界导数的激活都满足),则整个网络的 Lipschitz 常数满足 ‖f‖_Lip ≤ Πₗ‖Wₗ‖₂ (每层的谱范数之积)。因此 约束每层的谱范数 ≤1 就保证整个网络是 1-Lipschitz 。 Lipschitz 约束的含义 :‖f(x)−f(x')‖≤L‖x−x'‖,即输入的微小扰动(如对抗扰动)最多被放大 L 倍——这直接决定了 对抗鲁棒性 (若 L 小,则小扰动不会大幅改变输出)。 两种实现 :① 谱正则化 ——在损失中加入 Σₗ‖Wₗ‖₂(谱范数)作为惩罚项,用幂迭代估计 σ_max,软约束;② 谱归一化(Spectral Normalization) ——直接把每层权重除以它的谱范数(W←W/σ_max(W)),硬约束为 1-Lipschitz,这是 SN-GAN 与对抗训练的标准做法(计算便宜:每步一次幂迭代)。
- 实践要点:① 为什么优于 L2 ——L2 正则惩罚 ‖W‖_F²(所有奇异值平方和),会同时收缩大奇异值与小奇异值;而 Lipschitz 只由 最大 奇异值决定,故谱范数正则 更直接 地控制敏感度(L2 可能把大奇异值收缩得不够而小奇异值收缩过度)。② GAN 中的应用 ——WGAN 要求 critic 是 1-Lipschitz,最初用 weight clipping(粗暴、导致参数集中在边界),后改 WGAN-GP(梯度惩罚)与 SN(谱归一化);SN 因其高效稳定成为主流(SN-GAN)。③ 对抗鲁棒性 ——理论上,Lipschitz 常数小的模型对对抗扰动更鲁棒;实践中 谱归一化 + 对抗训练 是提升鲁棒性的标准组合。④ 其他应用 —— 流模型(Normalizing Flow) 需约束 Lipschitz 保证可逆性与稳定性; 扩散模型的 Lipschitz 约束 用于稳定性分析; 度量学习 中用 Lipschitz 约束保证嵌入的平滑性。⑤ 计算成本 ——幂迭代近似 σ_max 需每步额外几次矩阵-向量乘(成本约为前向的 5–10%),可接受;精确 SVD 太贵不可用。⑥ 局限 ——谱范数约束的是 最坏情况 的敏感度(全局上界),可能过于保守(牺牲正常样本的精度);实践中常用 近似约束 (如软惩罚)平衡。
- 谱范数正则惩罚最大奇异值
- 谱归一化(SN)直接除以谱范数
常见陷阱
- 用 L2 正则代替 Lipschitz 约束(两者不等价)
- 用精确 SVD 计算谱范数(应使用幂迭代)