M2-098M2: Classical Machine LearningRegularization (L1 / L2)Hard
⚡Flashcard Drill➔Mastery:

Spectral Norm Regularization and Lipschitz Continuity Constraints

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-099 →
📐 Mathematical Definition
∥f∥Lip=∏l∥Wl∥2,∥f(x)−f(x′)∥≤∥f∥Lip∥x−x′∥\|f\|_{Lip}=\prod_l\|W_l\|_2,\qquad \|f(x)-f(x')\|\le\|f\|_{Lip}\|x-x'\|
⚡ One-Sentence Core Answer
Bounding the spectral norm (maximum singular value) of each layer's weight matrix bounds the network's global Lipschitz constant, stabilizing GAN training and ensuring robustness.

📌 Key Takeaways

  • •
    Lipschitz bound: ∥f(x)−f(y)∥2≤L∥x−y∥2\|f(x) - f(y)\|_2 \le L \|x - y\|_2; for feedforward networks L≤∏l=1D∥Wl∥2L \le \prod_{l=1}^D \|W_l\|_2
  • •
    Spectral Normalization: WSN=W/σmax⁡(W)W_{\text{SN}} = W / \sigma_{\max}(W), enforcing ∥WSN∥2=1\|W_{\text{SN}}\|_2 = 1 via power iteration
  • •
    Application: prevents discriminator explosion in Wasserstein GANs and enhances adversarial robustness

📐 Mathematical Derivations

Mathematical Formulation: A linear transformation f(x)=Wxf(x) = W x has Lipschitz constant equal to its spectral norm: sup⁡x≠y∥W(x−y)∥2∥x−y∥2=σmax⁡(W)=∥W∥2\sup_{x \ne y} \frac{\|W(x - y)\|_2}{\|x - y\|_2} = \sigma_{\max}(W) = \|W\|_2.
For a deep neural network F(x)=WLϕ(WL−1…ϕ(W1x))F(x) = W_L \phi(W_{L-1} \dots \phi(W_1 x)), assuming activation functions ϕ\phi are 1-Lipschitz (e.g., ReLU, Leaky ReLU, GeLU), the composition satisfies: ∥F∥Lip≤∏l=1L∥Wl∥2\|F\|_{\text{Lip}} \le \prod_{l=1}^L \|W_l\|_2.
Spectral Normalization (Miyato et al., 2018): Normalizes weight matrices by their top singular value: Wˉ=Wσmax⁡(W)\bar{W} = \frac{W}{\sigma_{\max}(W)}. During each training step, σmax⁡(W)\sigma_{\max}(W) is estimated in O(1)O(1) via a single step of the power iteration method: v←WTu∥WTu∥v \leftarrow \frac{W^T u}{\|W^T u\|}, u←Wv∥Wv∥u \leftarrow \frac{W v}{\|W v\|}, with σmax⁡(W)≈uTWv\sigma_{\max}(W) \approx u^T W v.

🏭 Production Trade-offs

System design trade-offs: Spectral normalization provides a soft, gradient-friendly alternative to hard weight clipping in WGANs without degrading model representation capacity. It adds negligible computational overhead (∼1−2%\sim 1-2\%) compared to full SVD.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing the spectral norm ∥W∥2\|W\|_2 (maximum singular value) with the Frobenius norm ∥W∥F\|W\|_F (entrywise Euclidean norm)
  • ✕
    Using full SVD to compute spectral norms during every iteration instead of power iteration, causing severe training bottlenecks
🎯 Interviewer Follow-up Inquiries
  • ?
    How does power iteration compute the dominant singular value in a single forward-backward pass?
  • ?
    Why did Spectral Normalization replace gradient penalty (WGAN-GP) in large-scale generative models like BigGAN?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-097: Why Weight Decay Must Be Decoupled in Adam (AdamW)📋 Back to BankNext →M2-099: Stochastic Depth (DropPath) vs Standard Dropout: Mechanisms and Differences