M2-098M2: Classical Machine LearningRegularization (L1 / L2)Hard
⚡Flashcard Drill➔Mastery:
Spectral Norm Regularization and Lipschitz Continuity Constraints
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Bounding the spectral norm (maximum singular value) of each layer's weight matrix bounds the network's global Lipschitz constant, stabilizing GAN training and ensuring robustness.
📌 Key Takeaways
- •Lipschitz bound: ; for feedforward networks
- •Spectral Normalization: , enforcing via power iteration
- •Application: prevents discriminator explosion in Wasserstein GANs and enhances adversarial robustness
📐 Mathematical Derivations
Mathematical Formulation: A linear transformation has Lipschitz constant equal to its spectral norm: .
For a deep neural network , assuming activation functions are 1-Lipschitz (e.g., ReLU, Leaky ReLU, GeLU), the composition satisfies: .
Spectral Normalization (Miyato et al., 2018): Normalizes weight matrices by their top singular value: . During each training step, is estimated in via a single step of the power iteration method: , , with .
For a deep neural network , assuming activation functions are 1-Lipschitz (e.g., ReLU, Leaky ReLU, GeLU), the composition satisfies: .
Spectral Normalization (Miyato et al., 2018): Normalizes weight matrices by their top singular value: . During each training step, is estimated in via a single step of the power iteration method: , , with .
🏭 Production Trade-offs
System design trade-offs: Spectral normalization provides a soft, gradient-friendly alternative to hard weight clipping in WGANs without degrading model representation capacity. It adds negligible computational overhead () compared to full SVD.
⚠️ Common Interview Pitfalls
- ✕Confusing the spectral norm (maximum singular value) with the Frobenius norm (entrywise Euclidean norm)
- ✕Using full SVD to compute spectral norms during every iteration instead of power iteration, causing severe training bottlenecks
🎯 Interviewer Follow-up Inquiries
- ?How does power iteration compute the dominant singular value in a single forward-backward pass?
- ?Why did Spectral Normalization replace gradient penalty (WGAN-GP) in large-scale generative models like BigGAN?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚