M1-066M1: Mathematics & Statistics FundamentalsInformation TheoryHard
⚡Flashcard Drill➔Mastery:
Define Wasserstein Distance (Earth Mover's Distance) and Why WGAN Relies on It
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Wasserstein distance measures the minimum cost of transporting probability mass from one distribution to another; unlike KL and JS, it provides continuous, non-zero gradients even when distribution supports do not overlap.
📌 Key Takeaways
- •Optimal Transport Formulation: , where is the set of all joint couplings.
- •Kantorovich-Rubinstein Duality: , constrained to 1-Lipschitz functions.
- •Continuous everywhere: As parallel distributions shift, Wasserstein distance scales linearly with geometric distance, eliminating vanishing gradients.
📐 Mathematical Derivations
Toy example comparing JS and Wasserstein on disjoint supports: Let (Dirac mass at 0) and (Dirac mass at ). For , the supports are disjoint. (1) KL divergence: . (2) JS divergence: . The gradient everywhere, providing zero learning signal. (3) Wasserstein distance: The only coupling is , giving . The gradient is , providing a constant, stable gradient pulling toward 0 regardless of how far apart the distributions are.
🏭 Production Trade-offs
WGAN (Arjovsky et al., 2017) replaces the classification discriminator with a 1-Lipschitz 'critic' network , maximizing . Enforcing the 1-Lipschitz condition initially used weight clipping (), which caused capacity under-utilization; modern implementations use Gradient Penalty (WGAN-GP): adding penalty term along interpolations .
⚠️ Common Interview Pitfalls
- ✕Using weight clipping instead of gradient penalty (WGAN-GP) or spectral normalization, causing pathological weight saturation.
- ✕Interpreting the WGAN critic as a classifier predicting probabilities (critic output is unbounded real numbers, not probabilities).
🎯 Interviewer Follow-up Inquiries
- ?How does the Kantorovich-Rubinstein duality transform the intractable coupling infimum into a tractable 1-Lipschitz supremum?
- ?Why does spectral normalization in modern GANs and diffusion models enforce Lipschitz continuity?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚