M1-066M1: Mathematics & Statistics FundamentalsInformation TheoryHard
⚡Flashcard Drill➔Mastery:

Define Wasserstein Distance (Earth Mover's Distance) and Why WGAN Relies on It

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-067 →
📐 Mathematical Definition
W(p,q)=inf⁡γ∈Π(p,q)∫∥x−y∥ dγ(x,y)W(p,q)=\inf_{\gamma\in\Pi(p,q)}\int\|x-y\|\,d\gamma(x,y)
⚡ One-Sentence Core Answer
Wasserstein distance measures the minimum cost of transporting probability mass from one distribution to another; unlike KL and JS, it provides continuous, non-zero gradients even when distribution supports do not overlap.

📌 Key Takeaways

  • •
    Optimal Transport Formulation: W1(P,Q)=inf⁡γ∈Π(P,Q)E(x,y)∼γ[∥x−y∥]W_1(P, Q) = \inf_{\gamma \in \Pi(P, Q)} E_{(x, y)\sim \gamma}[\|x - y\|], where Π(P,Q)\Pi(P, Q) is the set of all joint couplings.
  • •
    Kantorovich-Rubinstein Duality: W1(P,Q)=sup⁡∥f∥L≤1Ex∼P[f(x)]−Ey∼Q[f(y)]W_1(P, Q) = \sup_{\|f\|_L \le 1} E_{x\sim P}[f(x)] - E_{y\sim Q}[f(y)], constrained to 1-Lipschitz functions.
  • •
    Continuous everywhere: As parallel distributions shift, Wasserstein distance scales linearly with geometric distance, eliminating vanishing gradients.

📐 Mathematical Derivations

Toy example comparing JS and Wasserstein on disjoint supports: Let P=δ0P = \delta_0 (Dirac mass at 0) and Qθ=δθQ_\theta = \delta_\theta (Dirac mass at θ\theta). For θ≠0\theta \ne 0, the supports are disjoint. (1) KL divergence: DKL(P∥Qθ)=+∞D_{\text{KL}}(P\parallel Q_\theta) = +\infty. (2) JS divergence: DJS(P∥Qθ)=log⁡2D_{\text{JS}}(P\parallel Q_\theta) = \log 2. The gradient ∂∂θDJS=0\frac{\partial}{\partial \theta} D_{\text{JS}} = 0 everywhere, providing zero learning signal. (3) Wasserstein distance: The only coupling is γ(x,y)=δ(x=0,y=θ)\gamma(x, y) = \delta(x=0, y=\theta), giving W1(P,Qθ)=∣θ−0∣=∣θ∣W_1(P, Q_\theta) = |\theta - 0| = |\theta|. The gradient is ∂∂θW1=sign(θ)≠0\frac{\partial}{\partial \theta} W_1 = \text{sign}(\theta) \ne 0, providing a constant, stable gradient pulling θ\theta toward 0 regardless of how far apart the distributions are.

🏭 Production Trade-offs

WGAN (Arjovsky et al., 2017) replaces the classification discriminator with a 1-Lipschitz 'critic' network fw(x)f_w(x), maximizing Ex∼P[fw(x)]−Ez∼pz[fw(Gθ(z))]E_{x\sim P}[f_w(x)] - E_{z\sim p_z}[f_w(G_\theta(z))]. Enforcing the 1-Lipschitz condition initially used weight clipping (w∈[−c,c]w \in [-c, c]), which caused capacity under-utilization; modern implementations use Gradient Penalty (WGAN-GP): adding penalty term λE[(∥∇x^f(x^)∥2−1)2]\lambda E[(\|\nabla_{\hat{x}} f(\hat{x})\|_2 - 1)^2] along interpolations x^\hat{x}.
⚠️ Common Interview Pitfalls
  • ✕
    Using weight clipping instead of gradient penalty (WGAN-GP) or spectral normalization, causing pathological weight saturation.
  • ✕
    Interpreting the WGAN critic as a classifier predicting probabilities (critic output is unbounded real numbers, not probabilities).
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the Kantorovich-Rubinstein duality transform the intractable coupling infimum into a tractable 1-Lipschitz supremum?
  • ?
    Why does spectral normalization in modern GANs and diffusion models enforce Lipschitz continuity?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-065: Explain Jensen-Shannon (JS) Divergence and Its Structural Improvements Over KL Divergence📋 Back to BankNext →M1-067: Explain the Principle of Maximum Entropy and Its Applications in Statistical Learning