WGAN & Wasserstein

WGAN replaces the JS divergence with the Wasserstein distance (earth-mover, via the Kantorovich-Rubinstein duality): $W(P_r, P_g) = \sup_{\Vert f \Vert_L \le 1} \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{z \sim p_z}[f(G(z))]$, requiring the critic $f$ to be 1-Lipschitz. The original WGAN enforces this via weight clipping, which distorts the distribution; WGAN-GP instead penalizes the gradient norm on linear interpolations $\hat{x}$ between real and fake samples: $\lambda \mathbb{E}_{\hat{x} \sim P_{\hat{x}}}\left[(\Vert \nabla_{\hat{x}} D(\hat{x}) \Vert_2 - 1)^2\right]$ (default $\lambda = 10$).

replacing standard GANs for stable image generation (e.g. DCGAN backbone + WGAN-GP loss); interviews compare Wasserstein vs JSD/KL and ask how the 1-Lipschitz constraint is implemented.

JSD saturates at $\log 2$ and its gradient vanishes almost everywhere when the supports of real and generated distributions do not overlap (common early in training), starving the generator; the Wasserstein distance is continuous and differentiable everywhere and gives meaningful gradients even for disjoint supports, stabilizing training and yielding an unsaturated loss usable as a convergence signal. Cost: the 1-Lipschitz constraint must be maintained — weight clipping causes exploding/vanishing gradients, so gradient penalty is the robust fix.

State the definition of Wasserstein distance and its Kantorovich-Rubinstein dual form?

Why do JSD/KL gradients vanish for non-overlapping supports while Wasserstein does not?

How is the 1-Lipschitz constraint enforced? Trade-offs of weight clipping vs gradient penalty?

Why is the WGAN-GP penalty minimized when $\Vert \nabla_{\hat{x}} D(\hat{x}) \Vert_2 = 1$?

Why does WGAN's critic omit sigmoid, and what does its loss value mean?