M1-065M1: Mathematics & Statistics FundamentalsInformation TheoryMedium
⚡Flashcard Drill➔Mastery:

Explain Jensen-Shannon (JS) Divergence and Its Structural Improvements Over KL Divergence

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-066 →
📐 Mathematical Definition
JS(p∥q)=12KL(p∥m)+12KL(q∥m),m=12(p+q)\mathrm{JS}(p\|q)=\tfrac12\mathrm{KL}(p\|m)+\tfrac12\mathrm{KL}(q\|m),\quad m=\tfrac12(p+q)
⚡ One-Sentence Core Answer
JS divergence symmetrizes KL divergence by evaluating distance to the mixture midpoint M=P+Q2M = \frac{P+Q}{2}; it is symmetric, bounded in [0,log⁡2][0, \log 2], and avoids infinite division-by-zero penalties.

📌 Key Takeaways

  • •
    Formulation: DJS(P∥Q)=12DKL(P∥M)+12DKL(Q∥M)D_{\text{JS}}(P\parallel Q) = \frac{1}{2}D_{\text{KL}}(P\parallel M) + \frac{1}{2}D_{\text{KL}}(Q\parallel M), where M=12(P+Q)M = \frac{1}{2}(P + Q).
  • •
    Symmetry: DJS(P∥Q)=DJS(Q∥P)D_{\text{JS}}(P\parallel Q) = D_{\text{JS}}(Q\parallel P).
  • •
    Boundedness: 0≤DJS(P∥Q)≤log⁡20 \le D_{\text{JS}}(P\parallel Q) \le \log 2 (or 1 if base-2 log is used).
  • •
    Metric property: The square root DJS\sqrt{D_{\text{JS}}} satisfies the triangle inequality, forming a true mathematical metric.

📐 Mathematical Derivations

Expanding using KL definitions: DJS(P∥Q)=12∑xP(x)log⁡P(x)M(x)+12∑xQ(x)log⁡Q(x)M(x)=H(M)−12(H(P)+H(Q))D_{\text{JS}}(P\parallel Q) = \frac{1}{2}\sum_x P(x)\log\frac{P(x)}{M(x)} + \frac{1}{2}\sum_x Q(x)\log\frac{Q(x)}{M(x)} = H(M) - \frac{1}{2}(H(P) + H(Q)). This reveals JS divergence as the Jensen information difference between the entropy of the mixture and the mixture of entropies. Because M(x)≥12P(x)M(x) \ge \frac{1}{2}P(x) and M(x)≥12Q(x)M(x) \ge \frac{1}{2}Q(x), the ratio P(x)M(x)≤2\frac{P(x)}{M(x)} \le 2 and Q(x)M(x)≤2\frac{Q(x)}{M(x)} \le 2. Thus, the terms inside the logarithm are strictly bounded above by 2, preventing the divergence from ever blowing up to +∞+\infty, even when PP and QQ have completely disjoint supports.

🏭 Production Trade-offs

Goodfellow et al. (2014) proved that the original Generative Adversarial Network (GAN) objective with an optimal discriminator D∗(x)=pdata(x)pdata(x)+pg(x)D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)} minimizes exactly 2DJS(pdata∥pg)−2log⁡22 D_{\text{JS}}(p_{\text{data}} \parallel p_g) - 2\log 2. However, when distributions reside on disjoint low-dimensional manifolds in high-dimensional space, DJSD_{\text{JS}} is a constant log⁡2\log 2, producing zero gradient for the generator (vanishing gradient), which motivated the Wasserstein GAN.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming JS divergence provides smooth gradients when distribution supports are completely disjoint (it saturates at constant log⁡2\log 2).
  • ✕
    Treating standard DJSD_{\text{JS}} as a metric without taking the square root DJS\sqrt{D_{\text{JS}}} (standard JS violates the triangle inequality).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does original GAN training suffer from vanishing generator gradients when discriminator is optimal?
  • ?
    How does Wasserstein distance solve the gradient vanishing problem that plagues JS divergence on disjoint manifolds?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-064: Define the Exponential Family of Distributions and Why It Is Foundational to GLMs and Deep Learning📋 Back to BankNext →M1-066: Define Wasserstein Distance (Earth Mover's Distance) and Why WGAN Relies on It