M1-065M1: Mathematics & Statistics FundamentalsInformation TheoryMedium
⚡Flashcard Drill➔Mastery:
Explain Jensen-Shannon (JS) Divergence and Its Structural Improvements Over KL Divergence
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
JS divergence symmetrizes KL divergence by evaluating distance to the mixture midpoint ; it is symmetric, bounded in , and avoids infinite division-by-zero penalties.
📌 Key Takeaways
- •Formulation: , where .
- •Symmetry: .
- •Boundedness: (or 1 if base-2 log is used).
- •Metric property: The square root satisfies the triangle inequality, forming a true mathematical metric.
📐 Mathematical Derivations
Expanding using KL definitions: . This reveals JS divergence as the Jensen information difference between the entropy of the mixture and the mixture of entropies. Because and , the ratio and . Thus, the terms inside the logarithm are strictly bounded above by 2, preventing the divergence from ever blowing up to , even when and have completely disjoint supports.
🏭 Production Trade-offs
Goodfellow et al. (2014) proved that the original Generative Adversarial Network (GAN) objective with an optimal discriminator minimizes exactly . However, when distributions reside on disjoint low-dimensional manifolds in high-dimensional space, is a constant , producing zero gradient for the generator (vanishing gradient), which motivated the Wasserstein GAN.
⚠️ Common Interview Pitfalls
- ✕Assuming JS divergence provides smooth gradients when distribution supports are completely disjoint (it saturates at constant ).
- ✕Treating standard as a metric without taking the square root (standard JS violates the triangle inequality).
🎯 Interviewer Follow-up Inquiries
- ?Why does original GAN training suffer from vanishing generator gradients when discriminator is optimal?
- ?How does Wasserstein distance solve the gradient vanishing problem that plagues JS divergence on disjoint manifolds?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚