M1-012M1: Mathematics & Statistics FundamentalsInformation TheoryMedium
⚡Flashcard Drill➔Mastery:
Explain Why KL Divergence is Asymmetric and Contrast the Optimization Behaviors of Forward vs. Reverse KL
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Forward KL is zero-avoiding (mode-covering), whereas Reverse KL is zero-forcing (mode-seeking).
📌 Key Takeaways
- •Forward KL: forces wherever to prevent infinite penalty.
- •Reverse KL: forces wherever to avoid severe penalties.
- •Supervised learning uses Forward KL; Variational Inference (VAE) and RLHF policy optimization use Reverse KL.
📐 Mathematical Derivations
Analyzing the objectives: (1) Forward KL: . If but , the integrand blows up to . Hence, is forced to stretch across the entire support of (mean-seeking / mode-covering), resulting in over-broad variances. (2) Reverse KL: . If while , the penalty diverges. Thus, collapses to a single dominant mode of where is safely high (mode-seeking), completely ignoring other modes.
🏭 Production Trade-offs
In generative models and LLM alignment: (1) Standard SFT minimizes cross-entropy, equivalent to Forward KL, causing LLMs to generate diverse but occasionally hallucinated output. (2) RLHF and DPO optimize policy against reference models using Reverse KL regularization , keeping responses tightly focused on high-reward modes. (3) In multimodal distributions, reverse KL risks mode collapse.
⚠️ Common Interview Pitfalls
- ✕Using Forward KL when fitting unimodal approximations to multimodal posteriors, yielding unrealistically wide densities over zero-probability troughs.
- ✕Confusing which distribution acts as the reference baseline in policy gradients.
🎯 Interviewer Follow-up Inquiries
- ?How does Jensen-Shannon Divergence (JSD) achieve symmetry between and ?
- ?Why is Reverse KL computationally tractable in Variational Inference when is unnormalized?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚