M1-011M1: Mathematics & Statistics FundamentalsInformation TheoryEasy
⚡Flashcard Drill➔Mastery:
Formulate Self-Information, Entropy, Cross-Entropy, and KL Divergence, and Clarify Their Mathematical Relationships
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Entropy measures inherent uncertainty, cross-entropy measures expected code length using an imperfect model, and KL divergence measures the excess penalty: .
📌 Key Takeaways
- •Self-information: ; less probable events convey greater informational surprise.
- •Shannon Entropy: ; expected surprise of distribution .
- •KL Divergence: (Gibbs' inequality).
- •Identity: .
📐 Mathematical Derivations
Starting from cross-entropy: . By Jensen's Inequality, since is strictly convex: , with equality if and only if almost everywhere.
🏭 Production Trade-offs
When training classification models where target is a constant one-hot ground-truth distribution, its entropy . Consequently, minimizing cross-entropy loss is mathematically identical to minimizing . In knowledge distillation or label smoothing, and soft targets supply non-zero entropy, regularizing model confidence.
⚠️ Common Interview Pitfalls
- ✕Assuming KL divergence is a true distance metric (it violates symmetry and the triangle inequality).
- ✕Allowing where , which triggers division by zero and infinite loss.
🎯 Interviewer Follow-up Inquiries
- ?Why does cross-entropy gradient with softmax simplify cleanly to ?
- ?What is the connection between cross-entropy minimization and maximum likelihood estimation (MLE)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚