Entropy, KL & Cross-Entropy

Entropy $H(X) = -\sum_x p(x)\log p(x)$ measures uncertainty (expected bits of an optimal code); KL divergence measures the discrepancy between two distributions, $D_{KL}(P \Vert Q) = \sum_x p(x)\log\frac{p(x)}{q(x)} \ge 0$, with equality iff $P = Q$. Non-negativity follows from Jensen: since $\log$ is concave, $D_{KL}(P \Vert Q) = -\mathbb{E}_p\left[\log\frac{q(x)}{p(x)}\right] \ge -\log\mathbb{E}_p\left[\frac{q(x)}{p(x)}\right] = -\log 1 = 0$. KL is asymmetric: for $P = \text{Bern}(0.5)$ and $Q = \text{Bern}(0.1)$, $D_{KL}(P \Vert Q) \approx 0.511$ while $D_{KL}(Q \Vert P) \approx 0.368$, and $D_{KL} \to \infty$ whenever $Q(x) = 0 < P(x)$. Cross-entropy satisfies $H(P,Q) = H(P) + D_{KL}(P \Vert Q)$.

distribution discrepancy is the central metric of probabilistic ML — cross-entropy loss in classification, the $D_{KL}(q \Vert p)$ term in VAEs, student-approximates-teacher in distillation, KL penalties constraining policy drift in RLHF; classic interview questions ("why cross-entropy loss", "prove KL non-negative") all start here.

unlike geometric distances such as L2, KL carries information-theoretic meaning (extra bits to encode $P$ with $Q$) and unifies with entropy and mutual information; but it is asymmetric and violates the triangle inequality, so symmetric alternatives (Jensen-Shannon divergence, Wasserstein distance) must be constructed when needed.

Test Your Knowledge: Practice Questions for "Entropy, KL & Cross-Entropy"

Single choice pitfall questions with instant feedback and mistake tracking.