Mutual Information

Mutual information quantifies how much one random variable tells us about another: $I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) = \sum_x \sum_y p(x,y)\log\frac{p(x,y)}{p(x)p(y)} \ge 0$. It is the KL divergence between the joint and the product of marginals, $I(X;Y) = D_{KL}(p(x,y) \Vert p(x)p(y))$, and $I(X;Y) = 0 \iff X \perp Y$. Intuitively $H(X|Y) = H(X) - I(X;Y)$: MI is exactly the reduction in uncertainty about $X$ after observing $Y$.

feature selection (pick features with highest MI to the label), information-gain splitting in decision trees, cluster/dimensionality-reduction quality evaluation, information-bottleneck analysis of generative models; interviews often start from "does zero correlation imply independence?"

correlation only captures linear dependence — e.g. for $Y = X^2$ the correlation is 0 while MI is large; MI captures arbitrary (including nonlinear) dependence and is invariant to monotone transformations, making it a strictly more complete dependence measure. The cost: estimation for continuous variables is hard (binning/kernel density or variational bounds), with notable upward bias at small sample sizes.

Define mutual information and give its three equivalent forms. Why is $I(X;Y) \ge 0$?

How does MI differ from the correlation coefficient? Why does zero correlation not imply independence?

What is the necessary and sufficient condition for $I(X;Y) = 0$? Relation with conditional entropy $H(X|Y)$?

Meaning of the chain rule $I(X,Y;Z) = I(X;Z) + I(Y;Z|X)$? How is it used in feature selection?

Given a joint table $p(x,y)$, compute MI by hand? How to estimate MI for continuous variables?