M1-013M1: Mathematics & Statistics FundamentalsInformation TheoryMedium
⚡Flashcard Drill➔Mastery:

Define Mutual Information (MI) and Pointwise Mutual Information (PMI), and Detail Their Practical ML Applications

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-014 →
📐 Mathematical Definition
I(X;Y)=∑x,yp(x,y)log⁡p(x,y)p(x)p(y),PMI(x,y)=log⁡p(x,y)p(x)p(y)I(X;Y)=\sum_{x,y}p(x,y)\log\frac{p(x,y)}{p(x)p(y)},\qquad \mathrm{PMI}(x,y)=\log\frac{p(x,y)}{p(x)p(y)}
⚡ One-Sentence Core Answer
PMI measures the co-occurrence association of specific outcomes, while Mutual Information is the expected PMI across all outcomes, quantifying total shared non-linear information.

📌 Key Takeaways

  • •
    PMI: PMI(x;y)=log⁡P(x,y)P(x)P(y)\text{PMI}(x; y) = \log \frac{P(x, y)}{P(x)P(y)}; positive if co-occurring more often than random chance.
  • •
    Mutual Information: I(X;Y)=DKL(P(X,Y)∥P(X)P(Y))=H(X)−H(X∣Y)I(X; Y) = D_{\text{KL}}(P(X, Y) \parallel P(X)P(Y)) = H(X) - H(X\mid Y).
  • •
    Captures arbitrary non-linear dependencies, unlike linear Pearson correlation.
  • •
    Key uses: feature selection, Word2Vec matrix factorization, and contrastive self-supervised learning (InfoNCE).

📐 Mathematical Derivations

From definition: I(X;Y)=∑x,yP(x,y)log⁡P(x,y)P(x)P(y)I(X; Y) = \sum_{x, y} P(x, y)\log \frac{P(x, y)}{P(x)P(y)}. Decomposing: ∑x,yP(x,y)[log⁡P(x,y)−log⁡P(x)−log⁡P(y)]=−∑xP(x)log⁡P(x)−(−∑x,yP(x,y)log⁡P(x∣y))=H(X)−H(X∣Y)\sum_{x, y} P(x, y)[\log P(x, y) - \log P(x) - \log P(y)] = -\sum_x P(x)\log P(x) - \left(-\sum_{x, y} P(x, y)\log P(x\mid y)\right) = H(X) - H(X\mid Y). Symmetrically, I(X;Y)=H(Y)−H(Y∣X)=H(X)+H(Y)−H(X,Y)I(X; Y) = H(Y) - H(Y\mid X) = H(X) + H(Y) - H(X, Y). If XX and YY are independent, P(x,y)=P(x)P(y)  ⟹  I(X;Y)=0P(x, y)=P(x)P(y) \implies I(X; Y)=0. Furthermore, 0≤I(X;Y)≤min⁡(H(X),H(Y))0 \le I(X; Y) \le \min(H(X), H(Y)).

🏭 Production Trade-offs

Levy & Goldberg (2014) proved that Word2Vec SGNS implicitly factorizes a shifted Positive PMI (PPMI) matrix PPMI(w,c)−log⁡k\text{PPMI}(w, c) - \log k. In contrastive learning (CPC, CLIP), the InfoNCE loss minimizes a variational lower bound on mutual information I(X;Y)≥log⁡(K)−LInfoNCEI(X; Y) \ge \log(K) - \mathcal{L}_{\text{InfoNCE}}. In feature selection, MI handles non-linear relationships that Pearson correlation misses, but estimating continuous high-dimensional MI requires non-trivial binning or neural estimators (MINE).
⚠️ Common Interview Pitfalls
  • ✕
    PMI suffers from severe bias toward extremely low-frequency rare words (addressed by Normalized PMI or frequency exponents).
  • ✕
    Using Pearson correlation when relationships are strongly non-linear (e.g., Y=X2Y = X^2 where ρ=0\rho=0 but I(X;Y)I(X; Y) is large).
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the InfoNCE loss function establish a lower bound on mutual information?
  • ?
    Why is high-dimensional mutual information estimation fundamentally prone to high sample variance?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-012: Explain Why KL Divergence is Asymmetric and Contrast the Optimization Behaviors of Forward vs. Reverse KL📋 Back to BankNext →M1-014: Explain the Theoretical Equivalence Between Cross-Entropy Loss Minimization and Maximum Likelihood Estimation (MLE)