M1-074M1: Mathematics & Statistics Fundamentals估计理论 (MLE/MAP)Medium
⚡Flashcard Drill➔Mastery:

Explain Fisher Information and the Cramér-Rao Lower Bound (CRLB)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-075 →
📐 Mathematical Definition
I(θ)=E[(∂log⁡p∂θ)2]=−E[∂2log⁡p∂θ2],Var(θ^)≥1nI(θ)I(\theta)=\mathbb E\Big[\Big(\frac{\partial\log p}{\partial\theta}\Big)^2\Big]=-\mathbb E\Big[\frac{\partial^2\log p}{\partial\theta^2}\Big],\qquad \mathrm{Var}(\hat\theta)\ge\frac{1}{nI(\theta)}
⚡ One-Sentence Core Answer
Fisher Information measures the amount of information that an observable random variable carries about an unknown parameter; the Cramér-Rao Lower Bound proves that no unbiased estimator can have variance smaller than the inverse Fisher Information: Var(θ^)≥I(θ)−1\text{Var}(\hat{\theta}) \ge I(\theta)^{-1}.

📌 Key Takeaways

  • •
    Score function: S(θ)=∇θlog⁡p(x∣θ)S(\theta) = \nabla_\theta \log p(x\mid \theta); expectation of score is always zero: E[S(θ)]=0E[S(\theta)] = 0.
  • •
    Fisher Information: I(θ)=E[(S(θ))(S(θ))T]=−E[∇θ2log⁡p(x∣θ)]I(\theta) = E[(S(\theta))(S(\theta))^T] = -E[\nabla^2_\theta \log p(x\mid \theta)]; expectation of negative Hessian.
  • •
    Cramér-Rao Lower Bound: For any unbiased estimator θ^\hat{\theta}, Cov(θ^)⪰I(θ)−1\text{Cov}(\hat{\theta}) \succeq I(\theta)^{-1}.
  • •
    Asymptotic Efficiency: MLE asymptotically achieves the CRLB: N(θ^MLE−θ0)→dN(0,I(θ0)−1)\sqrt{N}(\hat{\theta}_{\text{MLE}} - \theta_0) \xrightarrow{d} \mathcal{N}(0, I(\theta_0)^{-1}).

📐 Mathematical Derivations

Proof via Cauchy-Schwarz inequality: For 1D unbiased estimator θ^\hat{\theta} (E[θ^]=θE[\hat{\theta}] = \theta). Differentiating with respect to θ\theta: ddθ∫θ^(x)p(x∣θ)dx=∫θ^(x)∂log⁡p(x∣θ)∂θp(x∣θ)dx=E[θ^⋅S(θ)]=1\frac{d}{d\theta} \int \hat{\theta}(x) p(x\mid \theta)dx = \int \hat{\theta}(x) \frac{\partial \log p(x\mid \theta)}{\partial \theta} p(x\mid \theta)dx = E[\hat{\theta} \cdot S(\theta)] = 1. Since E[S(θ)]=0E[S(\theta)]=0, Cov(θ^,S(θ))=E[(θ^−θ)S(θ)]=E[θ^S]−θE[S]=1\text{Cov}(\hat{\theta}, S(\theta)) = E[(\hat{\theta} - \theta)S(\theta)] = E[\hat{\theta} S] - \theta E[S] = 1. By Cauchy-Schwarz inequality: Cov(θ^,S(θ))2≤Var(θ^)Var(S(θ))\text{Cov}(\hat{\theta}, S(\theta))^2 \le \text{Var}(\hat{\theta}) \text{Var}(S(\theta)). Since Var(S(θ))=E[S2]=I(θ)\text{Var}(S(\theta)) = E[S^2] = I(\theta), substituting gives: 1≤Var(θ^)I(θ)  ⟹  Var(θ^)≥1I(θ)1 \le \text{Var}(\hat{\theta}) I(\theta) \implies \text{Var}(\hat{\theta}) \ge \frac{1}{I(\theta)}.

🏭 Production Trade-offs

Fisher Information is foundational to modern ML: (1) Natural Gradient Descent (Amari, 1998): Parameter updates Δθ=−I(θ)−1∇θL\Delta \theta = -I(\theta)^{-1} \nabla_\theta \mathcal{L} take steepest descent steps with respect to the Riemannian manifold of probability distributions (invariant to reparameterization). (2) Elastic Weight Consolidation (EWC): Mitigates catastrophic forgetting in continual learning by regularizing parameter drift using the diagonal Fisher information Fi(θi−θA,i∗)2F_i (\theta_i - \theta_{A, i}^*)^2.
⚠️ Common Interview Pitfalls
  • ✕
    Applying CRLB to biased estimators without the derivative correction factor (1+b′(θ))2/I(θ)(1 + b'(\theta))^2 / I(\theta).
  • ✕
    Confusing the observed Fisher Information (negative Hessian evaluated at θ^\hat{\theta}) with expected Fisher Information.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Natural Gradient Descent relate to TRPO (Trust Region Policy Optimization) and K-FAC?
  • ?
    Why does the Fisher Information metric tensor define a Riemannian geometry via the second-order expansion of KL divergence?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-073: Explain FP32 Master Weights and Their Essential Role in Mixed Precision Training (FP16/BF16)📋 Back to BankNext →M1-075: Explain Shrinkage Estimators and the James-Stein Phenomenon