M1-039M1: Mathematics & Statistics Fundamentals估计理论 (MLE/MAP)Medium
⚡Flashcard Drill➔Mastery:

Define Statistical Consistency and Asymptotic Efficiency, and Assess Whether MLE Satisfies Both

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-040 →
📐 Mathematical Definition
n(θ^MLE−θ)→dN(0, I(θ)−1)\sqrt n(\hat\theta_{MLE}-\theta)\xrightarrow{d}\mathcal N(0,\ I(\theta)^{-1})
⚡ One-Sentence Core Answer
Consistency means estimates converge in probability to the true parameter as N→∞N\to\infty; efficiency means achieving the minimum possible variance (Cramér-Rao bound); MLE satisfies both under standard regularity conditions.

📌 Key Takeaways

  • •
    Consistency: θ^n→Pθ0\hat{\theta}_n \xrightarrow{P} \theta_0 as n→∞n \to \infty (Weak consistency).
  • •
    Asymptotic Efficiency: lim⁡n→∞nVar(θ^n)=I(θ0)−1\lim_{n\to\infty} n \text{Var}(\hat{\theta}_n) = I(\theta_0)^{-1} (achieves Cramér-Rao lower bound).
  • •
    Asymptotic Normality: n(θ^n−θ0)→dN(0,I(θ0)−1)\sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} \mathcal{N}(0, I(\theta_0)^{-1}).

📐 Mathematical Derivations

Proof sketch of consistency: By the Law of Large Numbers, the sample average log-likelihood 1nℓn(θ)→PEθ0[log⁡p(X∣θ)]\frac{1}{n}\ell_n(\theta) \xrightarrow{P} E_{\theta_0}[\log p(X\mid \theta)]. Maximizing this limit is equivalent to arg⁡max⁡θ(Eθ0[log⁡p(X∣θ)]−Eθ0[log⁡p(X∣θ0)])=arg⁡min⁡θDKL(pθ0∥pθ)\arg\max_\theta \left(E_{\theta_0}[\log p(X\mid \theta)] - E_{\theta_0}[\log p(X\mid \theta_0)]\right) = \arg\min_\theta D_{\text{KL}}(p_{\theta_0} \parallel p_\theta). Since DKL≥0D_{\text{KL}} \ge 0 with unique minimum 0 at pθ=pθ0p_\theta = p_{\theta_0} (assuming identifiability), θ^n\hat{\theta}_n must converge to θ0\theta_0. For efficiency, Taylor expanding the score function 0=∇ℓn(θ^n)≈∇ℓn(θ0)+∇2ℓn(θ0)(θ^n−θ0)0 = \nabla \ell_n(\hat{\theta}_n) \approx \nabla \ell_n(\theta_0) + \nabla^2 \ell_n(\theta_0)(\hat{\theta}_n - \theta_0) yields n(θ^n−θ0)≈(−1n∇2ℓn(θ0))−11n∇ℓn(θ0)→dI(θ0)−1N(0,I(θ0))=N(0,I(θ0)−1)\sqrt{n}(\hat{\theta}_n - \theta_0) \approx \left(-\frac{1}{n}\nabla^2 \ell_n(\theta_0)\right)^{-1} \frac{1}{\sqrt{n}}\nabla \ell_n(\theta_0) \xrightarrow{d} I(\theta_0)^{-1} \mathcal{N}(0, I(\theta_0)) = \mathcal{N}(0, I(\theta_0)^{-1}).

🏭 Production Trade-offs

These asymptotic properties justify MLE as the default estimator in applied science. However, consistency and efficiency hold only when regularity conditions are satisfied: (1) True parameter lies in the interior of parameter space. (2) Identifiability holds. (3) Support of p(x∣θ)p(x\mid \theta) does not depend on θ\theta. When support depends on θ\theta (e.g. U(0,θ)\mathcal{U}(0, \theta)), MLE converges at rate O(1/n)O(1/n) rather than O(1/n)O(1/\sqrt{n}) and is not asymptotically normal.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming MLE is always consistent (Neyman-Scott paradox shows MLE can be severely inconsistent when incidental parameters grow with NN).
  • ✕
    Confusing unbiasedness (finite sample property) with consistency (asymptotic property).
🎯 Interviewer Follow-up Inquiries
  • ?
    What is the Neyman-Scott paradox and how does it demonstrate that MLE can be completely inconsistent?
  • ?
    What regularity conditions are strictly required for the Cramér-Rao bound to hold?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-038: Explain the Bias-Variance Decomposition and Write the Decomposition for Expected Generalization Error📋 Back to BankNext →M1-040: Define Model Identifiability and Provide a Concrete Example of Non-Identifiability in Latent Variable Models