M1-036M1: Mathematics & Statistics Fundamentals估计理论 (MLE/MAP)Easy
⚡Flashcard Drill➔Mastery:

Define Maximum Likelihood Estimation (MLE) with a Complete End-to-End Derivation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-037 →
📐 Mathematical Definition
θ^MLE=arg⁡max⁡θ∑ilog⁡p(xi∣θ)\hat\theta_{MLE}=\arg\max_\theta\sum_i\log p(x_i\mid\theta)
⚡ One-Sentence Core Answer
MLE finds parameter values that maximize the probability of generating the observed sample dataset: θ^MLE=arg⁡max⁡θ∑i=1Nlog⁡p(xi∣θ)\hat{\theta}_{\text{MLE}} = \arg\max_\theta \sum_{i=1}^N \log p(x_i\mid \theta).

📌 Key Takeaways

  • •
    Principle: Treats data DD as fixed and parameter θ\theta as the variable to optimize.
  • •
    Log-likelihood: Logarithm transforms product of probabilities into a sum of log-densities, simplifying derivatives and avoiding underflow.
  • •
    Score function: First derivative ∇θℓ(θ)=0\nabla_\theta \ell(\theta) = 0; information matrix evaluated via negative Hessian.

📐 Mathematical Derivations

Complete example: Estimating Gaussian parameters θ=(μ,σ2)\theta = (\mu, \sigma^2) from i.i.d. Sample x1,…,xN∼N(μ,σ2)x_1, \dots, x_N \sim \mathcal{N}(\mu, \sigma^2). The likelihood is L(μ,σ2)=∏i=1N12πσ2exp⁡(−(xi−μ)22σ2)L(\mu, \sigma^2) = \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i - \mu)^2}{2\sigma^2}\right). The log-likelihood is ℓ(μ,σ2)=−N2log⁡(2π)−N2log⁡(σ2)−12σ2∑i=1N(xi−μ)2\ell(\mu, \sigma^2) = -\frac{N}{2}\log(2\pi) - \frac{N}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^N (x_i - \mu)^2. Setting ∂ℓ∂μ=1σ2∑i=1N(xi−μ)=0  ⟹  μ^MLE=1N∑i=1Nxi\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^N (x_i - \mu) = 0 \implies \hat{\mu}_{\text{MLE}} = \frac{1}{N}\sum_{i=1}^N x_i. Setting ∂ℓ∂σ2=−N2σ2+12(σ2)2∑i=1N(xi−μ)2=0  ⟹  σ^MLE2=1N∑i=1N(xi−μ^)2\frac{\partial \ell}{\partial \sigma^2} = -\frac{N}{2\sigma^2} + \frac{1}{2(\sigma^2)^2}\sum_{i=1}^N (x_i - \mu)^2 = 0 \implies \hat{\sigma}_{\text{MLE}}^2 = \frac{1}{N}\sum_{i=1}^N (x_i - \hat{\mu})^2. Note that σ^MLE2\hat{\sigma}_{\text{MLE}}^2 is biased (E[σ^2]=N−1Nσ2E[\hat{\sigma}^2] = \frac{N-1}{N}\sigma^2).

🏭 Production Trade-offs

MLE has optimal asymptotic properties under regularity conditions: it is asymptotically unbiased, asymptotically normal, and achieves the Cramér-Rao Lower Bound (asymptotically efficient). However, in finite small-sample regimes, MLE suffers from severe overfitting (e.g. Estimating CTR from 2 clicks out of 2 impressions yields p^=1.0\hat{p} = 1.0), necessitating Bayesian priors or smoothing.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming MLE is always unbiased (as demonstrated by Gaussian variance 1N\frac{1}{N} vs 1N−1\frac{1}{N-1}).
  • ✕
    Applying MLE when regularity conditions fail (e.g. Estimating support boundary θ\theta in U(0,θ)\mathcal{U}(0, \theta) where likelihood derivative is non-zero everywhere).
🎯 Interviewer Follow-up Inquiries
  • ?
    What is the MLE estimator for the upper bound of a Uniform distribution U(0,θ)\mathcal{U}(0, \theta)?
  • ?
    How does the Fisher Information matrix dictate the asymptotic covariance of MLE estimators?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-035: Enumerate Common Sources of Numerical Instability in Deep Learning and Their Standard Industrial Mitigations📋 Back to BankNext →M1-037: Define Maximum A Posteriori (MAP) Estimation and Explain Its Exact Connection to MLE and Regularization