M1-037M1: Mathematics & Statistics Fundamentals估计理论 (MLE/MAP)Easy
⚡Flashcard Drill➔Mastery:

Define Maximum A Posteriori (MAP) Estimation and Explain Its Exact Connection to MLE and Regularization

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-038 →
📐 Mathematical Definition
θ^MAP=arg⁡max⁡θ[log⁡p(D∣θ)+log⁡p(θ)]\hat\theta_{MAP}=\arg\max_\theta\big[\log p(D\mid\theta)+\log p(\theta)\big]
⚡ One-Sentence Core Answer
MAP maximizes the posterior mode: θ^MAP=arg⁡max⁡[log⁡p(D∣θ)+log⁡p(θ)]\hat{\theta}_{\text{MAP}} = \arg\max [\log p(D\mid \theta) + \log p(\theta)]; a Gaussian prior corresponds to L2 regularization (Ridge), while a Laplace prior corresponds to L1 (Lasso).

📌 Key Takeaways

  • •
    Formulation: θ^MAP=arg⁡max⁡θ[ℓ(θ)+log⁡p(θ)]\hat{\theta}_{\text{MAP}} = \arg\max_\theta [\ell(\theta) + \log p(\theta)].
  • •
    Equivalence to MLE: Under a uniform / uninformative prior p(θ)=constp(\theta) = \text{const}, MAP is identical to MLE.
  • •
    Regularization equivalence: Negative log-prior acts exactly as the regularization penalty λR(θ)\lambda R(\theta).

📐 Mathematical Derivations

Bayes' theorem gives p(θ∣D)=p(D∣θ)p(θ)p(D)p(\theta\mid D) = \frac{p(D\mid \theta)p(\theta)}{p(D)}. Taking the log and dropping the constant denominator: arg⁡max⁡θlog⁡p(θ∣D)=arg⁡max⁡θ[log⁡p(D∣θ)+log⁡p(θ)]=arg⁡min⁡θ[−ℓ(θ)−log⁡p(θ)]\arg\max_\theta \log p(\theta\mid D) = \arg\max_\theta [\log p(D\mid \theta) + \log p(\theta)] = \arg\min_\theta [-\ell(\theta) - \log p(\theta)]. Case 1 (Gaussian Prior): If θ∼N(0,τ2I)\theta \sim \mathcal{N}(0, \tau^2 I), then −log⁡p(θ)=12τ2∥θ∥22+const-\log p(\theta) = \frac{1}{2\tau^2}\|\theta\|_2^2 + \text{const}, which is precisely L2 weight decay with λ=1τ2\lambda = \frac{1}{\tau^2}. Case 2 (Laplace Prior): If θj∼Laplace(0,b)\theta_j \sim \text{Laplace}(0, b), then −log⁡p(θ)=1b∥θ∥1+const-\log p(\theta) = \frac{1}{b}\|\theta\|_1 + \text{const}, which is precisely L1 Lasso regularization with λ=1b\lambda = \frac{1}{b}.

🏭 Production Trade-offs

MAP strikes a middle ground: it incorporates domain prior knowledge to regularize against overfitting like Bayesian methods, but computes only a single point estimate via optimization, avoiding intractable posterior integrations. However, MAP is not invariant to reparameterization: transforming θ=g(ϕ)\theta = g(\phi) changes the mode because the Jacobian determinant scales the density, unlike full Bayesian posterior expectation.
⚠️ Common Interview Pitfalls
  • ✕
    Believing MAP is fully Bayesian (MAP discards posterior variance and uncertainty, providing only a point estimate).
  • ✕
    Assuming MAP is invariant under non-linear coordinate transformations.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the MAP estimate not invariant under change of variables, whereas MLE is invariant?
  • ?
    How does Empirical Bayes estimate the prior hyperparameters τ2\tau^2 directly from marginal data?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-036: Define Maximum Likelihood Estimation (MLE) with a Complete End-to-End Derivation📋 Back to BankNext →M1-038: Explain the Bias-Variance Decomposition and Write the Decomposition for Expected Generalization Error