M1-002M1: Mathematics & Statistics FundamentalsProbability FoundationsEasy
⚡Flashcard Drill➔Mastery:

Formulate Bayes' Theorem and Detail the Respective Roles of Prior, Likelihood, and Posterior

📐 Mathematical Definition
P(θ∣D)=P(D∣θ)P(θ)P(D),posterior∝likelihood×priorP(\theta\mid D)=\frac{P(D\mid\theta)P(\theta)}{P(D)},\qquad \text{posterior}\propto\text{likelihood}\times\text{prior}
⚡ One-Sentence Core Answer
Posterior is proportional to Likelihood times Prior; the prior encodes existing beliefs, the likelihood is provided by empirical data, and the posterior reflects updated knowledge.

📌 Key Takeaways

  • •
    The denominator P(D)P(D) serves as the normalizing constant (marginal likelihood / evidence).
  • •
    Conjugate priors ensure the posterior resides in the same parametric family as the prior, enabling closed-form algebraic updates.
  • •
    MAP selects the posterior mode, whereas MLE selects the likelihood mode (equivalent to MAP under a uniform prior).

📐 Mathematical Derivations

Bayes' Theorem states P(θ∣D)=P(D∣θ)P(θ)P(D)=P(D∣θ)P(θ)∫P(D∣θ′)P(θ′)dθ′P(\theta\mid D) = \frac{P(D\mid \theta)P(\theta)}{P(D)} = \frac{P(D\mid \theta)P(\theta)}{\int P(D\mid \theta')P(\theta')d\theta'}. Here, P(θ)P(\theta) denotes the prior distribution representing epistemic uncertainty before observing data. The likelihood function L(θ)=P(D∣θ)L(\theta)=P(D\mid \theta) quantifies how plausibly the observed sample DD was generated under parameter configuration θ\theta. The evidence P(D)P(D) guarantees that the posterior density integrates to 1. In logarithmic form, log⁡P(θ∣D)=log⁡P(D∣θ)+log⁡P(θ)−const\log P(\theta\mid D) = \log P(D\mid \theta) + \log P(\theta) - \text{const}, illustrating that Bayesian updating is an additive log-evidence accumulation process.

🏭 Production Trade-offs

In industrial production: (1) Under small-sample regimes (e.g., cold-start CTR prediction in ad auctions), an empirical Bayes prior prevents severe overfitting by pulling noisy estimates toward category averages (shrinkage). (2) As sample size N→∞N\to \infty, the likelihood dominates by the Bernstein-von Mises theorem, causing Bayesian and frequentist MLE estimators to asymptotically converge. Conjugacy offers O(1)O(1) streaming updates, while non-conjugate complex models require compute-heavy approximations like MCMC or SVI.
⚠️ Common Interview Pitfalls
  • ✕
    Overconfidence caused by excessively rigid priors (e.g., Dirac delta or zero variance) preventing data from shifting belief.
  • ✕
    Ignoring the computational bottleneck of the evidence denominator in continuous high-dimensional parameter spaces.
🎯 Interviewer Follow-up Inquiries
  • ?
    Under what exact conditions do MAP and MLE produce mathematically identical point estimates?
  • ?
    How do conjugate priors differ in multi-parameter exponential families versus 1D distributions?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

Explore the comprehensive technical article, exam cards, and global architecture tree.

← PreviousM1-001: Explain Conditional Probability and the Law of Total Probability, and Why They Underpin Bayesian Inference📋Back to BankNext →M1-003: Define Independence vs. Conditional Independence, and Why Naive Bayes Works Despite Strong Violations
中文速览与面试要点

写出贝叶斯定理,并说明先验、似然、后验各自的角色。

后验 ∝ 似然 × 先验;先验编码已有信念,似然由数据提供,后验是更新后的信念。

核心要点

  • 贝叶斯定理把'学习'形式化为信念更新:posterior ∝ likelihood × prior。三个量的角色清晰可分—— 先验 p(θ) 编码在看到数据前对参数的信念(正则化/领域知识的载体); 似然 p(D|θ) 由数据与生成模型假设决定,是唯一与数据相关的项; 后验 p(θ|D) 是二者的乘积再归一化。分母 p(D)=∫p(D|θ)p(θ)dθ 与 θ 无关,因此常被省略为比例式。关键性质:数据量增加时,似然项的对数随 N 线性增长而先验固定,故先验影响被逐渐稀释,后验趋向 MLE——这是'数据够多时正则化不重要'的形式化表述。共轭先验(Beta-Bernoulli、Dirichlet-Multinomial、Normal-Normal)使后验与先验同族,可解析更新,是 A/B 测试中 Thompson Sampling 的基础。
  • 两个易被追问的区分:① MAP 不是'贝叶斯最优' ——MAP 只取后验众数一个点,丢弃了后验的不确定性;贝叶斯决策理论要求对后验做积分得到后验预测分布 p(y|x,D)=∫p(y|x,θ)p(θ|D)dθ,这在高维下不可解,故工程上退化为 MAP(等价于加正则的 MLE)。② 先验的选择是主观的但可检验 ——弱信息先验(如 Jeffreys 先验)尽量减少主观影响;若后验对先验敏感,说明数据信息不足,应报告先验敏感性分析。
  • 分母 P(D) 是归一化常数(证据)
  • 共轭先验使后验与先验同族,可解析更新
  • MAP = 取后验众数;MLE = 取似然众数(等价于均匀先验)

常见陷阱

  • 把后验分布与后验预测分布混淆(前者关于参数,后者关于新数据)
  • 认为 MAP 就是贝叶斯方法——它丢失了不确定性量化