Bayesian Inference

Bayesian inference treats the parameter $\theta$ as a random variable and combines the prior $p(\theta)$ with the likelihood $p(x|\theta)$ into a posterior distribution: $p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)} = \frac{p(x|\theta)p(\theta)}{\int p(x|\theta)p(\theta)\,d\theta}$, i.e. posterior $\propto$ likelihood $\times$ prior, where the evidence $p(x)$ is merely a normalizing constant. Conjugate priors keep the posterior in the same family and analytically tractable: Beta prior + Binomial likelihood → Beta posterior; Normal prior + Normal likelihood → Normal posterior; Gamma prior + Poisson likelihood → Gamma posterior.

small-sample estimation, online learning (sequentially reusing the posterior as the next prior), uncertainty quantification (posterior variance = parameter uncertainty), Bayesian neural networks and Bayesian optimization; interviews typically open with the MLE / MAP / posterior comparison.

the frequentist framework only yields point estimates and confidence intervals, with no way to inject prior knowledge or express parameter uncertainty; Bayes gives a full posterior distribution where the prior regularizes small-sample estimates against overfitting. The cost: the posterior integral usually has no closed form — requiring conjugate priors, MCMC, or variational approximations.

Test Your Knowledge: Practice Questions for "Bayesian Inference"

Single choice pitfall questions with instant feedback and mistake tracking.