Bayesian Inference

Bayesian inference treats the parameter $\theta$ as a random variable and combines the prior $p(\theta)$ with the likelihood $p(x|\theta)$ into a posterior distribution: $p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)} = \frac{p(x|\theta)p(\theta)}{\int p(x|\theta)p(\theta)\,d\theta}$, i.e. posterior $\propto$ likelihood $\times$ prior, where the evidence $p(x)$ is merely a normalizing constant. Conjugate priors keep the posterior in the same family and analytically tractable: Beta prior + Binomial likelihood → Beta posterior; Normal prior + Normal likelihood → Normal posterior; Gamma prior + Poisson likelihood → Gamma posterior.

small-sample estimation, online learning (sequentially reusing the posterior as the next prior), uncertainty quantification (posterior variance = parameter uncertainty), Bayesian neural networks and Bayesian optimization; interviews typically open with the MLE / MAP / posterior comparison.

the frequentist framework only yields point estimates and confidence intervals, with no way to inject prior knowledge or express parameter uncertainty; Bayes gives a full posterior distribution where the prior regularizes small-sample estimates against overfitting. The cost: the posterior integral usually has no closed form — requiring conjugate priors, MCMC, or variational approximations.

Write the Bayesian posterior formula and explain each term? How does the posterior change with data size?

Differences among MLE, MAP and Bayesian inference? What does MAP estimate from the posterior?

What is a conjugate prior? Give the Beta-Binomial and Normal-Normal examples.

Bayesian advantages over MLE with small samples? How does the prior act as regularization?

Approximation methods when the posterior is intractable (MCMC, variational inference, Laplace approximation)?