Point Estimation

Point estimation maps samples to a single parameter value: Maximum Likelihood Estimation (MLE) picks the parameter that maximizes the probability of the observed data, $\hat{\theta} = \arg\max_{\theta} \prod_{i=1}^{n} p(x_i \mid \theta)$; since log is monotonic, one instead maximizes the log-likelihood $\ell(\theta) = \sum_i \log p(x_i \mid \theta)$ and sets its derivative to zero.

the entry point for fitting nearly every parametric model — normal parameters, logistic regression, the M-step of EM; interviews typically ask for the normal-mean MLE derivation, Fisher information & CRLB, and MLE vs method of moments.

unlike the method of moments, which matches only the first $k$ sample moments (biased in small samples, possibly inefficient), MLE is asymptotically unbiased, consistent and efficient under regularity conditions, achieving the information-theoretic lower bound:

$\text{Var}(\hat{\theta}) \ge \frac{1}{I(\theta)}$ (Cramér–Rao bound), with Fisher information $I(\theta) = -\mathbb{E}[\frac{\partial^2}{\partial \theta^2} \log L(\theta)]$; the price is finite-sample bias (e.g. $\hat{\sigma}^2$ needs the $n-1$ correction) and numerical optimization.

Test Your Knowledge: Practice Questions for "Point Estimation"

Single choice pitfall questions with instant feedback and mistake tracking.