Point Estimation

Point estimation maps samples to a single parameter value: Maximum Likelihood Estimation (MLE) picks the parameter that maximizes the probability of the observed data, $\hat{\theta} = \arg\max_{\theta} \prod_{i=1}^{n} p(x_i \mid \theta)$; since log is monotonic, one instead maximizes the log-likelihood $\ell(\theta) = \sum_i \log p(x_i \mid \theta)$ and sets its derivative to zero.

the entry point for fitting nearly every parametric model — normal parameters, logistic regression, the M-step of EM; interviews typically ask for the normal-mean MLE derivation, Fisher information & CRLB, and MLE vs method of moments.

unlike the method of moments, which matches only the first $k$ sample moments (biased in small samples, possibly inefficient), MLE is asymptotically unbiased, consistent and efficient under regularity conditions, achieving the information-theoretic lower bound:

$\text{Var}(\hat{\theta}) \ge \frac{1}{I(\theta)}$ (Cramér–Rao bound), with Fisher information $I(\theta) = -\mathbb{E}[\frac{\partial^2}{\partial \theta^2} \log L(\theta)]$; the price is finite-sample bias (e.g. $\hat{\sigma}^2$ needs the $n-1$ correction) and numerical optimization.

Write the MLE definition and derive the MLE of a normal mean.

Why take the log-likelihood for MLE? Does it change the optimum?

Meaning of the Cramér–Rao bound and the definition of Fisher information?

MLE vs method of moments: differences, pros and cons?

Asymptotic properties of MLE (consistency, asymptotic normality, efficiency)?