M1-038M1: Mathematics & Statistics Fundamentals估计理论 (MLE/MAP)Medium
⚡Flashcard Drill➔Mastery:

Explain the Bias-Variance Decomposition and Write the Decomposition for Expected Generalization Error

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-039 →
📐 Mathematical Definition
E[(y−f^)2]=Bias2[f^]+Var[f^]+σ2\mathbb E[(y-\hat f)^2]=\mathrm{Bias}^2[\hat f]+\mathrm{Var}[\hat f]+\sigma^2
⚡ One-Sentence Core Answer
Expected prediction error decomposes into Bias squared (underfitting from wrong model assumptions), Variance (overfitting from sensitivity to training data noise), and Irreducible Error (inherent label noise).

📌 Key Takeaways

  • •
    Decomposition: EPE(x)=Bias2(f^(x))+Var(f^(x))+σϵ2\text{EPE}(x) = \text{Bias}^2(\hat{f}(x)) + \text{Var}(\hat{f}(x)) + \sigma_\epsilon^2.
  • •
    Bias: (E[f^(x)]−f(x))2(E[\hat{f}(x)] - f(x))^2; deviation of average model prediction from ground truth.
  • •
    Variance: E[(f^(x)−E[f^(x)])2]E[(\hat{f}(x) - E[\hat{f}(x)])^2]; variability of model predictions across different training sets.
  • •
    Irreducible error: σϵ2=Var(ϵ)\sigma_\epsilon^2 = \text{Var}(\epsilon); inherent noise in data generating process.

📐 Mathematical Derivations

Let y=f(x)+ϵy = f(x) + \epsilon with E[ϵ]=0,Var(ϵ)=σ2E[\epsilon]=0, \text{Var}(\epsilon)=\sigma^2. The expected squared error of estimator f^\hat{f} at point xx over all datasets DD is E[(y−f^(x))2]=E[(f(x)+ϵ−f^(x))2]=E[((f(x)−E[f^(x)])+(E[f^(x)]−f^(x))+ϵ)2]E[(y - \hat{f}(x))^2] = E[(f(x) + \epsilon - \hat{f}(x))^2] = E[((f(x) - E[\hat{f}(x)]) + (E[\hat{f}(x)] - \hat{f}(x)) + \epsilon)^2]. Expanding the square: cross-terms with ϵ\epsilon vanish because ϵ\epsilon is independent of training data (E[ϵ]=0E[\epsilon]=0). The cross-term between (f−E[f^])(f - E[\hat{f}]) (a constant) and (E[f^]−f^)(E[\hat{f}] - \hat{f}) vanishes because E[E[f^]−f^]=0E[E[\hat{f}] - \hat{f}] = 0. What remains is: (f(x)−E[f^(x)])2+E[(f^(x)−E[f^(x)])2]+E[ϵ2]=Bias2+Variance+σ2(f(x) - E[\hat{f}(x)])^2 + E[(\hat{f}(x) - E[\hat{f}(x)])^2] + E[\epsilon^2] = \text{Bias}^2 + \text{Variance} + \sigma^2.

🏭 Production Trade-offs

The classical bias-variance tradeoff: Increasing model capacity (parameters, depth) reduces bias but increases variance. Modern deep learning exhibits the 'Double Descent' phenomenon (Belkin et al., 2019): beyond the interpolation threshold where parameters exceed data points (P>NP > N), variance peaks and then paradoxically decreases due to the implicit regularization of gradient descent finding minimum-norm interpolators.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing sample variance of predictions on a single test set with the model variance over different hypothetical training sets.
  • ✕
    Believing deep neural networks always suffer from catastrophic variance in overparameterized regimes.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Bagging (Random Forest) reduce variance without increasing bias?
  • ?
    How does Boosting (GBDT) primarily reduce bias while keeping variance bounded?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-037: Define Maximum A Posteriori (MAP) Estimation and Explain Its Exact Connection to MLE and Regularization📋 Back to BankNext →M1-039: Define Statistical Consistency and Asymptotic Efficiency, and Assess Whether MLE Satisfies Both