M1-038M1: Mathematics & Statistics Fundamentals估计理论 (MLE/MAP)Medium
⚡Flashcard Drill➔Mastery:
Explain the Bias-Variance Decomposition and Write the Decomposition for Expected Generalization Error
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Expected prediction error decomposes into Bias squared (underfitting from wrong model assumptions), Variance (overfitting from sensitivity to training data noise), and Irreducible Error (inherent label noise).
📌 Key Takeaways
- •Decomposition: .
- •Bias: ; deviation of average model prediction from ground truth.
- •Variance: ; variability of model predictions across different training sets.
- •Irreducible error: ; inherent noise in data generating process.
📐 Mathematical Derivations
Let with . The expected squared error of estimator at point over all datasets is . Expanding the square: cross-terms with vanish because is independent of training data (). The cross-term between (a constant) and vanishes because . What remains is: .
🏭 Production Trade-offs
The classical bias-variance tradeoff: Increasing model capacity (parameters, depth) reduces bias but increases variance. Modern deep learning exhibits the 'Double Descent' phenomenon (Belkin et al., 2019): beyond the interpolation threshold where parameters exceed data points (), variance peaks and then paradoxically decreases due to the implicit regularization of gradient descent finding minimum-norm interpolators.
⚠️ Common Interview Pitfalls
- ✕Confusing sample variance of predictions on a single test set with the model variance over different hypothetical training sets.
- ✕Believing deep neural networks always suffer from catastrophic variance in overparameterized regimes.
🎯 Interviewer Follow-up Inquiries
- ?How does Bagging (Random Forest) reduce variance without increasing bias?
- ?How does Boosting (GBDT) primarily reduce bias while keeping variance bounded?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚