M2-102M2: Classical Machine LearningBias-Variance Tradeoff & Model SelectionMedium
⚡Flashcard Drill➔Mastery:
How Model Ensembling Impacts Bias and Variance
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Bagging reduces variance through parallel averaging; Boosting reduces bias through sequential residual fitting; Stacking optimizes combinations of diverse base hypotheses.
📌 Key Takeaways
- •Bagging: ; reduces variance, leaves bias unchanged
- •Boosting: constructs additive expansion targeting residuals; shrinks bias
- •Stacking: trains a meta-learner on out-of-fold base predictions, combining diverse inductive biases
📐 Mathematical Derivations
Mathematical Decomposition:
① Bagging Variance Reduction: Let base learners each have variance and pairwise correlation . The variance of the ensemble mean is: . As , the second term vanishes, leaving the irreducible floor . Decorrelating base learners (reducing via feature subsampling in Random Forest) directly minimizes this floor.
② Boosting Bias Reduction: Each sequential learner fits the pseudo-residuals , performing functional gradient descent that systematically contracts approximation bias at rate .
① Bagging Variance Reduction: Let base learners each have variance and pairwise correlation . The variance of the ensemble mean is: . As , the second term vanishes, leaving the irreducible floor . Decorrelating base learners (reducing via feature subsampling in Random Forest) directly minimizes this floor.
② Boosting Bias Reduction: Each sequential learner fits the pseudo-residuals , performing functional gradient descent that systematically contracts approximation bias at rate .
🏭 Production Trade-offs
Base learner pairing: Pair Bagging with high-variance, low-bias learners (deep, unpruned decision trees). Pair Boosting with low-variance, weak learners (shallow trees, decision stumps). In Stacking, ensure diverse algorithmic families (e.g., GBDT + Ridge + Neural Net) rather than homogenous models.
⚠️ Common Interview Pitfalls
- ✕Using Bagging on high-bias linear models, expecting variance reduction to improve underfitting
- ✕Ensembling identical or highly correlated models (), which yields near-zero variance reduction
🎯 Interviewer Follow-up Inquiries
- ?Why does Random Forest randomly select a subset of features at each split, and how does that affect in the variance formula?
- ?Why can boosting overfit noisy data if iterations continue without early stopping?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚