M2-102M2: Classical Machine LearningBias-Variance Tradeoff & Model SelectionMedium
⚡Flashcard Drill➔Mastery:

How Model Ensembling Impacts Bias and Variance

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-103 →
📐 Mathematical Definition
Var(fˉ)=ρσ2+1−ρBσ2,Bias(fˉ)≈Bias(f)\mathrm{Var}(\bar f)=\rho\sigma^2+\frac{1-\rho}{B}\sigma^2,\qquad \mathrm{Bias}(\bar f)\approx\mathrm{Bias}(f)
⚡ One-Sentence Core Answer
Bagging reduces variance through parallel averaging; Boosting reduces bias through sequential residual fitting; Stacking optimizes combinations of diverse base hypotheses.

📌 Key Takeaways

  • •
    Bagging: Var(fˉ)=ρσ2+1−ρBσ2\text{Var}(\bar{f}) = \rho \sigma^2 + \frac{1-\rho}{B} \sigma^2; reduces variance, leaves bias unchanged
  • •
    Boosting: constructs additive expansion Fm(x)=Fm−1(x)+γmhm(x)F_m(x) = F_{m-1}(x) + \gamma_m h_m(x) targeting residuals; shrinks bias
  • •
    Stacking: trains a meta-learner on out-of-fold base predictions, combining diverse inductive biases

📐 Mathematical Derivations

Mathematical Decomposition:
① Bagging Variance Reduction: Let BB base learners each have variance σ2\sigma^2 and pairwise correlation ρ\rho. The variance of the ensemble mean fˉ=1B∑i=1Bfi\bar{f} = \frac{1}{B} \sum_{i=1}^B f_i is: Var(fˉ)=ρσ2+1−ρBσ2\text{Var}(\bar{f}) = \rho \sigma^2 + \frac{1 - \rho}{B} \sigma^2. As B→∞B \to \infty, the second term vanishes, leaving the irreducible floor ρσ2\rho \sigma^2. Decorrelating base learners (reducing ρ\rho via feature subsampling in Random Forest) directly minimizes this floor.
② Boosting Bias Reduction: Each sequential learner hmh_m fits the pseudo-residuals rim=−[∂L(yi,F(xi))∂F(xi)]F=Fm−1r_{im} = -\left[\frac{\partial \mathcal{L}(y_i, F(x_i))}{\partial F(x_i)}\right]_{F=F_{m-1}}, performing functional gradient descent that systematically contracts approximation bias at rate O(e−cm)O(e^{-c m}).

🏭 Production Trade-offs

Base learner pairing: Pair Bagging with high-variance, low-bias learners (deep, unpruned decision trees). Pair Boosting with low-variance, weak learners (shallow trees, decision stumps). In Stacking, ensure diverse algorithmic families (e.g., GBDT + Ridge + Neural Net) rather than homogenous models.
⚠️ Common Interview Pitfalls
  • ✕
    Using Bagging on high-bias linear models, expecting variance reduction to improve underfitting
  • ✕
    Ensembling identical or highly correlated models (ρ≈1\rho \approx 1), which yields near-zero variance reduction
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Random Forest randomly select a subset of features at each split, and how does that affect ρ\rho in the variance formula?
  • ?
    Why can boosting overfit noisy data if iterations continue without early stopping?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-101: Learning Curves vs Validation Curves: Formulations and Diagnostic Workflows📋 Back to BankNext →M2-103: When Does Adding More Data Fail to Improve Model Performance?