Ensemble Comparison

The three schools of ensemble learning. Bagging (random forests): base learners train independently and in parallel, each on its own bootstrap sample, voting/averaging at inference — base learners should be strong (deep, unpruned trees); it mainly reduces variance. Boosting (AdaBoost / GBDT / XGBoost / LightGBM): base learners are sequential and dependent, each round focusing on the previous residuals (negative-gradient pseudo-residuals) or misclassified samples (amplified weights), combined additively with weights — base learners should be weak (shallow trees); it mainly reduces bias. Stacking: heterogeneous base learners plus a meta-learner; the base learners' outputs (probabilities or classes) become meta-features for a second-level model (typically logistic regression or a linear model) that learns the optimal combination — it does not require homogeneous base learners and exploits model diversity to reduce both bias and variance; meta-features must be generated via cross-validation/OOF folds, otherwise the base learners have already seen the samples and the meta-features leak, overfitting the meta-model. Theory: bias-variance decomposition $\text{Err} = \text{bias}^2 + \text{variance} + \sigma_\epsilon^2$; averaging $k$ learners with variance $\sigma^2$ and pairwise correlation $\rho$ gives $\text{Var}\left(\bar{f}\right) = \rho \sigma^2 + \frac{1 - \rho}{k} \sigma^2$ — bagging reduces variance by lowering $\rho$ via sample/feature randomization and growing $k$, leaving bias unchanged; boosting reduces bias directly by fitting negative-gradient residuals round by round (weak learners are bias-dominated); stacking breaks error correlation through heterogeneity and lets a meta-learner learn the combination weights. Selection rule of thumb: high-variance settings (little data, strong models) → bagging; high-bias settings (weak models, underfitting) → boosting; multiple complementary strong models → stacking.

which ensemble to choose, why random forests resist overfitting while GBDT needs careful tuning, and why stacking uses OOF are high-frequency comparison questions — a memorized comparison table gives you a standard answer framework for model-selection interviews.

a single model is either bias-dominated or variance-dominated — the three schools offer three complementary combination strategies: bagging pushes high variance down to the $\rho \sigma^2$ level via randomization + averaging; boosting grinds down the bias of weak models via residual-focused additive modeling; stacking stitches complementary strong models together with a meta-learner, typically gaining 1–2 more points than plain voting/averaging — all three share the same bias-variance decomposition, making the gains explainable.

Comparison table: Bagging vs Boosting vs Stacking across base-learner strength (strong/weak/heterogeneous), parallelism (parallel/sequential/two-stage), data usage (bootstrap/weighted/OOF), bias vs variance reduction, and representative algorithms

Bias-variance decomposition and the average variance $\text{Var}(\bar{f}) = \rho\sigma^2 + \frac{1-\rho}{k}\sigma^2$: why does bagging reduce variance but boosting reduce bias?

Why does bagging need strong base learners (deep unpruned trees) while boosting needs weak ones (shallow)? What breaks if reversed?

Essential difference of stacking: heterogeneous models + meta-learner; why must meta-features come from OOF folds (leakage prevention)?

Scenario selection: which ensemble for high-variance vs high-bias underfitting? When does stacking pay off most?