Ensemble Comparison

The three schools of ensemble learning. Bagging (random forests): base learners train independently and in parallel, each on its own bootstrap sample, voting/averaging at inference — base learners should be strong (deep, unpruned trees); it mainly reduces variance. Boosting (AdaBoost / GBDT / XGBoost / LightGBM): base learners are sequential and dependent, each round focusing on the previous residuals (negative-gradient pseudo-residuals) or misclassified samples (amplified weights), combined additively with weights — base learners should be weak (shallow trees); it mainly reduces bias. Stacking: heterogeneous base learners plus a meta-learner; the base learners' outputs (probabilities or classes) become meta-features for a second-level model (typically logistic regression or a linear model) that learns the optimal combination — it does not require homogeneous base learners and exploits model diversity to reduce both bias and variance; meta-features must be generated via cross-validation/OOF folds, otherwise the base learners have already seen the samples and the meta-features leak, overfitting the meta-model. Theory: bias-variance decomposition $\text{Err} = \text{bias}^2 + \text{variance} + \sigma_\epsilon^2$; averaging $k$ learners with variance $\sigma^2$ and pairwise correlation $\rho$ gives $\text{Var}\left(\bar{f}\right) = \rho \sigma^2 + \frac{1 - \rho}{k} \sigma^2$ — bagging reduces variance by lowering $\rho$ via sample/feature randomization and growing $k$, leaving bias unchanged; boosting reduces bias directly by fitting negative-gradient residuals round by round (weak learners are bias-dominated); stacking breaks error correlation through heterogeneity and lets a meta-learner learn the combination weights. Selection rule of thumb: high-variance settings (little data, strong models) → bagging; high-bias settings (weak models, underfitting) → boosting; multiple complementary strong models → stacking.

which ensemble to choose, why random forests resist overfitting while GBDT needs careful tuning, and why stacking uses OOF are high-frequency comparison questions — a memorized comparison table gives you a standard answer framework for model-selection interviews.

a single model is either bias-dominated or variance-dominated — the three schools offer three complementary combination strategies: bagging pushes high variance down to the $\rho \sigma^2$ level via randomization + averaging; boosting grinds down the bias of weak models via residual-focused additive modeling; stacking stitches complementary strong models together with a meta-learner, typically gaining 1–2 more points than plain voting/averaging — all three share the same bias-variance decomposition, making the gains explainable.

Test Your Knowledge: Practice Questions for "Ensemble Comparison"

Single choice pitfall questions with instant feedback and mistake tracking.