M2-027M2: Classical Machine Learning集成方法 (Bagging/RF)Easy
⚡Flashcard Drill➔Mastery:
Compare the Core Theoretical and Architectural Differences Between Bagging and Boosting
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Bagging builds independent, deep trees in parallel on bootstrap samples to reduce variance; Boosting builds sequential, shallow trees iteratively on gradient residuals to reduce bias.
📌 Key Takeaways
- •Training Paradigm: Bagging is parallel and independent; Boosting is sequential and dependent ( corrects errors of ).
- •Base Learners: Bagging uses high-capacity, low-bias deep learners (unpruned trees); Boosting uses low-capacity, low-variance weak learners (shallow stumps).
- •Primary Mechanism: Bagging averages predictions to reduce variance: ; Boosting performs gradient descent in function space to reduce bias.
📐 Mathematical Derivations
Variance reduction in Bagging: Let trees have individual prediction variance with pairwise correlation . The variance of the ensemble average is: . As , the second term vanishes, leaving . This proves that the performance bound of Bagging is dictated entirely by tree correlation , motivating Random Forest feature subsampling to drive . In Boosting, the sequential update is where , shrinking residual bias exponentially.
🏭 Production Trade-offs
Industrial selection: (1) Random Forest (Bagging): Highly robust to hyperparameters, impossible to overfit by simply adding more trees, embarrassingly parallel, ideal for noisy datasets. (2) GBDT / LightGBM (Boosting): Consistently achieves higher predictive accuracy on clean tabular benchmarks, but requires careful tuning of learning rate , tree depth, and early stopping to prevent overfitting.
⚠️ Common Interview Pitfalls
- ✕Growing shallow trees in Random Forest (Bagging requires low-bias base learners; shallow trees cause severe underfitting).
- ✕Setting learning rate too high in GBDT without early stopping, causing rapid overfitting.
🎯 Interviewer Follow-up Inquiries
- ?Why can Random Forest never overfit simply by increasing the number of trees ?
- ?How does Gradient Boosting interpret sequential tree fitting as functional gradient descent in function space?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚