M2-023M2: Classical Machine LearningDecision TreesEasy
⚡Flashcard Drill➔Mastery:
Detail Pre-Pruning and Post-Pruning Strategies to Prevent Overfitting in Decision Trees
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Decision trees prevent overfitting via Pre-Pruning (stopping growth early via depth and sample limits) and Post-Pruning (growing a full tree and pruning subtrees using Cost-Complexity Pruning).
📌 Key Takeaways
- •Pre-Pruning parameters:
max_depth(restricts tree levels),min_samples_split(minimum samples to allow split),min_samples_leaf(minimum samples per terminal leaf),min_impurity_decrease. - •Post-Pruning (Cost-Complexity Pruning): Minimizes , balancing empirical misclassification against tree leaf count .
- •Why Post-Pruning is superior: Pre-pruning suffers from 'myopia' (stops early before reaching split combinations that together provide high gain); post-pruning evaluates full context.
📐 Mathematical Derivations
Minimal Cost-Complexity Pruning (Breiman et al., 1984): For any subtree rooted at node , the cost of collapsing into a single leaf node is . The cost of keeping the subtree is . The subtree is preferred as long as . Setting these equal defines the critical threshold . As complexity penalty increases from 0 to , it generates a nested sequence of pruned subtrees . Cross-validation is used to select optimal (parameter
ccp_alpha in scikit-learn).🏭 Production Trade-offs
In industrial pipelines: (1) Standalone decision trees require aggressive pre-pruning or post-pruning to prevent memorizing training noise. (2) In Random Forests, individual trees are deliberately grown deep and unpruned (maximizing individual tree capacity and diversity) because variance is subsequently eliminated by ensemble averaging. (3) In GBDT, trees are shallow (depth 3 to 6) weak learners.
⚠️ Common Interview Pitfalls
- ✕Setting
min_samples_leaftoo small in noisy datasets (leads to leaves fitting individual outlier observations). - ✕Relying solely on
max_depthwithout tuningmin_samples_split.
🎯 Interviewer Follow-up Inquiries
- ?How does scikit-learn
CostComplexityPruningcompute the sequence of effective alphas? - ?Why are shallow trees (depth 3-6) preferred in Gradient Boosting while deep trees are preferred in Random Forests?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚