M2-025M2: Classical Machine LearningDecision TreesMedium
⚡Flashcard Drill➔Mastery:
Explain the Bias-Variance Profile of Decision Trees and Why Individual Trees Suffer from High Variance
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
A fully grown decision tree has low bias but high variance; small perturbations in training data flip top-level splits, cascading down to alter the entire tree structure and producing erratic predictions.
📌 Key Takeaways
- •Low Bias: Given sufficient depth, a decision tree can partition feature space until every leaf contains a single observation, achieving zero training error.
- •High Variance: Top-level greedy splits are highly sensitive to sample noise; swapping a few data points changes the root split, restructuring all subtrees.
- •Hierarchical Error Propagation: An error made at the root node cannot be corrected by lower nodes.
📐 Mathematical Derivations
Hierarchical instability mechanics: At the root node, candidate features and might have near-identical impurity reductions: and . In dataset , is selected, and all subsequent splits partition within sub-regions of . In a slightly perturbed dataset , noise elevates to , causing to be selected as root. The entire downstream geometric partition is completely disjoint from . In terms of variance: . Because and assign observations to completely different leaves with different local averages, prediction variance is massive.
🏭 Production Trade-offs
This fundamental bias-variance profile dictates ensemble design: (1) Bagging / Random Forests take low-bias, high-variance deep trees and average them together, reducing variance by factor without increasing bias. (2) Boosting (GBDT) takes high-bias, low-variance shallow stumps and sequentially fits residuals, reducing bias step-by-step.
⚠️ Common Interview Pitfalls
- ✕Deploying a single deep, unpruned decision tree to production (guarantees overfitting and high inference variance).
- ✕Attempting to reduce variance of a single tree by collecting more features (adds more noise to greedy split selection).
🎯 Interviewer Follow-up Inquiries
- ?Why does averaging independent estimators with variance reduce variance to ?
- ?How does Random Forest feature subsampling () decorrelate individual trees to maximize variance reduction?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚