M2-025M2: Classical Machine LearningDecision TreesMedium
⚡Flashcard Drill➔Mastery:

Explain the Bias-Variance Profile of Decision Trees and Why Individual Trees Suffer from High Variance

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-026 →
📐 Mathematical Definition
Var↑ with depth\mathrm{Var}\uparrow\ \text{with depth}
⚡ One-Sentence Core Answer
A fully grown decision tree has low bias but high variance; small perturbations in training data flip top-level splits, cascading down to alter the entire tree structure and producing erratic predictions.

📌 Key Takeaways

  • •
    Low Bias: Given sufficient depth, a decision tree can partition feature space until every leaf contains a single observation, achieving zero training error.
  • •
    High Variance: Top-level greedy splits are highly sensitive to sample noise; swapping a few data points changes the root split, restructuring all subtrees.
  • •
    Hierarchical Error Propagation: An error made at the root node cannot be corrected by lower nodes.

📐 Mathematical Derivations

Hierarchical instability mechanics: At the root node, candidate features x1x_1 and x2x_2 might have near-identical impurity reductions: ΔI(x1)=0.421\Delta I(x_1) = 0.421 and ΔI(x2)=0.419\Delta I(x_2) = 0.419. In dataset D1D_1, x1x_1 is selected, and all subsequent splits partition within sub-regions of x1x_1. In a slightly perturbed dataset D2D_2, noise elevates x2x_2 to ΔI(x2)=0.422\Delta I(x_2) = 0.422, causing x2x_2 to be selected as root. The entire downstream geometric partition is completely disjoint from D1D_1. In terms of variance: Var(f^(x))=E[(f^(x;D)−E[f^(x)])2]\text{Var}(\hat{f}(x)) = E[(\hat{f}(x; D) - E[\hat{f}(x)])^2]. Because f^(x;D1)\hat{f}(x; D_1) and f^(x;D2)\hat{f}(x; D_2) assign observations to completely different leaves with different local averages, prediction variance is massive.

🏭 Production Trade-offs

This fundamental bias-variance profile dictates ensemble design: (1) Bagging / Random Forests take low-bias, high-variance deep trees and average them together, reducing variance by factor 1/B1/B without increasing bias. (2) Boosting (GBDT) takes high-bias, low-variance shallow stumps and sequentially fits residuals, reducing bias step-by-step.
⚠️ Common Interview Pitfalls
  • ✕
    Deploying a single deep, unpruned decision tree to production (guarantees overfitting and high inference variance).
  • ✕
    Attempting to reduce variance of a single tree by collecting more features (adds more noise to greedy split selection).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does averaging BB independent estimators with variance σ2\sigma^2 reduce variance to σ2/B\sigma^2 / B?
  • ?
    How does Random Forest feature subsampling (m=dm = \sqrt{d}) decorrelate individual trees to maximize variance reduction?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-024: Explain Why Decision Trees are Invariant to Monotonic Transformations and Their Inductive Bias📋 Back to BankNext →M2-026: Compare Industrial Strategies for Handling Missing Values and Categorical Features in Tree Models