M2-022M2: Classical Machine LearningDecision TreesEasy
⚡Flashcard Drill➔Mastery:
Explain Decision Tree Splitting Criteria: Information Gain, Gain Ratio, and Gini Impurity
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Decision trees split nodes to maximize the reduction in impurity: Information Gain uses Shannon Entropy, Gain Ratio normalizes by split entropy, and Gini Impurity uses quadratic misclassification probability.
📌 Key Takeaways
- •Entropy (ID3): ; Information Gain .
- •Gain Ratio (C4.5): , penalizing high-cardinality features.
- •Gini Impurity (CART): ; computationally faster (no logarithms), standard in scikit-learn.
📐 Mathematical Derivations
Gini Impurity interpretation: . This represents the probability that a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the class distribution. Comparison with Entropy: Taylor expanding around : . Thus . In binary classification with : is a parabola with maximum at ; Entropy peaks at . Both are strictly concave, ensuring that any split strictly reduces expected impurity.
🏭 Production Trade-offs
Algorithmic choices: (1) ID3 vs C4.5: ID3 suffers from extreme bias toward high-cardinality features (e.g. Splitting on
User_ID creates pure leaves with , but zero generalization power). C4.5 divides by to penalize broad fan-outs. (2) CART: Uses Gini impurity and strictly binary splits, eliminating the need for expensive logarithmic evaluations and producing balanced binary trees.⚠️ Common Interview Pitfalls
- ✕Using Information Gain on high-cardinality IDs or timestamps without Gain Ratio normalization.
- ✕Believing Gini and Entropy produce substantially different decision trees (in 98% of cases, both select identical split points).
🎯 Interviewer Follow-up Inquiries
- ?Why does CART enforce strictly binary splits while C4.5 allows multi-way splits?
- ?What is the variance reduction split criterion used for regression trees in CART?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚