M2-062M2: Classical Machine LearningFeature EngineeringEasy
⚡Flashcard Drill➔Mastery:
Common Categorical Feature Encoding Methods and Applicable Scenarios
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
One-hot for low cardinality, target encoding for high cardinality, frequency encoding for prevalence, hashing for unbounded streams, and embeddings for deep architectures.
📌 Key Takeaways
- •One-hot: low cardinality (< 20 levels), linear models and neural networks
- •Target encoding: high cardinality, requires out-of-fold smoothing to prevent leakage
- •Embeddings: dense representation capable of capturing semantic similarities in deep learning
📐 Mathematical Derivations
Trade-offs across categorical encodings: ① One-Hot Encoding: Generates an indicator vector per category. Suitable for low cardinality (). In tree models, it degrades efficiency as splits become sparse and unbalanced. ② Target Encoding: Encodes categories as the smoothed expectation of the target , where . Crucial: Must use out-of-fold (OOF) cross-fitting and additive noise to prevent target leakage. ③ Frequency/Count Encoding: Maps categories to their frequency . Effective when prevalence correlates with the target. ④ Feature Hashing (Hashing Trick): Uses hash functions to project categories into a fixed-size space . Handles open vocabularies and streaming inputs at the cost of hash collisions. ⑤ Entity Embeddings: Maps high-cardinality discrete IDs to continuous vectors via lookup tables, capturing non-linear relationships.
🏭 Production Trade-offs
Engineering trade-offs: ① Tree Models vs Linear Models: Gradient boosted trees (LightGBM/CatBoost) handle integer/categorical splits natively without one-hot expansion. Linear models require one-hot or target encoding. ② High-Cardinality Cold Start: Target encoding and frequency encoding require a robust fallback (global prior) for unseen categories. ③ Dimension vs Collision: In hashing, balance memory footprint against collision rates.
⚠️ Common Interview Pitfalls
- ✕Using global target statistics without out-of-fold cross-validation, causing severe target leakage
- ✕Applying one-hot encoding to features with thousands of levels, causing dimensionality explosion and memory exhaustion
🎯 Interviewer Follow-up Inquiries
- ?How does CatBoost prevent target leakage in its ordered target encoding implementation?
- ?When would you choose frequency encoding over target encoding?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚