M2-062M2: Classical Machine LearningFeature EngineeringEasy
⚡Flashcard Drill➔Mastery:

Common Categorical Feature Encoding Methods and Applicable Scenarios

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-063 →
📐 Mathematical Definition
target enc: yˉcat with smoothing\text{target enc}:\ \bar y_{\text{cat}}\ \text{with smoothing}
⚡ One-Sentence Core Answer
One-hot for low cardinality, target encoding for high cardinality, frequency encoding for prevalence, hashing for unbounded streams, and embeddings for deep architectures.

📌 Key Takeaways

  • •
    One-hot: low cardinality (< 20 levels), linear models and neural networks
  • •
    Target encoding: high cardinality, requires out-of-fold smoothing to prevent leakage
  • •
    Embeddings: dense representation capable of capturing semantic similarities in deep learning

📐 Mathematical Derivations

Trade-offs across categorical encodings: ① One-Hot Encoding: Generates an indicator vector per category. Suitable for low cardinality (<20< 20). In tree models, it degrades efficiency as splits become sparse and unbalanced. ② Target Encoding: Encodes categories as the smoothed expectation of the target S^c=λ(nc)yˉc+(1−λ(nc))yˉ\hat{S}_c = \lambda(n_c) \bar{y}_c + (1 - \lambda(n_c)) \bar{y}, where λ(nc)=ncnc+m\lambda(n_c) = \frac{n_c}{n_c + m}. Crucial: Must use out-of-fold (OOF) cross-fitting and additive noise to prevent target leakage. ③ Frequency/Count Encoding: Maps categories to their frequency nc/Nn_c / N. Effective when prevalence correlates with the target. ④ Feature Hashing (Hashing Trick): Uses hash functions to project categories into a fixed-size space h(x)(modB)h(x) \pmod B. Handles open vocabularies and streaming inputs at the cost of hash collisions. ⑤ Entity Embeddings: Maps high-cardinality discrete IDs to continuous vectors via lookup tables, capturing non-linear relationships.

🏭 Production Trade-offs

Engineering trade-offs: ① Tree Models vs Linear Models: Gradient boosted trees (LightGBM/CatBoost) handle integer/categorical splits natively without one-hot expansion. Linear models require one-hot or target encoding. ② High-Cardinality Cold Start: Target encoding and frequency encoding require a robust fallback (global prior) for unseen categories. ③ Dimension vs Collision: In hashing, balance memory footprint against collision rates.
⚠️ Common Interview Pitfalls
  • ✕
    Using global target statistics without out-of-fold cross-validation, causing severe target leakage
  • ✕
    Applying one-hot encoding to features with thousands of levels, causing dimensionality explosion and memory exhaustion
🎯 Interviewer Follow-up Inquiries
  • ?
    How does CatBoost prevent target leakage in its ordered target encoding implementation?
  • ?
    When would you choose frequency encoding over target encoding?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-061: How to Choose Target Dimensions in Dimensionality Reduction? Variance Ratio Method and Limitations📋 Back to BankNext →M2-063: Why Bin Numerical Features? Common Binning Strategies and Use Cases