M2-112M2: Classical Machine LearningFeature EngineeringMedium
⚡Flashcard Drill➔Mastery:

Target Encoding Implementation Details: Out-of-Fold Cross-Fitting and Smoothing

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-113 →
📐 Mathematical Definition
enc(c)=nc⋅yˉc+m⋅yˉglobalnc+m\text{enc}(c)=\frac{n_c\cdot\bar y_c+m\cdot\bar y_{global}}{n_c+m}
⚡ One-Sentence Core Answer
Target encoding maps categories to expected target values; prevent severe overfitting using Empirical Bayes smoothing and strict out-of-fold cross-fitting.

📌 Key Takeaways

  • •
    Smoothing formula: S^c=λ(nc)yˉc+(1−λ(nc))yˉglobal\hat{S}_c = \lambda(n_c) \bar{y}_c + (1 - \lambda(n_c)) \bar{y}_{\text{global}} with sigmoid or m-estimate weight
  • •
    Overfitting risk: rare categories with nc=1n_c=1 trivially leak the target label without smoothing
  • •
    Out-of-fold (OOF) cross-fitting: encode training fold kk using statistics computed strictly on the other K−1K-1 folds

📐 Mathematical Derivations

Mathematical Formulation:
① Smoothing (Empirical Bayes / Micci-Barreca):
x^c=λ(nc)yˉc+(1−λ(nc))μglobal\hat{x}_c = \lambda(n_c) \bar{y}_c + (1 - \lambda(n_c)) \mu_{\text{global}}, where yˉc=1nc∑i∈cyi\bar{y}_c = \frac{1}{n_c} \sum_{i \in c} y_i and μ=1N∑i=1Nyi\mu = \frac{1}{N} \sum_{i=1}^N y_i.
Smoothing weight function: λ(nc)=11+e−(nc−k)/f\lambda(n_c) = \frac{1}{1 + e^{-(n_c - k)/f}} or mm-estimate λ(nc)=ncnc+m\lambda(n_c) = \frac{n_c}{n_c + m}. When sample count ncn_c is small, λ→0\lambda \to 0, pulling the estimate toward the global prior μ\mu. When ncn_c is large, λ→1\lambda \to 1, trusting empirical category mean yˉc\bar{y}_c.
② Out-of-Fold (OOF) Computation:
Partition training set into KK folds. For each fold kk:
Compute yˉc(−k)\bar{y}_c^{(-k)} using only data from D∖DkD \setminus D_k. Apply this mapping to encode instances in DkD_k. For test data, encode using global statistics from all KK folds.

🏭 Production Trade-offs

Industrial implementation: In high-cardinality tabular datasets (e.g., zip codes, merchant IDs), smoothed OOF target encoding outperforms one-hot encoding by orders of magnitude in both memory footprint and tree split efficiency.
⚠️ Common Interview Pitfalls
  • ✕
    Computing target encoding globally on the entire training set without out-of-fold partitioning, causing catastrophic target leakage
  • ✕
    Omitting smoothing on rare categories, allowing single positive instances to be encoded as pure 1.01.0
🎯 Interviewer Follow-up Inquiries
  • ?
    How does CatBoost's ordered target encoding eliminate the need for standard K-fold cross-fitting?
  • ?
    Why must small additive Gaussian noise often be injected into target-encoded features during training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-111: Differences Between GBDT Built-in Feature Importance and TreeSHAP Values📋 Back to BankNext →M2-113: Text Features: Comparing TF-IDF and Dense Embedding Representations