Target Encoding

Target encoding encodes high-cardinality categorical features (user_id, city, item ID) with the mean target per category. Smoothing formula: $TE = \lambda(n_i) \cdot \text{mean}(y_i) + (1 - \lambda(n_i)) \cdot \text{global\_mean}$, where the confidence weight is $\lambda(n_i) = \frac{n_i}{n_i + m}$ — the larger the category count $n_i$, the more the within-category mean is trusted, and the smoothing strength $m$ controls how hard the estimate shrinks toward the global mean, protecting small groups from noisy means. Leakage risk: computing group means on the full dataset makes both train and validation features carry target information (inflated CV, production collapse) — the biggest trap of TE. Fixes: out-of-fold (OOF) encoding — for each sample, compute the mean from the folds that did not contain it; CatBoost goes further with Ordered Target Statistics — compute statistics using only history samples preceding the current one, structurally eliminating leakage.

high-cardinality categorical features with tree models (GBDT/LightGBM) and Kaggle competitions; interviews ask “why not one-hot”, “how to prevent TE leakage”, “how to choose m”.

one-hot explodes dimensionality for high cardinality (1M user_ids become 1M columns, memory and sparsity disaster); raw group-mean encoding leaks the target and overfits small groups; smoothing shrinkage plus OOF/Ordered encoding keeps the category-target association while controlling leakage and overfitting — CatBoost replaces one-hot with Ordered encoding precisely so categorical features can join training directly.

Test Your Knowledge: Practice Questions for "Target Encoding"

Single choice pitfall questions with instant feedback and mistake tracking.