Label Smoothing

label smoothing mixes the one-hot label with a uniform distribution: $y' = y(1-\epsilon) + \frac{\epsilon}{K}$, where $K$ is the number of classes and $\epsilon$ is typically 0.1; the true class gets $1-\epsilon+\frac{\epsilon}{K}$ and every other class gets $\frac{\epsilon}{K}$. It is equivalent to adding a KL penalty toward the uniform distribution to the cross-entropy loss.

standard for ImageNet-style classification (Inception / ResNet v2 families), Transformer machine translation, and LLM pretraining/finetuning to suppress overconfidence and improve calibration. Interviews ask for the formula, why it prevents overfitting, and its side effect on distillation.

it prevents extreme logits (confidence saturating to 1) on one-hot labels, improving generalization and calibration. Classic side effect (Müller et al. 2019, 'When Does Label Smoothing Help?'): smoothing flattens the relative logit information ('dark knowledge') of the teacher, so student distillation gains drop noticeably — when distillation is planned, the teacher is usually trained without label smoothing.

Write the label-smoothing formula $y' = y(1-\epsilon) + \frac{\epsilon}{K}$, explain the roles of $\epsilon$ and $K$, and state the usual value of $\epsilon$.

Why does label smoothing prevent overconfidence? How does it affect logit magnitudes and model calibration?

Why does label smoothing hurt knowledge distillation? What information in the teacher's output does it destroy?

What regularization term is label smoothing equivalent to adding to cross-entropy? Write the corresponding loss form.

In which scenarios should label smoothing be avoided? How does it interact with temperature scaling?