Transfer Learning

Transfer learning initializes a target task with weights pretrained on a large dataset (e.g. ImageNet). The freezing strategy follows feature hierarchy: shallow layers capture generic low-level features (edges/textures/colors) while deep layers encode task-specific semantics, so the norm is freezing early layers and training only the last few. Feature extraction freezes the whole backbone and trains only a new head (backbone lr = 0); fine-tuning continues training all or some parameters with a small learning rate. Catastrophic forgetting — new-task updates overwrite old-task knowledge — is mitigated by a smaller lr, partial freezing, or parameter regularization such as EWC.

the standard starting point for CV/NLP tasks with scarce labels (ImageNet-pretrained CNNs, instruction/domain fine-tuning of LLMs); interviews always ask freeze vs fine-tune.

with few labels, random initialization overfits and training from scratch needs millions of samples; transfer reuses generic representations learned on big data, so a few thousand images suffice and sample demand drops by orders of magnitude. Rule of thumb: little data + similar distribution → feature extraction; more data or large domain shift → fine-tuning; the costs — distribution mismatch and catastrophic forgetting — must be managed.

When to use feature extraction vs fine-tuning? What are the deciding factors?

Why are shallow features generic while deep features are task-specific?

Causes of catastrophic forgetting and how to mitigate it?

Why is fine-tuning's learning rate usually far smaller than from-scratch training?

What to do when the pretraining distribution differs significantly from the target?