Transfer learning and pre-training

In short: In transfer learning you start with a model already trained for a big task and adapt it to your own with little data. That saves data, time and compute cost.

In more detail: Early layers of an image network recognise edges and textures that are useful in almost every task. Only the last layers need to learn anew.

In detail

• Procedure: load pre-trained model → replace the last layer → either train only this one (freeze the others) or fine-tune everything with a small learning rate.
• Images: pre-training on ImageNet, then plant recognition with 500 of your own photos.
• Language: pre-training on web texts, then task adaptation or prompting, see Fine-tuning and LoRA.
• Why it works: pre-trained models have learned general features.
• Risks: biases and licences of the base model are inherited, “catastrophic forgetting” when fine-tuning.
• Today: almost all AI applications start with a foundation model, see Hugging Face.

See also: Convolutional networks (CNN) for images, How a language model is trained