Contrastive and self-supervised learning

In short: In self-supervised learning the system creates its learning task from the data itself. Contrastive learning pulls similar examples together in vector space and pushes dissimilar ones apart.

In more detail: This makes it possible to learn from huge amounts of unlabelled data, which the internet has in abundance.

In detail

• Language models: predict the next token or fill gaps (masked model).
• Images: reconstruct patches (MAE) or map two views of the same image close, others far (SimCLR, DINO).
• Contrastive loss (InfoNCE): positive pairs attract, negative ones repel. CLIP learns image-text pairs this way, see Vision transformer and CLIP.
• Sentence embeddings: two wordings of the same meaning should lie close. Important for search, see Vector databases and similarity search.
• Negative examples: too easy negatives teach nothing, “hard negatives” (almost similar) bring quality.
• Advantage: hardly any labelling needed, good general representations.
• Disadvantage: large batches and much compute.

See also: Kinds of learning: supervised, unsupervised, reinforcement, Word2Vec and the history of embeddings