Word2Vec and the history of embeddings

In short: Word2Vec (2013) showed that words can be represented as vectors in which meaning is a direction: king − man + woman ≈ queen. Modern embeddings descend from this idea.

In more detail: The model learns from the neighbourhood of words in text: words that appear in similar surroundings get similar vectors (“you know a word by the company it keeps”).

In detail

• Word2Vec: two variants: skip-gram (predict surroundings from word) and CBOW (word from surroundings). Result: one fixed vector per word.
• GloVe: similar result from global word counts.
• Problem: one vector per word knows no context (“bank” at a river or money).
• Contextual embeddings: ELMo and BERT deliver different vectors per sentence. In transformers each token becomes a new vector with context in every layer.
• Sentence embeddings: one vector for a whole sentence or paragraph, trained with contrastive methods, basis of search, see Vector databases and similarity search.
• Bias: embeddings mirror stereotypes in the data (doctor – man, nurse – woman), see Bias, fairness and data quality.
• Today: the first module of a language model is an embedding table with one row per token, see Embedding (vector representation).

See also: Embedding (vector representation), Tokenisation: BPE and vocabulary