Encoder, decoder and the model families (BERT, GPT, T5)

In short: There are three designs of transformers: encoder only (BERT, understands text), decoder only (GPT, generates text) and encoder-decoder (T5, turns input into output). Almost all of today’s chat models are pure decoders.

In more detail: The difference lies in the attention mask: encoders look in both directions, decoders only to the left.

In detail

• Encoder (BERT, 2018): bidirectional. Trained by fill-in-the-blank texts (“masked language model”). Good for classification, search and embeddings. Generates no text.
• Decoder (GPT): causal, predicts the next token. Generates text, the basis of ChatGPT, Claude, Llama, Gemini and others.
• Encoder-decoder (T5, BART, original transformer): the encoder reads the input, the decoder writes the output and looks into the encoder via cross-attention. Good for translation and summarisation.
• Why decoders dominate: one single training (“next token”) on huge amounts of text that can be used with prompts for almost any task.
• Embedding models: mostly encoders or converted decoders, see Embedding (vector representation).
• Whisper: encoder-decoder for speech to text.

See also: Transformer architecture, How a language model generates text: step by step