How a language model is trained

In short: A language model is trained in stages: pre-training (predicting the next token on huge amounts of text), post-training (following instructions) and fine-tuning with human feedback. Pre-training takes almost all of the compute.

In more detail: In pre-training the model reads trillions of tokens from web pages, books, code, papers. At every position it should predict the next token. The error is the cross-entropy. In doing so it learns grammar, facts, patterns and reasoning schemes because they help prediction.

In detail

• Data: filtered, de-duplicated, mixed in quality tiers. Web crawls (Common Crawl), books, Wikipedia, code (GitHub), scientific texts, synthetic data.
• Scale: current large models see ten trillion tokens and more.
• Compute: roughly 6 · parameters · tokens floating-point operations (FLOPs). A 70-billion model with 15 trillion tokens needs about 6·10²⁴ FLOPs, weeks on thousands of GPUs.
• Stages:
  1. Pre-training (self-supervised).
  2. Context extension (longer examples).
  3. Instruction tuning and chat format (following directions).
  4. RLHF and preference optimisation (helpful, honest, harmless).
  5. Reasoning models and chain of thought (reinforcement learning on tasks with verifiable solutions).
• Scaling: more parameters, data and compute improve the loss predictably, see Scaling laws.
• Knowledge cutoff: the model knows only what was known up to the training data.
• Hardware: Training at scale: GPUs, parallelism and memory.

See also: How a language model generates text: step by step, Fine-tuning and LoRA