Scaling laws

In short: Scaling laws describe that a language model’s loss falls predictably when parameters, training data and compute are increased together. The relationship follows roughly a power law.

In more detail: The OpenAI paper (2020) and DeepMind’s “Chinchilla” study (2022) showed that for a given compute budget model size and data amount should grow in balance: about 20 tokens per parameter.

In detail

• Power law: doubling the effort yields each time an equal percentage improvement of the loss, with diminishing returns.
• Chinchilla optimum: for 70 billion parameters about 1.4 trillion tokens. Many earlier models were too big and undertrained.
• Beyond the optimum: today smaller models are trained far longer (Llama 3 with 8 billion parameters on 15 trillion tokens) because inference is then cheaper. Training costs once, use costs permanently.
• Emergence: some abilities (multi-step arithmetic) seem to appear only from a certain size. Whether this is real or a measurement artefact is disputed.
• New axis: more compute at use time (thinking), see Reasoning models and chain of thought.
• Limits: high-quality texts are finite (“data wall”). Synthetic data and multiple epochs help.
• Costs: the frontier demands billion-dollar investment, see Energy consumption of AI.

See also: How a language model is trained, Parameter (AI model)