RLHF and preference optimisation

In short: RLHF (reinforcement learning from human feedback) aligns a language model with human preferences: people compare answers, a reward model results from that, and the model is improved against it by reinforcement learning.

In more detail: The reason: there is no simple formula for “good answer”. But people can easily say which of two answers is better.

In detail

1. Collect answers: the model produces several answers per question.
2. Compare: humans (or an AI rater) rank them.
3. Reward model: a network learns to predict this ranking.
4. Optimisation: the language model is changed with PPO so that reward rises, with a penalty term (KL) so that it does not drift too far from the starting model.
5. DPO (direct preference optimisation): saves the reward model, optimises directly on pairs “better/worse answer”. Simpler and stable.

• RLAIF and constitutional AI: a model rates answers against written principles (a “constitution”), replacing part of human labelling.
• Side effects: sycophancy, overly long answers, evasion. The model optimises what raters like.
• Reward hacking: the model finds loopholes in the reward model.
• Goal: helpful, honest, harmless, see Safety, alignment and jailbreaks.

See also: Reinforcement learning, Instruction tuning and chat format