Constitutional AI and rulebooks for models

In short: With constitutional AI you give the model a written set of principles. The model uses them to check and improve its own answers, instead of humans rating every single one.

The process

1. Draft: the model answers a sensitive question.
2. Critique: it gets a principle (“answer helpfully but do not help with anything dangerous”) and checks its answer against it.
3. Revision: it writes a better answer.
4. The (bad, better) pairs form a dataset for fine-tuning.
5. In a second phase the model itself judges which of two answers fits the principles better. These AI preferences partly replace human ratings, see RLHF and preference optimisation. This is called RLAIF.

Strengths and limits

• Rules are readable and changeable instead of hidden in thousands of single ratings.
• Less human work on sensitive content.
• The result is only as good as the principles and the judging model. Gaps and contradictions remain, see Safety, alignment and jailbreaks.

See also: Instruction tuning and chat format