Instruction tuning and chat format

In short: A pre-trained model merely continues texts. In instruction tuning (SFT, supervised fine-tuning) it learns from examples of question and good answer to follow instructions and reply as an assistant.

In more detail: A base model given “How do I bake a cake?” might continue with more questions. After tuning it answers with instructions.

In detail

• Data: tens of thousands to millions of examples from human-written answers and filtered model answers.
• Chat format: the text is divided into roles: system (behaviour), user, assistant. Special tokens mark the roles. The model learns to write only as “assistant”.
• System prompt: the operator’s instructions placed before the conversation.
• Loss on answers only: in SFT only the assistant part is scored.
• Little data suffices: quality beats quantity (“LIMA” observation): the knowledge is already in pre-training, tuning only shapes behaviour.
• Afterwards: preference learning, see RLHF and preference optimisation.
• Alternative: own fine-tuning for specialist fields, see Fine-tuning and LoRA.

See also: Prompt engineering, How a language model is trained