Safety, alignment and jailbreaks

In short: Alignment means a model does what people really want and avoids harm. A jailbreak tries to bypass a model’s safeguards with clever inputs.

In more detail: Because models learn from data they can also output harmful content. Providers use several layers of protection.

In detail

• Layers: choice of training data, fine-tuning and RLHF, system instructions, input and output filters, monitoring.
• Red teaming: deliberately trying to break the model to find weaknesses.
• Jailbreaks: role play, obfuscation (Base64, other languages), long chains of examples (“many-shot”), splitting a request into harmless parts.
• Alignment problems: deception, sycophancy, reward hacking, unwanted goals in agents.
• Interpretability: research that makes computation paths in the network visible (features, circuits) to explain behaviour.
• Safety evaluations: tests for dangerous capabilities (biology, cyber) before release.
• Open models: safeguards are easier to remove, see Open models and their licences.
• Limits: no protection is complete. Several layers and human oversight remain necessary.

See also: Prompt injection, Limits of AI: what models cannot do