Constitutional AI: teaching models to self-improve

2024

Most AI companies train their models using RLHF (Reinforcement Learning from Human Feedback): humans rate outputs, and the model learns to produce higher-rated responses. It works, but it's expensive and can encode human biases.

Anthropic took a different approach: Constitutional AI (CAI). Instead of relying solely on human raters, they gave the model a "constitution" (a set of principles like "be helpful," "be honest," "avoid harm") and trained it to critique and revise its own outputs.

The process works in two stages. First, the model generates a response. Then it's asked: "Does this response follow the constitution? If not, revise it." The model's self-critiques become training data for the next iteration.

This approach scales better than human feedback alone. You can update the constitution (add new principles, remove outdated ones) without retraining from scratch. And it makes the model's values more transparent: they're literally written down.

Critics worried it might make models overly cautious. But in practice, Constitutional AI produced models that were both more capable and more aligned, a rare combination that suggested the safety-versus-capability tradeoff might be a false dichotomy.