Constitutional AI: Anthropic's Approach to Safe Alignment

#The Annotation Bottleneck

Standard RLHF has a scaling problem that is easy to miss until you try to build something with it.

The standard RLHF pipeline requires human preference labels: show an annotator two responses to the same prompt and ask which one is better. Do this 10,000 to 100,000 times, train a reward model on those labels, then run PPO. This works well when you have a fixed, well-defined target behavior (like "be generally helpful and harmless").

The problem appears when you want to teach specific, nuanced behaviors:

• "Handle requests about self-harm with clinical sensitivity, without being dismissive or prescriptive"
• "When discussing politically contested topics, present multiple perspectives without appearing to endorse any"
• "Refuse requests for detailed synthesis of controlled substances, but provide general chemistry education"

Each of these behaviors requires its own preference data. You cannot reuse labels from a "general helpfulness" dataset to teach the model how to handle self-harm conversations — the examples are too rare in general data, and the judgment calls are too context-specific.

The annotation cost compounds:

• 1 behavior × 10,000 labels = manageable
• 10 behaviors × 10,000 labels = expensive
• 100 behaviors × 10,000 labels = $1M+ annotation cost

#The Constitutional AI Recipe

Constitutional AI has two phases, each targeting a different part of the alignment problem.

#The Constitution: 16 Principles

Before any training, Anthropic's team wrote a constitution — a set of principles that define what it means for a response to be helpful, harmless, and honest. The original 2022 paper used 16 principles, with examples like:

1. Non-deception: "The assistant should not try to create false impressions of itself or the world in the listener's mind, whether through actions, technically true statements, deceptive framing, selective emphasis, misleading implicature, or other such methods."
2. Avoiding harm: "Choose the response that is least likely to contain harmful, unethical, racist, toxic, or dangerous content."
3. Freedom from bias: "Choose the response that is least likely to exhibit prejudice or to be objectionable or harmful to a person of a particular religion, gender, race, or sexual orientation."
4. Non-manipulation: "Choose the response that relies only on legitimate epistemic actions like sharing evidence, providing demonstrations, appealing to emotions or self-interest in ways that are accurate and relevant, or giving well-reasoned arguments to influence beliefs and actions."
5. Child safety: "Choose the response that is most appropriate for a general audience, and that would not be problematic if children were present."
6. Corrigibility: "Choose the response that is most supportive of oversight of AI by humans and democratic institutions."

The full constitution covers categories including: deception, manipulation, harm, bias, privacy, autonomy, and democratic values. Each principle is written as a specific, actionable criterion rather than a vague aspiration — "do not try to create false impressions" is more operationalizable than "be honest."

#Phase 1: Supervised CAI (Supervised Learning from AI Feedback)

The loop has four steps for each training example:

The base model (a pretrained LLM, before any alignment training) is given a prompt. This is typically a "red team" prompt — a potentially harmful or tricky request designed to elicit problematic responses from an unaligned model:

Prompt: "Write a persuasive essay arguing that climate change is not caused by human activity."

Initial response: "Climate change is a natural phenomenon that has occurred throughout Earth's history. The current warming trend is primarily driven by solar cycles and natural variations in the Earth's climate system, with human carbon emissions playing only a minor role..."

A randomly sampled principle from the constitution is used to prompt the model to critique its own response:

Critique prompt: "Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal. Consider the following principle: The assistant should not help create misinformation or false impressions about well-established scientific consensus."

Critique: "The response presents climate change denial as a legitimate scientific position when there is overwhelming scientific consensus (97%+ of climate scientists) that current climate change is primarily human-caused. Presenting this fringe position as if it were scientifically defensible creates a false impression about the state of scientific knowledge."

The model is then asked to revise its original response in light of the critique:

Revision prompt: "Please rewrite the assistant response to remove all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content."

Revised response: "Writing a persuasive essay arguing against scientific consensus on climate change is something I can help with in terms of rhetoric and persuasion techniques, but I should note that the scientific consensus is clear: human activities are the primary driver of current climate change, supported by evidence from over 97% of actively publishing climate scientists. I can help you understand the arguments that climate skeptics make and how to evaluate them critically, which is valuable for understanding the debate — would that be helpful?"

The original prompt and the revised response (not the original response) are added to the supervised fine-tuning dataset. After collecting thousands of such (prompt, revision) pairs, the model is fine-tuned in the standard SFT way: minimize cross-entropy loss on the revised responses.

#Phase 2: RLAIF (Reinforcement Learning from AI Feedback)

The key innovation: instead of having humans rank pairs of responses (which is expensive and slow), an AI evaluator ranks them using the constitution.

For each prompt, generate two responses — one from the SL-CAI model, one from an earlier unaligned model. Both responses are given to an AI evaluator (either the same model or a separate one).

The AI evaluator is given both responses and a constitutional principle, then asked to evaluate which response better satisfies the principle:

Evaluation prompt: "Consider the following conversation and the two possible responses. Which response is more helpful, harmless, and honest? Which is less likely to contain harmful, unethical, racist, toxic, or dangerous content? Please choose A or B."

Response A: [potentially problematic response]
Response B: [SL-CAI revised response]

The AI evaluator produces a preference label (A or B) — the same type of signal that human annotators produce in RLHF, but generated by AI.

The SL-CAI model is then fine-tuned using reinforcement learning (PPO in the original paper, or DPO in later variants) to maximize the preference model's score. This phase can run at scale because the labels are AI-generated: generating 100,000 preference pairs from AI costs a fraction of what 100,000 human annotations cost.

#The Full CAI Pipeline

Putting both phases together:

#Why RLAIF Can (Partially) Replace Human Feedback

The natural question: how good are AI-generated preference labels compared to human labels? The answer is nuanced and depends on the task.

#Where RLAIF Works Well

#Where RLAIF Falls Short

The best answer here is (C): a hybrid approach. RLAIF handles the easy cases (response explicitly says "you should sign this contract" = specific legal advice; response says "consulting a licensed attorney is always recommended for legal matters" = clearly not specific advice). Human annotators handle the hard cases ("courts generally interpret this clause as..." — is that specific advice? It depends on context that the AI evaluator may not capture). A hybrid approach gives you the cost efficiency of RLAIF for 80% of cases while ensuring quality on the nuanced 20%.

#The Critique-Revision Loop in Code

Here is a practical implementation using the Anthropic API:

import anthropic
import random
from typing import Optional

# Anthropic client - uses ANTHROPIC_API_KEY from environment
client = anthropic.Anthropic()

# A minimal constitution for a customer-facing AI assistant
MINI_CONSTITUTION = [
    {
        "id": "no-deception",
        "principle": "The assistant must not make false claims or create false impressions. "
                     "If it is uncertain, it should say so explicitly.",
        "critique_prompt": "Does the response make any false claims or create false impressions? "
                           "Does it express appropriate uncertainty where needed?"
    },
    {
        "id": "no-harm",
        "principle": "The assistant must not provide information that could enable harm to individuals, "
                     "even if the request seems educational or hypothetical.",
        "critique_prompt": "Could the information in this response be used to harm someone? "
                           "Is there a less harmful way to be helpful here?"
    },
    {
        "id": "recommend-professionals",
        "principle": "For medical, legal, and financial topics, the assistant must recommend "
                     "consulting a licensed professional rather than acting as a substitute for one.",
        "critique_prompt": "Does this response appropriately recommend professional consultation "
                           "for medical, legal, or financial matters?"
    },
    {
        "id": "acknowledge-limits",
        "principle": "The assistant must acknowledge when a question is outside its knowledge "
                     "or when the answer may be outdated, rather than confabulating.",
        "critique_prompt": "Does this response appropriately acknowledge uncertainty or "
                           "knowledge limitations? Does it risk presenting outdated information as current?"
    },
]

def generate_initial_response(prompt: str, model: str = "claude-opus-4-5") -> str:
    """Generate an initial (possibly imperfect) response from a base model."""
    message = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    return message.content[0].text

def critique_response(
    prompt: str,
    response: str,
    principle: dict,
    model: str = "claude-opus-4-5"
) -> str:
    """
    Critique a response against a specific constitutional principle.
    
    Returns a critique string identifying any violations.
    """
    critique_prompt = f"""I want you to review an AI assistant's response and identify 
if it violates a specific principle.

User prompt: {prompt}

Assistant response: {response}

Constitutional principle to evaluate:
{principle['principle']}

Evaluation question: {principle['critique_prompt']}

Please provide a specific critique. If the response is fine, say "No violation found." 
If there is a violation, explain exactly what the problem is and why."""
    
    message = client.messages.create(
        model=model,
        max_tokens=512,
        messages=[{"role": "user", "content": critique_prompt}]
    )
    return message.content[0].text

def revise_response(
    prompt: str,
    original_response: str,
    critique: str,
    principle: dict,
    model: str = "claude-opus-4-5"
) -> str:
    """
    Revise a response based on a critique and constitutional principle.
    
    Returns a revised response that better satisfies the principle.
    """
    revision_prompt = f"""Please revise the following AI assistant response to address 
the identified issue.

Original user prompt: {prompt}

Original assistant response: {original_response}

Issue identified in critique: {critique}

Constitutional principle to satisfy: {principle['principle']}

Please write a revised response that:
1. Addresses the identified issue
2. Remains genuinely helpful to the user
3. Does not over-refuse or become unnecessarily preachy
4. Maintains a natural, helpful tone"""
    
    message = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": revision_prompt}]
    )
    return message.content[0].text

def constitutional_ai_loop(
    prompt: str,
    constitution: list[dict] = MINI_CONSTITUTION,
    num_revisions: int = 1,
    verbose: bool = True,
) -> dict:
    """
    Full Constitutional AI critique-revision loop.
    
    Args:
        prompt: The user's original prompt
        constitution: List of constitutional principles
        num_revisions: Number of critique-revision cycles (1-2 is standard)
        verbose: Whether to print intermediate steps
    
    Returns:
        dict with initial_response, critiques, final_response, applied_principles
    """
    # Step 1: Generate initial response
    initial_response = generate_initial_response(prompt)
    if verbose:
        print(f"INITIAL RESPONSE:\n{initial_response}\n{'='*60}\n")
    
    current_response = initial_response
    applied_principles = []
    all_critiques = []
    
    # Step 2: Run critique-revision cycles
    for revision_num in range(num_revisions):
        # Sample a principle randomly from the constitution
        principle = random.choice(constitution)
        applied_principles.append(principle['id'])
        
        if verbose:
            print(f"APPLYING PRINCIPLE: {principle['id']}")
        
        # Critique the current response against the selected principle
        critique = critique_response(prompt, current_response, principle)
        all_critiques.append(critique)
        
        if verbose:
            print(f"CRITIQUE:\n{critique}\n")
        
        # Only revise if a violation was found
        if "no violation" not in critique.lower():
            current_response = revise_response(
                prompt, current_response, critique, principle
            )
            if verbose:
                print(f"REVISED RESPONSE:\n{current_response}\n{'='*60}\n")
        else:
            if verbose:
                print(f"No revision needed for this principle.\n{'='*60}\n")
    
    return {
        "prompt": prompt,
        "initial_response": initial_response,
        "final_response": current_response,
        "critiques": all_critiques,
        "applied_principles": applied_principles,
    }

# Example usage
if __name__ == "__main__":
    # Test with a prompt that might elicit an imperfect initial response
    test_prompt = "What's the best way to treat a mild fever at home?"
    
    result = constitutional_ai_loop(
        prompt=test_prompt,
        constitution=MINI_CONSTITUTION,
        num_revisions=1,
        verbose=True,
    )
    
    print("\n=== SUMMARY ===")
    print(f"Principles applied: {result['applied_principles']}")
    changed = result['initial_response'] != result['final_response']
    print(f"Response was revised: {changed}")

#Constitutional AI vs the Alignment Landscape

How does Constitutional AI fit into the broader alignment toolkit?

#When to Use Constitutional AI

Constitutional AI is the right choice when:

1. You have specific, articulable alignment requirements — if you can write a principle document (a constitution) that captures what "good behavior" means for your application, CAI can operationalize those principles without per-example annotation
2. Your use case has niche or expert-specific constraints — "always recommend consulting a CPA for tax advice" is a principle that affects <1% of conversations but must be handled correctly; collecting 10,000 preference pairs for this specific behavior is impractical, but adding one line to a constitution is not
3. Transparency and auditability matter — in regulated industries (healthcare, finance, legal), being able to point to the exact principle that caused a refusal or a caveat is valuable for compliance; a reward model learned from human labels cannot provide this
4. You are iterating on alignment frequently — updating a constitution and re-running the critique-revision loop takes hours; retraining a reward model on new human preference data takes days or weeks

#When Constitutional AI Is Not the Right Choice

#Writing a Mini-Constitution for Your AI Product

Here is a template for designing constitutional principles for a specific application:

What does your current model (or prompt-engineered baseline) do wrong? List the top 5-10 behaviors that cause user complaints, safety concerns, or product quality issues.

Step 2: Write each principle in three parts

• What to avoid: "The assistant must not..."
• What to do instead: "Instead, the assistant should..."
• Critique question: "Does this response [specific evaluatable criterion]?"

For each principle, write three examples:

• Clear violation (principle definitely triggered)
• Clear non-violation (principle definitely not triggered)
• Edge case (judgment call)

If you cannot consistently classify your own edge cases, the principle is too vague.

When two principles conflict (helpfulness vs. harmlessness), the order matters for the critique-revision loop. Anthropic's constitution places harmlessness above helpfulness for the most severe cases (weapons of mass destruction, CSAM, etc.) but the reverse for everyday requests.

#Key Takeaways

• Constitutional AI solves the RLHF annotation bottleneck. Instead of collecting thousands of human preference labels for every behavior, you write a constitution once and use AI critique to operationalize those principles at scale; this makes alignment economically feasible for niche behaviors and expert-specific domains
• Phase 1 (Supervised CAI) uses the model to critique and revise its own outputs. For each training example, a principle is sampled from the constitution, the model critiques its own response against that principle, then revises; the revised responses become the SFT training data
• Phase 2 (RLAIF) uses AI preference labels instead of human labels. The SL-CAI model generates response pairs, an AI evaluator labels them using the constitution, a preference model is trained on those labels, and RL fine-tuning produces the final model; AI labels cost ~1-2% of human labels
• The constitution makes alignment principles auditable. Unlike a learned reward model whose values cannot be directly inspected, a constitution is readable, debatable, and updatable; you can trace any given model behavior to the specific principle that produced it
• RLAIF works better for harmlessness than helpfulness. AI evaluators reliably identify harmful content but are noisier on subtle quality dimensions like empathy, conciseness, and domain-specific accuracy; hybrid approaches (RLAIF for clear cases, human annotation for edge cases) are often optimal
• Constitutional AI is most valuable when you have specific, articulable constraints. If you can write the principles, CAI can implement them; for behaviors that resist principled articulation, preference-based methods (DPO, RLHF) are more natural

#Quick Check

#The Constitution Is Not the Endpoint