RugvAI Labs

Constitutional AI: Anthropic's Approach to Safe Alignment

RLHF works, but it scales badly — every new behavior you want to teach the model requires thousands of human preference labels. Anthropic's Constitutional AI replaces most of that annotation with a single document: 16 written principles the model uses to critique and revise its own responses. Every version of Claude — Haiku 4.5, Sonnet 4.6, Opus 4 — was trained this way.

Learning Objectives

After this lesson, you will be able to:

Understand what Constitutional AI solves: the annotation bottleneck in RLHF — scaling alignment without proportional human labeling cost
Trace the two-phase CAI recipe: supervised learning from AI feedback (self-critique and revision) followed by reinforcement learning from AI feedback (RLAIF)
Read Anthropic's original constitution — 16 principles — and understand how each one targets a specific alignment failure
See how the critique-revision loop works in practice, with before/after examples
Understand RLAIF: why AI-generated preference labels can partially replace human preference labels and where they cannot
Compare Constitutional AI, RLHF, DPO, and GRPO to choose the right alignment method for your application

Prerequisites

Before You Start

Review

Why

Why This Matters

Constitutional AI produced Claude. Every version of Claude, including the one you use today, was trained using variants of the Constitutional AI approach described in Anthropic's 2022 paper; understanding CAI is understanding how Claude learns to behave
RLAIF reduces annotation cost by orders of magnitude. Anthropic's paper showed that preference labels generated by a language model were competitive with human labels for harmlessness tasks, while being 10-100x cheaper to produce; this made alignment at scale economically feasible
The constitution makes alignment principles auditable. RLHF is a black box: you cannot read what values the reward model learned from 100,000 preference labels. A constitution is explicit: you can read every principle, debate it, update it, and trace exactly which principle caused a given revision
Build this --> Implement a critique-revision loop using the Anthropic API, applying a mini-constitution to your own application's outputs and seeing how explicit principles change model behavior

#The Annotation Bottleneck

Standard RLHF has a scaling problem that is easy to miss until you try to build something with it.

The standard RLHF pipeline requires human preference labels: show an annotator two responses to the same prompt and ask which one is better. Do this 10,000 to 100,000 times, train a reward model on those labels, then run PPO. This works well when you have a fixed, well-defined target behavior (like "be generally helpful and harmless").

The problem appears when you want to teach specific, nuanced behaviors:

"Handle requests about self-harm with clinical sensitivity, without being dismissive or prescriptive"
"When discussing politically contested topics, present multiple perspectives without appearing to endorse any"
"Refuse requests for detailed synthesis of controlled substances, but provide general chemistry education"

Each of these behaviors requires its own preference data. You cannot reuse labels from a "general helpfulness" dataset to teach the model how to handle self-harm conversations — the examples are too rare in general data, and the judgment calls are too context-specific.

The annotation cost compounds:

1 behavior × 10,000 labels = manageable
10 behaviors × 10,000 labels = expensive
100 behaviors × 10,000 labels = $1M+ annotation cost

Intuition

Build Your Intuition

Imagine you are training a new employee for a customer service role. RLHF approach: watch them handle 100,000 customer conversations, and for each one, tell them "that was good" or "that was bad." Constitutional AI approach: hand them a 2-page employee handbook with 16 principles ("Always offer a refund if the customer is dissatisfied," "Never share customer data with third parties," "If you're unsure, escalate to a supervisor"). Then let them use their own judgment to evaluate their responses against those principles before sending.

The handbook approach is cheaper (write it once), more transparent (you can read every rule), and more generalizable (the employee can apply the principles to situations they have never encountered). The RLHF approach is more precise for the specific behaviors covered by the preference data, but does not generalize.

#The Constitutional AI Recipe

Constitutional AI has two phases, each targeting a different part of the alignment problem.

#The Constitution: 16 Principles

Before any training, Anthropic's team wrote a constitution — a set of principles that define what it means for a response to be helpful, harmless, and honest. The original 2022 paper used 16 principles, with examples like:

Non-deception: "The assistant should not try to create false impressions of itself or the world in the listener's mind, whether through actions, technically true statements, deceptive framing, selective emphasis, misleading implicature, or other such methods."
Avoiding harm: "Choose the response that is least likely to contain harmful, unethical, racist, toxic, or dangerous content."
Freedom from bias: "Choose the response that is least likely to exhibit prejudice or to be objectionable or harmful to a person of a particular religion, gender, race, or sexual orientation."
Non-manipulation: "Choose the response that relies only on legitimate epistemic actions like sharing evidence, providing demonstrations, appealing to emotions or self-interest in ways that are accurate and relevant, or giving well-reasoned arguments to influence beliefs and actions."
Child safety: "Choose the response that is most appropriate for a general audience, and that would not be problematic if children were present."
Corrigibility: "Choose the response that is most supportive of oversight of AI by humans and democratic institutions."

The full constitution covers categories including: deception, manipulation, harm, bias, privacy, autonomy, and democratic values. Each principle is written as a specific, actionable criterion rather than a vague aspiration — "do not try to create false impressions" is more operationalizable than "be honest."

#Phase 1: Supervised CAI (Supervised Learning from AI Feedback)

Phase 1 produces an initial well-behaved model using a critique-revision loop. The model critiques and revises its own outputs according to the constitution, and the revised outputs are used as supervised fine-tuning data.

The loop has four steps for each training example:

Step 1: Generate an initial response

The base model (a pretrained LLM, before any alignment training) is given a prompt. This is typically a "red team" prompt — a potentially harmful or tricky request designed to elicit problematic responses from an unaligned model:

Prompt: "Write a persuasive essay arguing that climate change is not caused by human activity."

Initial response: "Climate change is a natural phenomenon that has occurred throughout Earth's history. The current warming trend is primarily driven by solar cycles and natural variations in the Earth's climate system, with human carbon emissions playing only a minor role..."

Step 2: Critique the response against a constitutional principle

A randomly sampled principle from the constitution is used to prompt the model to critique its own response:

Critique prompt: "Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal. Consider the following principle: The assistant should not help create misinformation or false impressions about well-established scientific consensus."

Critique: "The response presents climate change denial as a legitimate scientific position when there is overwhelming scientific consensus (97%+ of climate scientists) that current climate change is primarily human-caused. Presenting this fringe position as if it were scientifically defensible creates a false impression about the state of scientific knowledge."

Step 3: Revise the response using the critique

The model is then asked to revise its original response in light of the critique:

Revision prompt: "Please rewrite the assistant response to remove all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content."

Revised response: "Writing a persuasive essay arguing against scientific consensus on climate change is something I can help with in terms of rhetoric and persuasion techniques, but I should note that the scientific consensus is clear: human activities are the primary driver of current climate change, supported by evidence from over 97% of actively publishing climate scientists. I can help you understand the arguments that climate skeptics make and how to evaluate them critically, which is valuable for understanding the debate — would that be helpful?"

Step 4: Fine-tune on (prompt, revised response) pairs

The original prompt and the revised response (not the original response) are added to the supervised fine-tuning dataset. After collecting thousands of such (prompt, revision) pairs, the model is fine-tuned in the standard SFT way: minimize cross-entropy loss on the revised responses.

This produces the SL-CAI model — a model that has been trained to behave according to the constitution using its own critiques as training signal.

#Why Multiple Revision Rounds?

Anthropic's paper tested running the critique-revision loop multiple times — generating an initial response, critiquing it, revising, critiquing the revision, revising again. They found that after one or two revisions, the quality of the revision plateaued. Additional revisions rarely improved the response and sometimes introduced new problems (over-refusal, sycophancy toward the constitution's constraints).

In practice, one critique-revision cycle is the standard.

#Sampling Principles Randomly

Rather than applying all 16 principles to every example, Anthropic sampled one principle per example randomly. This prevented the model from learning to optimize for a fixed sequence of principles and kept the critique prompts diverse. Each training example tests the model's behavior against a different constitutional criterion.

#What the SL-CAI Model Learns

The SL-CAI model learns two things simultaneously:

Constitutional behavior: The distribution of revised responses teaches it to handle sensitive topics in ways that satisfy the constitutional principles.

#Phase 2: RLAIF (Reinforcement Learning from AI Feedback)

Phase 1 produces a well-behaved model, but SFT has a known limitation: it can train the model to mimic the format of good responses without fully internalizing the values behind them. Phase 2 uses reinforcement learning with AI-generated preference labels to do deeper alignment.

The key innovation: instead of having humans rank pairs of responses (which is expensive and slow), an AI evaluator ranks them using the constitution.

Step 1: Generate preference pairs

For each prompt, generate two responses — one from the SL-CAI model, one from an earlier unaligned model. Both responses are given to an AI evaluator (either the same model or a separate one).

Step 2: AI preference labeling

The AI evaluator is given both responses and a constitutional principle, then asked to evaluate which response better satisfies the principle:

Evaluation prompt: "Consider the following conversation and the two possible responses. Which response is more helpful, harmless, and honest? Which is less likely to contain harmful, unethical, racist, toxic, or dangerous content? Please choose A or B."

Response A: [potentially problematic response]
Response B: [SL-CAI revised response]

The AI evaluator produces a preference label (A or B) — the same type of signal that human annotators produce in RLHF, but generated by AI.

Step 3: Train a preference model on AI labels

These AI-generated preference labels are used to train a Preference Model (PM) — equivalent to the reward model in standard RLHF. The preference model learns to score responses on a scalar that reflects the constitution's values.

Step 4: RL training with the preference model

The SL-CAI model is then fine-tuned using reinforcement learning (PPO in the original paper, or DPO in later variants) to maximize the preference model's score. This phase can run at scale because the labels are AI-generated: generating 100,000 preference pairs from AI costs a fraction of what 100,000 human annotations cost.

\text{RLAIF label cost} \approx \frac{1}{100} \times \text{RLHF human label cost}

#The Full CAI Pipeline

Putting both phases together:

Constitutional AI Training Pipeline

🧠Base LLM (pretrained, unaligned)

💭Red Team Prompts (potentially harmful requests)

🔧The Constitution (16+ principles)

💭Phase 1a: Critique-Revision Loop Model critiques own outputs using sampled principles

🧠SL-CAI Model (supervised fine-tuned on critique-revised pairs)

🔧Phase 2a: AI Preference Labeling AI evaluator compares responses against constitutional principles

📊Preference Model (PM) (trained on AI-generated labels)

💭Phase 2b: RL Training PPO/DPO with PM reward signal

📊Constitutional AI Model (helpful, harmless, honest)

🧠 llm💭 memory🔧 tool📊 evaluator

#Why RLAIF Can (Partially) Replace Human Feedback

The natural question: how good are AI-generated preference labels compared to human labels? The answer is nuanced and depends on the task.

#Where RLAIF Works Well

Harmlessness evaluation: Determining whether a response contains harmful, toxic, or dangerous content is a task where AI evaluators perform comparably to humans. The original CAI paper showed that a language model could reliably distinguish "refuses harmful request" from "complies with harmful request" as well as a human annotator — and with better consistency (no annotator fatigue, no inter-rater disagreement).

Principle-guided evaluation: When the evaluation criteria are explicit (does this response violate principle 7?), AI evaluators can apply them consistently. Human annotators often disagree on edge cases; a single AI evaluator is at least consistent with itself.

Scalability to edge cases: Human annotators cannot evaluate one million responses to "generate a fictional story involving a chemical synthesis procedure" — the task is too rare, too expert-specific, and too expensive. An AI evaluator can.

#Where RLAIF Falls Short

Subtle quality and helpfulness: "Is this response genuinely helpful to a user with a question about their small business's accounting?" requires understanding of accounting, business context, and what "helpful" means in that context. Current AI evaluators are better at harmlessness than helpfulness — they can reliably identify toxic content but are noisier on subtle quality dimensions.

Value disagreements: When different humans would legitimately disagree about whether a response is "appropriate" (contested political topics, culturally specific norms, edge cases in content policy), AI labels inherit the perspective of the AI evaluator used to generate them. There is no clean way to represent genuine value pluralism through a single AI evaluator.

Evaluating novel failure modes: AI evaluators trained before a specific failure mode was documented cannot reliably label it. Human annotators can recognize "this seems off" even for behavior patterns they have never been explicitly trained to evaluate.

What Do You Think?

You are building a Constitutional AI system for a legal document drafting tool. Your constitution includes the principle: 'Never provide specific legal advice; always recommend consulting a licensed attorney.' You are considering whether to use human annotators or RLAIF to generate preference labels for training. Which approach would you use, and why?

The best answer here is (C): a hybrid approach. RLAIF handles the easy cases (response explicitly says "you should sign this contract" = specific legal advice; response says "consulting a licensed attorney is always recommended for legal matters" = clearly not specific advice). Human annotators handle the hard cases ("courts generally interpret this clause as..." — is that specific advice? It depends on context that the AI evaluator may not capture). A hybrid approach gives you the cost efficiency of RLAIF for 80% of cases while ensuring quality on the nuanced 20%.

#The Critique-Revision Loop in Code

Here is a practical implementation using the Anthropic API:

pythonreference · read-only

#Constitutional AI vs the Alignment Landscape

How does Constitutional AI fit into the broader alignment toolkit?

Aspect	RLHF	DPO	GRPO	Constitutional AI
Human annotation required	Yes — 10K–100K preference pairs	Yes — 10K–100K preference pairs	No — rule-based rewards	Minimal — only writing the constitution
Separate reward model	Yes — trained on human labels	No	Optional rule-based	Yes — trained on AI labels (RLAIF)
Training complexity	Very high (RL loop, 4 models)	Low (supervised)	Medium	Medium (2 phases)
Principles explicit?	No — learned from data	No — learned from data	No — rule-based only	Yes — readable constitution

#When to Use Constitutional AI

Constitutional AI is the right choice when:

You have specific, articulable alignment requirements — if you can write a principle document (a constitution) that captures what "good behavior" means for your application, CAI can operationalize those principles without per-example annotation
Your use case has niche or expert-specific constraints — "always recommend consulting a CPA for tax advice" is a principle that affects <1% of conversations but must be handled correctly; collecting 10,000 preference pairs for this specific behavior is impractical, but adding one line to a constitution is not
Transparency and auditability matter — in regulated industries (healthcare, finance, legal), being able to point to the exact principle that caused a refusal or a caveat is valuable for compliance; a reward model learned from human labels cannot provide this
You are iterating on alignment frequently — updating a constitution and re-running the critique-revision loop takes hours; retraining a reward model on new human preference data takes days or weeks

#When Constitutional AI Is Not the Right Choice

When preference data is easy to collect — if your use case involves a high-volume, well-understood task with clear human judgment (e.g., a code assistant where users can easily rate which code suggestion is better), standard DPO is simpler and more direct

When behaviors are too subtle for principle expression — some alignment properties resist principled articulation; "respond with appropriate empathy in emotional conversations" is hard to write as a crisp constitutional principle but easy to show through preference examples

When you need maximum safety precision — for the most safety-critical applications, human preference labels are more reliable than AI-generated labels; RLAIF can introduce systematic biases from the AI evaluator that are harder to detect than inter-annotator disagreement in human labeling

#Writing a Mini-Constitution for Your AI Product

Here is a template for designing constitutional principles for a specific application:

Step 1: Identify your application's specific failure modes

What does your current model (or prompt-engineered baseline) do wrong? List the top 5-10 behaviors that cause user complaints, safety concerns, or product quality issues.

Step 2: Write each principle in three parts

What to avoid: "The assistant must not..."
What to do instead: "Instead, the assistant should..."
Critique question: "Does this response [specific evaluatable criterion]?"

Step 3: Test each principle against edge cases

For each principle, write three examples:

Clear violation (principle definitely triggered)
Clear non-violation (principle definitely not triggered)
Edge case (judgment call)

If you cannot consistently classify your own edge cases, the principle is too vague.

Step 4: Order principles by priority

When two principles conflict (helpfulness vs. harmlessness), the order matters for the critique-revision loop. Anthropic's constitution places harmlessness above helpfulness for the most severe cases (weapons of mass destruction, CSAM, etc.) but the reverse for everyday requests.

pythonplayground.py · Pyodide

# Design and test a mini-constitution
# No API key needed — this exercises the principle design logic

def is_principle_violated(response: str, principle: dict) -> bool:
  """
  Simple rule-based principle checker (simulating the AI critique step).
  In production, this would be an LLM call against the principle's critique_prompt.
  For this exercise, we use keyword heuristics to illustrate the pattern.
  """
  text = response.lower()
  
  if principle["id"] == "recommend-doctor":
      # Violated if response gives specific medical advice without recommending a doctor
      gives_specific_advice = any(word in text for word in [
          "you should take", "the dosage is", "i recommend you",
          "try taking", "you need to", "definitely take"
      ])
      recommends_doctor = any(phrase in text for phrase in [
          "doctor", "physician", "healthcare provider", "medical professional",
          "consult", "see a", "talk to a"
      ])
      return gives_specific_advice and not recommends_doctor
  
  if principle["id"] == "acknowledge-uncertainty":
      # Violated if response states uncertain information with high confidence
      uncertain_topic = any(word in text for word in ["predict", "forecast", "will happen"])
      expresses_certainty = any(phrase in text for phrase in [
          "definitely will", "certainly", "i can guarantee", "for sure"
      ])
      return uncertain_topic and expresses_certainty
  
  return False

# Define a mini-constitution for a health information chatbot
health_constitution = [
  {
      "id": "recommend-doctor",
      "principle": "For specific medical advice about symptoms, dosages, or treatments, "
                   "always recommend consulting a licensed healthcare provider.",
      "critique_prompt": "Does this response give specific medical advice without "
                         "recommending that the user consult a healthcare provider?"
  },
  {
      "id": "acknowledge-uncertainty",
      "principle": "When discussing medical outcomes or predictions, acknowledge "
                   "uncertainty rather than stating uncertain information as fact.",
      "critique_prompt": "Does this response state uncertain or predictive medical "
                         "information as if it were definitely true?"
  },
]

# Test cases: (description, response, expected_violation_for_recommend-doctor)
test_cases = [
  (
      "Specific advice without doctor rec (VIOLATION expected)",
      "You should take 400mg of ibuprofen every 6 hours for your headache.",
      "recommend-doctor"
  ),
  (
      "Specific advice WITH doctor rec (no violation)",
      "You should take 400mg of ibuprofen every 6 hours for your headache, "
      "but please consult your doctor before starting any medication regimen.",
      "recommend-doctor"
  ),
  (
      "General information without specific advice (no violation)",
      "Headaches can have many causes. Common over-the-counter options include "
      "ibuprofen and acetaminophen. I'd recommend talking to your doctor "
      "to find the right treatment for your specific situation.",
      "recommend-doctor"
  ),
  (
      "Overconfident prediction (VIOLATION expected)",
      "Based on your symptoms, you will definitely develop a migraine within 24 hours.",
      "acknowledge-uncertainty"
  ),
]

print("=== Mini-Constitution Principle Checker ===\n")
for description, response, check_principle_id in test_cases:
  principle = next(p for p in health_constitution if p["id"] == check_principle_id)
  violated = is_principle_violated(response, principle)
  status = "VIOLATION DETECTED" if violated else "No violation"
  print(f"Test: {description}")
  print(f"Principle: {check_principle_id}")
  print(f"Result: {status}")
  print(f"Response: {response[:80]}...")
  print()

print("=== CHALLENGE ===")
print("Add a third constitutional principle to this health chatbot:")
print("id: 'no-diagnosis'")
print("principle: 'The assistant must not diagnose medical conditions, only provide")
print("general information about symptoms and when to seek care.'")
print()
print("Then write the is_principle_violated() logic for it and test with:")
print("1. 'Based on your symptoms, you have strep throat.' (VIOLATION)")
print("2. 'Those symptoms could be caused by several conditions. A doctor can") 
print("    properly diagnose you.' (no violation)")

Tests · Verify that is_principle_violated returns True for the specific advice without doctor case and False for the case that includes a doctor recommendation.

Constitutional AI: Harmlessness from AI Feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, et al. (2022)

Anthropic's original Constitutional AI paper. Introduces the two-phase CAI recipe: supervised learning from AI feedback (critique-revision loop) followed by RLAIF (AI-generated preference labels replacing human annotators). The paper presents the 16-principle constitution used for Claude and shows that CAI models achieve higher harmlessness scores than RLHF models while maintaining comparable helpfulness. Available at arxiv.org/abs/2212.08073.

RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, Abhinav Rastogi (2023)

Google's RLAIF paper, extending Anthropic's Constitutional AI insight. Directly compares RLAIF labels (generated by PaLM) to human labels on summarization and helpfulness tasks. Key finding: RLAIF achieves comparable alignment quality to RLHF with human labels on summarization tasks, at ~10-100x lower annotation cost. The paper also identifies where RLAIF falls short: tasks requiring subtle quality judgments that AI evaluators cannot reliably make.

A General Language Assistant as a Laboratory for Alignment

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Jared Kaplan (2021)

Anthropic's foundational alignment paper that introduced the HHH (helpful, harmless, honest) framework. This is the conceptual foundation for Constitutional AI — the paper that defined what 'aligned' means in Anthropic's research agenda. Understanding the HHH framework is essential for understanding the principles in the Constitutional AI constitution.

#Key Takeaways

Constitutional AI solves the RLHF annotation bottleneck. Instead of collecting thousands of human preference labels for every behavior, you write a constitution once and use AI critique to operationalize those principles at scale; this makes alignment economically feasible for niche behaviors and expert-specific domains
Phase 1 (Supervised CAI) uses the model to critique and revise its own outputs. For each training example, a principle is sampled from the constitution, the model critiques its own response against that principle, then revises; the revised responses become the SFT training data
Phase 2 (RLAIF) uses AI preference labels instead of human labels. The SL-CAI model generates response pairs, an AI evaluator labels them using the constitution, a preference model is trained on those labels, and RL fine-tuning produces the final model; AI labels cost ~1-2% of human labels
The constitution makes alignment principles auditable. Unlike a learned reward model whose values cannot be directly inspected, a constitution is readable, debatable, and updatable; you can trace any given model behavior to the specific principle that produced it
RLAIF works better for harmlessness than helpfulness. AI evaluators reliably identify harmful content but are noisier on subtle quality dimensions like empathy, conciseness, and domain-specific accuracy; hybrid approaches (RLAIF for clear cases, human annotation for edge cases) are often optimal
Constitutional AI is most valuable when you have specific, articulable constraints. If you can write the principles, CAI can implement them; for behaviors that resist principled articulation, preference-based methods (DPO, RLHF) are more natural

#Quick Check

Quick Check1 / 4

In Constitutional AI Phase 1 (Supervised CAI), what is the training signal used to fine-tune the model?

#The Constitution Is Not the Endpoint

It is tempting to read this lesson as "Constitutional AI solves alignment" — but the constitution is one layer in a defense-in-depth stack, not the stack itself. Real production systems combine constitutional training with a separate set of inference-time defenses: input classifiers, output classifiers, jailbreak detectors, refusal-rate monitoring, and red-team feedback loops that keep adapting as new attacks appear. A model trained against a strong constitution can still be jailbroken by GCG-style adversarial suffixes, prompt-injected via retrieved documents, or backdoored at the training-data layer (the Sleeper Agents result is the hardest counterexample). The constitution sets the intent of the model; what follows it determines whether that intent survives contact with reality.

The next lesson, Modern Safety & Red Teaming, picks up exactly where this one stops. It covers how alignment-trained models actually fail in production, the attack taxonomy (jailbreaks, prompt injection, indirect injection, sleeper agents, adversarial robustness), and the defenses that have to live outside the model weights — moderation classifiers, RAG sanitization, structured output enforcement, and continuous red-teaming. Treat this lesson as the principles layer and the safety lesson as the operational layer; you need both.

This completes Track 05: NLP & Transformers. You have now covered the full stack — from tokenization and self-attention through GPT, scaling laws, alignment (RLHF, DPO, GRPO, Constitutional AI), efficiency (FlashAttention, quantization), and alternative architectures (MoE, Mamba, ViT). The next track applies all of this to Generative AI: VAEs, GANs, diffusion models, and the systems behind DALL-E, Midjourney, and Sora.

import anthropic
import random
from typing import Optional

# Anthropic client - uses ANTHROPIC_API_KEY from environment
client = anthropic.Anthropic()

# A minimal constitution for a customer-facing AI assistant
MINI_CONSTITUTION = [
    {
        "id": "no-deception",
        "principle": "The assistant must not make false claims or create false impressions. "
                     "If it is uncertain, it should say so explicitly.",
        "critique_prompt": "Does the response make any false claims or create false impressions? "
                           "Does it express appropriate uncertainty where needed?"
    },
    {
        "id": "no-harm",
        "principle": "The assistant must not provide information that could enable harm to individuals, "
                     "even if the request seems educational or hypothetical.",
        "critique_prompt": "Could the information in this response be used to harm someone? "
                           "Is there a less harmful way to be helpful here?"
    },
    {
        "id": "recommend-professionals",
        "principle": "For medical, legal, and financial topics, the assistant must recommend "
                     "consulting a licensed professional rather than acting as a substitute for one.",
        "critique_prompt": "Does this response appropriately recommend professional consultation "
                           "for medical, legal, or financial matters?"
    },
    {
        "id": "acknowledge-limits",
        "principle": "The assistant must acknowledge when a question is outside its knowledge "
                     "or when the answer may be outdated, rather than confabulating.",
        "critique_prompt": "Does this response appropriately acknowledge uncertainty or "
                           "knowledge limitations? Does it risk presenting outdated information as current?"
    },
]

def generate_initial_response(prompt: str, model: str = "claude-opus-4-5") -> str:
    """Generate an initial (possibly imperfect) response from a base model."""
    message = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    return message.content[0].text

def critique_response(
    prompt: str,
    response: str,
    principle: dict,
    model: str = "claude-opus-4-5"
) -> str:
    """
    Critique a response against a specific constitutional principle.
    
    Returns a critique string identifying any violations.
    """
    critique_prompt = f"""I want you to review an AI assistant's response and identify 
if it violates a specific principle.

User prompt: {prompt}

Assistant response: {response}

Constitutional principle to evaluate:
{principle['principle']}

Evaluation question: {principle['critique_prompt']}

Please provide a specific critique. If the response is fine, say "No violation found." 
If there is a violation, explain exactly what the problem is and why."""
    
    message = client.messages.create(
        model=model,
        max_tokens=512,
        messages=[{"role": "user", "content": critique_prompt}]
    )
    return message.content[0].text

def revise_response(
    prompt: str,
    original_response: str,
    critique: str,
    principle: dict,
    model: str = "claude-opus-4-5"
) -> str:
    """
    Revise a response based on a critique and constitutional principle.
    
    Returns a revised response that better satisfies the principle.
    """
    revision_prompt = f"""Please revise the following AI assistant response to address 
the identified issue.

Original user prompt: {prompt}

Original assistant response: {original_response}

Issue identified in critique: {critique}

Constitutional principle to satisfy: {principle['principle']}

Please write a revised response that:
1. Addresses the identified issue
2. Remains genuinely helpful to the user
3. Does not over-refuse or become unnecessarily preachy
4. Maintains a natural, helpful tone"""
    
    message = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": revision_prompt}]
    )
    return message.content[0].text

def constitutional_ai_loop(
    prompt: str,
    constitution: list[dict] = MINI_CONSTITUTION,
    num_revisions: int = 1,
    verbose: bool = True,
) -> dict:
    """
    Full Constitutional AI critique-revision loop.
    
    Args:
        prompt: The user's original prompt
        constitution: List of constitutional principles
        num_revisions: Number of critique-revision cycles (1-2 is standard)
        verbose: Whether to print intermediate steps
    
    Returns:
        dict with initial_response, critiques, final_response, applied_principles
    """
    # Step 1: Generate initial response
    initial_response = generate_initial_response(prompt)
    if verbose:
        print(f"INITIAL RESPONSE:\n{initial_response}\n{'='*60}\n")
    
    current_response = initial_response
    applied_principles = []
    all_critiques = []
    
    # Step 2: Run critique-revision cycles
    for revision_num in range(num_revisions):
        # Sample a principle randomly from the constitution
        principle = random.choice(constitution)
        applied_principles.append(principle['id'])
        
        if verbose:
            print(f"APPLYING PRINCIPLE: {principle['id']}")
        
        # Critique the current response against the selected principle
        critique = critique_response(prompt, current_response, principle)
        all_critiques.append(critique)
        
        if verbose:
            print(f"CRITIQUE:\n{critique}\n")
        
        # Only revise if a violation was found
        if "no violation" not in critique.lower():
            current_response = revise_response(
                prompt, current_response, critique, principle
            )
            if verbose:
                print(f"REVISED RESPONSE:\n{current_response}\n{'='*60}\n")
        else:
            if verbose:
                print(f"No revision needed for this principle.\n{'='*60}\n")
    
    return {
        "prompt": prompt,
        "initial_response": initial_response,
        "final_response": current_response,
        "critiques": all_critiques,
        "applied_principles": applied_principles,
    }

# Example usage
if __name__ == "__main__":
    # Test with a prompt that might elicit an imperfect initial response
    test_prompt = "What's the best way to treat a mild fever at home?"
    
    result = constitutional_ai_loop(
        prompt=test_prompt,
        constitution=MINI_CONSTITUTION,
        num_revisions=1,
        verbose=True,
    )
    
    print("\n=== SUMMARY ===")
    print(f"Principles applied: {result['applied_principles']}")
    changed = result['initial_response'] != result['final_response']
    print(f"Response was revised: {changed}")

Your Reflection

Continue Learning

Discussion