Reflexion & Self-Critique

The LLM gets the answer wrong. You ask "are you sure?" and it correctly finds its own mistake. That's not a bug — that's a 10–30% quality lift on hard tasks, hiding in plain sight, at the cost of 2x tokens and ~30% latency. Reflexion (Shinn 2023) is the canonical pattern; o1, DeepSeek-R1, and Claude's extended thinking are variants.

Learning Objectives

After this lesson, you will be able to:

Explain why adding a self-critique step lifts agent quality 10-30% on hard tasks — at the cost of 30% latency and 2x token usage
Implement Reflexion-style verbal RL: agent attempts → critic evaluates → reflection note added to memory → retry
Pick when self-critique is worth the cost (code generation, math reasoning, multi-hop QA) vs when ReAct alone is enough
Avoid the common pitfalls: critic that always agrees, infinite reflection loops, reflections that don't change behavior

Prerequisites

Before You Start

Review

Why

Why This Matters

Why Reflexion (Shinn 2023) became the canonical "verbal RL" technique — without parameter updates, an LLM agent can iteratively improve via natural-language self-feedback; Shinn et al. showed 22% absolute improvement on HumanEval coding tasks
How Anthropic's Constitutional AI is essentially Reflexion at scale — the model evaluates its own outputs against a constitution and revises; this loop powers Claude's self-correction behavior
Why o1-style "thinking longer" is a Reflexion variant — OpenAI o1 and DeepSeek-R1 train models to generate explicit reasoning + self-critique chains; the same pattern users deploy at runtime
Build this → Wrap a code-generation agent with a critic loop: attempt → run tests → if failing, generate reflection on what went wrong → retry with reflection in context; benchmark on 50 LeetCode problems vs single-attempt baseline

#The Core Idea

#Anatomy of Reflexion: Three Roles, One Memory

Shinn 2023 names three distinct roles inside a Reflexion agent. Conflating them is the most common implementation mistake.

Component	Job	Common implementation
Actor	Attempt the task. Produces an action / output.	The "main" LLM call, usually a capable model (Claude Opus, GPT-4o). Sees the task plus all prior reflections in its context.
Evaluator	Score the actor's output. Pass/fail, possibly with structured feedback.	Either (a) external code (unit tests, regex match, ground-truth comparison), or (b) an LLM critic prompt that returns JSON like `{score, passing, notes}`.
Self-reflection	Write a verbal lesson learned from the failed attempt. Compresses (action, evaluation) into one or two sentences of actionable advice.	A separate LLM call with a reflection-specific prompt. Output is appended to memory.

The fourth ingredient is the memory itself — a list of natural-language reflections retained across trials. Crucially, reflections do NOT replace the failing actions in memory; they are additional context. On trial t, the actor sees: original task + the running reflection list. It cannot see the raw failing outputs from earlier trials (Shinn argues this prevents the model from anchoring on its prior bad answer).

#The Reflexion Loop

1. Actor: attempt the task → action_1
2. Evaluator: score the result
   - Pass → return result
   - Fail → continue
3. Self-reflection: write a lesson on why action_1 failed
4. Memory: append the reflection to a persistent reflection list
5. Actor (retry): attempt the task with reflections in context → action_2
6. Repeat from step 2 (with max retries)

A worked example on a math problem makes the roles concrete:

Trial 1
  Task:   "What is the smallest positive integer divisible by 1..10?"
  Actor:  "It's 2 * 3 * 5 * 7 = 210." (uses unique primes <= 10)
  Eval:   FAIL (expected 2520; got 210)
  Reflect: "I only multiplied prime factors. I must use the LCM, which means
            including the highest power of each prime <= 10: 2^3, 3^2, 5, 7
            = 8 * 9 * 5 * 7 = 2520."
Trial 2  (memory now contains the trial-1 reflection)
  Actor:  "LCM(1..10) = 2^3 * 3^2 * 5 * 7 = 2520."
  Eval:   PASS

The reflection isn't just "be more careful" — it names the specific misconception (primes vs. prime powers) and the specific fix. That specificity is what makes trial 2 different from trial 1.

What Do You Think?

In the math example above, suppose Trial 1's reflection had been only 'I made an arithmetic error.' What's the most likely Trial 2 outcome?

\text{Reflexion: } a_t = \pi(s_t, M_{\le t}) \quad \text{where } M_t = M_{t-1} \cup \{r_t\}, \quad r_t = \text{Reflector}(a_{t-1}, e_{t-1})

#Implementation: Code-Gen Reflexion

pythonplayground.py · Pyodide

# Reflexion loop on code generation: attempt → run tests → reflect on failure → retry
import json
import subprocess
import tempfile
from openai import OpenAI

client = OpenAI()

def evaluate_code(code: str, test_cases: list[dict]) -> dict:
  """Run code against test cases. Returns {'passed': bool, 'failures': [...]}"""
  failures = []
  with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
      f.write(code)
      tmp_path = f.name
  for tc in test_cases:
      try:
          test_code = f"{code}\n\nresult = {tc['call']}\nassert result == {tc['expected']!r}, f'Got {{result}}, expected {tc['expected']!r}'"
          result = subprocess.run(["python", "-c", test_code], capture_output=True, text=True, timeout=5)
          if result.returncode != 0:
              failures.append({"test": tc, "error": result.stderr[-500:]})
      except Exception as e:
          failures.append({"test": tc, "error": str(e)[-500:]})
  return {"passed": len(failures) == 0, "failures": failures}

def reflexion(task: str, test_cases: list, max_attempts: int = 3) -> str:
  reflections = []
  last_code = None
  for attempt in range(max_attempts):
      # Actor: generate code (with reflections in context)
      sys = "You are a Python coding assistant. Generate complete code for the task."
      if reflections:
          sys += "\n\nReflections from previous failed attempts:\n" + "\n".join(f"- {r}" for r in reflections)
      response = client.chat.completions.create(
          model="gpt-4o-mini",
          messages=[
              {"role": "system", "content": sys},
              {"role": "user", "content": f"Task: {task}\nReturn ONLY the Python code, no markdown."}
          ],
      )
      code = response.choices[0].message.content
      # Strip markdown if any
      if "```python" in code:
          code = code.split("```python")[1].split("```")[0]
      elif "```" in code:
          code = code.split("```")[1].split("```")[0]
      last_code = code

# Evaluator: run tests
      result = evaluate_code(code, test_cases)
      print(f"\n=== Attempt {attempt + 1} ===")
      print(f"Code:\n{code[:300]}{'...' if len(code) > 300 else ''}")
      print(f"Result: {'PASS' if result['passed'] else 'FAIL'}")
      if result["passed"]:
          return code

# Reflector: write lesson learned
      reflection_prompt = f"""You attempted this task:
{task}

Your code:
{code}

These tests failed:
{json.dumps(result['failures'], indent=2)}

Write ONE concise reflection (1-3 sentences) about what went wrong and how to fix it. Don't write code; just analyze the error pattern."""
      ref_response = client.chat.completions.create(
          model="gpt-4o-mini",
          messages=[{"role": "user", "content": reflection_prompt}],
      )
      reflection = ref_response.choices[0].message.content.strip()
      reflections.append(reflection)
      print(f"Reflection: {reflection}")

return last_code  # Best effort

# --- Test ---
task = "Write a function 'is_palindrome(s: str) -> bool' that checks if string is a palindrome (case-insensitive, ignoring spaces and punctuation)."
test_cases = [
  {"call": "is_palindrome('racecar')", "expected": True},
  {"call": "is_palindrome('A man a plan a canal Panama')", "expected": True},
  {"call": "is_palindrome('hello')", "expected": False},
  {"call": "is_palindrome('Was it a car or a cat I saw?')", "expected": True},
]

result = reflexion(task, test_cases, max_attempts=3)
print("\nFinal code:\n", result)

print("\nCHALLENGE: Compare with single-attempt (no Reflexion).")
print("Run 50 random LeetCode-style tasks; measure pass-rate delta.")

Tests · Verify Reflexion catches cases where the first attempt forgets case-insensitivity or punctuation handling. Verify it converges within 3 attempts on most palindrome cases.

#When to Use Self-Critique

Setting	Use Reflexion?
Code generation with executable tests	Yes — clear pass/fail signal, big quality lift
Math reasoning (GSM8K, MATH)	Yes — wrong-answer catches via re-derivation
Multi-hop QA with verifiable facts	Yes — fact-check loop catches hallucinations
Open-ended creative writing	No — no clear pass/fail; subjective
Latency-critical (< 2s)	No — Reflexion adds 30%+ latency
Tasks where first-pass succeeds 90%+	No — overhead not worth it

#Pitfalls

#Self-Refine: Reflexion Without an External Evaluator

The closely related Self-Refine technique (Madaan, Tandon, Gupta, Hallinan, et al., 2023) drops the external evaluator entirely. The same LLM generates an output, then critiques its own output, then revises — all from prompts alone, no ground truth required.

The pipeline is:

output_0 = actor(task)
for k in range(max_iters):
    feedback_k = self_critic(task, output_k)
    if feedback_k.contains("no changes needed"):
        return output_k
    output_{k+1} = actor(task, output_k, feedback_k)

Where Reflexion needs a pass/fail signal (tests, ground truth, downstream environment), Self-Refine works on any open-ended task — summarization, code style, story generation. Madaan et al. reported 5-40% gains across math, code, dialogue, and constrained generation tasks, without any external feedback channel.

Two practical differences to keep in mind:

Risk of degradation. Without ground truth, the critic can push the output in the wrong direction. A safety check: keep output_0 and only commit to output_k if the critic explicitly approves it; otherwise return the earliest "good enough" version.
Domain mismatch. Self-Refine shines on tasks with clear stylistic or structural criteria (e.g., "make this code more readable"). It struggles when correctness is what you want — there, you still need Reflexion with executable checks.

Quick check

Your team wants to improve the quality of LLM-generated meeting summaries. There's no ground-truth 'correct summary'. Which pattern fits?

#Retention of Reflections Across Trials

A subtle but important design choice: how long do reflections live? Three regimes:

Memory regime	Behavior	When to use
Per-task	Reflections cleared at the end of each task. Each task starts fresh.	Default. Reflections from "factor a polynomial" don't help "translate to Spanish".
Cross-trial within episode	Reflections persist across trials of the SAME task, then clear.	Original Reflexion paper setup. Standard.
Persistent across tasks	Reflections from every task aggregated into a long-term memory.	Advanced — risks unbounded growth, requires summarization/retrieval.

In practice, the cross-trial-within-episode regime is what most production code does. Persistent cross-task reflections sound powerful but introduce a new failure mode: reflections from unrelated tasks contaminating context ("I should remember the array was 0-indexed last time" applied to a task that has nothing to do with arrays). If you do go persistent, gate retrieval by task similarity or summarize aggressively.

#Reflexion vs o1/o3: Train-Time vs Inference-Time

The 2024-2026 frontier: train models to do Reflexion internally rather than wrapping them with an external loop. OpenAI o1 generates a long internal "reasoning chain" that includes self-critique steps — Reflexion built into the weights.

For most production: external Reflexion loops still win for tasks where you have explicit ground truth (tests, fact-check sources). For exploratory reasoning: o1/o3-class models with built-in critique are simpler.

#Key Takeaways

Reflexion adds a self-critique step that lifts agent quality 10-30% on hard tasks — at 30% latency cost
Loop: attempt → evaluate → reflect → retry with reflection in context. Hard cap at 3-5 attempts.
Use Reflexion when: clear pass/fail signal exists (tests, ground truth) and one-shot fails 30%+ of the time
Avoid Reflexion when: latency-critical, subjective tasks, first-pass succeeds 90%+
o1/o3-style models bake Reflexion into the weights. For non-verifiable reasoning, prefer those over external loops

#Quick Check

Quick Check1 / 2

Why does Reflexion improve LLM agent quality without any model retraining?

What Do You Think?

A team adds Reflexion to a customer-support chatbot. Quality on hard tickets goes up 12%, but the median p95 latency climbs from 2s to 6s. What's the most defensible production decision?

Reflexion lets agents learn from mistakes within a session. Next: Anthropic's catalog of workflow patterns that prescribe WHEN to use ReAct vs Reflexion vs structured chains.

Reflexion & Self-Critique

Learning Objectives

After this lesson, you will be able to:

Explain why adding a self-critique step lifts agent quality 10-30% on hard tasks — at the cost of 30% latency and 2x token usage
Implement Reflexion-style verbal RL: agent attempts → critic evaluates → reflection note added to memory → retry
Pick when self-critique is worth the cost (code generation, math reasoning, multi-hop QA) vs when ReAct alone is enough
Avoid the common pitfalls: critic that always agrees, infinite reflection loops, reflections that don't change behavior

Prerequisites

Before You Start

Review

Why

Why This Matters

Why Reflexion (Shinn 2023) became the canonical "verbal RL" technique — without parameter updates, an LLM agent can iteratively improve via natural-language self-feedback; Shinn et al. showed 22% absolute improvement on HumanEval coding tasks
How Anthropic's Constitutional AI is essentially Reflexion at scale — the model evaluates its own outputs against a constitution and revises; this loop powers Claude's self-correction behavior
Why o1-style "thinking longer" is a Reflexion variant — OpenAI o1 and DeepSeek-R1 train models to generate explicit reasoning + self-critique chains; the same pattern users deploy at runtime
Build this → Wrap a code-generation agent with a critic loop: attempt → run tests → if failing, generate reflection on what went wrong → retry with reflection in context; benchmark on 50 LeetCode problems vs single-attempt baseline

#The Core Idea

#Anatomy of Reflexion: Three Roles, One Memory

Shinn 2023 names three distinct roles inside a Reflexion agent. Conflating them is the most common implementation mistake.

Component	Job	Common implementation
Actor	Attempt the task. Produces an action / output.	The "main" LLM call, usually a capable model (Claude Opus, GPT-4o). Sees the task plus all prior reflections in its context.
Evaluator	Score the actor's output. Pass/fail, possibly with structured feedback.	Either (a) external code (unit tests, regex match, ground-truth comparison), or (b) an LLM critic prompt that returns JSON like `{score, passing, notes}`.
Self-reflection	Write a verbal lesson learned from the failed attempt. Compresses (action, evaluation) into one or two sentences of actionable advice.	A separate LLM call with a reflection-specific prompt. Output is appended to memory.

#The Reflexion Loop

1. Actor: attempt the task → action_1
2. Evaluator: score the result
   - Pass → return result
   - Fail → continue
3. Self-reflection: write a lesson on why action_1 failed
4. Memory: append the reflection to a persistent reflection list
5. Actor (retry): attempt the task with reflections in context → action_2
6. Repeat from step 2 (with max retries)

A worked example on a math problem makes the roles concrete:

Trial 1
  Task:   "What is the smallest positive integer divisible by 1..10?"
  Actor:  "It's 2 * 3 * 5 * 7 = 210." (uses unique primes <= 10)
  Eval:   FAIL (expected 2520; got 210)
  Reflect: "I only multiplied prime factors. I must use the LCM, which means
            including the highest power of each prime <= 10: 2^3, 3^2, 5, 7
            = 8 * 9 * 5 * 7 = 2520."
Trial 2  (memory now contains the trial-1 reflection)
  Actor:  "LCM(1..10) = 2^3 * 3^2 * 5 * 7 = 2520."
  Eval:   PASS

The reflection isn't just "be more careful" — it names the specific misconception (primes vs. prime powers) and the specific fix. That specificity is what makes trial 2 different from trial 1.

What Do You Think?

In the math example above, suppose Trial 1's reflection had been only 'I made an arithmetic error.' What's the most likely Trial 2 outcome?

\text{Reflexion: } a_t = \pi(s_t, M_{\le t}) \quad \text{where } M_t = M_{t-1} \cup \{r_t\}, \quad r_t = \text{Reflector}(a_{t-1}, e_{t-1})

#Implementation: Code-Gen Reflexion

pythonplayground.py · Pyodide

# Reflexion loop on code generation: attempt → run tests → reflect on failure → retry
import json
import subprocess
import tempfile
from openai import OpenAI

client = OpenAI()

# Reflector: write lesson learned
      reflection_prompt = f"""You attempted this task:
{task}

Your code:
{code}

These tests failed:
{json.dumps(result['failures'], indent=2)}

return last_code  # Best effort

result = reflexion(task, test_cases, max_attempts=3)
print("\nFinal code:\n", result)

print("\nCHALLENGE: Compare with single-attempt (no Reflexion).")
print("Run 50 random LeetCode-style tasks; measure pass-rate delta.")

Tests · Verify Reflexion catches cases where the first attempt forgets case-insensitivity or punctuation handling. Verify it converges within 3 attempts on most palindrome cases.

#When to Use Self-Critique

Setting	Use Reflexion?
Code generation with executable tests	Yes — clear pass/fail signal, big quality lift
Math reasoning (GSM8K, MATH)	Yes — wrong-answer catches via re-derivation
Multi-hop QA with verifiable facts	Yes — fact-check loop catches hallucinations
Open-ended creative writing	No — no clear pass/fail; subjective
Latency-critical (< 2s)	No — Reflexion adds 30%+ latency
Tasks where first-pass succeeds 90%+	No — overhead not worth it

#Pitfalls

#Self-Refine: Reflexion Without an External Evaluator

The pipeline is:

output_0 = actor(task)
for k in range(max_iters):
    feedback_k = self_critic(task, output_k)
    if feedback_k.contains("no changes needed"):
        return output_k
    output_{k+1} = actor(task, output_k, feedback_k)

Two practical differences to keep in mind:

Risk of degradation. Without ground truth, the critic can push the output in the wrong direction. A safety check: keep output_0 and only commit to output_k if the critic explicitly approves it; otherwise return the earliest "good enough" version.
Domain mismatch. Self-Refine shines on tasks with clear stylistic or structural criteria (e.g., "make this code more readable"). It struggles when correctness is what you want — there, you still need Reflexion with executable checks.

Quick check

Your team wants to improve the quality of LLM-generated meeting summaries. There's no ground-truth 'correct summary'. Which pattern fits?

#Retention of Reflections Across Trials

A subtle but important design choice: how long do reflections live? Three regimes:

Memory regime	Behavior	When to use
Per-task	Reflections cleared at the end of each task. Each task starts fresh.	Default. Reflections from "factor a polynomial" don't help "translate to Spanish".
Cross-trial within episode	Reflections persist across trials of the SAME task, then clear.	Original Reflexion paper setup. Standard.
Persistent across tasks	Reflections from every task aggregated into a long-term memory.	Advanced — risks unbounded growth, requires summarization/retrieval.

#Reflexion vs o1/o3: Train-Time vs Inference-Time

#Key Takeaways

Reflexion adds a self-critique step that lifts agent quality 10-30% on hard tasks — at 30% latency cost
Loop: attempt → evaluate → reflect → retry with reflection in context. Hard cap at 3-5 attempts.
Use Reflexion when: clear pass/fail signal exists (tests, ground truth) and one-shot fails 30%+ of the time
Avoid Reflexion when: latency-critical, subjective tasks, first-pass succeeds 90%+
o1/o3-style models bake Reflexion into the weights. For non-verifiable reasoning, prefer those over external loops

#Quick Check

Quick Check1 / 2

Why does Reflexion improve LLM agent quality without any model retraining?

What Do You Think?

A team adds Reflexion to a customer-support chatbot. Quality on hard tickets goes up 12%, but the median p95 latency climbs from 2s to 6s. What's the most defensible production decision?

Reflexion lets agents learn from mistakes within a session. Next: Anthropic's catalog of workflow patterns that prescribe WHEN to use ReAct vs Reflexion vs structured chains.

Reflexion & Self-Critique

After this lesson, you will be able to:

Before You Start

#The Core Idea

#Anatomy of Reflexion: Three Roles, One Memory

#The Reflexion Loop

#Implementation: Code-Gen Reflexion

#When to Use Self-Critique

#Pitfalls

#Self-Refine: Reflexion Without an External Evaluator

#Retention of Reflections Across Trials

#Reflexion vs o1/o3: Train-Time vs Inference-Time

#Key Takeaways

#Quick Check

Deep Dive

Your Reflection

Continue Learning

Discussion

Reflexion & Self-Critique

After this lesson, you will be able to:

Before You Start

#The Core Idea

#Anatomy of Reflexion: Three Roles, One Memory

#The Reflexion Loop

#Implementation: Code-Gen Reflexion

#When to Use Self-Critique

#Pitfalls

#Self-Refine: Reflexion Without an External Evaluator

#Retention of Reflections Across Trials

#Reflexion vs o1/o3: Train-Time vs Inference-Time

#Key Takeaways

#Quick Check

Deep Dive

Your Reflection

Continue Learning

Discussion