ReAct decides one step at a time. Planning decides ten. The difference matters when a single bad early step poisons all the downstream ones — fix a typo three files into a refactor and you'd better have planned the whole refactor first. This lesson covers chain-of-thought, tree-of-thought, plan-and-execute, and when each is worth its ~30% latency cost.
Learning Objectives
After this lesson, you will be able to:
Understand how agents break big, complex tasks into smaller, manageable to-do lists (subtask trees)
Explain why telling the AI to 'think step by step' (chain-of-thought) dramatically improves accuracy -- and the research behind it
Compare tree-of-thought (exploring multiple paths at once like a maze) with linear chain-of-thought (following one path at a time)
Describe the plan-and-execute pattern: make a plan, carry it out step by step, and adjust the plan when things change
Explain how reflection lets agents look at their own work and say 'wait, I can do better' before finalizing
Planning is the skill that separates agents that stumble around from agents that get things done efficiently. If you have ever written a to-do list before starting a big project, you already understand the core idea. Now you will see how AI does the same thing.
Try it! Open ChatGPT and try two prompts. First: "What is 17 times 28?" (it might get it wrong). Second: "What is 17 times 28? Think step by step." Watch how the second prompt produces explicit reasoning ("17 times 20 is 340, 17 times 8 is 136, 340 + 136 = 476") and gets the right answer. You just experienced chain-of-thought prompting -- the simplest and most powerful reasoning technique.
A single tool call can answer "What is the weather?" But what about "Write a technical blog post about transformers"? That task requires researching the topic, organizing the information, structuring an outline, writing sections, reviewing for accuracy, and editing for clarity. No single tool call or LLM generation can handle it. The agent needs a plan.
Task decomposition is the process of converting a complex goal into a tree of simpler subtasks. Each subtask should be something the agent can accomplish in one or a few steps.
Consider "Write a technical blog post about transformers":
Research -- search for key papers, read existing explanations, identify core concepts
Outline -- organize concepts into logical sections, decide on analogies
Draft -- write each section, include code examples, add diagrams descriptions
Edit -- fix issues found in review, polish prose, add introduction and conclusion
Each of these subtasks can be further decomposed. "Research" becomes: search for the original "Attention Is All You Need" paper, find visual explanations, gather code examples. This recursive decomposition continues until every leaf node is a concrete, executable action -- a single tool call or a single generation.
The quality of decomposition matters enormously. Too coarse (one giant step: "write the post") and the agent tries to do everything at once, producing mediocre results. Too fine (50 micro-steps: "write the first sentence of paragraph 3") and the agent drowns in coordination overhead. The sweet spot is usually 3-7 top-level subtasks, each decomposable into 2-4 actions.
There are several ways agents can decompose tasks:
Top-down decomposition: Start with the final goal and recursively break it into smaller pieces. "Write a blog post" becomes [Research, Outline, Draft, Review, Edit]. Each of those breaks down further. This is the most common and reliable approach.
Bottom-up composition: Start with the available tools and capabilities, then figure out how to combine them to achieve the goal. "I have search, write, and review tools -- I can search first, then write, then review." This works when the tools constrain the possible approaches.
Analogical decomposition: Think of a similar task you have solved before and adapt its plan. "Writing a blog post is like writing a report: both need research, structure, drafting, and editing." LLMs do this naturally when given examples of previously decomposed tasks.
Iterative refinement: Start with a rough plan, execute the first step, and refine the remaining steps based on what you learn. This is the most adaptive approach but requires the most reasoning at each step.
The simplest and most widely used reasoning strategy is chain-of-thought (CoT) prompting. The idea is almost embarrassingly simple: you tell the LLM to "think step by step" and it produces dramatically better answers.
Without CoT: "What is 17 times 28?" -- the LLM might guess incorrectly, outputting something like 456 or 496.
With CoT: "What is 17 times 28? Let's think step by step." -- the LLM writes out: "17 times 20 is 340. 17 times 8 is 136. 340 plus 136 is 476." Correct.
Why does this work? Because the intermediate reasoning steps serve as "scratch paper" for the model. Each token generated becomes context for the next token. By writing out partial computations, the model can maintain and build upon intermediate results rather than trying to leap to the answer in one step. The model is not doing anything different internally -- it is just giving itself room to work.
For agents, chain-of-thought is the default reasoning mode. Every "Thought" step in the ReAct loop is essentially chain-of-thought reasoning applied to action selection. The agent thinks: "I need weather data. I have a weather tool. The user wants San Francisco. I should call get_weather with location San Francisco." Each sentence builds on the previous one, making the final decision more reliable.
Zero-shot CoT: Simply append "Let's think step by step" to the prompt. The model generates its own reasoning steps. This is the simplest approach and works surprisingly well -- the original research showed it improved accuracy on math problems by over 50%.
Few-shot CoT: Provide 2-3 examples of step-by-step reasoning before the actual question. Each example shows the full chain: question, reasoning steps, answer. This is more reliable because the examples demonstrate the exact format and depth of reasoning you expect. The model follows the pattern it sees.
For agent systems, zero-shot CoT is more practical because you cannot predict what tasks the agent will face. But for specialized agents with predictable task types (always doing math, always analyzing code), few-shot CoT in the system prompt produces consistently better reasoning.
Chain-of-thought is linear -- one step follows the next in a single path. But some problems have branching solutions where the first approach might be a dead end. Tree-of-thought (ToT) extends CoT by exploring multiple reasoning paths simultaneously and evaluating which ones are most promising.
Try it: Tree-of-Thought — watch branches expand, get evaluated, and get prunedInteractive
Loading visualization...
Imagine you are writing an essay. Chain-of-thought writes one draft, start to finish. Tree-of-thought generates three different opening paragraphs, evaluates which one is strongest, then continues from that winner. At each branching point, it generates multiple options, scores them, and prunes the losers -- like a chess player considering multiple moves before committing.
The process at each step:
Generate -- produce 2-4 candidate continuations
Evaluate -- score each candidate on quality, relevance, and feasibility
Select -- keep the best candidate(s), discard the rest
Expand -- continue from the selected path(s)
Backtrack -- if all paths from a node fail, go back and try discarded alternatives
This is powerful but expensive. Each branch requires LLM calls. A tree with a branching factor of 3 and a depth of 5 requires up to 3 to the 5th power = 243 evaluations. In practice, ToT is reserved for high-stakes tasks where getting the right answer is more important than speed or cost -- things like mathematical proofs, puzzle solving, and critical code generation.
A lighter-weight alternative to full tree-of-thought is best-of-N sampling. Instead of building a tree with evaluation at each step, you simply generate N complete solutions independently and pick the best one. Generate 3 drafts of the blog post introduction, evaluate all 3, keep the best. This captures some of the diversity benefit of ToT without the complexity of tree management and backtracking.
Best-of-N is especially effective when combined with a strong evaluator. If you have a reliable way to score outputs (automated tests for code, rubric-based evaluation for writing), best-of-N often matches ToT quality at a fraction of the implementation complexity.
What Do You Think?
An agent makes a plan with 10 steps and step 3 fails. What should it do?
What Do You Think?
You are deciding between Chain-of-Thought, Tree-of-Thought, and a reasoning model (o3 or Claude with extended thinking) for an agent that solves competition math. Which is the strongest default in 2025?
The best answer is to replan. Steps 4-10 might have depended on step 3's output, so blindly skipping it would cause downstream failures. Starting over wastes all the progress from steps 1-2. And stopping immediately gives up too easily. A good agent examines why step 3 failed, considers alternatives (different tool? different approach? different data source?), and adjusts the remaining plan accordingly. This adaptive replanning is what separates robust agents from brittle scripts.
The plan-and-execute pattern separates planning from execution into distinct phases. This is one of the most effective patterns for complex multi-step tasks.
Phase 1 -- Plan: The agent generates a complete step-by-step plan for the task. No tools are called yet. The plan is pure reasoning -- a roadmap of what needs to happen and in what order.
Phase 2 -- Execute: The agent executes each step in order, calling tools and generating text as needed. After each step, it checks whether the plan still makes sense given what it has learned.
Phase 3 -- Replan: If a step fails or produces unexpected results, the agent revises the remaining plan. It does not start over -- it adapts. The progress from completed steps is preserved.
This pattern is more robust than pure ReAct for complex tasks because the upfront planning phase produces a more coherent strategy. ReAct agents sometimes wander -- they solve the immediate subproblem without considering the bigger picture. Plan-and-execute agents maintain a global view of where they are and where they need to go.
Not every failure requires replanning. Here is a decision framework:
Retry when the failure is transient: network timeout, rate limit, temporary API error. Wait briefly and try the same step again (with a retry limit of 2-3 attempts).
Adapt when the failure is permanent but contained: the API you wanted is down, but an alternative exists. Swap the tool and continue with the rest of the plan.
Replan when the failure changes the strategy: the data you expected does not exist, the approach you planned is not feasible, or you learned something that invalidates your assumptions. Generate a new plan for the remaining steps.
Escalate when the failure is beyond the agent's ability to handle: required permissions are missing, the task is ambiguous, or every approach has failed. Return what you have and explain what blocked progress.
Select a complex scenario above and observe how the agent decomposes it into phases. Watch the Think and Plan phases especially -- notice how the agent considers its options before acting, and how it evaluates results before deciding what comes next.
Figure
Three steps cycle continuously. Thought: the agent states what it needs — "I need 2025 spending data." Action: it issues a concrete tool call — search('US renewable energy spending 2025'). Observation: the result comes back — "$215 billion." That observation feeds the next Thought, and the loop turns again until the goal is met. The cycle is the mechanism; a single pass through it is just a function call.
The user asks: "How much did the US spend on renewable energy in 2025, and what percentage increase is that from 2020?" This requires two pieces of data, a calculation, and a synthesis. No single tool call can answer it -- the agent needs to reason and act in alternation.
The agent generates its first reasoning trace: "I need two numbers: US renewable energy spending in 2025 and in 2020. Let me search for the 2025 figure first, since it is the primary question. I will search for the 2020 figure after."
#Step 3: Action -- search("US renewable energy spending 2025")
The agent acts on its reasoning by calling the search tool: search("US renewable energy spending 2025"). The thought determined the action. Without the thought step, the agent might have searched for both years at once with a vague query, getting worse results.
The search returns: "The US invested $215 billion in renewable energy in 2025 according to BloombergNEF's annual report." The agent reads this result as its next observation. New data enters the reasoning loop.
The agent reasons again: "I now have the 2025 figure: $215 billion. I still need the 2020 figure to calculate the percentage increase. Let me search for that." Each thought builds on prior observations, making the next action more targeted.
#Step 6: Action -- search("US renewable energy spending 2020")
The agent searches again: search("US renewable energy spending 2020"). The observation returns: "US clean energy investment was $85 billion in 2020." Now the agent has both numbers.
The agent reasons: "$215B in 2025 vs $85B in 2020. Percentage increase = ((215 - 85) / 85) * 100." It calls: calculate("((215 - 85) / 85) * 100"). The tool returns: 152.94. Rather than risk mental math errors, the agent used a tool for precision.
The agent synthesizes everything: "The US spent approximately $215 billion on renewable energy in 2025, representing a 153% increase from the $85 billion spent in 2020." Each ReAct cycle -- Thought, Action, Observation -- built upon the last. The interleaving of reasoning and action produced a factual, verified, calculated answer that neither pure reasoning nor pure action could have achieved alone.
Reflection is when an agent evaluates its own output and decides whether it is good enough. This is one of the most powerful techniques in modern agent design, and it is conceptually simple: after producing an output, the agent is prompted to critique that output.
The key insight is that LLMs are often better at evaluating text than generating it on the first try. The model that wrote a mediocre introduction can usually identify what is wrong with it when asked "What are the weaknesses of this introduction?" The gap between generation quality and evaluation quality is the window where reflection creates value.
The agent receives: "Write a technical blog post about transformers." It generates a plan: [Research the topic, Create an outline, Write a draft, Review and edit, Finalize].
The agent calls web_search("transformer architecture deep learning explained") and gathers key concepts: self-attention, multi-head attention, positional encoding, encoder-decoder architecture, the "Attention Is All You Need" paper.
From the research, the agent creates an outline: Introduction with analogy, self-attention mechanism explained, multi-head attention, positional encoding, the encoder-decoder structure, practical applications, code example.
The agent writes the full blog post. It covers all the technical details accurately. But the result reads like a textbook -- dense equations from the second paragraph, no analogies, no code until the very end, and a dry opening that would lose most readers in seconds.
The agent critiques its own draft: "The introduction jumps straight into math -- a blog post should hook readers first with an accessible analogy. The section on multi-head attention uses notation without explaining it. There is no code example until the end, but readers expect to see code early. The conclusion just restates the introduction."
The agent adjusts its plan based on the critique: rewrite the intro with the "translating a sentence by looking at every word simultaneously" analogy, add inline code snippets after each concept, simplify the multi-head attention section, write a new conclusion with forward-looking applications. The revised post is significantly better -- because the agent caught its own mistakes.
Role shift. When generating, the model is in "producer" mode -- it tries to fill space and sound knowledgeable. When critiquing, it shifts to "evaluator" mode -- it compares the output against quality standards. These are different skills, and the evaluator mode is often stronger.
Fresh perspective. The critique prompt provides a different frame. "Write a blog post" and "What are the weaknesses of this blog post?" activate different parts of the model's training data -- the first draws on writing examples, the second draws on editing and review examples.
Specific feedback. The critique produces concrete, actionable feedback ("the intro is too technical") rather than vague improvements. This specificity makes the revision step effective.
The reflection pattern can also be applied recursively: reflect on the reflection. But in practice, one round of reflection captures most of the value. Two rounds show diminishing returns. Three or more rounds often lead to over-editing where the agent keeps changing things without improving them.
A common production pattern pairs a "writer" role with a "critic" role, even within a single agent:
Writer pass: Generate the output with a "write the best version you can" prompt
Critic pass: Switch to a critic persona with a "find every weakness, error, and area for improvement" prompt
Revision pass: Give the writer the original output plus the critic's feedback with a "revise based on this feedback" prompt
This three-pass approach typically costs 3x the tokens of a single generation but produces outputs that rate significantly higher on quality assessments. For tasks where quality matters more than speed (published content, customer-facing responses, code that will be deployed), the tradeoff is worthwhile.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan (2023)
Extends chain-of-thought prompting into a tree search over reasoning paths. The agent generates multiple candidate thoughts at each step, evaluates them using the LLM as a heuristic, and backtracks when needed -- like a human considering and discarding approaches before settling on the best one. Significantly outperforms CoT on tasks requiring exploration.
When should you use each strategy? Here is a practical guide:
Chain-of-thought -- Use for most tasks. Simple, fast, and effective. Good for: answering questions, single-step tool use, straightforward analysis. Cost: 1x (single linear pass).
Tree-of-thought -- Use for tasks with multiple viable approaches where picking the wrong one wastes significant effort. Good for: mathematical proofs, creative writing with specific constraints, code architecture decisions. Cost: 3-10x (branching and evaluation).
Plan-and-execute -- Use for complex multi-step tasks where maintaining a global view matters. Good for: research reports, multi-tool workflows, anything with more than 3-4 steps. Cost: 1.5-2x (planning overhead + execution).
Reflection -- Use as an add-on to any strategy when output quality is more important than speed. Good for: writing, code review, analysis where errors are costly. Cost: 2x per reflection round (generate + critique + revise).
In practice, most production agents use plan-and-execute with chain-of-thought reasoning at each step, plus one round of reflection on high-stakes outputs. Tree-of-thought is reserved for specialized applications where the branching cost is justified by the value of finding the optimal path.
Consider Claude Code (the tool you might be using right now). When given a coding task, it follows a plan-and-execute pattern:
Understand -- Read relevant files, understand the codebase structure
Plan -- Decide which files to modify, in what order, and what changes to make
Execute -- Write code, one file at a time, testing as it goes
Verify -- Run tests, check for build errors, review its own changes
Replan -- If tests fail or the build breaks, diagnose the issue and adjust
This is plan-and-execute with reflection (the verify step) and adaptive replanning (the replan step). The same patterns that sound abstract in theory are the foundation of tools you use every day.
Understanding how planning goes wrong is just as important as understanding how it works:
Goal drift: The agent starts with a clear goal but gradually shifts focus as it encounters interesting but irrelevant information during research. The plan was "write a blog post about transformers" but the agent spends 80% of its steps exploring tangential topics about RNNs and LSTMs.
Sunk cost persistence: The agent has invested many steps in a failing approach and keeps trying instead of replanning. "I have already searched 5 databases and found nothing, but I will try 5 more" instead of reconsidering whether the information exists.
Plan rigidity: The agent treats its initial plan as sacred and executes it mechanically even when intermediate results clearly suggest a different approach would work better.
Premature optimization: The agent spends excessive effort perfecting an early step (researching every possible source, writing and rewriting the outline) when a "good enough" version would let it make progress and refine later.
Context window overflow: Long plans with many steps accumulate so much context (tool results, reasoning traces, intermediate outputs) that the LLM's context window fills up, causing it to forget the original plan or earlier findings.
Each of these can be mitigated with guardrails: goal reminders injected every N steps, plan progress checks, context summarization at regular intervals, and the hard operational limits discussed in the Architect View.
The most effective mitigation for planning failures is progress checkpoints. After every 3-5 steps, inject a checkpoint prompt:
"Original goal: [goal]. Steps completed: [list]. Steps remaining: [list]. Am I making progress toward the goal? Should I continue with the current plan or adjust?"
This simple technique prevents goal drift, catches sunk cost persistence early, and forces the agent to re-evaluate its plan against the original objective. It adds a small token overhead but dramatically improves completion rates on complex tasks.
Another powerful technique is context compression: periodically summarize the conversation history into a compact format that preserves key findings while freeing up context window space. Instead of the agent reading 50 pages of raw tool output, it reads a 1-page summary of what was learned. This prevents context window overflow while keeping the agent informed.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Tests · Run the ReAct loop with a multi-step question. Verify the agent uses search for data gathering, calculate for math, and answer for the final response. Confirm it completes in 4 steps.
Chain-of-thought prompting dramatically improves reasoning. Simply asking the model to "think step by step" forces it to decompose problems into manageable pieces, reducing errors on complex tasks
Tree-of-thought explores multiple reasoning paths. Instead of a single linear chain, branching exploration considers alternative approaches and evaluates them, finding better solutions for problems with multiple valid strategies
Plan-and-execute separates strategy from tactics. The agent first creates a high-level plan, then executes each step, and replans when reality diverges from the plan, combining strategic thinking with tactical flexibility
Reflection enables self-improvement. By critiquing its own output and identifying errors, an agent can iteratively refine its answers without external feedback, catching mistakes that a single pass would miss
What is the key advantage of tree-of-thought over chain-of-thought reasoning?
You now understand how a single agent reasons and plans -- from simple chain-of-thought to branching tree-of-thought to the full plan-execute-replan cycle, with reflection as a quality multiplier. But a capable agent is also a dangerous agent. Next up: Agent Safety, where we build the guardrails that keep powerful agents from going off the rails.