o1 doesn't think faster — it thinks LONGER. Reasoning models trade inference cost for capability, generating 10,000–100,000 hidden chain-of-thought tokens before answering. DeepSeek R1 proved you could do it for 95% less money than OpenAI. Claude Opus 4's "extended thinking" mode does the same. This is the most important paradigm shift in AI since the transformer itself.
Learning Objectives
After this lesson, you will be able to:
Understand the big shift: instead of training bigger models, let the same model think longer on hard problems
See how reasoning models generate step-by-step thinking (and check their own work) as a learned behavior, not a prompt trick
Compare step-level feedback vs. answer-level feedback and understand why checking each step produces more reliable reasoning
Know when to use a reasoning model vs. a standard LLM based on difficulty, speed needs, and cost
For years, the recipe for better AI was simple: train bigger models on more data. GPT-2 to GPT-3 to GPT-4 -- each generation was bigger and trained longer. This is training-time scaling: invest more compute during training, get a better model.
But there is a ceiling. Training runs already cost hundreds of millions of dollars. Data is running out. And a model trained for months still generates every answer in the same fixed amount of time.
Try it! Here is a tricky question: "A farmer has 17 sheep. All but 9 die. How many sheep does the farmer have left?" Try answering instantly without pausing. Did you say 8? Now re-read it carefully. The answer is 9 -- "all but 9" means 9 survive. That moment of re-reading and catching your mistake is exactly what reasoning models do automatically.
Test-time compute scaling flips the paradigm: instead of making the model bigger, let the same model think longer on hard problems. A $100M model that spends 30 seconds reasoning can outperform a $1B model that answers in 0.5 seconds.
Reasoning models are not fundamentally different architectures. They are standard transformer-based LLMs that have been trained to generate extended reasoning traces before producing a final answer.
Input: "A farmer has 17 sheep. All but 9 die. How many sheep does the farmer have left?"
A standard LLM might immediately answer "8" (17 - 9 = 8). But the correct answer is 9 -- "all but 9" means 9 survive. This requires careful reading, not fast arithmetic.
The reasoning model generates internal "thinking" tokens:
"Let me parse this carefully. 'All but 9 die.' This means every sheep EXCEPT 9 dies. So 9 sheep survive. Wait -- my first instinct was to compute 17 - 9 = 8, but that would mean 'all but 9 die' means '9 die,' which is not what it says. 'All but 9' means 'all except 9.' So the answer is 9."
These tokens are generated autoregressively, just like any other text. The model is literally "talking to itself."
"Let me verify: if 9 sheep are left, then 17 - 9 = 8 sheep died. Does 'all but 9 die' mean '8 die and 9 survive'? Yes, that is correct. The answer is 9."
This self-verification is a learned behavior -- the model was trained to check its reasoning, not just produce an answer.
Chain-of-thought prompting (from the Prompt Engineering lesson): You tell a standard LLM to "think step by step." The model tries to follow your instruction, but it was not specifically trained for this. Quality varies wildly.
Reasoning model behavior: The model has been trained (via RL and/or supervised fine-tuning) to ALWAYS reason before answering hard questions. The chain of thought is a core capability, not a prompting trick. The model decides how much to reason based on problem difficulty.
Try it: Watch an agent reason step by stepInteractive
How do you train a model to reason? Three main approaches have emerged:
#Approach 1: Supervised Fine-Tuning on Reasoning Traces
Collect human-written step-by-step solutions to hard problems. Fine-tune the model on these examples. The model learns to mimic the format and style of careful reasoning.
Limitation: You are limited by the quality and diversity of human reasoning. Humans are expensive and slow at writing detailed reasoning traces.
#Approach 2: Reinforcement Learning with Outcome Reward
Train the model using RL where the reward is based only on the final answer. The model is free to develop whatever internal reasoning strategy produces correct answers. This is how DeepSeek-R1 was trained.
R={+1−1if final answer is correctif final answer is incorrect
Remarkable finding: DeepSeek-R1 discovered extended reasoning, self-verification, and backtracking behaviors purely through RL -- without any human demonstrations. The model learned that "thinking before answering" is a good strategy because it leads to more rewards.
#Approach 3: Process Reward Models (Step-Level Feedback)
Instead of only rewarding the final answer, provide feedback on each step of the reasoning.
Rprocess=t=1∑Trtwhere rt=quality of step t
Why process reward models are better
An outcome reward model cannot distinguish between a correct answer obtained by luck and one obtained by sound reasoning.
A process reward model rewards good reasoning even when the final answer is wrong (due to a minor calculation error), and penalizes bad reasoning even when the final answer happens to be correct.
This produces more reliable reasoning -- the model develops trustworthy thinking patterns rather than learning to get lucky.
Reasoning models use 10-100x more tokens per query than standard models. At $15/million output tokens, a query that generates 2000 reasoning tokens costs 30x more than a standard 65-token response. For high-value tasks (medical diagnosis, code generation for production), this is easily worth it. For low-value tasks (casual chat, content filtering), it is wasteful.
Users expect chatbots to respond in under 2 seconds. A reasoning model that thinks for 30 seconds before responding breaks this expectation. Solutions include:
Streaming the reasoning so users see the model "thinking" (like o1's summary)
Routing easy questions to standard models and hard questions to reasoning models
Progressive display showing intermediate conclusions as the model reasons
Training-time scaling follows a power law: to double performance, you need roughly 10x more training compute. This is becoming prohibitively expensive.
Test-time compute scaling offers a more efficient path: let a moderately-trained model think longer on hard problems. This is analogous to how human expertise works -- a good student with 30 minutes outperforms a mediocre student with 5 minutes, even if the mediocre student "studied more" (was trained on more data).
Tests · Verify that the standard model gives $0.10 (incorrect). Verify that the reasoning model gives $0.05 (correct). Verify that ball + bat = $1.10 and bat - ball = $1.00 when ball = $0.05.
Reasoning models spend variable compute per question -- they "think longer" on harder problems by generating extended chains of thought, self-verification, and backtracking; this is test-time compute scaling
Training-time scaling is hitting diminishing returns; test-time scaling offers a new frontier -- instead of always making models bigger, let models think deeper when it matters; a well-trained model with 30 seconds of reasoning can outperform a much larger model that answers instantly
Process reward models produce more reliable reasoning than outcome reward models -- rewarding each step of reasoning (not just the final answer) trains models to develop trustworthy thinking patterns, not just lucky guesses
Reasoning models are NOT always better -- they are slower (10-60 seconds vs 0.5 seconds) and more expensive (10-100x more tokens); use them for hard math, logic, and coding; use standard models for simple queries, creative tasks, and latency-sensitive applications
The future is smart routing -- production systems will classify query difficulty and route to the appropriate model (fast and cheap vs slow and accurate), giving users the best tradeoff for each request
What is the key difference between chain-of-thought prompting and reasoning models like o1?
This lesson completes the NLP and Transformers track. You now understand the full arc: from tokenization to attention to transformers to GPT to prompt engineering to reasoning models. Next, explore how these models are aligned with human preferences in the Reinforcement Learning track, or dive into building applications with them in the RAG and Agents tracks.
After the reasoning chain is complete, the model produces the final answer: 9 sheep.
The reasoning tokens may or may not be visible to the user depending on the system design. OpenAI's o1 shows a summary of the reasoning. Claude's extended thinking can optionally display the full chain.
Easy questions -> small, fast model (Claude Haiku, GPT-4o mini)
Medium questions -> standard model (Claude Sonnet, GPT-4o)
Hard questions -> reasoning model (Claude with extended thinking, o1)
This gives you the best of all worlds: fast responses for easy questions, deep reasoning for hard ones, and optimized cost.