DeepMind beat Lee Sedol with AlphaGo. OpenAI beat Dota 2 world champions with OpenAI Five. AlphaFold cracked protein folding. ChatGPT became a household name. All four use reinforcement learning. RL is the framework for any AI that learns by doing — and after this lesson, the agent-environment loop will be something you see everywhere in modern AI.
Learning Objectives
After this lesson, you will be able to:
Understand the agent-environment loop and its five parts: state (what the agent sees), action (what it does), reward (the score), policy (the strategy), and transition (what happens next)
See how sequential decision-making gets formalized as a Markov Decision Process (MDP) -- the math behind "what I do now affects what happens later"
Know the MDP components (S, A, R, P, γ) and the Bellman equation that recursively defines the value of a state in terms of future states
Distinguish when RL is the right tool vs. supervised or unsupervised learning, and understand the unique challenges of credit assignment and exploration in RL
Over time, the dog learns: sitting leads to treats. It starts sitting more often. It has learned a policy -- a mapping from situations to actions -- entirely through trial and error, guided by rewards.
That is reinforcement learning. No labeled dataset. No explicit instructions. Just an agent interacting with an environment, receiving rewards, and gradually figuring out what works.
Reinforcement learning (RL) is the third pillar of machine learning, alongside supervised and unsupervised learning. It is fundamentally different from both. In supervised learning, a teacher provides the correct answer for every input. In unsupervised learning, the algorithm finds structure in unlabeled data. In RL, an agent learns by taking actions in an environment and receiving rewards that signal how well it is doing.
This is how humans learn most things. Nobody gave you a labeled dataset for riding a bicycle. You tried, fell, adjusted, tried again, and eventually learned. RL formalizes this intuition into mathematics.
Every RL problem has the same structure: an agent interacts with an environment in a loop.
The agent observes the current state of the environment
The agent chooses an action based on that state
The environment transitions to a new state
The environment gives the agent a reward signal
Repeat from step 1
This loop continues until the episode ends (the game is over, the robot reaches its goal, the conversation concludes) or forever in continuing tasks.
Try it! Think of a game you have played recently -- chess, a video game, even rock-paper-scissors. Can you identify the state (what you see), the action (what you choose), and the reward (win/lose/score)? Every game is an RL problem. You have been doing reinforcement learning your whole life without knowing it.
The agent perceives the current state of the environment. In a grid world, this is the agent's position. In a self-driving car, it includes speed, GPS coordinates, and camera images. The state is the agent's window into the world -- everything it knows right now.
The agent has no knowledge of the future. It must decide what to do based solely on this snapshot.
Given the current state, the agent uses its policy to choose an action. Early in training, the policy is essentially random -- the agent has no idea what works. Later, as it learns from experience, the policy becomes increasingly strategic.
The action could be discrete (move left, right, up, down) or continuous (apply 3.7 Newtons of force at a 42-degree angle).
After the agent acts, the environment provides a reward signal -- a single number indicating how good or bad the outcome was. The reward encodes the task's objective. Reach the goal? +1. Fall off a cliff? -10. Just surviving? +0.01.
This is the only feedback the agent ever receives. No explanations, no corrections -- just a number.
The environment transitions to a new state s' as a consequence of the agent's action. The agent now finds itself in a different situation. In a grid world, it has moved to a new cell. In a game, the screen has changed.
This transition may be deterministic (same action always leads to same result) or stochastic (randomness is involved).
With the experience tuple (s, a, r, s'), the agent updates its internal knowledge. It adjusts its policy to make good actions (those that led to high reward) more likely and bad actions less likely. This is where the actual learning happens.
Different algorithms update differently -- Q-learning adjusts a value table, policy gradients adjust action probabilities, actor-critics update both.
The agent is now in state s'. It observes this new state, selects another action, receives another reward, transitions again, and learns again. This loop runs for hundreds, thousands, or millions of steps.
Over time, the agent's policy converges toward optimal behavior -- maximizing cumulative reward across the entire episode. What started as random stumbling becomes strategic, purposeful action.
State (S): A description of the current situation. In chess, the state is the board position. In a self-driving car, the state includes speed, position, nearby objects, traffic signals. The state must contain enough information for the agent to make a good decision.
Action (A): What the agent can do. In chess, the action is a legal move. In a video game, actions might be {left, right, jump, shoot}. The set of all possible actions is the action space -- it can be discrete (finite choices) or continuous (any value in a range, like steering angle).
Reward (R): A scalar signal from the environment indicating how good or bad the last action was. Win the game? +1. Lose? -1. Stay alive? +0.01. The reward function encodes the goal. The agent's entire purpose is to maximize cumulative reward over time.
Policy (pi): The agent's strategy -- a mapping from states to actions. A policy can be deterministic ("in state X, always do action Y") or stochastic ("in state X, do action Y with 70% probability and action Z with 30%"). The goal of RL is to find the optimal policy -- the one that maximizes expected total reward.
Transition function (T): How the environment changes in response to actions. In a deterministic environment, the same state-action pair always leads to the same next state. In a stochastic environment, there is randomness -- the same action might lead to different outcomes.
What Do You Think?
A robot is learning to walk. It takes a step and falls over. What role does the 'falling over' play in the RL framework?
Falling over serves two roles: the environment transitions to a new state (robot on the ground), and the agent likely receives a negative reward (penalty for falling). Together, these signals teach the agent to avoid whatever action caused the fall.
Figure
A circular diagram showing the agent-environment loop: the agent observes state s, selects action a, the environment returns reward r and next state s', and the cycle repeats -- with arrows flowing clockwise through observe, act, reward, transition.
In chess, the current board position tells you everything you need to know -- it does not matter how you got there. This is the Markov property. Many real-world problems are approximately Markov if the state representation is rich enough. If it is not, you can often stack multiple observations together (e.g., using the last 4 video game frames as the state) to make it approximately Markov.
Try it: Navigate the Grid WorldInteractive
Watch an RL agent learn to navigate a grid world. The agent starts with no knowledge and gradually discovers the optimal path through trial and error. Adjust the learning rate and discount factor to see how they affect learning. This is an MDP in action -- states are grid cells, actions are movements, and the reward is at the goal.
The agent does not just maximize immediate reward -- it maximizes the return, the total accumulated reward from the current time step onward. But should a reward received 100 steps from now count as much as a reward received right now?
Usually not. We apply a discount factor gamma (between 0 and 1) that makes future rewards worth progressively less:
Gt=Rt+1+γRt+2+γ2Rt+3+⋯=k=0∑∞γkRt+k+1
Why discount?
Mathematical convenience: The infinite sum converges when gamma < 1
Uncertainty: The further into the future, the less certain we are about predictions
Preference for sooner rewards: Just like in economics, a dollar today is worth more than a dollar tomorrow
Practical: Without discounting, the return in continuing tasks would be infinite
The agent needs to evaluate states and actions. "How good is it to be in this state?" and "How good is it to take this action in this state?" These questions are answered by value functions.
No supervisor: There is no oracle providing the correct action. The agent must discover good actions through trial and error.
Delayed reward: The consequences of an action may not be apparent for many steps. A chess move might only prove good or bad 30 moves later.
Non-stationary data: The agent's data distribution changes as it learns -- improving its policy changes which states it visits.
Exploration vs. exploitation: The agent must balance trying new things (exploration) with doing what it already knows works (exploitation).
What Do You Think?
Which of these problems is best suited for reinforcement learning rather than supervised learning?
Balancing a pole on a cart (CartPole) is a classic RL problem because: (1) there is no dataset of "correct" actions, (2) the agent must learn through interaction, (3) the reward (staying balanced) depends on a sequence of actions, and (4) the physics of the environment create a sequential decision-making challenge.
Episodic tasks have a clear end: a chess game terminates in checkmate or draw, an Atari game ends when the player dies. The agent's experience breaks naturally into episodes.
Continuing tasks go on forever: a thermostat controlling room temperature, a trading bot managing a portfolio, a recommendation system serving users. There is no terminal state.
Model-free: The agent learns directly from experience without building an internal model of the environment. "I do not know how the world works, but I know what works in the world." Q-learning and policy gradients are model-free.
Model-based: The agent learns a model of the environment (transition probabilities and rewards) and uses it to plan. "I have a mental model of the world, and I use it to simulate and plan ahead." AlphaGo uses a learned model for planning via Monte Carlo Tree Search.
On-policy: The agent learns about the policy it is currently following. It collects experience using its current strategy and updates that same strategy. SARSA and PPO are on-policy.
Off-policy: The agent can learn from data collected by a different policy. It can learn from old experience, from expert demonstrations, or from random exploration. Q-learning and DQN are off-policy.
Tests · Run 100 episodes with the random policy and compute average steps. Then run 100 with the optimal policy. The optimal should always take exactly 4 steps.
RL learns through trial-and-error interaction. Unlike supervised learning which needs labeled examples, RL agents discover optimal behavior by taking actions and receiving reward signals from the environment
The agent-environment loop has five core components. State (where am I), action (what can I do), reward (how good was that), policy (what should I do), and transition (what happens next) define every RL problem
MDPs formalize sequential decision-making. The Markov property says the future depends only on the current state, not the history of how you got there, which makes the math tractable
RL is the right tool when the environment is interactive. Games, robotics, recommendation systems, and dialogue optimization all involve sequential decisions with delayed rewards where RL excels
Interactive Lab
Drop an RL agent into a grid world, give it +10 for reaching the exit and -1 per step, and watch the agent-environment loop turn random flailing into deliberate navigation over a few hundred episodes.
In the RL agent-environment loop, which component determines the agent's strategy for choosing actions?
You now understand the fundamental framework of reinforcement learning -- the agent-environment loop, MDPs, rewards, and value functions. Next up: the exploration-exploitation dilemma -- should the agent try something new or stick with what it knows works?