Actor-critic is one of the most elegant ideas in RL: pair a performer with a coach. The actor tries things; the critic gives instant feedback ("that was above average" or "that was a mistake"). Together they learn far faster than either could alone. This is the architecture beneath PPO, which is beneath RLHF, which is beneath ChatGPT. Every modern deep RL algorithm is some flavor of actor-critic.
Learning Objectives
After this lesson, you will be able to:
Understand the actor-critic setup: one network makes decisions (actor), another evaluates them (critic)
Code A2C from scratch, including the actor loss, critic loss, and the entropy bonus that keeps the agent exploring
See how A3C runs multiple agents in parallel and why this speeds up training
Follow the evolution from A2C to A3C to IMPALA to PPO -- each one fixing the previous one's weakness
The previous lesson introduced REINFORCE: collect a complete episode, compute returns, update the policy. It works, but it has a crippling weakness -- high variance. A single trajectory is a noisy estimate of the expected gradient. You need thousands of episodes before the signal becomes reliable, which makes training painfully slow.
Actor-critic methods are the engineering response to this problem. Instead of waiting for the full episode return, a second neural network (the critic) gives the actor real-time feedback after every single step.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
Actor-Critic adds a critic network that provides real-time feedback: "given the current state, is this action better or worse than average?" The actor plays the game; the critic whispers after every move "that was above average" or "that was a mistake."
The actor acts. The critic critiques. Both learn together. The result: much lower variance, faster convergence, and the ability to learn without waiting for episodes to end.
To understand why actor-critic exists, let us be precise about what is wrong with REINFORCE.
Problem 1: Must wait for episode end. REINFORCE computes the return G_t only after the episode terminates. In long episodes, this means thousands of steps of environment interaction before a single gradient update. Real-world systems -- robots, live games, streaming services -- cannot afford this latency.
Problem 2: High variance. G_t is a Monte Carlo estimate. The same action in the same state can yield wildly different returns across episodes due to stochasticity in the environment and policy. High variance means the gradient direction is unreliable, requiring many samples to average out the noise.
Problem 3: No bootstrapping. REINFORCE does not reuse any information between episodes. It is purely Monte Carlo -- "how did the whole episode go?" A more efficient approach uses bootstrapping: use the current value estimate to reduce the horizon of the Monte Carlo estimate.
All three problems share a root cause: REINFORCE does not maintain any model of the world. Actor-critic fixes this by training a value network (the critic) alongside the policy.
Try it! Imagine you are coaching someone learning to cook. After a full meal (REINFORCE), you say "the dinner was a 7 out of 10" -- not very useful. But if you give feedback after each step ("the sauce seasoning was perfect, the pasta was overcooked"), they learn much faster. That per-step feedback is the advantage estimate in actor-critic.
An actor-critic system consists of two components:
Actor (policy network π_θ): Takes the current state as input and outputs a probability distribution over actions. For discrete actions, this is a softmax. For continuous actions, this outputs the mean and variance of a Gaussian.
Critic (value network V_φ): Takes the current state as input and outputs a single scalar -- the expected total discounted reward from this state onward. This is the state-value function V(s).
In practice, the actor and critic often share a backbone network -- the same lower-level feature extractor, with two separate output heads. This allows them to share learned representations of the environment.
The key quantity in actor-critic is the advantage function:
A(s,a)=Q(s,a)−V(s)
We never compute Q(s,a) directly in A2C. Instead, we estimate the advantage using the TD residual (one-step temporal difference error):
A^(st,at)=rt+γVϕ(st+1)−Vϕ(st)
This is what makes actor-critic efficient: instead of waiting for the episode to finish, the critic provides a one-step estimate of how good the action was, relative to the current value function. This allows updates after every single environment step.
A2C trains the actor and critic jointly, with separate loss terms:
Actor loss (policy gradient with advantage):
Lactor=−Et[logπθ(at∣st)⋅A^t]
Critic loss (mean squared error on value prediction):
Lcritic=Et[(rt+γVϕ(st+1)−Vϕ(st))2]
Entropy bonus (exploration regularization):
H(π)=−a∑π(a∣s)logπ(a∣s)
The full A2C loss combines all three:
LA2C=Lactor+c1Lcritic−βH(π)
What Do You Think?
You are comparing A2C and REINFORCE on the same CartPole task. Both eventually converge to the same optimal policy. A2C reaches it in 3x fewer environment steps. Why?
The critic's advantage estimates have much lower variance than REINFORCE's Monte Carlo returns. Lower variance means the gradient direction is more accurate on each step. More accurate gradients mean more learning per environment interaction. Additionally, bootstrapping (using V(s') instead of the full return) lets A2C update after every step rather than waiting for episode termination -- meaning it processes many more gradient updates in the same wall-clock time.
Run the policy for N steps across M parallel environments. Record (state, action, reward, next_state, done) for each step. With N=5 steps and M=16 environments, you have 80 transitions per update.
Unlike REINFORCE, you do NOT wait for episodes to end. Steps from the middle of an episode are just as useful because the critic handles the value estimation for the incomplete remainder.
Pass all collected states through the shared actor-critic network. Get action probabilities from the actor head and value estimates from the critic head. Also pass next_states through the critic to get V(s') for bootstrapping.
Because the actor and critic share a backbone, this is a single forward pass -- computationally efficient.
For each transition: advantage = reward + gamma * V(next_state) - V(state). This is the TD residual. Positive advantage means "the actual outcome beat what the critic expected." Negative means "worse than expected."
Detach the advantage from the computation graph before using it for the actor update. The actor should not backpropagate through the critic's computation -- that would create interference between the two learning objectives.
Compute the actor loss: negative log-probability of actions taken, weighted by advantages. Compute the critic loss: squared TD error between predicted values and bootstrap targets. Compute the entropy bonus: negative entropy of the action distribution.
Call total_loss.backward(). Because the actor and critic share the backbone, gradients from both heads flow back through the shared layers and combine naturally. The optimizer takes a single step that simultaneously improves the policy and the value estimate.
This is why the shared-backbone design is popular: one backward pass updates all parameters.
Collect the next batch of N steps. The policy is now slightly better; the value estimates are slightly more accurate; the entropy bonus ensures we haven't collapsed to a deterministic policy yet. Repeat until convergence.
A2C with N=5, M=16, and 1M total environment steps typically solves CartPole in under 10 seconds on a laptop.
A2C and A3C (Asynchronous Advantage Actor-Critic) differ in how they use parallel workers. Both run multiple environment instances simultaneously to collect diverse experiences, but they differ in when gradients are applied.
All workers run for N steps, then wait for each other. The gradients from all workers are averaged, and the shared model is updated once. Every worker sees the same model version before and after each update.
Worker 1: ----run---- WAIT ------run------ WAIT
Worker 2: ----run---- WAIT ------run------ WAIT
Worker 3: ---run--- WAIT ----run---- WAIT
| |
Average grads Average grads
Update model Update model
Pros: No gradient staleness; all workers use the same, up-to-date policy; simpler to reason about; more stable
Cons: The slowest worker sets the pace; GPU/CPU resources are idle during waits
Pros: No idle time; faster wall-clock training; CPU-friendly (A3C was designed for multi-core CPUs)
Cons: Gradient staleness -- Worker 1 may compute gradients with policy version t=100, but by the time they arrive, the shared model is at t=107. The gradients are stale.
What Do You Think?
A3C runs 16 workers. Worker #1 finishes 5 seconds before Worker #16. Worker #1 sends gradients computed with policy version t=100. But the learner is now at t=105 when the gradients arrive. Is this a problem?
This is gradient staleness, and it is a real concern. The gradients from Worker #1 were computed assuming the policy was at version t=100. If the policy has changed significantly by t=105, those gradients point in a direction that may not be helpful -- or may even be harmful. A3C accepts this tradeoff for speed. IMPALA explicitly corrects for it.
IMPALA (Importance Weighted Actor-Learner Architecture, DeepMind 2018) takes parallelism to production scale. The key innovation is full decoupling of acting and learning.
In A3C, each worker computes both environment steps AND gradients. In IMPALA, roles are separated:
Actors: Many (potentially thousands) lightweight processes that only run the environment and collect experience. No gradient computation.
Learner: A central GPU process that only does gradient updates. Receives experience streams from actors.
This decoupling means the learner is always doing useful work (training), and actors are always doing useful work (collecting data). Neither waits for the other.
The off-policy problem. Because actors are independent, by the time an actor's trajectory reaches the learner, the learner's policy may have moved several updates ahead. The actor was running an older policy than what the learner is currently training. This creates an off-policy learning problem -- the data does not match the current policy.
V-trace: the correction. IMPALA uses V-trace, an off-policy correction algorithm:
vs=V(xs)+t=s∑s+n−1γt−s(i=s∏t−1ci)δtV
V-trace clips the importance ratio to keep corrections stable, providing a principled way to learn from slightly off-policy data. This is critical at production scale where actor-learner lag is unavoidable.
Each algorithm in this lineage fixed a specific problem with its predecessor:
Algorithm
Year
Key Innovation
Problem Fixed
A2C
2016
Synchronous parallel workers
REINFORCE variance
A3C
2016
Asynchronous updates (no sync barrier)
A2C idle time
IMPALA
2018
Decoupled actors + V-trace correction
A3C gradient staleness at scale
PPO
2017
Clipped objective (trust region)
A2C/A3C unstable large updates
PPO and IMPALA are contemporaries that address different aspects of the same problem. PPO focuses on stability of updates (how far can the policy change per step). IMPALA focuses on scalability (how to train with thousands of actors). Modern production systems often combine both ideas.
Where each is used today
A2C: Standard baseline for discrete action spaces; clean, simple, well-understood
A3C: Largely replaced by A2C + GPU parallelism; still used in CPU-only settings
IMPALA: Large-scale gaming AI, Atari57, DeepMind Control Suite; anywhere with thousands of actors
PPO: LLM alignment (RLHF), robotics, most academic RL research -- the dominant algorithm
Tests · After 300 episodes, the last-30 average episode length should be greater than the first-30 average — that's the signal that the actor and critic are actually learning. The advantage values should also shrink in magnitude as the critic catches up to the true returns.
The advantage function A(s,a) = Q(s,a) − V(s) is the core signal. By measuring how much better an action is than the average in a given state, the advantage provides a centered, lower-variance gradient signal compared to raw returns; this single formula is the beating heart of every modern actor-critic algorithm
A2C uses synchronous parallel workers for stable updates. All environments finish their rollout before gradients are averaged and applied; this guarantees every worker uses the same policy version, trading some wall-clock speed for stability and predictability
A3C achieves faster wall-clock time at the cost of gradient staleness. Asynchronous workers send gradients immediately without waiting; the policy advances while those gradients are in flight, meaning each update is computed against a slightly outdated policy
IMPALA decouples actors from learners with V-trace correction. Thousands of actors send trajectories to a central GPU learner; V-trace importance-sampling weights correct for the off-policy gap between the actor's old policy and the learner's current policy, enabling production-scale training
Interactive Lab
Watch the actor and critic networks learn side by side — see policy probabilities shift as the critic's value estimates sharpen, and observe the advantage signal flowing between them in real time.
Toggle between the actor's policy distribution and the critic's value heatmap on the same gridworld — internalize why actor-critic methods carry two complementary views of the world.
What is the advantage function A(s,a) and why is it used instead of the raw return G_t?
Actor-critic gives us a stable, low-variance policy gradient algorithm. But it still suffers from destructively large policy updates -- one bad gradient step can ruin weeks of training. Next up: Proximal Policy Optimization (PPO) -- which fixes this with a single elegant idea: clip the objective to prevent any single update from moving the policy too far.