Up to now, actions were simple choices: left, right, jump. But Boston Dynamics' Atlas, Tesla's Optimus, every legged robot in MuJoCo — they all need precise continuous control: exact joint torques, smooth velocities, microsecond timing. SAC and TD3 are the algorithms purpose-built for that world, and they're why robotics in 2026 actually works.
Learning Objectives
After this lesson, you will be able to:
Understand maximum entropy RL: the SAC objective maximizes reward PLUS entropy at every step (J = E[Σ γ^t (r_t + α·H(π(·|s_t)))]), making the policy as random as possible while still collecting reward
See SAC's twin-critic trick: two independent Q-networks trained together prevent the Q-value overestimation that made DDPG unstable, using min(Q1, Q2) as a pessimistic target
Understand why continuous action spaces require fundamentally different algorithms than discrete ones -- and why PPO, SAC, and TD3 occupy distinct niches in the algorithm zoo
Know how SAC's automatic entropy tuning treats α as a Lagrange multiplier to maintain a target entropy level, removing the biggest hyperparameter from the algorithm
Everything we have built so far -- DQN, A2C, PPO -- handles discrete action spaces naturally. The agent chooses from a finite set: left/right, jump/stay, one of 18 Atari button combinations. This works when actions are countable.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
Now consider a robot arm with 7 joints. Each joint needs a torque command -- a continuous real number, say in [-1, 1]. The action space is a 7-dimensional continuous space. You cannot enumerate "all possible actions" and compute Q-values for each. There are infinitely many.
PPO handles this with a Gaussian policy: output the mean and variance of a Gaussian, then sample a continuous action. This works, but PPO was designed for on-policy learning (it discards data after each update). For continuous control tasks with expensive environment interactions -- a real robot arm, a physics simulator costing 1 second per step -- sample efficiency matters enormously.
SAC (Soft Actor-Critic) and TD3 (Twin Delayed DDPG) are off-policy algorithms specifically designed for continuous control. They are more sample efficient than PPO because they reuse data in a replay buffer, and they handle the instabilities that plagued earlier off-policy continuous control algorithms.
The entropy term H(π(·|s)) = -∫ π(a|s) log π(a|s) da measures how spread out the policy is. A deterministic policy has zero entropy. A uniform random policy has maximum entropy.
Try it! Hold your arm out and try to touch a specific spot on your desk using only one rigid motion path. Now try reaching the same spot but allowing your wrist, elbow, and shoulder to move freely. The second approach is more flexible and robust -- if someone bumps your elbow, you can still reach the target. That flexibility is what entropy regularization gives SAC.
What does this give us?
Better exploration: The agent is incentivized to try many different actions, naturally discovering diverse strategies
Robustness: Policies that maximize entropy tend to generalize across variations in the environment (they haven't over-committed to one path)
Multiple optimal policies: If two strategies achieve the same reward, MaxEnt prefers the one that remains more uncertain -- maintaining optionality
Automatic entropy tuning: α can be treated as a Lagrange multiplier and automatically tuned to maintain a target entropy level
Adjust the entropy coefficient and see how it reshapes the policy between sharp exploitation and broad exploration.
Unlike PPO's discrete softmax, SAC's policy outputs parameters of a Gaussian distribution over actions:
πθ(a∣s)=N(μθ(s),σθ2(s))
Actions are squashed through tanh to map from ℝ to [-1, 1], matching the typical action bounds of continuous control environments. This requires a correction to the log-probability (Jacobian of the tanh transformation):
logπ(a∣s)=logN(u∣μ,σ)−i=1∑dlog(1−tanh2(ui))
#The Overestimation Bug: Why One Critic Is Not Enough
Before the twin-critic trick, the dominant off-policy continuous control algorithm was DDPG (Deep Deterministic Policy Gradient, 2016). DDPG combined Q-learning with a deterministic policy and a replay buffer. It worked on some tasks but was notoriously unstable and hard to tune.
The root cause: Q-value overestimation. When you update the policy to maximize Q(s, π(s)), you are training the policy on the outputs of the Q-network. But the Q-network has approximation errors -- it overestimates Q-values in parts of the state-action space that the policy rarely visits (because there is little training data there). The policy exploits these overestimated values, taking actions that the Q-network rates highly but which are actually poor. This creates a feedback loop:
In practice, DDPG Q-values would spike upward during training and then catastrophically crash, producing an unstable policy.
TD3 and SAC both solve this with the same elegant fix: train two independent Q-networks and use the minimum of their predictions as the target.
y=r+γ⋅min(Q1(s′,a′),Q2(s′,a′))
Why does this work? Each Q-network has independent random initialization and samples different mini-batches for training. Their overestimation errors are uncorrelated. Taking the minimum is statistically likely to be closer to the true value than either individual estimate:
If Q1 overestimates: min(Q1, Q2) = Q2 (which may be closer to the truth)
If Q2 overestimates: min(Q1, Q2) = Q1 (which may be closer to the truth)
If both overestimate: min is still less extreme than either individually
The minimum biases the estimate downward (pessimistic), but slight underestimation is far preferable to the divergent overestimation that destroyed DDPG.
Both Q-networks are trained with the same target y. Both are kept in sync. Both have corresponding target networks (soft-updated copies used only for computing targets, not for gradient updates):
SAC trains four networks simultaneously (plus two target networks):
Policy network π_θ: maps states to Gaussian parameters (μ, σ)
Q-network 1 Q_φ1(s,a): estimates Q-value for state-action pairs
Q-network 2 Q_φ2(s,a): estimates Q-value (independent from Q1)
Target Q-networks Q_φ1_target, Q_φ2_target: slow-moving copies for stable targets
SAC does not have a separate value network V(s) in the modern version (Haarnoja et al. 2019 simplified the original formulation -- V(s) can be computed analytically from Q and the policy entropy).
Replay Buffer: unlike on-policy methods (PPO, A2C), SAC stores all transitions in a large replay buffer and samples random mini-batches for training. This makes SAC off-policy and dramatically increases sample efficiency.
Run the current policy in the environment, storing (state, action, reward, next_state, done) transitions in a replay buffer (typically 1M capacity). At the start of training, explore randomly for a warmup period (e.g., 10,000 steps) before using the policy.
Unlike PPO (which discards data after each update), SAC keeps ALL collected experience. A transition from 500,000 steps ago can still be sampled and used for training today.
Sample a random mini-batch of B=256 transitions from the replay buffer. Random sampling breaks temporal correlations between consecutive transitions, which stabilizes training. This is the same insight as DQN's experience replay.
The mini-batch contains diverse transitions from different stages of training -- some from early random exploration, some from the current near-optimal policy. This diversity is what makes off-policy learning powerful.
For each transition, compute the target value:
y = reward + gamma * min(Q1_target(s', a'), Q2_target(s', a')) - alpha * log_pi(a'|s')
where a' is sampled fresh from the current policy at the next state. The entropy term (-alpha * log_pi) is SAC's key addition -- it penalizes low-entropy targets, incentivizing exploration.
The min of twin critics prevents Q-value overestimation. The target networks (not the live networks) are used here -- this prevents the "chasing your own tail" instability.
Train Q1 and Q2 independently to minimize MSE against target y:
L_Q = (Q(s,a) - y)^2
Both networks receive the same target y. Their independent random initializations and mini-batch sampling ensure their errors remain uncorrelated, keeping the min trick effective.
The policy is updated to maximize Q-value plus entropy:
L_pi = E[alpha * log_pi(a|s) - Q(s,a)]
where a is sampled from pi(.|s) using the reparameterization trick (so gradients flow back to the policy network). The policy is trained to produce actions that Q-networks rate highly AND maintain high entropy.
Note: use Q1 only (not min of Q1, Q2) for the policy gradient -- using the min would underestimate gradient quality. Only the minimum is needed for targets; the policy update uses a single Q estimate.
Optionally, tune alpha automatically by treating it as a Lagrange multiplier with target entropy H_target = -dim(A):
L_alpha = alpha * (-log_pi(a|s) - H_target)
If the policy's entropy is below H_target (too deterministic), increase alpha to encourage more exploration. If above H_target (too random), decrease alpha. This removes the need to manually tune the exploration-exploitation balance.
Slowly blend the live Q-network weights into the target networks with rate tau=0.005:
phi_target = tau * phi + (1-tau) * phi_target
The target networks move slowly and smoothly, preventing the training target from oscillating. This is Polyak averaging -- the same technique used in DDPG and TD3.
TD3 (Twin Delayed Deep Deterministic Policy Gradient, Fujimoto et al. 2018) makes the same two fixes as SAC (twin critics + delayed updates) but with a deterministic policy instead of a stochastic one. No entropy bonus, no Gaussian -- the policy outputs a single action directly.
TD3 also uses delayed policy updates: update the critics every step, but only update the actor every d=2 steps. This gives the critics time to stabilize before the actor uses their estimates, reducing the policy-value interaction instability.
A robot arm needs to control 7 joint angles simultaneously. You try PPO with discrete action bins (3 bins per joint = 3^7 = 2,187 action combinations). It fails to learn anything useful. Why is SAC a better fit?
The combinatorial explosion is the primary issue: 2,187 discrete actions means the Q-function has to learn precise values for thousands of action bins, most of which are never visited. SAC outputs a continuous 7-dimensional Gaussian directly. The policy gradient flows through the continuous action space smoothly, without the need to enumerate all possible actions.
Additionally, joint angles are physically continuous -- a torque of 0.501 Nm and 0.500 Nm are nearly identical. Discrete bins throw away this structure. SAC's Gaussian policy exploits it naturally.
Algorithm
Action Space
Policy Type
Sample Efficiency
Key Strength
PPO
Discrete or continuous
Stochastic
Medium
Stability, simplicity, RLHF
SAC
Continuous
Stochastic (Gaussian)
High
Sample efficiency, robustness
TD3
Continuous
Deterministic
High
Simpler to tune than SAC
DDPG
Continuous
Deterministic
Medium
Predecessor to TD3/SAC
Use PPO when
Action space is discrete (Atari, board games, text tokens)
You need a well-understood algorithm with minimal hyperparameter tuning
Training in massively parallel environments (thousands of game copies)
Alignment/RLHF pipelines
Use SAC when
Continuous action space (robot joints, motor torques, steering angles)
Sample efficiency is critical (real robot, expensive simulator)
You want automatic exploration without manual tuning of exploration parameters
Standard benchmark: MuJoCo locomotion (HalfCheetah, Ant, Humanoid)
Use TD3 when
Continuous action space, deterministic policy is preferred (predictable behavior)
You need a simpler implementation than SAC (no entropy term, no reparameterization)
Tests · After 500 steps, single Q average should be significantly above 1.0 (overestimation). Twin Q average should stay within 0.1 of 1.0. Print the percentage error: (avg_q - 1.0) / 1.0 * 100 for both methods.
One of SAC's most practical features is that α (the entropy coefficient) can be automatically tuned without manual hyperparameter search. The idea: treat the entropy constraint as a Lagrange optimization problem.
We want the policy entropy to be at least H_target = -dim(A) (a rule of thumb: the negative of the number of action dimensions). α is adjusted to enforce this constraint:
Lα=Ea∼π[−αlogπ(a∣s)−αHtarget]
In practice, automatic entropy tuning removes the biggest hyperparameter from SAC -- you no longer need to search over α values. This makes SAC easier to apply to new tasks.
Maximum entropy RL adds an entropy bonus at every step. SAC maximizes reward PLUS entropy, incentivizing the policy to remain stochastic and exploratory rather than collapsing to a brittle deterministic strategy; this is what makes SAC robust to environment variations and local optima
Twin critics prevent the Q-value overestimation that caused DDPG to diverge. Two independently initialized Q-networks have uncorrelated estimation errors; using min(Q1_target, Q2_target) as the TD target introduces a conservative downward bias that cancels the systematic upward bias from the max operator in the policy update
SAC is off-policy: it reuses experience from a large replay buffer. Unlike PPO which discards data after each update, SAC stores all transitions and samples random mini-batches; this makes SAC 5-10x more sample efficient than PPO on continuous control benchmarks
Choose SAC for continuous control, PPO for discrete actions and RLHF. The two algorithms occupy complementary niches: SAC's Gaussian policy and off-policy replay shine in robotics and physics simulation; PPO's simplicity, stability, and on-policy nature make it the standard for LLM alignment and massively parallel discrete environments
What does the entropy term in SAC's objective function accomplish?
SAC and TD3 solve continuous control with off-policy efficiency and twin-critic stability. The next frontier: what if the agent could plan ahead, building a model of the world and simulating futures before acting? That is model-based RL -- coming next.