You have a petabyte of logs — driving data, recommendation impressions, medical records — but you can't run a live experiment. Welcome to offline RL: the setting where standard Q-learning fails catastrophically because the network hallucinates high values for actions it's never seen. CQL and IQL fixed it with one line of regularization, and now YouTube, TikTok, Tesla, and every recommender at scale runs on offline RL ideas.
Learning Objectives
After this lesson, you will be able to:
Explain why naive Q-learning on offline data fails — the distributional shift problem and out-of-distribution Q-value over-estimation
Use Conservative Q-Learning (CQL) and Implicit Q-Learning (IQL) as the production-grade offline RL algorithms in 2026
Frame RL as supervised sequence prediction with Decision Transformers and the return-conditioning trick
Pick offline RL when interaction is expensive, dangerous, or impossible — and know when offline-then-online finetuning beats either alone
Don't worry if "offline RL" sounds intimidating — it's just RL where you can't interact with the environment, only learn from a fixed dataset. The challenge isn't conceptual; it's that standard RL algorithms break in this setting and you need specific fixes.
#The Fundamental Problem: OOD Action Overestimation
Why doesn't standard Q-learning work offline? Because it queries Q(s, a) at actions that never appeared in the data, and the network confidently hallucinates high values.
Standard Q update: y=r+γa′maxQθ(s′,a′)⇒argmax may pick OOD action with overestimated Q
This isn't a small bug — it's catastrophic. Naive DQN trained offline on D4RL Walker2d-medium achieves return ~10 (random ≈ 0, expert ≈ 4500). The agent confidently picks actions it's never seen.
Two families of fixes
Constrain the policy to behave like the data (BCQ, BEAR, AWAC) — the "behavior-regularized" family
Constrain the Q-values to be conservative for OOD actions (CQL, IQL) — the "value-regularized" family
The 2024-2026 winner: value-regularized methods, specifically CQL and IQL.
Kostrikov 2021 IQL takes a different approach: never query Q at OOD actions in the first place. It uses expectile regression to fit a value function V(s) that approximates max_a Q(s, a) over the dataset's actions only.
Chen et al. 2021 reframed RL completely: treat (state, action, return-to-go) tuples as a sequence, train a GPT-style transformer on it. At inference, condition on a high target return and sample actions autoregressively.
Decision Transformer simplifies the algorithmic stack — no Q networks, no value bootstrap, no conservative penalties — and works well on Atari and locomotion benchmarks. The 2023+ "Trajectory Transformer" and "Gato" papers extended this to multi-task generalist agents.
You have logged data from a recommender system: 100M (user_features, recommended_item, watch_time) tuples. The data is from a previous deployed model. What's the right offline RL algorithm?
The answer: CQL or IQL are the right call. BC ignores reward signal entirely. Naive DQN will overestimate Q for never-shown items. Decision Transformer can work but requires careful return conditioning. CQL/IQL are designed for exactly this setting.
Method
Best for
Avoid when
CQL
Discrete or continuous actions; need strong OOD safety
Hyperparameter α is hard to tune
IQL
Continuous actions, simpler ops
Can underperform CQL on noisy datasets
Decision Transformer
Long-horizon tasks where sequence modeling is natural
Naive Q-learning fails offline because of OOD action overestimation. Q networks confidently extrapolate to high values on actions never seen in the data, and bootstrapping amplifies the error
CQL adds one conservative penalty that pushes Q-values down for OOD actions and up for in-data actions; α controls strength
IQL avoids the OOD problem entirely by using expectile regression on V(s) over dataset actions only — never queries Q at OOD actions during target computation
Decision Transformer reframes RL as supervised sequence modeling. Train a GPT on (return-to-go, state, action) tuples, condition on high target return at inference
Offline-to-online finetuning is the production pattern — pretrain offline on logs, briefly finetune online with safe constraints; the recipe behind Tesla FSD, recommender systems, and modern robotics
Why does standard DQN trained offline on D4RL Walker2d-medium fail catastrophically (return ~10 vs expert ~4500)?
You now know how to do RL when you can't even talk to the environment — the setting that matters for medical, robotic, and recommender-system applications. Next: RL for LLMs — how the same algorithms turned GPT-3 into ChatGPT and o1.