Hierarchical RL & Options: Reasoning at Multiple Time Scales
Cooking a meal takes ten thousand muscle commands. Writing a novel takes a million keystrokes. Navigating a new building takes hundreds of footsteps. Flat RL — the kind that picks an atomic action every tick — drowns in long horizons and starves on sparse rewards. Humans don't operate that way. We chunk: "drive to work" is one decision, not seven thousand steering corrections. Hierarchical RL is the math of that chunking, and the Options framework (Sutton, Precup, Singh 1999) is its foundation.
Learning Objectives
After this lesson, you will be able to:
- Explain why long-horizon, sparse-reward problems break flat RL, and how temporal abstraction fixes it
- Define an Option as a (initiation set, intra-option policy, termination function) triple — the formal object that lets agents act over variable time spans
- Derive the semi-Markov Bellman equation for option-augmented MDPs and recognize when it reduces to vanilla Bellman
- Use the Option-Critic architecture (Bacon 2017) to learn options end-to-end from policy gradients, including the termination gradient
- Distinguish hand-designed, subgoal-discovered, and end-to-end-learned options; pick the one that matches your problem's structure
- Connect modern hierarchical agents (FeUdal Networks, HIRO, Voyager, SayCan) back to the 1999 Options framework
Prerequisites
Before You Start
Don't worry if "hierarchical RL" sounds like a different paradigm — it's not. It's just RL where some of your "actions" are sub-policies that run for many steps. The math is a straightforward generalization of the Bellman equation, and the intuition is exactly how you already plan your day.
#The Long-Horizon Problem
A flat RL agent — one that picks an atomic action at every tick — struggles when:
- The horizon is long. A 10,000-step episode means the policy gradient has 10,000 sources of variance per trajectory. Variance grows with horizon; sample efficiency collapses.
- Rewards are sparse. If the only reward is +1 at the goal and 0 everywhere else, the agent must accidentally stumble onto the goal before it can learn anything. In a maze with 1,000 states, this can take millions of episodes.
- Subtasks repeat. "Open the door" appears in a hundred different tasks. A flat agent re-learns it from scratch each time. There's no mechanism for skill reuse.
Humans solve all three by thinking hierarchically. "Make coffee" is one decision. "Walk to the kitchen" is a sub-decision. "Move the left leg forward 30 cm" is a sub-sub-decision. Each level has its own time scale, its own state representation, and its own (much shorter) horizon.
#A Concrete Example: Making Coffee
Imagine a household robot making a cup of coffee. The flat-RL view:
- 10,000 motor commands over 5 minutes (60 Hz control).
- Reward: +1 if a cup of coffee ends up on the counter, 0 otherwise.
- Number of trajectories needed: astronomical.
The hierarchical view, with three levels:
| Level | Time scale | Decisions | Example |
|---|---|---|---|
| High-level | once per minute | 5 | "go to kitchen", "open machine", "fill water", "press button", "wait" |
| Mid-level | once per second | ~300 | "walk forward", "turn left", "grasp handle", "rotate wrist" |
| Low-level | every tick (60 Hz) | ~18,000 | individual joint torques |
At the high level the horizon is 5, not 18,000. The credit assignment problem is now tractable: when the coffee fails, did the failure happen during "go to kitchen" or "press button"? That's a 5-way attribution, not a 18,000-way one.
What Do You Think?
Option duration is a random variable τ (some options take 1 step, some take 50). In the semi-Markov Bellman equation for options, what power of γ multiplies the next-state value Q(s', ω')?
#The Options Framework (Sutton, Precup, Singh 1999)
The canonical formalization. An option ω is a temporally extended action, defined by three things:
- Initiation set
I_ω ⊂ S— the states where the option is even allowed to start. ("Open the fridge" only initiates if you're near the fridge.) - Intra-option policy
π_ω(a | s)— the policy that picks primitive actions while the option is running. - Termination function
β_ω(s) ∈ [0, 1]— the probability that the option terminates in state s. When β fires, control returns to the high-level policy.
The high-level decision rule, called the policy over options μ(ω | s), picks an option whenever the previous one terminates.
A primitive action is just the trivial case:
I_a = S, π_a always picks a, β_a(s) = 1 for all s. Every primitive MDP is a special case of the options framework where every option terminates after exactly one step.#Semi-Markov MDPs
When the agent uses options, decisions don't happen every tick — they happen at option boundaries. Between boundaries, an arbitrary number of primitive steps elapse. This makes the augmented decision process a Semi-Markov Decision Process (SMDP):
- The "time" between high-level decisions is a random variable τ (the option's duration).
- Discounting must account for the variable interval — a 50-step option discounts the next value by γ⁵⁰; a 2-step option discounts by γ².
- The Bellman equation gets a τ-shaped exponent.
When every option is a one-step primitive (τ = 1 always), this collapses exactly to the vanilla MDP Bellman equation
Q(s, a) = E[R + γ max_a' Q(s', a')]. The options framework is a strict generalization of standard RL.#Option Discovery: Three Approaches
The options framework gives us the math but not the options themselves. Where do they come from?
#1. Hand-designed
The earliest approach (and still common in robotics). A domain expert writes down a small library of options: "grasp", "navigate-to-target", "open-door". Each has a known initiation set, a hand-coded or trained policy, and a hand-coded termination condition.
Pros: interpretable, debuggable, ships fast.
Cons: doesn't scale, doesn't transfer, expert effort dominates.
#2. Subgoal-based discovery
Identify bottleneck states — states with high betweenness centrality in the state graph. These are "doorways" that many trajectories must pass through. Then automatically construct options that reach these bottlenecks from anywhere in the surrounding region.
Classical examples: Şimşek & Barto 2009 ("Skill characterization based on betweenness"), McGovern & Barto 2001 ("Automatic discovery of subgoals").
Quick check
In a 4-room gridworld with narrow doorways between rooms, why are the doorway states 'bottlenecks' worth turning into option subgoals?
#3. End-to-end (Option-Critic)
Learn the options directly from policy gradients, jointly with the policy over options. This is the modern deep-RL approach, introduced in Bacon, Harb, and Precup's 2017 AAAI paper "The Option-Critic Architecture".
#The Option-Critic Architecture (Bacon 2017)
Three neural networks, all trained end-to-end:
- Policy over options
μ_θ(ω | s)— picks an option when the current one terminates. - Intra-option policies
π_{ω, ϕ}(a | s)— one head per option, picks primitive actions. - Termination functions
β_{ω, ψ}(s)— one head per option, outputs the probability of terminating.
All three are trained jointly via policy gradient. The three gradients are:
Intra-option policy gradient — standard policy gradient applied at every primitive step, using the option's own Q-value as a baseline:
Termination gradient — the key novelty. The termination function β should fire when the current option is worse than the best alternative available at s':
Policy-over-options gradient — standard policy gradient over the discrete-action set of options.
What Do You Think?
In Option-Critic, the termination function gradient is ∇β(s') · (Q(s', ω) - V(s')). When the advantage (Q - V) at termination is POSITIVE, what should β do?
#FeUdal Networks (Vezhnevets 2017)
A different architectural take on hierarchy: instead of an option triple, use a manager–worker decomposition.
- Manager runs at a slow time scale (every c steps, e.g., c=10). Outputs a goal g_t in a learned latent space.
- Worker runs at every primitive step. Receives the manager's goal and the current state; outputs primitive actions.
- Manager's reward is the external environment reward, accumulated over c-step windows.
- Worker's reward is an intrinsic reward measuring how well the worker moved the latent state in the direction the manager pointed (cosine similarity between actual latent transition and manager's goal direction).
This is end-to-end differentiable — both manager and worker are trained with policy gradients.
Quick check
Why does the FeUdal Networks manager receive a temporally-EXTENDED reward (accumulated over c steps), instead of getting the per-step environment reward?
#HIRO: Goal-Conditioned Hierarchies for Robotics
Nachum et al. 2018 introduced HIRO (HIerarchical Reinforcement learning with Off-policy correction), which became the standard for hierarchical robotics in the late 2010s.
- High-level policy outputs a goal g (a target state, e.g., a target position in 3D space) every k steps.
- Low-level policy receives (state, goal) and learns to reach the goal as fast as possible.
- Off-policy correction is the key trick: as the low-level policy improves, the high-level's old transition data becomes stale (the low level used to be bad at reaching goals, so the high level "saw" a different transition function then). HIRO relabels old high-level transitions with the goal that would have produced the observed trajectory under the current low-level policy.
HIRO works on real robotics tasks (Ant maze, manipulation) where flat methods fail. It's the workhorse hierarchy in many 2020-2024 robot-learning systems.
#The 4-Room Gridworld: Hands-On
Time to make the math concrete. Classical 4-room domain (Sutton, Precup, Singh 1999, Figure 1): an 11×11 grid divided into four rooms by walls, with four narrow doorways connecting adjacent rooms. The goal is a single cell. Reward: +1 at goal, 0 elsewhere. Episode ends at goal.
Flat Q-learning treats every cell as a state and learns Q(s, a) over four primitive actions (N, S, E, W). It takes many episodes because the only reward is at one cell, and exploration is slow.
2-option hierarchy. Define two options corresponding to "go to the doorway of the room currently containing the agent" — one option per room exit direction. Each option has:
- Initiation set: states inside the relevant room.
- Intra-option policy: hand-trained or hand-coded to navigate toward the doorway.
- Termination: when the doorway is reached.
The high-level Q-learner now picks options (effectively "leave through doorway X"). The state-action space at the high level is ~4 cells × ~4 options = 16 entries instead of 104 cells × 4 actions = 416 entries. Convergence is dramatically faster.
Loading visualization...
Two things to watch in the output:
- Convergence speed. The flat agent's first 50 episodes are very expensive (random walks through 104 cells, only +1 at the goal). The option agent's first 50 episodes are short because each option carries the agent halfway across the map.
- Asymptotic performance. Once the flat agent learns the optimal path, its per-episode step count drops to ~20. The option agent's asymptotic step count is comparable — the win is in sample efficiency, not in final policy quality. This is the canonical trade-off of hierarchical RL.
#Modern Hierarchical RL: LLMs as the High Level
The biggest 2022-2026 shift: the high-level policy is now usually an LLM. Two landmark systems:
SayCan (Ahn et al. 2022, Google). "Do As I Can, Not As I Say." An LLM proposes high-level options ("pick up the sponge", "open the drawer") given a natural-language task. A learned value function (the "Can" part) scores each proposal for feasibility in the current environment. The two scores are multiplied: the agent picks the option that's both linguistically appropriate AND physically possible.
Voyager (Wang et al. 2023, NVIDIA). A Minecraft agent that uses GPT-4 to generate an option library on the fly. Each "skill" (e.g., "mine wood with stone axe") is a snippet of code; the LLM writes the code, tests it in the game, refines if it fails, and adds successful skills to a growing library. After playing for 24 hours, Voyager has accumulated hundreds of reusable skills — exactly the "skill library" that hand-designed options promised in 1999, but now generated automatically by the LLM.
Reasoning-model LLMsReasoning ModelsReasoning models are LLMs trained to perform extended chain-of-thought reasoning before producing a final answer, improving performance on complex tasks.Learn more → are increasingly used as the high-level planner — the chain-of-thought trace becomes the explicit hierarchy, and the RL policy executes each step.
The key insight: the Options framework was the right abstraction; we just didn't have a good way to generate the options. LLMs solved the generation problem, and the 1999 math now powers a generation of embodied agents.
#When Hierarchical RL Wins
Use hierarchy when:
- Horizon is long. >1,000 primitive steps per episode is the rough threshold. Below that, flat RL's credit assignment is tractable.
- Rewards are sparse. If the only reward is at the goal and the goal is far, hierarchy gives you intermediate "rewards" (subtask completion) for free.
- Subtasks repeat. "Navigate to X" appears across many tasks. Learn it once as an option, reuse forever.
- There are discrete bottleneck states. Doorways, mode switches, phase transitions. These are natural option subgoals.
- You have a natural high-level planner. An LLM, a domain expert, or a task graph that gives you the option set for free.
#When Hierarchical RL Doesn't Help (or Hurts)
What Do You Think?
In which of these regimes does FLAT RL actually beat hierarchical RL?
#Connection to Multi-Task and Meta-RL
Options are, fundamentally, a transferable skill abstraction. Once you've learned a "navigate to doorway" option in one task, you can reuse it in every task that involves leaving a room. This makes the options framework deeply related to:
- Successor features (Barreto et al. 2017). A linear decomposition of value functions that lets you compose previously-learned policies into new ones — options are a special case where each "feature" is a subtask completion indicator.
- Universal value function approximators (Schaul et al. 2015). A single network V(s, g) that gives the value of being in state s when pursuing goal g. HIRO and FeUdal Networks are special cases.
- Meta-RL. Algorithms like MAML (Finn 2017) and RL² (Duan 2016) try to learn a prior over policies that adapts quickly to new tasks. Hierarchical methods get a similar effect more directly: the option library is the transferable knowledge.
In 2026, the line between "hierarchical RL", "skill learning", "meta-RL", and "LLM-as-planner" has blurred almost completely. They're all instances of the same idea: learn a small set of reusable behaviors, then compose them.
Recap
Key Takeaways
- An OPTION is a temporally extended action defined by three things: an initiation set (where it can start), an intra-option policy (what it does while running), and a termination function (when it stops). Primitive actions are the special case where the option terminates after one step.
- The SMDP Bellman equation generalizes vanilla Bellman: Q(s, ω) = E[Σ γ^k R_k + γ^τ max_ω' Q(s', ω')] where τ is the random option duration. The γ^τ exponent accounts for the variable time scale between high-level decisions.
- Option-Critic (Bacon 2017) learns options end-to-end via three policy gradients: intra-option policy gradient (standard PG inside an option), termination gradient (terminate when the current option is below the best alternative — advantage Q(s', ω) − V(s')), and policy-over-options gradient (standard PG over the discrete option set).
- FeUdal Networks (Vezhnevets 2017) and HIRO (Nachum 2018) use a manager–worker decomposition with goal-conditioned hierarchies. The manager gets temporally-extended reward; the worker gets intrinsic reward for following the manager's goals.
- Modern hierarchical agents (SayCan 2022, Voyager 2023) use LLMs as the high-level option generator. The 1999 Options framework supplies the math; LLMs supply the option library.
- Hierarchy wins on long horizons, sparse rewards, and repeated subtasks. It LOSES on short, dense, smooth problems where the overhead of option boundaries outweighs the credit-assignment win.