Hierarchical Reinforcement Learning

Pateria, Shubham; Subagdja, Budhitama; Tan, Ah-Hwee; Quek, Chai

doi:10.1145/3453160

Cited by 207 publications

(90 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hierarchical RL (Pateria et al, 2022) is a way to learn, plan, and represent knowledge with temporal abstraction at multiple levels, with a long history, e.g., options (Sutton et al, 1999). Hierarchical RL is an approach for issues of sparse rewards and/or long horizons, with exploration in the space of high-level goals.…”

Section: More Challengesmentioning

confidence: 99%

Reinforcement Learning in Practice: Opportunities and Challenges

Li¹

2022

Preprint

View full text Add to dashboard Cite

This article is a gentle discussion about the field of reinforcement learning for real life, about opportunities and challenges, with perspectives and without technical details, touching a broad range of topics. The article is based on both historical and recent research papers, surveys, tutorials, talks, blogs, and books. Various groups of readers, like researchers, engineers, students, managers, investors, officers, and people wanting to know more about the field, may find the article interesting.In this article, we first give a brief introduction to reinforcement learning (RL), and its relationship with deep learning, machine learning and AI. Then we discuss opportunities of RL, in particular, applications in products and services, games, recommender systems, robotics, transportation, economics and finance, healthcare, education, combinatorial optimization, computer systems, and science and engineering. The we discuss challenges, in particular, 1) foundation, 2) representation, 3) reward, 4) model, simulation, planning, and benchmarks, 5) learning to learn a.k.a. meta-learning, 6) off-policy/offline learning, 7) software development and deployment, 8) business perspectives, and 9) more challenges. We conclude with a discussion, attempting to answer: "Why has RL not been widely adopted in practice yet?" and "When is RL helpful?".

show abstract

Section: More Challengesmentioning

confidence: 99%

Reinforcement Learning in Practice: Opportunities and Challenges

Li¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We propose a two level Feudal Reinforcement Learning agent shown in Figure 2. Feudal Reinforcement Learning (Feudal RL) is a type of Hierarchical Reinforcement Learning where a high-level controller sets a subtask that is executed by a lower-level controller [Dayan and Hinton, 1992;Pateria et al, 2021]. The state space S = S h × S o consists of the positions and velocities of the agent's joints as well as the position and velocity of the projectile ball.…”

Section: R(s) =mentioning

confidence: 99%

Hierarchical Reinforcement Learning of Locomotion Policies in Response to Approaching Objects: A Preliminary Study

Yu¹,

Rammohan²,

Zheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Animals such as rabbits and birds can instantly generate locomotion behavior in reaction to a dynamic, approaching object, such as a person or a rock, despite having possibly never seen the object before and having limited perception of the object's properties. Recently, deep reinforcement learning has enabled complex kinematic systems such as humanoid robots to successfully move from point A to point B. Inspired by the observation of the innate reactive behavior of animals in nature, we hope to extend this progress in robot locomotion to settings where external, dynamic objects are involved whose properties are partially observable to the robot. As a first step toward this goal, we build a simulation environment in MuJoCo where a legged robot must avoid getting hit by a ball moving toward it. We explore whether prior locomotion experiences that animals typically possess benefit the learning of a reactive control policy under a proposed hierarchical reinforcement learning framework. Preliminary results support the claim that the learning becomes more efficient using this hierarchical reinforcement learning method, even when partial observability (radius-based object visibility) is taken into account.

show abstract

“…Different from previous work on HRL [21,31] and symbolic planning [28,15], in this work we use ILP to learn a symbolic state transition model consisting of learned clauses. The ILP method adopted is a refinement-based ∂ILP which integrates refinement operation [5] and ∂ILP [10] together.…”

Section: Learning Symbolic Transition Model Via Ilpmentioning

confidence: 99%

Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming

Xu,

Fekri

2021

Preprint

View full text Add to dashboard Cite

Recently deep reinforcement learning has achieved tremendous success in wide ranges of applications. However, it notoriously lacks data-efficiency and interpretability. Data-efficiency is important as interacting with the environment is expensive. Further, interpretability can increase the transparency of the blackbox-style deep RL models and hence gain trust from the users. In this work, we propose a new hierarchical framework via symbolic RL, leveraging a symbolic transition model to improve the data-efficiency and introduce the interpretability for learned policy. This framework consists of a high-level agent, a subtask solver and a symbolic transition model. Without assuming any prior knowledge on the state transition, we adopt inductive logic programming (ILP) to learn the rules of symbolic state transitions, introducing interpretability and making the learned behavior understandable to users. In empirical experiments, we confirmed that the proposed framework offers approximately between 30% to 40% more data efficiency over previous methods.

show abstract

Hierarchical Reinforcement Learning

Cited by 207 publications

References 31 publications

Reinforcement Learning in Practice: Opportunities and Challenges

Reinforcement Learning in Practice: Opportunities and Challenges

Hierarchical Reinforcement Learning of Locomotion Policies in Response to Approaching Objects: A Preliminary Study

Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming

Contact Info

Product

Resources

About