MDMD options discovery for accelerating exploration in sparse-reward domains

Zhu, Xianchao; Zhang, Ruiyuan; Zhu, William

doi:10.1016/j.knosys.2022.108151

Cited by 5 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare the proposed algorithms with Q-Learning (Watkins and Dayan 1992), BC (Simsek and Barreto 2008), SCC (Kazemitabar et al 2018), and MMSC (Setyawan et al 2022). Since the main goal of point option methods (Machado et al 2017;Jinnai et al 2019;Zhu et al 2022) is not making the agent reach the goal faster (in terms of steps or decisions), we argue that these studies are not directly comparable to ours. For SCC and BC, experiments are run with the same RL parameters as those given in Section 5.1.1.…”

Section: Experiments and Resultsmentioning

confidence: 86%

“…This operation is repeated until the given number of options is reached. Zhu et al (2022) propose a new option discovery method, Min Degree and Max Distance (MDMD) options, that accelerate the exploration by reducing the expected cover time of the environment in sparse reward problems. Unlike (Jinnai et al 2019), MDMD does not compute the Laplacian matrix's eigenvector.…”

Section: Related Workmentioning

confidence: 99%

“…Studies that construct point options (Machado et al 2017;Jinnai et al 2019;Zhu et al 2022) utilize a subclass of the option framework. Two states for the initiation set and the termination condition are sufficient for constructing a point option whereas two sets of states should be found for constructing options.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

et al. 2023

View full text Add to dashboard Cite

With the growing state/action space, learning a satisfactory policy for regular Reinforcement Learning (RL) algorithms such as flat Q-learning becomes quickly infeasible. One possible solution to handle such cases is to employ hierarchical RL (HRL). In this work, we present two methods to autonomously construct (1) skills (ASKA) and (2) arbitrarily elaborate superskills or complexes through defining an arbitrary number of hierarchies in HRL (ASKAC) over a graph-based iteratively-growing environment model. We employ dynamic community detection (DCD) in detecting subgoals since DCD considers local changes only over the partially growing graphs and lowers the time complexity of the subgoal detection where groups of environment states (i.e., subenvironments) are modeled by communities from the graph theory. DCD's drawback is oversegmentation where it mispartitions a subenvironment further into smaller components. To maintain the robustness of ASKAC against DCD's possible oversegmentation we introduce the concept of skill coupling. Skill coupling does not only robustly solve the oversegmentation issue, but it also improves HRL by building up more elaborate complexes (i.e., skill compositions) obtained at an arbitrary number of hierarchies and reduces the number of decisions leading to the goal employing these complexes. In addition to the experiments that investigate the effect of parameters, proposed methods are experimentally evaluated in grid world and taxi driver benchmark environments.

show abstract

Section: Experiments and Resultsmentioning

confidence: 86%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

et al. 2023

View full text Add to dashboard Cite

show abstract

“…This approach enhances exploration and transfer capabilities. Most representatively, the options framework may be the most common formalism that allows agents to reason regarding extended actions [56][57][58][59][60]. This framework models courses of action as options, which can accelerate learning in different ways, allowing, for example, faster credit assignment, planning, transfer learning, and better exploration.…”

Section: Hierarchical Reinforcement Learningmentioning

confidence: 99%

Efficient relation extraction via quantum reinforcement learning

Zhu,

Mu,

Wang

et al. 2024

Complex Intell. Syst.

Self Cite

View full text Add to dashboard Cite

Most existing relation extraction methods only determine the relation type after identifying all entities, thus not fully modeling the interaction between relation-type recognition and entity mention detection. This article introduces a novel paradigm for relation extraction by treating relevant entities as parameters of relations and harnessing the strong expressive capabilities and acceleration advantages of quantum computing to address the relation extraction task. In this article, we develop a quantum hierarchical reinforcement learning approach to enhance the interaction between relation-type recognition and entity mention detection. The entire relation extraction process is broken down into a hierarchical structure of two layers of quantum reinforcement learning strategies dedicated to relation detection and entity extraction, demonstrating greater feasibility and expressiveness, especially when dealing with superimposed relations. Our proposed method outperforms existing approaches through experimental evaluations on commonly used public datasets, mainly showcasing its significant advantages in extracting superimposed relationships.

show abstract

“…Thus, temporal abstraction is the process by which agents learn the temporal structure of tasks in a way that can cut down cognitive load and enhance the generalization ability of jobs with shared structure 40 – 45 . One formal approach to addressing this form of abstraction is the options framework 39 , 46 – 50 . Agents using options seek to learn a set of policies related to different subtasks, along with their initiation and termination conditions.…”

Section: Introductionmentioning

confidence: 99%

Temporally extended successor feature neural episodic control

Zhu

2024

Sci Rep

Self Cite

View full text Add to dashboard Cite

One of the long-term goals of reinforcement learning is to build intelligent agents capable of rapidly learning and flexibly transferring skills, similar to humans and animals. In this paper, we introduce an episodic control framework based on the temporal expansion of subsequent features to achieve these goals, which we refer to as Temporally Extended Successor Feature Neural Episodic Control (TESFNEC). This method has shown impressive results in significantly improving sample efficiency and elegantly reusing previously learned strategies. Crucially, this model enhances agent training by incorporating episodic memory, significantly reducing the number of iterations required to learn the optimal policy. Furthermore, we adopt the temporal expansion of successor features a technique to capture the expected state transition dynamics of actions. This form of temporal abstraction does not entail learning a top-down hierarchy of task structures but focuses on the bottom-up combination of actions and action repetitions. Thus, our approach directly considers the temporal scope of sequences of temporally extended actions without requiring predefined or domain-specific options. Experimental results in the two-dimensional object collection environment demonstrate that the method proposed in this paper optimizes learning policies faster than baseline reinforcement learning approaches, leading to higher average returns.

show abstract

MDMD options discovery for accelerating exploration in sparse-reward domains

Cited by 5 publications

References 18 publications

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

Efficient relation extraction via quantum reinforcement learning

Temporally extended successor feature neural episodic control

Contact Info

Product

Resources

About