Constructing Temporally Extended Actions through Incremental Community Detection

Xu, Xing; Mei, Yang; Li, Ge

doi:10.1155/2018/2085721

Cited by 3 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For a skill to be obtained autonomously, the components of the triplet ⟨I, , ⟩ are needed. Therefore, there are many HRL studies that focus on detecting subgoals and/or defining initiation sets (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017;McGovern and Barto 2001;Stolle and Precup 2002;Şimşek and Barto 2004a;Ghafoorian et al 2013;Daniel et al 2016;Cockcroft et al 2020). Graph theoretic approaches are widely used for subgoal detection (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017).…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, there are many HRL studies that focus on detecting subgoals and/or defining initiation sets (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017;McGovern and Barto 2001;Stolle and Precup 2002;Şimşek and Barto 2004a;Ghafoorian et al 2013;Daniel et al 2016;Cockcroft et al 2020). Graph theoretic approaches are widely used for subgoal detection (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017). Some of these studies apply different community detection algorithms to identify initiation sets besides detecting subgoals (Xu et al 2018;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Farahani and Mozayani 2019).…”

Section: Related Workmentioning

confidence: 99%

“…Graph theoretic approaches are widely used for subgoal detection (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017). Some of these studies apply different community detection algorithms to identify initiation sets besides detecting subgoals (Xu et al 2018;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Farahani and Mozayani 2019). For example, Label Propagation (LP) (Raghavan et al 2007), an agglomerative clustering approach, is used to detect communities in Davoodabadi and Beigy (2011a), Farahani and Mozayani (2019).…”

Section: Related Workmentioning

confidence: 99%

“…A memory-based RL agent progressively constructs a partial model of the environment it interacts with in a dynamic fashion, where the partial model grows more similar to the actual environment as learning proceeds. Thus, subgoals are also detected in a dynamic manner (Xu et al 2018). In RL, environments may be effectively modeled using graphs (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019).…”

Section: Introductionmentioning

confidence: 99%

“…Thus, subgoals are also detected in a dynamic manner (Xu et al 2018). In RL, environments may be effectively modeled using graphs (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019). The connection points of subenvironments may be good candidates for subgoals by their nature.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

et al. 2023

View full text Add to dashboard Cite

With the growing state/action space, learning a satisfactory policy for regular Reinforcement Learning (RL) algorithms such as flat Q-learning becomes quickly infeasible. One possible solution to handle such cases is to employ hierarchical RL (HRL). In this work, we present two methods to autonomously construct (1) skills (ASKA) and (2) arbitrarily elaborate superskills or complexes through defining an arbitrary number of hierarchies in HRL (ASKAC) over a graph-based iteratively-growing environment model. We employ dynamic community detection (DCD) in detecting subgoals since DCD considers local changes only over the partially growing graphs and lowers the time complexity of the subgoal detection where groups of environment states (i.e., subenvironments) are modeled by communities from the graph theory. DCD's drawback is oversegmentation where it mispartitions a subenvironment further into smaller components. To maintain the robustness of ASKAC against DCD's possible oversegmentation we introduce the concept of skill coupling. Skill coupling does not only robustly solve the oversegmentation issue, but it also improves HRL by building up more elaborate complexes (i.e., skill compositions) obtained at an arbitrary number of hierarchies and reduces the number of decisions leading to the goal employing these complexes. In addition to the experiments that investigate the effect of parameters, proposed methods are experimentally evaluated in grid world and taxi driver benchmark environments.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

et al. 2023

View full text Add to dashboard Cite

show abstract

An agent with a sense of direction for option discovery in hierarchical reinforcement learning

Koudad,

Merzoug,

Benamar

2024

Int. J. Model. Simul. Sci. Comput.

View full text Add to dashboard Cite

Several approaches have been proposed to remedy the dimensionality problem that reinforcement learning (RL) suffers from. Among the solutions, hierarchical reinforcement learning (HRL) consists of dividing an RL problem into sub-problems called options or abstract actions. Discovering abstract actions or options for HRL is challenging, multiple approaches are proposed. In this paper, we present a new approach, an agent with direction sense for an automatic option discovery. Our agent uses its direction sense to discover shortcuts and shortest paths between states that he has already visited, he detects bottlenecks for building termination conditions and initiation states for options. Thus, at the learning step, the agent uses his previous experience of exploration in parallel with an intrinsically motivated learning. The options discovered are task-independent and could be used for new tasks. Experimental results on maze problems and Tic-tac-toe game indicate better results compared with flat RL and another RL approach in general and special cases.

show abstract

Learning Multi-Level Skill Hierarchies with Graphwave

Bele,

Žabkar

2024

Proceedings of the 10th Student Computing Research Symposium (SCORES’24)

View full text Add to dashboard Cite

We introduce a novel framework for learning multi-level skill hi-erarchies in reinforcement learning environments by leveraging structural similarities in state-space graphs. To obtain structural embeddings, we use the Graphwave algorithm, which places struc-turally similar states in close proximity in the latent space. In the latent space, we perform hierarchical clustering of states while respecting the topology of the state-space graph. At different levels of the hierarchy we learn the options that represent the skills; a skill at each level of the hierarchy is defined using the skills from the level below. We compare our approach with the state-of-the-art method across several environments. Our results show that structural embeddings can speed up option learning significantly in certain domains.

show abstract

Constructing Temporally Extended Actions through Incremental Community Detection

Cited by 3 publications

References 20 publications

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

An agent with a sense of direction for option discovery in hierarchical reinforcement learning

Learning Multi-Level Skill Hierarchies with Graphwave

Contact Info

Product

Resources

About