A graph-theoretic approach toward autonomous skill acquisition in reinforcement learning

Kazemitabar, Seyed Jalal; Taghizadeh, Nasrin; Beigy, Hamid

doi:10.1007/s12530-017-9193-9

Cited by 4 publications

(12 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Continuously approximating the edge weights relies on exploiting historical information. This is similar to the SCC based method [19] which takes advantage of both graphical approach and frequency based approach. The expected advantage is to help locating subgoals more accurately.…”

Section: Discussionmentioning

confidence: 99%

“…Combining neural network training, it shows effectiveness for complex environments like Atari games. The work presented in [19] finds subgoals in linear time based on forming Strongly Connected Component (SCC) of the graph. What is unique is that this method also exploits historical data to help improve the performance.…”

Section: Literature Reviewmentioning

confidence: 99%

“…These two branches of approaches do not have a clear border. Actually some approaches can take advantage of both of them, such as the SCC based approach [19]. The approach in [13] also operates on the state transition graph but concentrates more on some metric in the context of the ant colony optimization.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Compared to some existing graphical based option generation algorithms, employing Louvain algorithm is of advantage on computation time. For instance, the betweenness centrality computation following [26] should be O ( nm ) and O ( nm + n 2 log⁡ n ) for unweighted and weighted graph, respectively; LPA grows like O ( km ) where k is the sum of the algorithm's internal label propagation iterations; and the SCC based method [19] is a linear time algorithm with an O ( m + n ) complexity.…”

Section: Generating Options From Communitiesmentioning

confidence: 99%

See 3 more Smart Citations

Constructing Temporally Extended Actions through Incremental Community Detection

Mei

2018

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Hierarchical reinforcement learning works on temporally extended actions or skills to facilitate learning. How to automatically form such abstraction is challenging, and many efforts tackle this issue in the options framework. While various approaches exist to construct options from different perspectives, few of them concentrate on options' adaptability during learning. This paper presents an algorithm to create options and enhance their quality online. Both aspects operate on detected communities of the learning environment's state transition graph. We first construct options from initial samples as the basis of online learning. Then a rule-based community revision algorithm is proposed to update graph partitions, based on which existing options can be continuously tuned. Experimental results in two problems indicate that options from initial samples may perform poorly in more complex environments, and our presented strategy can effectively improve options and get better results compared with flat reinforcement learning.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Generating Options From Communitiesmentioning

confidence: 99%

See 2 more Smart Citations

Constructing Temporally Extended Actions through Incremental Community Detection

Mei

2018

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Thus, subgoals are also detected in a dynamic manner (Xu et al 2018). In RL, environments may be effectively modeled using graphs (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019). The connection points of subenvironments may be good candidates for subgoals by their nature.…”

Section: Introductionmentioning

confidence: 99%

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

et al. 2023

View full text Add to dashboard Cite

With the growing state/action space, learning a satisfactory policy for regular Reinforcement Learning (RL) algorithms such as flat Q-learning becomes quickly infeasible. One possible solution to handle such cases is to employ hierarchical RL (HRL). In this work, we present two methods to autonomously construct (1) skills (ASKA) and (2) arbitrarily elaborate superskills or complexes through defining an arbitrary number of hierarchies in HRL (ASKAC) over a graph-based iteratively-growing environment model. We employ dynamic community detection (DCD) in detecting subgoals since DCD considers local changes only over the partially growing graphs and lowers the time complexity of the subgoal detection where groups of environment states (i.e., subenvironments) are modeled by communities from the graph theory. DCD's drawback is oversegmentation where it mispartitions a subenvironment further into smaller components. To maintain the robustness of ASKAC against DCD's possible oversegmentation we introduce the concept of skill coupling. Skill coupling does not only robustly solve the oversegmentation issue, but it also improves HRL by building up more elaborate complexes (i.e., skill compositions) obtained at an arbitrary number of hierarchies and reduces the number of decisions leading to the goal employing these complexes. In addition to the experiments that investigate the effect of parameters, proposed methods are experimentally evaluated in grid world and taxi driver benchmark environments.

show abstract

Cellular Automata, Learning Automata, and Cellular Learning Automata for Optimization

Kordestani

Mirsaleh

Rezvanian

et al. 2021

Intelligent Systems Reference Library

View full text Add to dashboard Cite

A graph-theoretic approach toward autonomous skill acquisition in reinforcement learning

Cited by 4 publications

References 19 publications

Constructing Temporally Extended Actions through Incremental Community Detection

Constructing Temporally Extended Actions through Incremental Community Detection

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

Cellular Automata, Learning Automata, and Cellular Learning Automata for Optimization

Contact Info

Product

Resources

About