Behavioral Hierarchy: Exploration and Representation

Barto, Andrew G.; Konidaris, George; Vigorito, Christopher M.

doi:10.1007/978-3-642-39875-9_2

Cited by 15 publications

(11 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…sink and stove both off), or because they are truly irrelevant to subsequent activities. In saltatory MB-HRL, these points can allow the planning process to attend only to a small core set of environmental features, further lightening the computational load (see [ 11 , 22 ]). A special but important case of such abstraction involves continuous state spaces, where saltatory MB-HRL can permit discretization (see [ 23 ]).…”

Section: Model-based Hierarchical Reinforcement Learning: Computationmentioning

confidence: 99%

“…In this article, we consider the relationship between model-based RL and another form of RL that has also recently become a topic of discussion in cognitive science and neuroscience, namely hierarchical reinforcement learning (HRL) [9][10][11]. The basic idea in HRL is to augment the set of actions available to the agent to include a set of temporally extended multi-action subroutines or skills.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Model-based hierarchical reinforcement learning and human action control

Botvinick

Weinstein

2014

Phil. Trans. R. Soc. B

141

150

View full text Add to dashboard Cite

Recent work has reawakened interest in goal-directed or ‘model-based’ choice, where decisions are based on prospective evaluation of potential action outcomes. Concurrently, there has been growing attention to the role of hierarchy in decision-making and action control. We focus here on the intersection between these two areas of interest, considering the topic of hierarchical model-based control. To characterize this form of action control, we draw on the computational framework of hierarchical reinforcement learning, using this to interpret recent empirical findings. The resulting picture reveals how hierarchical model-based mechanisms might play a special and pivotal role in human decision-making, dramatically extending the scope and complexity of human behaviour.

show abstract

Section: Model-based Hierarchical Reinforcement Learning: Computationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Model-based hierarchical reinforcement learning and human action control

Botvinick

Weinstein

2014

Phil. Trans. R. Soc. B

141

150

View full text Add to dashboard Cite

show abstract

“…As opposed to classical methods enabling robots to learn tool-use, as (Brown and Sammut, 2012) or (Schillaci et al, 2012), which consider tools as objects with affordances to learn using a symbolic representation, (Forestier and Oudeyer, 2016) does not necessitate this formalism and learns tool-use using simply parameterized skills, leveraging on a pre-defined task hierarchy. Barto et al (2013) showed that building complex actions made of lower-level actions according to the task hierarchy can bootstrap exploration by reaching interesting outcomes more rapidly. Temporal abstraction has also proven to enhance the learning efficiency of a deep reinforcement learner in Kulkarni et al (2016).…”

Section: Introductionmentioning

confidence: 99%

Learning a Set of Interrelated Tasks by Using a Succession of Motor Policies for a Socially Guided Intrinsically Motivated Learner

2019

View full text Add to dashboard Cite

We aim at a robot capable to learn sequences of actions to achieve a field of complex tasks. In this paper, we are considering the learning of a set of interrelated complex tasks hierarchically organized. To learn this high-dimensional mapping between a continuous high-dimensional space of tasks and an infinite dimensional space of unbounded sequences of actions, we introduce a new framework called “procedures”, which enables the autonomous discovery of how to combine previously learned skills in order to learn increasingly complex combinations of motor policies. We propose an active learning algorithmic architecture, capable of organizing its learning process in order to achieve a field of complex tasks by learning sequences of primitive motor policies. Based on heuristics of active imitation learning, goal-babbling and strategic learning using intrinsic motivation, our algorithmic architecture leverages our procedures framework to actively decide during its learning process which outcome to focus on and which exploration strategy to apply. We show on a simulated environment that our new architecture is capable of tackling the learning of complex motor policies by adapting the complexity of its policies to the task at hand. We also show that our “procedures” enable the learning agent to discover the task hierarchy and exploit his experience of previously learned skills to learn new complex tasks.

show abstract

“…One area of future research is to augment no-cost rules with techniques used to increase the effectiveness of with-cost rules in very large state spaces. These techniques include the development of state abstractions and behavioral hierarchies (Sutton et al, 1999 ; Dietterich, 2000 ; Barto and Mahadevan, 2003 ; Ravindran and Barto, 2003 ; Mahadevan, 2010 ; Osentoski and Mahadevan, 2010 ; Barto et al, 2013a ) which should be applicable, in principle, to the no-cost rules we use here. We expect any limitations from scaling of our no-cost rules to be similar to those of with-cost RL rules.…”

Section: Discussionmentioning

confidence: 99%

Finding minimal action sequences with a simple evaluation of actions

Shah

Gurney

2014

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Animals are able to discover the minimal number of actions that achieves an outcome (the minimal action sequence). In most accounts of this, actions are associated with a measure of behavior that is higher for actions that lead to the outcome with a shorter action sequence, and learning mechanisms find the actions associated with the highest measure. In this sense, previous accounts focus on more than the simple binary signal of “was the outcome achieved?”; they focus on “how well was the outcome achieved?” However, such mechanisms may not govern all types of behavioral development. In particular, in the process of action discovery (Redgrave and Gurney, 2006), actions are reinforced if they simply lead to a salient outcome because biological reinforcement signals occur too quickly to evaluate the consequences of an action beyond an indication of the outcome's occurrence. Thus, action discovery mechanisms focus on the simple evaluation of “was the outcome achieved?” and not “how well was the outcome achieved?” Notwithstanding this impoverishment of information, can the process of action discovery find the minimal action sequence? We address this question by implementing computational mechanisms, referred to in this paper as no-cost learning rules, in which each action that leads to the outcome is associated with the same measure of behavior. No-cost rules focus on “was the outcome achieved?” and are consistent with action discovery. No-cost rules discover the minimal action sequence in simulated tasks and execute it for a substantial amount of time. Extensive training, however, results in extraneous actions, suggesting that a separate process (which has been proposed in action discovery) must attenuate learning if no-cost rules participate in behavioral development. We describe how no-cost rules develop behavior, what happens when attenuation is disrupted, and relate the new mechanisms to wider computational and biological context.

show abstract

Behavioral Hierarchy: Exploration and Representation

Cited by 15 publications

References 57 publications

Model-based hierarchical reinforcement learning and human action control

Model-based hierarchical reinforcement learning and human action control

Learning a Set of Interrelated Tasks by Using a Succession of Motor Policies for a Socially Guided Intrinsically Motivated Learner

Finding minimal action sequences with a simple evaluation of actions

Contact Info

Product

Resources

About