Divide and Conquer: Hierarchical Reinforcement Learning and Task Decomposition in Humans

Diuk, Carlos; Schapiro, Anna C.; Córdova, Natalia; Ribas-Fernandes, José J F; Niv, Yaron; Botvinick, Matthew

doi:10.1007/978-3-642-39875-9_12

Cited by 25 publications

(18 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent behavioral and fMRI work suggest that human planners do indeed identify bottlenecks in task space, using these as subgoals for planning. In a study by Solway et al (in press) (see Diuk, Schapiro, Cordova, & Botvinick, 2013), participants were presented with a set of "landmarks" (e.g., post office, school) and learned their adjacency relations within a fictive town. These adjacencies were based on a graph with strong community structure, with one bottleneck location linking two clusters of nodes with dense internal connections.…”

Section: Hierarchical Structurementioning

confidence: 99%

The Computational and Neural Basis of Cognitive Control: Charted Territory and New Frontiers

2014

View full text Add to dashboard Cite

Cognitive control has long been one of the most active areas of computational modeling work in cognitive science. The focus on computational models as a medium for specifying and developing theory predates the PDP books, and cognitive control was not one of the areas on which they focused. However, the framework they provided has injected work on cognitive control with new energy and new ideas. On the occasion of the books' anniversary, we review computational modeling in the study of cognitive control, with a focus on the influence that the PDP approach has brought to bear in this area. Rather than providing a comprehensive review, we offer a framework for thinking about past and future modeling efforts in this domain. We define control in terms of the optimal parameterization of task processing. From this vantage point, the development of control systems in the brain can be seen as responding to the structure of naturalistic tasks, through the filter of the brain systems with which control directly interfaces. This perspective lays open a set of fascinating but difficult research questions, which together define an important frontier for future computational research.

show abstract

Section: Hierarchical Structurementioning

confidence: 99%

The Computational and Neural Basis of Cognitive Control: Charted Territory and New Frontiers

2014

View full text Add to dashboard Cite

show abstract

“…It would be interesting and important to explore how this model could incorporate more autonomous methods for constructing the hierarchical structure. This is one of the major open problems in HRL, and is an active area of research, especially in regards to how this could occur in a neural system [ 2 , 25 , 40 , 53 , 68 ].…”

Section: Discussionmentioning

confidence: 99%

A neural model of hierarchical reinforcement learning

Rasmussen¹,

Voelker

Eliasmith

2017

PLoS ONE

View full text Add to dashboard Cite

We develop a novel, biologically detailed neural model of reinforcement learning (RL) processes in the brain. This model incorporates a broad range of biological features that pose challenges to neural RL, such as temporally extended action sequences, continuous environments involving unknown time delays, and noisy/imprecise computations. Most significantly, we expand the model into the realm of hierarchical reinforcement learning (HRL), which divides the RL process into a hierarchy of actions at different levels of abstraction. Here we implement all the major components of HRL in a neural model that captures a variety of known anatomical and physiological properties of the brain. We demonstrate the performance of the model in a range of different environments, in order to emphasize the aim of understanding the brain's general reinforcement learning ability. These results show that the model compares well to previous modelling work and demonstrates improved performance as a result of its hierarchical ability. We also show that the model's behaviour is consistent with available data on human hierarchical RL, and generate several novel predictions.

show abstract

“…There are several types of sequential tasks studied in the literature (Ruh et al, 2010 ; Diuk et al, 2013 ; Friedel et al, 2014 ; Huys et al, 2015 ). In this context, decision trees present an interesting scenario, because they naturally embody the most basic structures of a sequential decision-making situation (Fu and Anderson, 2006 ) while allowing for a clear description of the task structure.…”

Section: Discussionmentioning

confidence: 99%

Modeling Search Behaviors during the Acquisition of Expertise in a Sequential Decision-Making Task

Moënne‐Loccoz

Vergara

López

et al. 2017

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Our daily interaction with the world is plagued of situations in which we develop expertise through self-motivated repetition of the same task. In many of these interactions, and especially when dealing with computer and machine interfaces, we must deal with sequences of decisions and actions. For instance, when drawing cash from an ATM machine, choices are presented in a step-by-step fashion and a specific sequence of choices must be performed in order to produce the expected outcome. But, as we become experts in the use of such interfaces, is it possible to identify specific search and learning strategies? And if so, can we use this information to predict future actions? In addition to better understanding the cognitive processes underlying sequential decision making, this could allow building adaptive interfaces that can facilitate interaction at different moments of the learning curve. Here we tackle the question of modeling sequential decision-making behavior in a simple human-computer interface that instantiates a 4-level binary decision tree (BDT) task. We record behavioral data from voluntary participants while they attempt to solve the task. Using a Hidden Markov Model-based approach that capitalizes on the hierarchical structure of behavior, we then model their performance during the interaction. Our results show that partitioning the problem space into a small set of hierarchically related stereotyped strategies can potentially capture a host of individual decision making policies. This allows us to follow how participants learn and develop expertise in the use of the interface. Moreover, using a Mixture of Experts based on these stereotyped strategies, the model is able to predict the behavior of participants that master the task.

show abstract

Divide and Conquer: Hierarchical Reinforcement Learning and Task Decomposition in Humans

Cited by 25 publications

References 33 publications

The Computational and Neural Basis of Cognitive Control: Charted Territory and New Frontiers

The Computational and Neural Basis of Cognitive Control: Charted Territory and New Frontiers

A neural model of hierarchical reinforcement learning

Modeling Search Behaviors during the Acquisition of Expertise in a Sequential Decision-Making Task

Contact Info

Product

Resources

About