Reinforcement learning, efficient coding, and the statistics of natural tasks

Botvinick, Matthew; Weinstein, Ari; Solway, Alec; Barto, Andrew G.

doi:10.1016/j.cobeha.2015.08.009

Cited by 51 publications

(39 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We predict that a number of other high-and mid-level object and scene properties are linked to these statistics, which would be consistent with the hypothesis that cortical tuning to statistical regularities produces representations that are informative to a broad range of stimulus properties. Similar statistical principles may underlie the cortical representations that support other high-level cognitive functions, such as object categorization 57 , face recognition 58 , and reinforcement learning 59 .…”

Section: Natural Statistics and Cortical Representationmentioning

confidence: 90%

Object representations in the human brain reflect the co-occurrence statistics of vision and language

Bonner

Epstein

2020

Preprint

View full text Add to dashboard Cite

A central regularity of visual perception is the co-occurrence of objects in the natural environment. Here we use machine learning and fMRI to test the hypothesis that object cooccurrence statistics are encoded in the human visual system and elicited by the perception of individual objects. We identified low-dimensional representations that capture the latent statistical structure of object co-occurrence in real-world scenes, and we mapped these statistical representations onto voxelwise fMRI responses during object viewing. We found that cortical responses to single objects were predicted by the statistical ensembles in which they typically occur, and that this link between objects and their visual contexts was made most strongly in the anterior portion of the scene-selective parahippocampal place area. In contrast, a languagebased statistical model of the co-occurrence of object names in written text predicted responses in neighboring regions of object-selective visual cortex. Together, these findings show that the sensory coding of objects in the human brain reflects the latent statistics of object context in visual and linguistic experience.

show abstract

Section: Natural Statistics and Cortical Representationmentioning

confidence: 90%

Object representations in the human brain reflect the co-occurrence statistics of vision and language

Bonner

Epstein

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Our results add to a growing literature on structure learning , extending its principles to the domain of cognitive control policies (Braun, Mehring, & Wolpert, 2010; Collins & Frank, 2013; Gershman, Blei, & Niv, 2010; Huys et al, 2015; Rougier, Noelle, Braver, Cohen, & O’Reilly, 2005). Structure learning refers to our ability to identify and leverage invariant structure in the space of natural tasks to improve learning of novel tasks (Botvinick et al, 2009, 2015; Gershman & Niv, 2010; Tenenbaum, Kemp, Griffiths, & Goodman, 2011). …”

Section: Discussionmentioning

confidence: 99%

“…Just like different real-world tasks often share stimulus-response-outcome contingencies, they also share other forms dynamic structure (Botvinick, Weinstein, Solway, & Barto, 2015; Schank & Abelson, 1977). Such shared structure affords an opportunity for generalization of internal control policies.…”

Section: Introductionmentioning

confidence: 99%

Learning and transfer of working memory gating policies

Bhandari

Badre

2018

Cognition

View full text Add to dashboard Cite

knowledge about the tasks we encounter enables us to rapidly and flexibly adapt to novel task contexts. Previous research has focused primarily on abstract rules that leverage shared structure in stimulus-response (S-R) mappings as the basis of such task knowledge. Here we provide evidence that working memory (WM) gating policies – a type of control policy required for internal control of WM during a task – constitute a form of abstract task knowledge that can be transferred across contexts. In two experiments, we report specific evidence for the transfer of selective WM gating policies across changes of task context. We show that this transfer is not tied to shared structure in S-R mappings, but instead in the dynamic structure of the task. Collectively, our results highlight the importance of WM gating policies in particular, and control policies in general, as a key component of the task knowledge that supports flexible behavior and task generalization.

show abstract

“…It is well known in fields like AI and RL that finding appropriate task decomposition is key to solving complex problems [4][5][6]. The contribution of this paper to the literature is identifying a mapping between different control problems and corresponding information measures, all of which can be used within a unitary method to derive task-relevant decompositions or clusters.…”

Section: Discussionmentioning

confidence: 99%

“…In artificial intelligence (AI), reinforcement learning (RL) and related areas, it has been long known that learning to code the problem space efficiently is paramount to finding efficient solutions [4][5][6]. While in several practical AI and RL applications, the so-called state-space is provided by programmers by hand, for example in the form of a grid of discrete cells that cover the whole space, it is not guaranteed that this is the most efficient representation to solve the problem at hand.…”

Section: Introductionmentioning

confidence: 99%

Nonparametric Problem-Space Clustering: Learning Efficient Codes for Cognitive Control Tasks

Maisto

Rigoli

Pezzulo

2016

Entropy

View full text Add to dashboard Cite

Abstract:We present an information-theoretic method permitting one to find structure in a problem space (here, in a spatial navigation domain) and cluster it in ways that are convenient to solve different classes of control problems, which include planning a path to a goal from a known or an unknown location, achieving multiple goals and exploring a novel environment. Our generative nonparametric approach, called the generative embedded Chinese restaurant process (geCRP), extends the family of Chinese restaurant process (CRP) models by introducing a parameterizable notion of distance (or kernel) between the states to be clustered together. By using different kernels, such as the the conditional probability or joint probability of two states, the same geCRP method clusters the environment in ways that are more sensitive to different control-related information, such as goal, sub-goal and path information. We perform a series of simulations in three scenarios-an open space, a grid world with four rooms and a maze having the same structure as the Hanoi Tower-in order to illustrate the characteristics of the different clusters (obtained using different kernels) and their relative benefits for solving planning and control problems.

show abstract

Reinforcement learning, efficient coding, and the statistics of natural tasks

Cited by 51 publications

References 54 publications

Object representations in the human brain reflect the co-occurrence statistics of vision and language

Object representations in the human brain reflect the co-occurrence statistics of vision and language

Learning and transfer of working memory gating policies

Nonparametric Problem-Space Clustering: Learning Efficient Codes for Cognitive Control Tasks

Contact Info

Product

Resources

About