Deep Reinforcement Learning and Its Neuroscientific Implications

Botvinick, Matthew; Wang, Jane X.; Dabney, Will; Miller, Kevin J; Kurth‐Nelson, Zeb

doi:10.1016/j.neuron.2020.06.014

Cited by 167 publications

(133 citation statements)

References 146 publications

(194 reference statements)

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…Learning and building artificial intelligence (AI) agents capable of interacting with their environment are major objectives in the fields of ML and AI. Deep artificial neural networks (73) have demonstrated great success over the recent years, particularly in the domains of image recognition, natural language processing, and deep reinforcement learning (74,75). Three challenges currently hamper further progress in the theoretical understanding of deep neural networks.…”

Section: Odor-background Segregation: a Joint Effect Of Temporal And mentioning

confidence: 99%

A spiking neural program for sensorimotor control during foraging in flying insects

Rapp

Nawrot

2020

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Foraging is a vital behavioral task for living organisms. Behavioral strategies and abstract mathematical models thereof have been described in detail for various species. To explore the link between underlying neural circuits and computational principles, we present how a biologically detailed neural circuit model of the insect mushroom body implements sensory processing, learning, and motor control. We focus on cast and surge strategies employed by flying insects when foraging within turbulent odor plumes. Using a spike-based plasticity rule, the model rapidly learns to associate individual olfactory sensory cues paired with food in a classical conditioning paradigm. We show that, without retraining, the system dynamically recalls memories to detect relevant cues in complex sensory scenes. Accumulation of this sensory evidence on short time scales generates cast-and-surge motor commands. Our generic systems approach predicts that population sparseness facilitates learning, while temporal sparseness is required for dynamic memory recall and precise behavioral control. Our work successfully combines biological computational principles with spike-based machine learning. It shows how knowledge transfer from static to arbitrary complex dynamic conditions can be achieved by foraging insects and may serve as inspiration for agent-based machine learning.

show abstract

Section: Odor-background Segregation: a Joint Effect Of Temporal And mentioning

confidence: 99%

A spiking neural program for sensorimotor control during foraging in flying insects

Rapp

Nawrot

2020

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…By mapping between language and the corresponding neural activity, recent studies have harnessed machine learning to decode linguistic information from the brain [24][25][26][27][28][29][30][31][32] . Taking a step forward, some theoretical and empirical work, especially in visual neuroscience 18,[33][34][35] , but recently also in language [25][26][27][28][29][30][31]36 , posits that deep neural networks may provide a new modeling framework to study neural computations in biological neural networks. To substantiate the call for this paradigm shift, we provide new behavioral and neural evidence for the connection between prediction-based DLMs and the human brain as they process natural speech.…”

Section: Introductionmentioning

confidence: 99%

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

Goldstein

Zada

Buchnik

et al. 2020

Preprint

View full text Add to dashboard Cite

Departing from rule-based linguistic models, advances in deep learning resulted in a new type of autoregressive deep language models (DLMs). These models are trained using a self-supervised next word prediction task. We provide empirical evidence for the connection between autoregressive DLMs and the human language faculty using spoken narrative and electrocorticographic recordings. Behaviorally, we demonstrate that humans have a remarkable capacity for word prediction in natural contexts, and that, given a sufficient context window, DLMs can attain human-level prediction performance. Leveraging on DLM embeddings we demonstrate that the brain constantly and spontaneously predicts the identity of the next word in natural speech, hundreds of milliseconds before they are perceived. Finally, we demonstrate that contextual embeddings derived from autoregressive DLMs capture neural representations of the unique, context-specific meaning of words in the narrative. Our findings suggest that DLMs provides a novel biologically feasible computational framework for studying the neural basis of language.

show abstract

“…When learning was not complete after the first success trial, furthermore, the data showed substantial though not complete preservation of the frontal explore state.While reward prediction error is an integral part of many learning processes, additional processes may be critical in rapid, model-based role learning. Our results are reminiscent of recent work on deep reinforcement learning, where meta-learning can allow rapid hypothesis-driven experimentation to replace slow parameter tuning(33).Evidently, a learned task model must govern the shift in frontal state that accompanies and may implement the switch from explore to exploit. An important open question is how this model is created.…”

mentioning

confidence: 64%

A one-shot learning signal in monkey prefrontal cortex

Achterberg

Kadohisa

Watanabe

et al. 2020

Preprint

View full text Add to dashboard Cite

Much animal learning is slow, with cumulative changes in behavior driven by reward prediction errors. When the abstract structure of a problem is known, however, both animals and formal learning models can rapidly attach new items to their roles within this structure, sometimes in a single trial. Frontal cortex is likely to play a key role in this process. To examine information seeking and use in a known problem structure, we trained monkeys in a novel explore/exploit task, requiring the animal first to test objects for their association with reward, then, once rewarded objects were found, to re-select them on further trials for further rewards. Many cells in the frontal cortex showed an explore/exploit preference, changing activity in a signal trial to align with one-shot learning in the monkeys’ behaviour. In contrast to this binary switch, these cells showed little evidence of continuous changes linked to expectancy or prediction error. Explore/exploit preferences were independent for two stages of the trial, object selection and receipt of feedback. Within an established task structure, frontal activity may control the separate operations of explore and exploit, switching in one trial between the two.Significance statementMuch animal learning is slow, with cumulative changes in behavior driven by reward prediction errors. When the abstract structure a problem is known, however, both animals and formal learning models can rapidly attach new items to their roles within this structure. To address transitions in neural activity during one-shot learning, we trained monkeys in an explore/exploit task using familiar objects and a highly familiar task structure. In contrast to continuous changes reflecting expectancy or prediction error, frontal neurons showed a binary, one-shot switch between explore and exploit. Within an established task structure, frontal activity may control the separate operations of exploring alternative objects to establish their current role, then exploiting this knowledge for further reward.

show abstract

Deep Reinforcement Learning and Its Neuroscientific Implications

Cited by 167 publications

References 146 publications

A spiking neural program for sensorimotor control during foraging in flying insects

A spiking neural program for sensorimotor control during foraging in flying insects

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

A one-shot learning signal in monkey prefrontal cortex

Contact Info

Product

Resources

About