Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

Cross, Logan; Cockburn, Jeff; Yue, Yisong; O’Doherty, John P.

doi:10.1016/j.neuron.2020.11.021

Cited by 42 publications

(55 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The segmentation results of the MSCC-MDF established in this research were compared with those of CNN [ 21 ], fully CNN (FCN) [ 22 ], SegNet [ 23 ], and deep Q-network (DQN) [ 24 ] ( Figure 9 ). The Dice coefficient of the MSCC-MDF model was lower than other algorithms, and the difference was considerable ( P < 0.05).…”

Section: Resultsmentioning

confidence: 99%

Convolutional Neural Network Intelligent Segmentation Algorithm-Based Magnetic Resonance Imaging in Diagnosis of Nasopharyngeal Carcinoma Foci

Wang

Gong

Zhang

et al. 2021

Contrast Media & Molecular Imaging

View full text Add to dashboard Cite

The aim of this study was to explore the adoption value of convolutional neural network- (CNN-) based magnetic resonance imaging (MRI) image intelligent segmentation model in the identification of nasopharyngeal carcinoma (NPC) lesions. The multisequence cross convolutional (MSCC) method was used in the complex convolutional network algorithm to establish the intelligent segmentation model two-dimensional (2D) ResUNet for the MRI image of the NPC lesion. Moreover, a multisequence multidimensional fusion segmentation model (MSCC-MDF) was further established. With 45 patients with NPC as the research objects, the Dice coefficient, Hausdorff distance (HD), and percentage of area difference (PAD) were calculated to evaluate the segmentation effect of MRI lesions. The results showed that the 2D-ResUNet model processed by MSCC had the largest Dice coefficient of 0.792 ± 0.045 for segmenting the tumor lesions of NPC, and it also had the smallest HD and PAD, which were 5.94 ± 0.41 mm and 15.96 ± 1.232%, respectively. When batch size = 5, the convergence curve was relatively gentle, and the convergence speed was the best. The largest Dice coefficient of MSCC-MDF model segmenting NPC tumor lesions was 0.896 ± 0.09, and its HD and PAD were the smallest, which were 5.07 ± 0.54 mm and 14.41 ± 1.33%, respectively. Its Dice coefficient was lower than other algorithms ( P < 0.05 ), but HD and PAD were significantly higher than other algorithms ( P < 0.05 ). To sum up, the MSCC-MDF model significantly improved the segmentation performance of MRI lesions in NPC patients, which provided a reference for the diagnosis of NPC.

show abstract

Section: Resultsmentioning

confidence: 99%

Convolutional Neural Network Intelligent Segmentation Algorithm-Based Magnetic Resonance Imaging in Diagnosis of Nasopharyngeal Carcinoma Foci

Wang

Gong

Zhang

et al. 2021

Contrast Media & Molecular Imaging

View full text Add to dashboard Cite

show abstract

“…Reinforcement learning is a particular area of machine learning concerned with how intelligent agents should select actions in order to maximize a prescribed reward through trial-and-error interactions within a dynamic environment 42 . DRL is the combination of reinforcement learning and deep learning to make these trial-and-error decisions 43 . At the mesoscopic scale detailed below, model agents seek to have a desired phenotype at each simulation time step.…”

Section: Methodsmentioning

confidence: 99%

Multiscale computational modeling of cancer growth using features derived from microCT images

Zangooei

Margolis

Hoyt

2021

Sci Rep

View full text Add to dashboard Cite

Advances in medical imaging technologies now allow noninvasive image acquisition from individual patients at high spatiotemporal resolutions. A relatively new effort of predictive oncology is to develop a paradigm for forecasting the future status of an individual tumor given initial conditions and an appropriate mathematical model. The objective of this study was to introduce a comprehensive multiscale computational method to predict cancer and microvascular network growth patterns. A rectangular lattice-based model was designed so different evolutionary scenarios could be simulated and for predicting the impact of diffusible factors on tumor morphology and size. Further, the model allows prediction-based simulation of cell and microvascular behavior. Like a single cell, each agent is fully realized within the model and interactions are governed in part by machine learning methods. This multiscale computational model was developed and incorporated input information from in vivo microscale computed tomography (microCT) images acquired from breast cancer-bearing mice. It was found that as the difference between expansion of the cancer cell population and microvascular network increases, cells undergo proliferation and migration with a greater probability compared to other phenotypes. Overall, multiscale computational model agreed with both theoretical expectations and experimental findings (microCT images) not used during model training.

show abstract

“…An emerging body of work has used DRL to train ANNs that solve tasks closely inspired by tasks from neuroscience. For instance, agents have been trained to study learning and dynamics in the motor-cortex [Song et al, 2020;Weinstein and Botvinick, 2017], time encoding in the hippocampus [Lin and Richards, 2021], rewardbased learning and meta-learning in the pre-frontal cortex [Botvinick et al, 2019;Song et al, 2017;Wang et al, 2018], and task-associated representations across multiple brain areas [Cross et al, 2021]. There have been several recent perspectives articulating the relevance of this emerging algorithmic toolkit to neuroscience [Botvinick et al, 2020;Gershman and Ölveczky, 2020] and ethology [Crosby, 2020].…”

Section: Related Workmentioning

confidence: 99%

Emergent behavior and neural dynamics in artificial agents tracking turbulent plumes

Singh¹,

Breugel²,

N.³

et al. 2021

Preprint

View full text Add to dashboard Cite

Tracking a turbulent plume to locate its source is a complex control problem because it requires multi-sensory integration and must be robust to intermittent odors, changing wind direction, and variable plume statistics. This task is routinely performed by flying insects, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behavior have been studied in detail in many experimental studies. Here, we take a complementary in silico approach, using artificial agents trained with reinforcement learning to develop an integrated understanding of the behaviors and neural computations that support plume tracking. Specifically, we use deep reinforcement learning (DRL) to train recurrent neural network (RNN) agents to locate the source of simulated turbulent plumes. Interestingly, the agents' emergent behaviors resemble those of flying insects, and the RNNs learn to represent task-relevant variables, such as head direction and time since last odor encounter. Our analyses suggest an intriguing experimentally testable hypothesis for tracking plumes in changing wind direction-that agents follow local plume shape rather than the current wind direction. While reflexive short-memory behaviors are sufficient for tracking plumes in constant wind, longer timescales of memory are essential for tracking plumes that switch direction. At the level of neural dynamics, the RNNs' population activity is low-dimensional and organized into distinct dynamical structures, with some correspondence to behavioral modules. Our in silico approach provides key intuitions for turbulent plume tracking strategies and motivates future targeted experimental and theoretical developments.

show abstract

Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

Cited by 42 publications

References 59 publications

Convolutional Neural Network Intelligent Segmentation Algorithm-Based Magnetic Resonance Imaging in Diagnosis of Nasopharyngeal Carcinoma Foci

Convolutional Neural Network Intelligent Segmentation Algorithm-Based Magnetic Resonance Imaging in Diagnosis of Nasopharyngeal Carcinoma Foci

Multiscale computational modeling of cancer growth using features derived from microCT images

Emergent behavior and neural dynamics in artificial agents tracking turbulent plumes

Contact Info

Product

Resources

About