CoBeL-RL: A neuroscience-oriented simulation framework for complex behavior and learning

Abstract

Reinforcement learning (RL) has become a popular paradigm for modeling animal behavior, analyzing neuronal representations, and studying their emergence during learning. This development has been fueled by advances in understanding the role of RL in both the brain and artificial intelligence. However, while in machine learning a set of tools and standardized benchmarks facilitate the development of new methods and their comparison to existing ones, in neuroscience, the software infrastructure is much more fragmented. Even if sharing theoretical principles, computational studies rarely share software frameworks, thereby impeding the integration or comparison of different results. Machine learning tools are also difficult to port to computational neuroscience since the experimental requirements are usually not well aligned. To address these challenges we introduce CoBeL-RL, a closed-loop simulator of complex behavior and learning based on RL and deep neural networks. It provides a neuroscience-oriented framework for efficiently setting up and running simulations. CoBeL-RL offers a set of virtual environments, e.g. T-maze and Morris water maze, which can be simulated at different levels of abstraction, e.g. a simple gridworld or a 3D environment with complex visual stimuli, and set up using intuitive GUI tools. A range of RL algorithms, e.g. Dyna-Q and deep Q-network algorithms, is provided and can be easily extended. CoBeL-RL provides tools for monitoring and analyzing behavior and unit activity, and allows for fine-grained control of the simulation via interfaces to relevant points in its closed-loop. In summary, CoBeL-RL fills an important gap in the software toolbox of computational neuroscience.

A model of hippocampal replay driven by experience and environmental structure facilitates spatial learning

“…We implemented our model as well as all of our simulations and analysis code in Python 3 using the CoBeL-RL framework ( Walther et al, 2021 ; Diekmann et al, 2022 ) ( https://doi.org/10.5281/zenodo.5741291 ). The code for all simulations has been made available on Github: https://github.com/sencheng/-Mechanisms-and-Functions-of-Hippocampal-Replay (copy archived at Diekmann, 2023 ).…”

A Model of Hippocampal Replay Driven by Experience and Environmental Structure Facilitates Spatial Learning

“…We implemented our model as well as all of our simulations and analysis code in Python 3 using the CoBeL-RL framework (Walther et al, 2021; Diekmann et al, 2022) (https://doi.org/10.5281/zenodo.5741291). The code for all simulations has been made available on Github: https://github.com/sencheng/-Mechanisms-and-Functions-of-Hippocampal-Replay.…”