Context-dependent extinction learning emerging from raw sensory inputs: a reinforcement learning approach

Abstract

The context-dependence of extinction learning has been well studied and requires the hippocampus. However, the underlying neural mechanisms are still poorly understood. Using memory-driven reinforcement learning and deep neural networks, we developed a model that learns to navigate autonomously in biologically realistic virtual reality environments based on raw camera inputs alone. Neither is context represented explicitly in our model, nor is context change signaled. We find that memory-intact agents learn distinct context representations, and develop ABA renewal, whereas memory-impaired agents do not. These findings reproduce the behavior of control and hippocampal animals, respectively. We therefore propose that the role of the hippocampus in the context-dependence of extinction learning might stem from its function in episodic-like memory and not in context-representation per se. We conclude that context-dependence can emerge from raw visual inputs.

Navigation task and action space drive the emergence of egocentric and allocentric spatial representations

“…All simulations were carried out in virtual environments using the CoBeL-RL (Closed-loop simulator of complex behavior and learning based on reinforcement learning and deep neural networks) modeling framework [ 31 ]. In the guidance task, the agent started from a random location and had to navigate to a fixed, unmarked goal location within a square environment of size 2.75m × 2.75m, unless otherwise specified.…”

CoBeL-RL: A neuroscience-oriented simulation framework for complex behavior and learning

“…In this paper, we introduced CoBeL-RL, a RL framework oriented toward computational neuroscience, which provides a large range of environments, established RL models and analysis tools, and can be used to simulate a variety of behavioral tasks. Already, a set of computational studies focusing on explaining animal behavior (Walther et al, 2021 ; Zeng et al, 2022 ) as well as neural activity (Diekmann and Cheng, 2022 ; Vijayabaskaran and Cheng, 2022 ) have employed predecessor versions of CoBeL-RL. The framework has been expanded and refined since these earlier studies.…”

“…Additionally, CoBeL-RL has been used to understand the emergence of other spatial representations, e.g., head direction modulated cells, and their dependence on navigational strategy employed (Vijayabaskaran and Cheng, 2022 ). The initial version of the framework was used to analyze representational changes resulting from the learning of context-specific behavior in an extinction learning paradigm (Walther et al, 2021 ). CoBeL-RL currently only provides a small repertoire for the analysis of network activity with a focus on spatial representations.…”

“…A common way of solving the RL problem consists of inferring the value function, a mapping from state-action pairs to expected future reward, and selecting those actions which yield the highest values. The brain is thought to support this value-based type of RL (Schultz et al, 1997 ), and RL has been used to explain both human (Redish et al, 2007 ; Zhang et al, 2018 ) and animal behavior (Bathellier et al, 2013 ; Walther et al, 2021 ) (see Botvinick et al, 2020 for a recent review).…”

“…Here, we introduce the “ Closed-loop simulator of Co mplex Be havior and L earning based on R einforcement L earning and deep neural networks ,” or CoBeL-RL for short. The framework is based on the software that had been developed for studying the role of the hippocampus in spatial learning (Walther et al, 2021 ; Vijayabaskaran and Cheng, 2022 ) and further extends and unifies its functionality. The CoBeL-RL framework offers a range of reinforcement learning algorithms, e.g., Q-learning or deep Q-network, and environments commonly used in behavioral studies, e.g., T-maze or Morris water maze.…”