Modeling awake hippocampal reactivations with model-based bidirectional search

Abstract

Hippocampal offline reactivations during reward-based learning, usually categorized as replay events, have been found to be important for performance improvement over time and for memory consolidation. Recent computational work has linked these phenomena to the need to transform reward information into stateaction values for decision-making and to propagate it to all relevant states of the environment. Nevertheless, it is still unclear whether an integrated reinforcement learning mechanism could account for the variety of awake hippocampal reactivations, including variety in order (forward and reverse reactivated trajectories) and variety in the location where they occur (reward site or decision-point). Here we present a model-based bidirectional search model which accounts for a variety of hippocampal reactivations. The model combines forward trajectory sampling from current position and backward sampling through prioritized sweeping from states associated with large reward prediction errors until the two trajectories connect. This is repeated until stabilization of state-action values (convergence), which could explain why hippocampal reactivations drastically diminish when the animal's performance stabilizes. Simulations in a multiple T-maze task show that forward reactivations are prominently found at decision-points while backward reactivations are exclusively generated at reward sites. Finally, the model can generate imaginary trajectories that are not allowed to the agent during task performance. We raise some experimental predictions and implications for future studies of the role of the hippocampo-prefronto-striatal network in learning.

Replay in minds and machines

“…In a task setting with a stable goal, the replay buffer of a model-free learner will increasingly accumulate rewarded episodes while a model-based learner draws on a learned model to sample episodes in a more balanced fashion. The learning-related changes discussed here might therefore reflect a shift from a model-free to a model-based process with learning -although further data will be needed, and model-based and model-free replay might be difficult to disentangle experimentally (Khamassi and Girard, 2020). The third line of support comes from computational work that shows how backward replay can strengthen forward synaptic pathways through spike timing dependent plasticity (STDP) (Haga and Fukai, 2018) and thus support forward replay during sleep and active behavior (Johnson and Redish, 2007;Pfeiffer and Foster, 2013;Redish, 2015b, 2013).…”

“…One interpretation of these findings is that the hippocampus uses replay to evaluate all potential trajectories and the behaviorally relevant trajectory is instantiated in a different brain region. Furthermore, backward replay, which backpropagates value information from the goal location, and forward replay, which samples possible trajectories ahead of the animal, might connect their trajectories as proposed by models of bidirectional planning (Khamassi and Girard, 2020). Forward replay events have been shown to end at or close to the goal location (Pfeiffer and Foster, 2013) and might efficiently stop in states where value estimates have already been updated by a backward replay mechanism, as could be instantiated by prioritized sweeping (see Khamassi and Girard, 2020).…”

Reward prediction errors drive declarative learning irrespective of agency

“…However, to efficiently learn from replay, the brain has to decide which memories to replay. Hippocampal replay is sometimes considered to be modulated by unsigned RPEs (URPEs) where the absolute value of an RPE is computed (e.g., Khamassi & Girard, 2020;Momennejad et al, 2018;Roscow et al, 2019). However, some computational models argue for the importance of SRPE in hippocampal replay instead.…”

From spatial navigation via visual construction to episodic memory and imagination

“…with model-based bidirectional planning Khamassi and Girard (2020) address the same type of recall as B3, but offering new insights, since B3 do not address planning or the issue of reward: Forward reactivations are prominently found at decision-points while backward reactivations are exclusively generated at reward sites. Additionally, the model can generate imaginary trajectories that are not allowed to the agent during task performance.…”