Optimism and pessimism in optimised replay

Abstract

The replay of task-relevant trajectories is known to contribute to memory consolidation and improved task performance. A wide variety of experimental data show that the content of replayed sequences is highly specific and can be modulated by reward as well as other prominent task variables. However, the rules governing the choice of sequences to be replayed still remain poorly understood. One recent theoretical suggestion is that the prioritization of replay experiences in decision-making problems is based on their effect on the choice of action. We exploit this to address recent experimental data showing in a particular task that human subjects tended to replay sub-optimal outcomes that they later chose to avoid. We show that pessimistic replay is of benefit to forgetful agents experiencing large amounts of uncertainty in their models of the world. Further, we fit our model parameters to the individual subjects' choices and confirm that their replay choices were appropriate according to the proposed scheme.

Generative replay for compositional visual understanding in the prefrontal-hippocampal circuit

“…These results are in line with previous reports suggesting a role for replay when planning (Kurth-Nelson et al, 2016) and learning (Schuck & Niv, 2019) trajectories through an abstract state space in a RL problem and evidence from recordings in animals suggesting that replay can explore novel trajectories (Gupta et al, 2010;Ólafsdóttir et al, 2015). Our findings are in line with notions that generative replay provides a mechanism for efficiently learning and sampling from a generative model of the world (Foster, 2017), in line with a crucial role of replay in planning and RL (Antonov et al, 2021;Mattar & Daw, 2018) but also structure learning (Evans & Burgess, 2020). In the present study, we provide evidence for replay as a mechanism for testing hypotheses underlying compositional understanding.…”

A Model of Hippocampal Replay Driven by Experience and Environmental Structure Facilitates Spatial Learning

“…Also, as we will show below the default mode generates shortcuts replays as those found by Gupta et al (2010), whereas the reverse mode does not. Second, the recent discovery of so-called pessimistic replay, i.e., replay of experiences which are not optimal from a reinforcement learning perspective, show that suboptimal replay sequences occur in human brains (Eldar et al, 2020) for a good reason (Antonov et al, 2022).…”

“…the replay of sub-optimal experiences, in humans (Eldar et al, 2020) supports this. Interestingly, follow-up modeling work (Antonov et al, 2022) suggests that optimal replay occurs predominantly during initial trials and then switches to pessimistic replay. However, this switch is dependent on the utility of experiences (Mattar and Daw, 2018).…”

“…To test the optimality of SFMA’s different replay modes we computed for each step in each replay whether the reactivated experience was optimal, that is, it had the highest utility as defined by Mattar and Daw (2018). For simplicity we assumed uniform need as Antonov et al (2022) did.…”

Dopamine Enhances Model-Free Credit Assignment Through Boosting of Retrospective Model-Based Inference

“…More recent work has proposed a richer integration between MF and MB values (e.g., Keramati et al, 2016) as well as following an earlier influential suggestion as to how an MB system might train an MF system (Mattar and Daw, 2018;Sutton, 1991). More recent evidence has raised the possibility that this may occur during rest periods (Antonov et al, 2021;Liu et al, 2021a).…”