A neural network account of memory replay and knowledge consolidation

Abstract

Replay can consolidate memories by offline neural reactivation related to past experiences. Category knowledge is learned across multiple experiences and subsequently generalised to new situations. This ability to generalise is promoted by offline consolidation and replay during rest and sleep. However, aspects of replay are difficult to determine from neuroimaging studies alone. Here, we provide a comprehensive account of how category replay may work in the brain by simulating these processes in a neural network which assumed the functional roles of the human ventral visual stream and hippocampus. We showed that generative replay, akin to imagining entirely new instances of a category, facilitated generalisation to new experiences. This invites a reconsideration of the nature of replay more generally, and suggests that replay helps to prepare us for the future as much as remember the past. We simulated generative replay at different network locations finding it was most effective in later layers equivalent to the lateral occipital cortex, and less effective in layers corresponding to early visual cortex, thus drawing a distinction between the observation of replay in the brain and its relevance to consolidation. We modelled long-term memory consolidation in humans and found that category replay is most beneficial for newly acquired knowledge, at a time when generalisation is still poor, a finding which suggests replay helps us adapt to changes in our environment. Finally, we present a novel mechanism for the frequent observation that the brain selectively consolidates weaker information, and showed that a reinforcement learning process in which categories were replayed according to their contribution to network performance explains this well-documented phenomenon, thus reconceptualising replay as an active rather than passive process.

System alignment supports cross-domain learning and zero-shot generalisation

“…The benefits of system alignment may increase with problem size, as well as with time to consolidate system mappings in offline replay such as during sleep (cf. Barry & Love, 2021 ).…”

Generative replay for compositional visual understanding in the prefrontal-hippocampal circuit

“…Based on the involvement of the hippocampal-prefrontal circuit in the present visual construction task, we asked whether generative replay provides a candidate mechanism underlying flexible construction. It has been hypothesised that replay reflects sampling from a generative model of the world to facilitate inference (Foster, 2017), enable generalisation (Barry & Love, 2021), and train a recognition model (Dayan et al, 1995), thus providing a key mechanism for active hypothesis testing about possible futures an animal might encounter. Here, we tested whether replay is also involved in hypothesis testing about possible presents, i.e., correct ways of de-constructing a silhouette.…”

System alignment supports cross-domain learning and zero-shot generalisation

“…The benefits of system alignment may increase with problem size, as well as with time to consolidate system mappings in offline replay such as during sleep (cf. Barry and Love, 2021).…”