Stable maintenance of multiple representational formats in human visual short-term memory

Abstract

Visual short-term memory (VSTM) enables humans to form a stable and coherent representation of the external world. However, the nature and temporal dynamics of the neural representations in VSTM that support this stability are barely understood. Here we combined human intracranial electroencephalography (iEEG) recordings with analyses using deep neural networks and semantic models to probe the representational format and temporal dynamics of information in VSTM. We found clear evidence that VSTM maintenance occurred in two distinct representational formats which originated from different encoding periods. The first format derived from an early encoding period (250 to 770 ms) corresponded to higher-order visual representations. The second format originated from a late encoding period (1,000 to 1,980 ms) and contained abstract semantic representations. These representational formats were overall stable during maintenance, with no consistent transformation across time. Nevertheless, maintenance of both representational formats showed substantial arrhythmic fluctuations, i.e., waxing and waning in irregular intervals. The increases of the maintained representational formats were specific to the phases of hippocampal low-frequency activity. Our results demonstrate that human VSTM simultaneously maintains representations at different levels of processing, from higher-order visual information to abstract semantic representations, which are stably maintained via coupling to hippocampal low-frequency activity.

Post-encoding reactivation promotes one-shot learning of episodes in humans

“…Should post-encoding memory reinstatement be associated with the sustained increase neural activity during the delay we would expect it to be observed over extended portions of the offset delay period and not only at the beginning of it. More recently, it has been argued that such above-threshold delay-period activity may support functions other than information storage per se (D’Esposito and Postle, 2015) and the existence of other neural coding mechanisms such as “activity-silent” states (Stokes, 2015) and dynamic coding schemes (Liu et al, 2020). However, these neural representational formats are still susceptible to be identified with the implementation of multivariate decoding approaches, such as the one implemented in the current design, thereby rendering unlikely they were unobservable throughout the delay period in our study.…”

Pre-existing long-term memory facilitates the formation of visual short-term memory

“…Rich behavioral data suggest that semantic information about an object or scene and its associations can be rapidly accessed as early as 100-300 ms following stimulus presentation (see Potter, 2012 for a summary). A recent study based on direct recording from the human brain with intracranial electroencephalography provides some evidence to support these behavioral observations (Liu et al, 2020). Liu et al (2020) examined the correlations between neural signals recorded during VSTM formation with visual features of everyday objects derived from a deep neural network (DNN) and with semantic features of these objects.…”

“…A recent study based on direct recording from the human brain with intracranial electroencephalography provides some evidence to support these behavioral observations (Liu et al, 2020). Liu et al (2020) examined the correlations between neural signals recorded during VSTM formation with visual features of everyday objects derived from a deep neural network (DNN) and with semantic features of these objects. This approach reveals the amount of visual and semantic contributions during VSTM formation.…”

Pre-existing long-term memory facilitates the formation of visual short-term memory

“…Rich behavioral observations suggest that semantic information about an object or scene and its associations can be rapidly accessed as early as 100-300 ms following stimulus presentation (see Potter, 2012 for a summary). A recent study based on direct recording from the human brain with intracranial electroencephalography provides novel neural evidence to support these behavioral observations (Liu et al, 2020). Liu et al (2020) examined the correlations between neural signals recorded during VSTM formation with visual features of everyday objects derived from a deep neural network (DNN) and with semantic features of these objects.…”

“…A recent study based on direct recording from the human brain with intracranial electroencephalography provides novel neural evidence to support these behavioral observations (Liu et al, 2020). Liu et al (2020) examined the correlations between neural signals recorded during VSTM formation with visual features of everyday objects derived from a deep neural network (DNN) and with semantic features of these objects. This approach may therefore reveal the amount of visual and semantic contributions during VSTM formation.…”