Latency of modality-specific reactivation of auditory and visual information during episodic memory retrieval

Abstract

This study used magnetoencephalography (MEG) to examine the latency of modality-specific reactivation in the visual and auditory cortices during a recognition task to determine the effects of reactivation on episodic memory retrieval. Nine right-handed healthy young adults participated in the experiment. The experiment consisted of a word-encoding phase and two recognition phases. Three encoding conditions were included: encoding words alone (word-only) and encoding words presented with either related pictures (visual) or related sounds (auditory). The recognition task was conducted in the MEG scanner 15 min after the completion of the encoding phase. After the recognition test, a source-recognition task was given, in which participants were required to choose whether each recognition word was not presented or was presented with which information during the encoding phase. Word recognition in the auditory condition was higher than that in the word-only condition. Confidence-of-recognition scores (d') and the source-recognition test showed superior performance in both the visual and the auditory conditions compared with the word-only condition. An equivalent current dipoles analysis of MEG data indicated that higher equivalent current dipole amplitudes in the right fusiform gyrus occurred during the visual condition and in the superior temporal auditory cortices during the auditory condition, both 450-550 ms after onset of the recognition stimuli. Results suggest that reactivation of visual and auditory brain regions during recognition binds language with modality-specific information and that reactivation enhances confidence in one's recognition performance.

Cross-Modal Interaction Between Auditory and Visual Input Impacts Memory Retrieval

“…The finding that the concurrent presentation of meaningful auditory and visual cues enhances recognition of visual objects is consistent with prior work demonstrating facilitative effects of multisensory encoding ( Murray et al, 2004 , 2005 ; Lehmann and Murray, 2005 ; Thelen et al, 2012 , 2014 , 2015 ; Moran et al, 2013 ; Thelen and Murray, 2013 ; Heikkilä et al, 2015 ; Matusz et al, 2015 , 2017 ; Ueno et al, 2015 ). Previously, this type of memory enhancement has often been attributed to cross-modal interactions during the retrieval of visual information from memory, such that re-exposure to a previously-seen visual object initiates rapid re-activation of corresponding perceptual experiences.…”

“…Given that multisensory integration has been shown to enhance the salience and attentional processing of sensory stimuli ( Parkhurst et al, 2002 ; Santangelo and Spence, 2007 ; Matusz and Eimer, 2011 ; see Koelewijn et al, 2010 ; Talsma et al, 2010 for reviews), which can, in turn, strengthen memory encoding ( Salthouse et al, 1984 ; Evans and Baddeley, 2018 ), it would be reasonable to expect that auditory facilitation of visual attention and perception could translate to better visual memory as well. Indeed, audio-visual encoding can enhance recognition memory for visual objects ( Murray et al, 2004 , 2005 ; Lehmann and Murray, 2005 ; Thelen et al, 2012 , 2014 , 2015 ; Moran et al, 2013 ; Thelen and Murray, 2013 ; Heikkilä et al, 2015 ; Matusz et al, 2015 ; Ueno et al, 2015 ; see Matusz et al, 2017 for review). For instance, in a series of experiments utilizing a continuous recognition task (identifying “old” vs. “new” pictures), Matusz et al (2017) found that unimodal pictures (e.g., a cow) that were initially encoded along with a task irrelevant, but semantically congruent characteristic sound (e.g., “moo”) were later recognized with greater accuracy than unimodal stimuli or images paired with incongruent (e.g., “meow”) or neutral sounds (e.g., a tone).…”

Semantically Congruent Visual Information Can Improve Auditory Recognition Memory in Older Adults

“…This is the first study to show that older adults can benefit from multisensory memory cues. These findings are in line with previous studies conducted with young adults (Murray et al, 2004;Lehmann and Murray, 2005;Moran et al, 2013;Heikkilä et al, 2015Heikkilä et al, , 2017aThelen et al, 2015;Ueno et al, 2015) and with children (Heikkilä and Tiippana, 2016). There were differences in how younger and older adults utilized audiovisual information in memory encoding.…”

“…For example, the meowing of a cat is remembered better if presented with a picture of a cat during encoding, rather than alone or with a meaningless picture. This audiovisual congruency effect has been shown to exist for several stimulus combinations in young adults (Murray et al, 2004;Lehmann and Murray, 2005;Moran et al, 2013;Thelen et al, 2015;Ueno et al, 2015;Heikkilä et al, 2015, 2017a,b, but see Cohen et al, 2009 and in children (Heikkilä and Tiippana, 2016).…”

A multisensory perspective on object memory

“…Where the study design enabled the calculation of a more rigorous measure of sensory processing (the perceptual sensitivity parameter, d’, Macmillan & Creelman, 2004), these multisensory benefits were found to be even larger (i.e., 12% performance memory improvement; Matusz et al, 2015). Overall, these improvements have been seen across 6 studies involving more than 100 participants and exhibiting effect sizes ranging from small to large (η 2 p = 0.14–0.63; see Table 1 of Thelen and Murray, 2013 for details; see also Moran et al, 2013; for similar size of effects in studies involving setups with separate exposure and recall, see Heikkilä, Alho, Hyvönen, & Tiippana, 2015; Heikkilä & Tiippana, 2016; Naghavi, Eriksson, Larsson, & Nyberg, 2011; Ueno, Masumoto, Sutani, & Iwaki, 2015). …”