Auditory to Visual Cross-Modal Adaptation for Emotion: Psychophysical and Neural Correlates

Abstract

Adaptation is fundamental in sensory processing and has been studied extensively within the same sensory modality. However, little is known about adaptation across sensory modalities, especially in the context of high-level processing, such as the perception of emotion. Previous studies have shown that prolonged exposure to a face exhibiting one emotion, such as happiness, leads to contrastive biases in the perception of subsequently presented faces toward the opposite emotion, such as sadness. Such work has shown the importance of adaptation in calibrating face perception based on prior visual exposure. In the present study, we showed for the first time that emotion-laden sounds, like laughter, adapt the visual perception of emotional faces, that is, subjects more frequently perceived faces as sad after listening to a happy sound. Furthermore, via electroencephalography recordings and event-related potential analysis, we showed that there was a neural correlate underlying the perceptual bias: There was an attenuated response occurring at ∼ 400 ms to happy test faces and a quickened response to sad test faces, after exposure to a happy sound. Our results provide the first direct evidence for a behavioral cross-modal adaptation effect on the perception of facial emotion, and its neural correlate.

Bilingualism shapes the other race effect

“…We were curious as to whether the average listening or the average speaking proficiency measures were driving this association. As earlier mentioned, the FFA or the STS seems to be responsible for the perceptual narrowing observed during the ORE, and auditory signals seem capable of altering neural activity in both (Ethofer et al, 2006;Wang et al, 2016). We therefore hypothesised that listening proficiency might be better associated with the ORE than speaking proficiency.…”

“…These tests have been widely used to test both neurotypical (Arnell & Dube, 2015;Palermo et al, 2016) and neuropsychological (Bate et al, 2014;Burns et al, in press;Burns, Martin, Chan, & Xu, submitted;Burns, Tree, & Weidemann, 2014;Kirchner, Hatri, Heekeren, & Dziobek, 2011;O'Hearn, Schroer, Minshew, & Luna, 2010) populations' face memory abilities, and are particularly useful as they can reveal subtle differences in performance that other tests fail to yield (Duchaine & Nakayama, 2006). We anticipate that increasing bilingual proficiency will lead to a diminishing ORE, with listening proficiency in particular being more strongly linked than speaking proficiency due to the fact that auditory signals modulate activity in the FFA (Ethofer et al, 2006;Wang et al, 2016) and STS (Deen, Koldewyn, Kanwisher, & Saxe, 2015;Démonet, Thierry, & Cardebat, 2005).…”

“…The fusiform gyrus, a region of the brain commonly known as the fusiform face area (FFA) due to its highly specialised function of processing faces (Kanwisher, McDermott, & Chun, 1997), has been shown to exhibit diminished neural responses to other race faces, 87 indicating a possible neural locus for where perceptual narrowing occurs (Golby, Gabrieli, 88 Chiao, & Eberhardt, 2001). How experience with a second language may shift this narrowing is 89 at present unclear, but experience with auditory information has been shown to change the FFA's activity when viewing a face (Ethofer et al, 2006;Wang et al, 2016). This altering of activity is likely due to the auditory signal reaching the FFA via pathways through the amygdala (Ethofer et al, 2006) or voice selective brain regions (Von Kriegstein & Giraud, 2006).…”

Facilitation of Crossmodal Integration During Emotional Prediction in Methamphetamine Dependents

“…The prediction effect is important for crossmodal (Jessen and Kotz, 2013). Generally, emotional adaptation played a prediction role in crossmodal (Wang et al, 2017). We shall now investigate whether emotional adaptation can highly promote crossmodal integration and whether the difference between Vand A-leading cues can provide more helpful predictions of crossmodal integration for MADs.…”

“…Interestingly, emotional recognition of crossmodal in MADs could be highly prompted by A-leading cues, especially the fearful sound, yet this dominance ceased to exist within HCs. It is suggested that the prediction of fearful sound played an important role in the crossmodal of MADs (Wang et al, 2017). This asymmetry between the visual-and auditory-leading cues suggested that there were separate, selectively recruited networks (Cecere et al, 2017).…”