The integration of facial and vocal cues during emotional change perception: EEG markers

Abstract

The ability to detect emotional changes is of primary importance for social living. Though emotional signals are often conveyed by multiple modalities, how emotional changes in vocal and facial modalities integrate into a unified percept has yet to be directly investigated. To address this issue, we asked participants to detect emotional changes delivered by facial, vocal and facial-vocal expressions while behavioral responses and electroencephalogram were recorded. Behavioral results showed that bimodal emotional changes were detected with higher accuracy and shorter response latencies compared with each unimodal condition. Moreover, the detection of emotional change, regardless of modalities, was associated with enhanced amplitudes in the N2 and P3 component, as well as greater theta synchronization. More importantly, the P3 amplitudes and theta synchronization were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions. The superadditive responses in P3 amplitudes and theta synchronization were both positively correlated with the magnitude of the bimodal superadditivity in accuracy. These behavioral and electrophysiological data consistently illustrated an effect of audiovisual integration during the detection of emotional changes, which is most likely mediated by the P3 activity and theta oscillations in brain responses.

Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller

“…In a study on bimodal emotion integration, P300 amplitudes were larger for audiovisual emotion stimuli than for visual emotion stimuli; the authors suggested that the bimodal stimuli led to a “dual novelty” in the cognitive task comprising visual and auditory stimuli, which enabled subjects to actively process multisensory information (Chen, Han, et al, ). Similar findings were also reported in some studies on the sensitivity of P300 to emotional face–voice stimuli (Campanella et al, ), the integration of facial and vocal emotion perception (Chen, Pan, et al, ), and emotion recognition tasks (Liu et al, ). In addition, changes in age and sex features in voices have also been shown to increase subjects' attention to and perception of stimuli (Li et al, ).…”

“…In our study, the accuracy and RBR were significantly greater in the E‐AV spelling paradigm than those in the E‐V spelling paradigm at superposing one and two times; thus, the comparison of accuracy and RBR between the spelling paradigms was at superposing one and two times. From the Figure , we observed that the accuracy of the E‐AV was higher than that in 2012 (Jin et al, ) and 2016 (Chen, Pan, et al, ) at one and two superpositions and the RBR of the E‐AV was higher than that in the 2012, 2014 (Jin et al, ), and 2016 at one and two superpositions. The accuracy of the spelling paradigm is affected by several factors, such as the arrangement of the characters, visual angle (subjects spelled more accurately on a larger visual angle than on a smaller visual angle (Li, Nam, Shadden, & Johnson, ), SOA (increased SOA would result in a larger P300 amplitude to improve the classification accuracy [Lu, Speier, Hu, & Pouratian, ]), and other factors.…”

“…Chen, Han, Pan, Luo, and Wang () reported that P300 amplitudes were larger for bimodal stimuli (face and voice with emotion) than for the sum of two unimodal stimuli (face stimulus and voice stimulus). Other studies also showed increased P300 amplitudes for audiovisual emotion stimuli (Chen, Pan, et al, ; Chen, Pan, Wang, Zhang, & Yuan, ). In an fMRI study on emotional voices and faces, the bilateral posterior superior temporal gyrus (pSTG) and the right thalamus showed enhanced activation and strength of the BOLD response during bimodal conditions (Kreifelts, Ethofer, Grodd, Erb, & Wildgruber, ).…”

The Functional Role of Neural Oscillations in Non-Verbal Emotional Communication

“…Subsequent research by the same group showed that supra-additive increases in the STS occurred in both congruent and incongruent conditions (albeit later in the incongruent condition), suggesting automatic integration of emotional facial and vocal expressions (Hagan et al, 2013 ). Consistent with these findings, other studies have observed theta synchronization during the integration of facial and prosodic change (Chen et al, 2015 ). Together, these findings suggest that oscillatory activity in the alpha and theta frequency bands drive the integration of facial and vocal expressions.…”

Tracking the Sound of Human Affection: EEG Signals Reveal Online Decoding of Socio-Emotional Expression in Human Speech and Voice

“…In a study in which participants were presented with simultaneously presented vocal and facial expressions while being asked to detect the change of emotion from neutral to anger or happiness conveyed in voice or in face [20]. The P3 associated with the detection of the emotional categorical change in both voice and face was larger than the sum of the change in single channel (see also [21]). The N1 associated with the detection of early acoustic change was dependent on whether their atention was guided to the voice or the face, with the atention to the voice yielding to a N1 in bimodal change larger than the sum of the two single modal change conditions.…”