Evidence for the integration of audiovisual emotional information at the perceptual level of processing

When two become one: Temporally dynamic integration of the face and voice

“…Participants were more likely to misinterpret the face as the opposite sex when faces were accompanied by sex-atypical (M = 11.5%, SE = 1.0%) relative to sex-typical (M = 5.1%, SE = 0.8%) voices, t(40) = 5.38, p b .0001, a finding often cited as evidence of face-voice integration (e.g., Hietanen, Leppänen, Illi, & Surakka, 2004). To examine the temporal dynamics of this integration, we examined trials that were correctly categorized.…”

What is a nice smile like that doing in a place like this? Automatic affective responses to environments influence the recognition of facial expressions

“…There is plenty of experimental evidence showing that recognition of facial emotions is aVected by other concomitant emotional information signaled, for example, by vocal prosody (Hietanen, Leppänen, Illi, & Surakka, 2004), and by perceivers' own emotional states induced by Wlm clips, music (Niedenthal, Halberstadt, Margolin, & Innes-Ker, 2000), and odours (Leppänen & Hietanen, 2003). Previous results have also provided preliminary support for that seeing environmental scenes triggers rapid and automatic aVective responses and inXuences the recognition times of vocal (Korpela, Klemettilä & Hietanen, 2002) and facial (Hietanen & Korpela, 2004) expressions of emotions (anger and happiness).…”

Human Perception of Audio-Visual Synthetic Character Emotion Expression in the Presence of Ambiguous and Conflicting Information

“…A similar study was presented in [10]. In that work, evaluators were asked to rate 144 stimuli composed of still images and emotional speech using happy, angry, and neutral emotions.…”

“…However, when presented with conflicting emotional expressions, evaluators relied almost exclusively on the audio information. This work is novel in that it utilizes multiple sentences rather than a single word [8] or a single sentence [11], video segments rather than still images [6], [7], [9], [10], and a dimensional emotional evaluation (e.g., valence, activation, dominance) rather than a discrete emotional evaluation (e.g., angry, happy, sad, neutral) [6], [7], [9], [10]. The related work will be more fully discussed in Section II.…”