Audiovisual speech from emotionally expressive and lateralized faces

Abstract

Emotional expression and how it is lateralized across the two sides of the face may influence how we detect audiovisual speech. To investigate how these components interact we conducted experiments comparing the perception of sentences expressed with happy, sad, and neutral emotions. In addition we isolated the facial asymmetries for affective and speech processing by independently testing the two sides of a talker's face. These asymmetrical differences were exaggerated using dynamic facial chimeras in which left- or right-face halves were paired with their mirror image during speech production. Results suggest that there are facial asymmetries in audiovisual speech such that the right side of the face and right-facial chimeras supported better speech perception than their left-face counterparts. Affective information was also found to be critical in that happy expressions tended to improve speech performance on both sides of the face relative to all other emotions, whereas sad emotions generally inhibited visual speech information, particularly from the left side of the face. The results suggest that approach information may facilitate visual and auditory speech detection.

The Effect of Spectral Profile on the Intelligibility of Emotional Speech in Noise

“…The results of Gordon and Hibberts [1] are inconsistent with those of a more recent study [4] that examined a larger number of emotions (angry, disgust, fear, happy, sad, surprise and neutral). Dupuis and Pichora-Fuller [4] found that the intelligibility of happy and sad speech did not differ from neutral speech.…”

“…We considered two issues. (1). What acoustic correlate(s) of emotional expression best explains why speech expressing some emotions (e.g., happy) is more intelligible in noise than speech expressing other emotions (e.g., sad), even though each is mixed with noise at the same SNR.…”

“…The idea that emotionally expressive speech differentially engages a listener's attention has a relatively long history [3]. Based on this idea, Gordon and Hibberts [1] examined the effect of expressing a happy or sad emotion on speech intelligibility in noise (compared with neutral speech as a control). Happy and sad expressions were chosen based on the idea that a happy expression would engender feelings of approach, and so motivate listeners to attend more closely to stimuli, whereas a sad emotion would produce a withdrawal response and so lead listeners to pay less attention.…”

The Language, Tone and Prosody of Emotions: Neural Substrates and Dynamics of Spoken-Word Emotion Perception

“…The phonemic cues consist primarily of relatively fast spectral changes occurring within 50 ms speech segments, whereas the prosodic cues consist of slower spectral changes occurring over more than 200 ms speech segments (syllabic and suprasegmental range). Emotional speech confers processing advantages such as improved intelligibility in noise background as well as faster repetition time for words spoken with congruent emotional prosody (Nygaard and Queen, 2008; Gordon and Hibberts, 2011; Dupuis and Pichora-Fuller, 2014). …”

Intelligibility of Emotional Speech in Younger and Older Adults

“…Mullennix et al (2002) found that latencies for two tasks, matching judgment and phoneme identification in quiet, were negatively affected by variability in emotional prosody, suggesting that listeners benefit from repeated presentation of the same emotions. Gordon and Hibberts (2011) found that younger adults repeated sentences presented in noise more accurately when speech was spoken to portray happiness compared to sadness or neutral emotion. Gordon and Hibberts (2011) found that younger adults repeated sentences presented in noise more accurately when speech was spoken to portray happiness compared to sadness or neutral emotion.…”