Encoding Emotions in Speech with the Size Code

Abstract

Our current understanding of how emotions are expressed in speech is still very limited. Part of the difficulty has been the lack of understanding of the underlying mechanisms. Here we report the findings of a somewhat unconventional investigation of emotional speech. Instead of looking for direct acoustic correlates of multiple emotions, we tested a specific theory, the size code hypothesis of emotional speech, about two emotions – anger and happiness. According to the hypothesis, anger and happiness are conveyed in speech by exaggerating or understating the body size of the speaker. In two studies consisting of six experiments, we synthesized vowels with a three-dimensional articulatory synthesizer with parameter manipulations derived from the size code hypothesis, and asked Thai listeners to judge the body size and emotion of the speaker. Vowels synthesized with a longer vocal tract and lower F0 were mostly heard as from a larger person if the length and F0 differences were stationary, but from an angry person if the vocal tract was dynamically lengthened and F0 was dynamically lowered. The opposite was true for the perception of small body size and happiness. These results provide preliminary support for the size code hypothesis. They also point to potential benefits of theory-driven investigations in emotion research.

Humans attribute emotions to a robot that shows simple behavioural patterns borrowed from dog behaviour

“…While there are some data on decoding emotions based on body language and vocal signals in humans (Bänziger, Grandjean, & Scherer, 2009;Chuenwattanapranithi, 2008;de Gelder, 2009), so far the vast majority of studies on emotion recognition have focused on facial expressions (see e.g. Breazeal, 2003;Ekman, 1993).…”

Acoustic Feature Optimization Based on F-Ratio for Robust Speech Recognition

“…As can be observed from Fig. 15, although in the third row the two phonemes still have overlaps, some of them are attenuated, such as the upper part (21)(22)(23)(24). Only two phonemes from two types of speech are analyzed here.…”

“…The differences between neutral speech and each type of emotion affected speech differ in different frequency area, especially in the low and high frequencies. According to the size code hypothesis [22], when speech is produced with the happy emo- tion, the vocal tract of the speaker tends to be shortened, and the F 0 tend to be raised. The F-Ratio curves could reflect this affection slightly.…”

“…While the Mel FBE spectrograms may be able to space frequency more compactly for each speech unit, shown in the second row, in which the main part of spectrogram for each phoneme covers no more than half of the whole bands. However, for the Mel FBE spectrograms, there is much overlapped area between different phonemes, such as the upper part (15)(16)(17)(18)(19)(20)(21)(22)(23)(24), and it may be difficult to distinguish them on the frame level for the emotion affected speech. The F-Ratio based FBE spectrograms are proposed to solve this problem by emphasizing the resolution of frequencies with more discrepancies between different speech units, while attenuating the resolution of frequencies with more overlap between different speech units.…”

Auditory Communication in Domestic Dogs

“…Potential support for this explanation can be derived from findings that prosodic cues indicating anger and happiness in human speech may also be related to the acoustic size code discussed earlier in this chapter. Human listeners perceive synthetic vowels created with a dynamically lower F0 and smaller formant dispersion as being spoken in an angry voice, whilst vowels with a dynamically higher F0 and a larger formant dispersion are perceived as being spoken in a happy voice (Chuenwattanapranithi et al, 2008). This potential universality in ritualisation across mammal vocalisations may allow dogs to generalise their responses to specific prosodic cues, aiding their perception of certain emotions in the human voice.…”