Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion

Abstract

There has been considerable research into perceptible correlates of emotional state, but a very limited amount of the literature examines the acoustic correlates and other relevant aspects of emotion effects in human speech; in addition, the vocal emotion literature is almost totally separate from the main body of speech analysis literature. A discussion of the literature describing human vocal emotion, and its principal findings, are presented. The voice parameters affected by emotion are found to be of three main types: voice quality, utterance timing, and utterance pitch contour. These parameters are described both in general and in detail for a range of specific emotions. Current speech synthesizer technology is such that many of the parameters of human speech affected by emotion could be manipulated systematically in synthetic speech to produce a simulation of vocal emotion; application of the literature to construction of a system capable of producing synthetic speech with emotion is discussed.

Describing the emotional states that are expressed in speech

“…That has been called the palette theory (Scherer, 1984b). The term primary is still widely used, in lay parlance and in the speech literature (e.g., Murray and Arnott, 1993); but in fact, ÔpaletteÕ theories have very little support in modern emotion research (Ekman, 1999).…”

Perceiving affect from the voice and the face

“…Williams and Stevens (1972) concluded that the pitch contour is the best indicator ofthe emotional content ofan utterance. In their review ofthe literature, Murray and Arnott (1993) noted that the most commonly referenced vocal parameters are pitch (i.e., both the average value and range of the fundamental frequency), duration, intensity, and the undefined term voice quality.…”

Emotions, speech and the ASR framework

“…ÔAnxiousÕ utterances show segments that are shorter than average, with exception of voiceless plosives. Also in (Murray and Arnott, 1993), relations were shown between the emotion state and the duration of vowels and consonants. But in nearly all studies pitch and energy are the most commonly applied features to distinguish and classify emotion state (Murray and Arnott, 1993), or anyway to convey supra-textual information.…”

“…Also in (Murray and Arnott, 1993), relations were shown between the emotion state and the duration of vowels and consonants. But in nearly all studies pitch and energy are the most commonly applied features to distinguish and classify emotion state (Murray and Arnott, 1993), or anyway to convey supra-textual information. In Slaney and McRoberts, 1998, a study was conducted to automatically classify an utterance (spoken by a parent to a young infant) into three classes: approval, attention and prohibition.…”

“…For text-to-speech (TTS) systems, the advantage of Ôemotionally richÕ speech synthesis is evident (also see Murray and Arnott, 1993). The pragmatic value of TTS systems is mainly determined by two factors: the intelligibility of the speech that is produced, and the naturalness of the speech output.…”