Audio-Visual Prosody: Perception, Detection, and Synthesis of Prominence

Abstract

Abstract. In this chapter, we investigate the effects of facial prominence cues, in terms of gestures, when synthesized on animated talking heads. In the first study a speech intelligibility experiment is conducted, where speech quality is acoustically degraded, then the speech is presented to 12 subjects through a lip synchronized talking head carrying head-nods and eyebrow raising gestures. The experiment shows that perceiving visual prominence as gestures, synchronized with the auditory prominence, significantly increases speech intelligibility compared to when these gestures are randomly added to speech. We also present a study examining the perception of the behavior of the talking heads when gestures are added at pitch movements. Using eyegaze tracking technology and questionnaires for 10 moderately hearing impaired subjects, the results of the gaze data show that users look at the face in a similar fashion to when they look at a natural face when gestures are coupled with pitch movements opposed to when the face carries no gestures. From the questionnaires, the results also show that these gestures significantly increase the naturalness and helpfulness of the talking head.

Pre‐schoolers use head gestures rather than prosodic cues to highlight important information in speech

“…From a functional point of view, gestures convey meanings that can be referential (representing an entity or event deictically, iconically, or metaphorically) and non‐referential (signaling information structure, modal information, or discourse cohesion) (Kendon, 1980; McNeill, 2000; and many others thereafter). In the context of focus marking, head gestures are a specific type of body movements that serve a non‐referential meaning and can indicate focus quite consistently (Ambrazaitis & House, 2017; Esteve‐Gibert et al., 2017; Ishi et al., 2014), together with other movements such as eyebrow raising (Cavé et al., 1996; Dohen et al., 2006; Moubayed & Beskow, 2011) and manual beats (Roustan & Dohen, 2010).…”

Gesture/speech integration in the perception of prosodic emphasis

“…In a perception experiment involving computer synthesized animations, [14] and [15] showed that when headnods and eyebrow raises accompany prominent syllables, they can aid speech perception. On the other hand, it is not clear to 9th International Conference on Speech Prosody 2018 13-16 June 2018, Poznań, Poland them whether they will aid or hinder speech perception when they accompany non-prominent syllables.…”

Acoustic correlates of perceived syllable prominence in Spanish

“…Those acoustic differences which are perceptually salient could be primarely exploited by an automatic speech recognition [1,2] and understanding system [3]. Such knowledge appears to be also important for other areas of application: in computer-based pronunciation training, speech annotation for data-driven speech synthesis [4], speech summarization [5], speech comprehension for example through improving the syntactic parsing [6], or in audio-visual speech applications by driving gestures for avatars [7].…”