Audiovisual representation of prosody in expressive speech communication

Granström, Björn; House, David

doi:10.1016/j.specom.2005.02.017

Cited by 49 publications

(33 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although complementary, visual features have been shown to have a weaker cue strength than auditory ones for the distinction between statements and questions. This was observed for instance for Swedish (House, 2002;Granström & House, 2004) or English (Srinivasan & Massaro, 2003).…”

Section: Introductionmentioning

confidence: 64%

The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties

Cruz

Swerts

Frota

2017

Laboratory Phonology: Journal of the Association for Laboratory Phonology

View full text Add to dashboard Cite

Cruz, M., et al 2017 The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties. In this paper, we explore the role of intonation and visual cues in the perception of statements and questions in two varieties of European Portuguese-the standard (SEP) and the insular variety of Azores, Ponta Delgada (PtD)-previously shown to convey sentence type contrasts by different uses of intonational means and/or facial gestures, namely eyebrow movements. Forty native speakers (20 from each variety) were exposed to SEP and PtD stimuli in a perception task with three conditions (audio only, video only, and audiovisual). The audiovisual condition includes congruent and incongruent (both original and manipulated) stimuli, where there is either a match or a mismatch between the auditory and visual features as potential cues for a specific sentence type. We concluded that both SEP and PtD participants rely more on intonation than on eyebrow movement to identify sentence types, even when exposed to incongruent audiovisual stimuli. In the absence of audio information, unexpectedly, participants do not interpret eyebrow raising as a question marker, not even when perceiving stimuli from their native variety. When exposed to non-native audiovisual stimuli, both SEP and PtD participants present longer reaction times (RTs), especially for incongruent stimuli. Finally, although we confirm the strength of intonation over visual cues, RTs in the audiovisual condition are significantly shorter than in the audio condition, thus pointing to the relevance of visual cues for structural/linguistic marking.

show abstract

Section: Introductionmentioning

confidence: 64%

The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties

Cruz

Swerts

Frota

2017

Laboratory Phonology: Journal of the Association for Laboratory Phonology

View full text Add to dashboard Cite

show abstract

“…Moreover, the data elicitation method allowed to control for such idiosyncratically variable factors like degree of emphasis (e.g., Ladd and Morton, 1997;Liberman and Pierrehumbert, 1984) and personal involvement (e.g., Crespo-Sendra et al, 2010;Granström and House, 2005), thus increasing the comparability of the resulting pitch properties across the dataset. Experimental materials for this study followed the design of previous cross-linguistic experiments conducted by Grabe (1998).…”

Section: Data Elicitationmentioning

confidence: 99%

How Truncating Are ‘Truncating Languages'? Evidence from Russian and German

Rathcke¹

2017

Phonetica

View full text Add to dashboard Cite

Russian and German have pr eviously been described as ‘truncating‘, or cutting off target frequencies of the phrase-final pitch trajectories when the time available for voicing is compromised. However, supporting evidence is rare and limited to only a few pitch categories. This paper reports a production study conducted to document pitch adjustments to linguistic materials, in which the amount of voicing available for the realization of a pitch pattern varies from relatively long to extremely short. Productions of nuclear H+L*, H* and L*+H pitch accents followed by a low boundary tone were investigated in the two languages. The results of the study show that speakers of both ‘truncating languages' do not utilize truncation exclusively when accommodating to different segmental environments. On the contrary, they employ several strategies - among them is truncation but also compression and temporal re-alignment - to produce the target pitch categories under increasing time pressure. Given that speakers can systematically apply all three adjustment strategies to produce some pitch patterns (H* L% in German and Russian) while not using truncation in others (H+L* L% particularly in Russian), we question the effectiveness of the typological classification of these two languages as ‘truncating'. Moreover, the phonetic detail of truncation varies considerably, both across and within the two languages, indicating that truncation cannot be easily modeled as a unified phenomenon. The results further suggest that the phrase-final pitch adjustments are sensitive to the phonological composition of the tonal string and the status of a particular tonal event (associated vs. boundary tone), and do not apply to falling vs. rising pitch contours across the board, as previously put forward for German. Implications for the intonational phonology and prosodic typology are addressed in the discussion.

show abstract

“…Since head nods are found to be powerful cues to prominence in previous study [19], we focus on the synchronization of head nods with particular syllables in prosodic word. The temporal relationship are investigated between the peak points (i.e.…”

Section: Head Motion Correlation With Textual Informationmentioning

confidence: 99%

Head and facial gestures synthesis using PAD model for an expressive talking avatar

Jia

Zhang

et al. 2013

Multimed Tools Appl

View full text Add to dashboard Cite

This paper proposes to synthesize expressive head and facial gestures on talking avatar using the three dimensional pleasure-displeasure, arousal-nonarousal and dominancesubmissiveness (PAD) descriptors of semantic expressivity. The PAD model is adopted to bridge the gap between text semantics and visual motion features with three dimensions of pleasure-displeasure, arousal-nonarousal, and dominance-submissiveness. Based on the correlation analysis between PAD annotations and motion patterns derived from the head and facial motion database, we propose to build an explicit mapping from PAD descriptors to facial animation parameters with linear regression and neural networks for head motion and facial expression respectively. A PAD-driven talking avatar in text-to-visual-speech system is implemented by generating expressive head motions at the prosodic word level based on the (P, A) descriptors of lexical appraisal, and facial expressions at the sentence level according to the PAD descriptors of emotional information. A series of PAD reverse evaluation and comparative perceptual experiments shows that the head and facial gestures synthesized based on PAD model can significantly enhance the visual expressivity of talking avatar.

show abstract

Audiovisual representation of prosody in expressive speech communication

Cited by 49 publications

References 15 publications

The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties

The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties

How Truncating Are ‘Truncating Languages'? Evidence from Russian and German

Head and facial gestures synthesis using PAD model for an expressive talking avatar

Contact Info

Product

Resources

About