The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception

Krahmer, Emiel; Swerts, Marc

doi:10.1016/j.jml.2007.06.005

Cited by 322 publications

(312 citation statements)

References 72 publications

(82 reference statements)

Supporting

Mentioning

291

Contrasting

Unclassified

Order By: Relevance

“…The emphatic function of head nods has also been observed in perception studies. It has been found that eyebrow movements and head nods help listeners to perceive prominent events in speech (House et al, 2001;Krahmer and Swerts, 2007) and facilitate the recognition of prosodic contrastive focus (Dohen and Loevenbruck, 2004;Prieto et al, 2015). It has been proposed that the temporal patterns of the gesturespeech integration can be influenced by semantic and pragmatic reasons (e.g., Bergmann et al, 2011;Esteve-Gibert et al, 2014).…”

Section: Discussionmentioning

confidence: 99%

The timing of head movements: The role of prosodic heads and edges

Esteve-Gibert

Borràs-Comes

Asor

et al. 2017

The Journal of the Acoustical Society of America

Self Cite

View full text Add to dashboard Cite

This study examines the influence of the position of prosodic heads (accented syllables) and prosodic edges (prosodic word and intonational phrase boundaries) on the timing of head movements. Gesture movements and prosodic events tend to be temporally aligned in the discourse, the most prominent part of gestures typically being aligned with prosodically prominent syllables in speech. However, little is known about the impact of the position of intonational phrase boundaries on gesture-speech alignment patterns. Twenty-four Catalan speakers produced spontaneous (experiment 1) and semi-spontaneous head gestures with a confirmatory function (experiment 2), along with phrasefinal focused words in different prosodic conditions (stress-initial, stress-medial, and stress-final). Results showed (a) that the scope of head movements is the associated focused prosodic word, (b) that the left edge of the focused prosodic word determines where the interval of gesture prominence starts, and (c) that the speech-anchoring site for the gesture peak (or apex) depends both on the location of the accented syllable and the distance to the upcoming intonational phrase boundary. These results demonstrate that prosodic heads and edges have an impact on the timing of head movements, and therefore that prosodic structure plays a central role in the timing of co-speech gestures.

show abstract

Section: Discussionmentioning

confidence: 99%

The timing of head movements: The role of prosodic heads and edges

Esteve-Gibert

Borràs-Comes

Asor

et al. 2017

The Journal of the Acoustical Society of America

Self Cite

View full text Add to dashboard Cite

show abstract

“…1). Equating the audio across all conditions was important, to control for subtle differences in how producing gestures and actions might alter the acoustic properties of the accompanying speech (Krahmer and Swerts 2007). Because the lips were digitally obscured, it was not a problem to temporally align the speech with the visual gestures and actions.…”

Section: Methodsmentioning

confidence: 99%

The processing of speech, gesture, and action during language comprehension

et al. 2014

View full text Add to dashboard Cite

Hand gestures and speech form a single integrated system of meaning during language comprehension, but is gesture processed with speech in a unique fashion? We had subjects watch multimodal videos that presented auditory (words) and visual (gestures and actions on objects) information. Half of the subjects related the audio information to a written prime presented before the video, and the other half related the visual information to the written prime. For half of the multimodal video stimuli, the audio and visual information contents were congruent, and for the other half, they were incongruent. For all subjects, stimuli in which the gestures and actions were incongruent with the speech produced more errors and longer response times than did stimuli that were congruent, but this effect was less prominent for speech-action stimuli than for speech-gesture stimuli. However, subjects focusing on visual targets were more accurate when processing actions than gestures. These results suggest that although actions may be easier to process than gestures, gestures may be more tightly tied to the processing of accompanying speech.

show abstract

“…Eyebrow movements may function as a question marker in French (Purson et al, 1999), even when they are not necessarily coordinated with fundamental frequency changes (Cavé et al, 1996). The same visual cue also has a significant effect on the perception of focus (Krahmer et al, 2002) and prominence , 2006, 2008Krahmer & Swerts, 2007) in Dutch, and may help to distinguish between specific sentence types and pragmatic meanings across languages, such as between Dutch and Catalan (Borràs-Comes & Prieto, 2011;Borràs-Comes et al, 2014;Crespo-Sendra et al, 2013). For instance, Crespo-Sendra et al (2013) have shown that Catalan participants rely more on visual cues than Dutch participants in order to perceive the contrast between neutral and focused yes-no questions, and that this is related to the kind/richness of the respective intonational cues (Catalan speakers use the same intonational contour for both pragmatic meanings, with a different pitch range, whereas Dutch speakers use different intonational contours).…”

Section: Introductionmentioning

confidence: 99%

The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties

Cruz

Swerts

Frota

2017

Laboratory Phonology: Journal of the Association for Laboratory Phonology

Self Cite

View full text Add to dashboard Cite

Cruz, M., et al 2017 The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties. In this paper, we explore the role of intonation and visual cues in the perception of statements and questions in two varieties of European Portuguese-the standard (SEP) and the insular variety of Azores, Ponta Delgada (PtD)-previously shown to convey sentence type contrasts by different uses of intonational means and/or facial gestures, namely eyebrow movements. Forty native speakers (20 from each variety) were exposed to SEP and PtD stimuli in a perception task with three conditions (audio only, video only, and audiovisual). The audiovisual condition includes congruent and incongruent (both original and manipulated) stimuli, where there is either a match or a mismatch between the auditory and visual features as potential cues for a specific sentence type. We concluded that both SEP and PtD participants rely more on intonation than on eyebrow movement to identify sentence types, even when exposed to incongruent audiovisual stimuli. In the absence of audio information, unexpectedly, participants do not interpret eyebrow raising as a question marker, not even when perceiving stimuli from their native variety. When exposed to non-native audiovisual stimuli, both SEP and PtD participants present longer reaction times (RTs), especially for incongruent stimuli. Finally, although we confirm the strength of intonation over visual cues, RTs in the audiovisual condition are significantly shorter than in the audio condition, thus pointing to the relevance of visual cues for structural/linguistic marking.

show abstract

The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception

Cited by 322 publications

References 72 publications

The timing of head movements: The role of prosodic heads and edges

The timing of head movements: The role of prosodic heads and edges

The processing of speech, gesture, and action during language comprehension

The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties

Contact Info

Product

Resources

About