Gesture and speech in interaction: An overview

Wagner, Petra; Malisz, Zofia; Kopp, Stefan

doi:10.1016/j.specom.2013.09.008

Cited by 363 publications

(353 citation statements)

References 97 publications

Supporting

Mentioning

333

Contrasting

Unclassified

Order By: Relevance

“…Whenever the annotator doubted on this classification, a conservative criterion was used, meaning that utterances were coded as not being accompanied by a head gesture. The types of head movements that were included in the analyses were head nods (following Poggi et al, 2010, a head nod was any vertical head movement in which the head, after a slight tilt up, bends downward and then goes back to its starting point), upward movements (a head movement directed upward in the opposite direction from nodding), and head tilts (a head inclination or sideward movement) (see Wagner et al, 2014, for a complete overview of the head gesture forms). All selected sentences had the form of verb þ article þ noun/ adjective (the article being optional), as in the statement Porta barret "(S)he has a hat.…”

Section: Codingmentioning

confidence: 99%

“…Depending on the gesture and the way it is produced, this prominent part of the gesture can be either an interval, called "gesture stroke," or a peak in the gesture movement, called "gesture apex." Many studies have further investigated the specifics of this temporal alignment, revealing that gesture strokes and gesture apexes are aligned with stressed syllables in the speech stream (see Wagner et al, 2014, for a complete review). Interestingly, certain stressed syllables seem to attract more strongly the presence of co-speech gestures: gesture apexes (the peak of prominence in a gesture movement) are more frequently aligned with pitch-accented syllables and with focal pitch accents than with stressed syllables that have a lesser degree of prosodic emphasis (e.g., Alexanderson et al, 2013;De Ruiter, 1998;Ferre, 2014;Yasinnik et al, 2004).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The timing of head movements: The role of prosodic heads and edges

Esteve-Gibert

Borràs-Comes

Asor

et al. 2017

The Journal of the Acoustical Society of America

View full text Add to dashboard Cite

This study examines the influence of the position of prosodic heads (accented syllables) and prosodic edges (prosodic word and intonational phrase boundaries) on the timing of head movements. Gesture movements and prosodic events tend to be temporally aligned in the discourse, the most prominent part of gestures typically being aligned with prosodically prominent syllables in speech. However, little is known about the impact of the position of intonational phrase boundaries on gesture-speech alignment patterns. Twenty-four Catalan speakers produced spontaneous (experiment 1) and semi-spontaneous head gestures with a confirmatory function (experiment 2), along with phrasefinal focused words in different prosodic conditions (stress-initial, stress-medial, and stress-final). Results showed (a) that the scope of head movements is the associated focused prosodic word, (b) that the left edge of the focused prosodic word determines where the interval of gesture prominence starts, and (c) that the speech-anchoring site for the gesture peak (or apex) depends both on the location of the accented syllable and the distance to the upcoming intonational phrase boundary. These results demonstrate that prosodic heads and edges have an impact on the timing of head movements, and therefore that prosodic structure plays a central role in the timing of co-speech gestures.

show abstract

Section: Codingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The timing of head movements: The role of prosodic heads and edges

Esteve-Gibert

Borràs-Comes

Asor

et al. 2017

The Journal of the Acoustical Society of America

View full text Add to dashboard Cite

show abstract

“…Implications for models of speech and gesture production Over the years, various models of speech and gesture production have been proposed, including Krauss, Chen and Gottesman's (2000) Process model, Kita and Özyürek's (2003) Interface model, de Ruiter's (2000) Sketch model, and McNeill and Duncan's (2000) Growth Point theory (see e.g., Chu & Hagoort, 2014;Hostetter & Alibali, 2008;Wagner, Malisz, & Kopp, 2014, for recent comparisons and discussion). These models all seek to describe how speakers produce multimodal utterances and are concerned with issues such as the timing and integration of gesture and speech, and the role that gestures play in communication.…”

Section: On the Effects Of Visibilitymentioning

confidence: 99%

Reduction in gesture during the production of repeated references

Hoetjes

Koolen

Goudbeek

et al. 2015

Journal of Memory and Language

View full text Add to dashboard Cite

a b s t r a c tIn dialogue, repeated references contain fewer words (which are also acoustically reduced) and fewer gestures than initial ones. In this paper, we describe three experiments studying to what extent gesture reduction is comparable to other forms of linguistic reduction. Since previous studies showed conflicting findings for gesture rate, we systematically compare two measures of gesture rate: gesture rate per word and per semantic attribute (Experiment I). In addition, we ask whether repetition impacts the form of gestures, by manual annotation of a number of features (Experiment I), by studying gradient differences using a judgment test (Experiment II), and by investigating how effective initial and repeated gestures are at communicating information (Experiment III). The results revealed no reduction in terms of gesture rate per word, but a U-shaped reduction pattern for gesture rate per attribute. Gesture annotation showed no reliable effects of repetition on gesture form, yet participants judged gestures from repeated references as less precise than those from initial ones. Despite this gradient reduction, gestures from initial and repeated references were equally successful in communicating information. Besides effects of repetition, we found systematic effects of visibility on gesture production, with more, longer, larger and more communicative gestures when participants could see each other. We discuss the implications of our findings for gesture research and for models of speech and gesture production.

show abstract

“…Besides the acoustic signal during speech, the visual information related to facial expressions, hand gesture and body posture contributes significantly to the intelligibility of the message being transmitted, and to the perception of the actual meaning of the message. In addition, as pointed out in a recent survey about the interaction between gesture and speech [1], the parallel use of these modalities gives the listener access to complementary information not present in the acoustic signal by itself. For instance, when the speaker says "The dog is this tall" and simultaneously indicates with his hands the height of the dog, an extra information is being provided.…”

Section: Introductionmentioning

confidence: 99%

“…A thorough overview of existing multimodal corpora and the challenges and limits involved in corpus building, can be found in [2] and [3]. As pointed out in [1], building a multimodal corpus requires to make decisions about several issues such as the number and gender of the participants, the modality of the recording (monologue from scripted text or free speech, dialogue), the number and characteristics of the recording devices (single camera, multicamera, microphones, motion capture systems, devices capable of capturing depth information, like Microsoft Kinect), the languages being used (single language, or multilingual), the signals to be captured (audio, facial expressions, hands and arms gestures, body posture), the words and sentences to be recorded in the case of scripted text monologues, etc. Most of these decisions are influenced by the particular application intended for the corpus.…”

Section: Introductionmentioning

confidence: 99%

A French-Spanish Multimodal Speech Communication Corpus Incorporating Acoustic Data, Facial, Hands and Arms Gestures Information

Terissi¹,

Sad²,

Cerda³

et al. 2018

Interspeech 2018

View full text Add to dashboard Cite

A Bilingual Multimodal Speech Communication Corpus incorporating acoustic data as well as visual data related to face, hands and arms gestures during speech, is presented in this paper. This corpus comprises different speaking modalities, including scripted text speech, natural conversation, and free speech. The corpus has been compiled in two different languages, viz., French and Spanish. The experimental setups for the recording of the corpus, the acquisition protocols, and the employed equipment are described. Statistics regarding the number and gender of the speakers, number of words, number of sentences, and duration of the recording sessions, are also provided. Preliminary results from the analysis of the correlation among speech, head and hand movements during spontaneous speech are also presented in this paper, showing that acoustic prosodic features are related with head and hand gestures.

show abstract

Gesture and speech in interaction: An overview

Cited by 363 publications

References 97 publications

The timing of head movements: The role of prosodic heads and edges

The timing of head movements: The role of prosodic heads and edges

Reduction in gesture during the production of repeated references

A French-Spanish Multimodal Speech Communication Corpus Incorporating Acoustic Data, Facial, Hands and Arms Gestures Information

Contact Info

Product

Resources

About