52. Experimental methods in co-speech gesture research

Holler, Judith

doi:10.1515/9783110261318.837

Cited by 2 publications

(2 citation statements)

References 89 publications

(124 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, although several projects have developed multimodal data annotated with gestural information (for a discussion, see Wagner et al, 2014), few of these efforts have yet resulted in freely available annotated corpora for multiple languages such as in Koutsombogera and Vogel (2018). In parallel with this work, the linguistics community interested in co-speech gesture has created and annotated many multimodal data collections, but typically for experimental purposes and thus in very specific contexts (Holler, 2013).…”

Section: Introductionmentioning

confidence: 99%

Towards a Methodology Supporting Semiautomatic Annotation of HeadMovements in Video-recorded Conversations

Paggio¹,

Navarretta²,

Jongejan³

et al. 2021

Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop

View full text Add to dashboard Cite

We present a method to support the annotation of head movements in video-recorded conversations. Head movement segments from annotated multimodal data are used to train a model to detect head movements in unseen data. The resulting predicted movement sequences are uploaded to the ANVIL tool for postannotation editing. The automatically identified head movements and the original annotations are compared to assess the overlap between the two. This analysis showed that movement onsets were more easily detected than offsets, and pointed at a number of patterns in the mismatches between original annotations and model predictions that could be dealt with in general terms in post-annotation guidelines.

show abstract

Section: Introductionmentioning

confidence: 99%

Towards a Methodology Supporting Semiautomatic Annotation of HeadMovements in Video-recorded Conversations

Paggio¹,

Navarretta²,

Jongejan³

et al. 2021

Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop

View full text Add to dashboard Cite

show abstract

“…To illustrate, when asking "Can you pass me the coffee pot?," a speaker might gesture in a way that reflects the size and shape of the coffee pot, or the action of pouring coffee. Although there is evidence that iconic gestures facilitate language comprehension (see Dargue et al, 2019;Holler, 2013;Hostetter, 2011;Özyürek, 2009, for metaanalyses and reviews), little is known about the processing of iconic gestures in contexts where the entities being referred are, along with the speaker, in the listener's visual field. The aim of the present study is to explore how listeners allocate attention to scene elements in this context, whether gesture cues can facilitate comprehension, and whether these patterns change according to the ease or difficulty of understanding the auditory signal.…”

mentioning

confidence: 99%

Gesture and reference to objects in the here-and-now: Listeners’ use of gesture cues in quiet and in noise.

Saryazdi¹,

Chambers²

2022

Journal of Experimental Psychology: Learning, Memory, and Cogni

View full text Add to dashboard Cite

In face-to-face interaction, speakers spontaneously produce manual gestures that can facilitate listeners' comprehension of spoken language. The present study explores the factors affecting the uptake and influence of gesture cues in situations where a speaker is referring to objects visible to the listener. In this context, the listener's attention must be distributed across various scene regions, potentially reducing the ability to draw on and apply gesture cues in real time. In two experiments, the instruction provided by a speaker (e.g., "pick up the candy") was accompanied by an iconic grasp gesture (produced alongside the verb) that reflected the size/shape of the intended target. Effects on listeners' comprehension were compared with a no-gesture condition. Experiment 1 (audiovisual gating task) showed that, under simplified processing circumstances, gesture cues allowed earlier identification of intended targets. Experiment 2 (eye tracking) explored whether this facilitation is found in real-time comprehension, and whether attention to gesture information is influenced by the acoustic environment (quiet vs. background noise). Measures of gaze position showed that although the speaker's gesturing hand was rarely fixated directly, gestures did facilitate comprehension, particularly when the target object was smaller relative to alternatives. The magnitude of the gesture effect was greater in quiet than in noise, suggesting that the latter did not provoke listeners to increase attention to gesture to compensate for the challenging auditory signal. Together, the findings clarify how situational factors influence listeners' attention to visual information during real-time comprehension.

show abstract

52. Experimental methods in co-speech gesture research

Cited by 2 publications

References 89 publications

Towards a Methodology Supporting Semiautomatic Annotation of HeadMovements in Video-recorded Conversations

Towards a Methodology Supporting Semiautomatic Annotation of HeadMovements in Video-recorded Conversations

Gesture and reference to objects in the here-and-now: Listeners’ use of gesture cues in quiet and in noise.

Contact Info

Product

Resources

About