Gesture–speech physics: The biomechanical basis for the emergence of gesture–speech synchrony.

Pouw, Wim; Harrison, Steven J.; Dixon, James A.

doi:10.1037/xge0000646

Cited by 55 publications

(85 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, it has been found, in the pronunciation of the nonsense word "baba," when stress is put on the first (baba) or last syllable (baba), the pointing gesture's maximum extension is coordinated to align more closely with the stressed syllable (Rochet-Capellan, Laboissiere, Galvan, & Schwartz, 2008; see also Esteve-Gibert & Prieto, 2013;Krivokapić, Tiede, & Tyrone, 2017;Rusiewicz, Shaiman, Iverson, & Szuminsky, 2014). On the biomechanical level, it has further been found that gesture's moments of peak physical impetus (lasting~50 ms) entrain fundamental frequency and the amplitude envelope during phonation (Pouw, Harrison, & Dixon, 2019). Wim Pouw and James P. Trujillo contributed equally to the present manuscript and share first authorship Thus, the gesture-speech relationship is defined over many levels and timescales (for overviews, see Kendon, 2004;Wagner et al, 2014).…”

mentioning

confidence: 99%

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

2019

Self Cite

View full text Add to dashboard Cite

There is increasing evidence that hand gestures and speech synchronize their activity on multiple dimensions and timescales. For example, gesture's kinematic peaks (e.g., maximum speed) are coupled with prosodic markers in speech. Such coupling operates on very short timescales at the level of syllables (200 ms), and therefore requires high-resolution measurement of gesture kinematics and speech acoustics. High-resolution speech analysis is common for gesture studies, given that field's classic ties with (psycho)linguistics. However, the field has lagged behind in the objective study of gesture kinematics (e.g., as compared to research on instrumental action). Often kinematic peaks in gesture are measured by eye, where a "moment of maximum effort" is determined by several raters. In the present article, we provide a tutorial on more efficient methods to quantify the temporal properties of gesture kinematics, in which we focus on common challenges and possible solutions that come with the complexities of studying multimodal language. We further introduce and compare, using an actual gesture dataset (392 gesture events), the performance of two video-based motion-tracking methods (deep learning vs. pixel change) against a high-performance wired motion-tracking system (Polhemus Liberty). We show that the videography methods perform well in the temporal estimation of kinematic peaks, and thus provide a cheap alternative to expensive motion-tracking systems. We hope that the present article incites gesture researchers to embark on the widespread objective study of gesture kinematics and their relation to speech. Keywords Motion tracking. Video recording. Deep learning. Gesture and speech analysis. Multimodal language There is an increasing interest in the ways that cooccurring hand gestures and speech coordinate their activity (Esteve-Gibert & Guellaï, 2018; Wagner, Malisz, & Kopp, 2014). The nature of this coordination is exquisitely complex, as it operates on multiple levels and timescales. For example, on the semantic level, referential

show abstract

mentioning

confidence: 99%

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rusiewicz et al, 2013Rusiewicz et al, , 2014. Pouw and Dixon (2019) found that a Delayed Auditory Feedback actually increases the temporal alignment between gestures and speech. Lastly, Bergmann et al, (2011) found that gestures and speech were more temporally aligned when their semantic content was more similar.…”

Section: Temporal Alignmentmentioning

confidence: 99%

“…Moreover, this tight coupling has been conceptualized as gesture-speech synchronization (e.g. Iverson & Thelen, 1999;Pouw & Dixon, 2019;Treffner & Peter, 2002). Gestures and speech synchronize in time, semantic content, emphasis, and emotional valence (for a comprehensive review, see Wagner et al, 2014).…”

Section: Introductionmentioning

confidence: 99%

Easier said than done? Task difficulty’s influence on temporal alignment, semantic similarity, and complexity matching between gestures and speech

Jonge-Hoekstra¹,

Cox²,

Steen³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Gestures and speech are clearly synchronized in many ways. However, previous studies have shown the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture-speech mismatches which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech’s integration and synchronization. In the current study, we applied three different perspectives to investigate gesture-speech synchronization in an easy and a difficult task: temporal alignment, semantic similarity, and complexity matching. Participants engaged in a simple cognitive task, and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant’s speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal Detrended Fluctuation Analysis (MFDFA) was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture-speech synchronization in all three domains. We thereby extended the phenomenon of gesture-speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology) provides novel understanding about cognitive concepts in general, and about gesture-speech synchronization and task difficulty in specific.

show abstract

“…In a recent study, a possible candidate gesture-speech invariant was identified (Pouw, Harrison, et al, 2019). Namely, when steady-state vocalizing a vowel /a:/ (as in cinema), highimpetus whole-arm upper limb movements imparted positive peaks in the amplitude envelope and F0 of phonation acoustics.…”

mentioning

confidence: 99%

“…The dynamics of the hypothesized gesture-induced forces on the lower vocal tract are still unknown. We hypothesized that physical impulses produced by acceleration and decelerations in gesture, are not kept locally contained at the effector muscles (Pouw, Harrison, et al, 2019).…”

mentioning

confidence: 99%

Energy flows in gesture-speech physics: Exploratory findings and pre-registration of confirmatory analysis

Pouw¹,

Sa²,

Ne³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

A well-known phenomenon of multimodal language is the synchronous coupling of prosodic contours in speech with salient kinematic changes in co-speech hand-gesture motions. Invariably, such coupling has been rendered by psychologists to require a dedicated neural-cognitive mechanism preplanning speech and gesture trajectories. Recently, in a continuous vocalization task, it was found that acoustic peaks unintentionally appear in vocalizations when gesture motions reach peaks in physical impetus, suggesting a biomechanical basis for gesture-speech synchrony (Pouw, Harrison, & Dixon, 2019). However, from this rudimentary study it is still difficult to draw strong conclusions about gesture-speech dynamics in (more) complex speech and the precise biomechanical nature of these effects. Here we assess how the timing of physical impetus of a gesture relates to its effect on acoustic parameters of mono-syllabic consonant-vowel (CV) vocalization(/pa/). Furthermore, we assess how chest-wall kinematics is affected by gesturing, and whether this modulates the effect of gestures on acoustics. In the current exploratory analysis, we analyze a subset (N = 4) of an already collected dataset (N = 36), which serves as the basis for a pre-registration of the confirmatory analyses yet to be completed. Here we provide exploratory evidence that gestures affect acoustics (amplitude envelope and F0) as well as chest-wall kinematics during mono-syllabic vocalizations. These effects are more extreme when a gesture’s peak impetus occurs closer to the center of the vowel vocalization event. If the current findings can be replicated in confirmatory fashion, there is a more compelling case to be made that gesture-speech physics is important facet of multimodal synchrony.

show abstract

Gesture–speech physics: The biomechanical basis for the emergence of gesture–speech synchrony.

Cited by 55 publications

References 47 publications

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

Easier said than done? Task difficulty’s influence on temporal alignment, semantic similarity, and complexity matching between gestures and speech

Energy flows in gesture-speech physics: Exploratory findings and pre-registration of confirmatory analysis

Contact Info

Product

Resources

About