Gesture-Speech Physics: The Biomechanical Basis for the Emergence of Gesture-Speech Synchrony

Abstract

Hand gestures and speech move in a common rhythm, as exemplified by the synchrony between prosodic contrasts in gesture movement (e.g., peak velocity; maximum effort) and speech (e.g., peaks in fundamental frequency; F0). This joined rhythmic activity is hypothesized to have a variable set of functions, ranging from self-serving cognitive benefits for the gesturer, to communicational advantages that support listeners’ understanding. However, gesture-speech synchrony has been invariably understood as a “neural-cognitive” achievement; i.e., gesture and speech are coupled through neural-cognitive mediation. Yet, it is possible that gesture-speech synchrony emerges out of resonating forces that travel through a common physical medium – the body. In the current pre-registered study, we show that movements with relatively high physical impact affect phonation in a way that accommodates gesture-speech synchrony. Beating with one arm or two arms at a rhythmic pace led to acoustic peaks in the fundamental frequency (F0) and the amplitude envelope of phonation that were entrained with the rhythm of movement. Such effects were not found for upper limb movements with lower physical impetus (wrist movements), nor when participants were phonating without movement. We further provide evidence that postural stability is contributing to the effect of movement on phonation, as entrainment of movement and phonation was more pronounced when participants were standing as opposed to sitting. The current findings suggest that gesture-speech synchrony emerges from biomechanical constraints, potentially obviating the need for a cognitive predictive mechanism that ties gesture and speech in synchrony.

Acoustic specification of upper limb movement in voicing: Exploratory data report and pre-registration

“…Such peaks in phonation were observed at the moment at which posturally destabilizing forces of the arm movements were highest and at which the body counteracted such forces by tensioning of the muscles in anticipatory fashion. Indeed, we found that anticipatory postural adjustments played a role in movementphonation resonance, as peaks in phonation were more pronounced when participants were standing versus sitting (Pouw, Harrison, et al, 2018b). Previous research has found that, when standing, the forces produced during upper-limb movements are more destabilizing; as a result, thus the body must be more forcefully tensioned via the myofascial-skeletal network so as to keep postural integrity (Cordo & Nashner, 1982).…”

“…In the current paper, we will explore the possibility that visual information from gesture is but one of its (communicatively meaningful) products. Recently, we have found that upper-limb movements with relatively high physical impetus produce prominent but non-intentional changes in voice quality (Pouw, Harrison, & Dixon, 2018b). Specifically, we found peaks in the fundamental frequency (F0) and the amplitude envelope (energy in the signal) of continuous phonation of the vowel /a:/ when participants made high-impact movements that recruited the entire arm (see Figure 1) but not when producing low-4 impact wrist movements or when standing still.…”

Stabilizing Speech Production through Gesture-Speech Coordination

“…To provide a mechanism for self-entrainment, Pouw, Harrison, & Dixon (2018) found that when gestures have higher physical impetus on the body, peaks in the amplitude envelope and the fundamental frequency of phonation occur that increase over time. Such peaks are directly related to prosodic dimensions of speech and have been used as such in the study of the gesture and speech prosody correlation (for a review see Wagner et al, 2014).…”

Beat gestures influence which speech sounds you hear

“…In contrast to iconic gestures, beat gestures are not related to the spoken message on a semantic level, but only on a temporal level: their apex is aligned to vowel onset in lexically stressed syllables carrying prosodic emphasis 20,21,34,54 . Neurobiological studies suggest that listeners may tune oscillatory activity in the alpha and theta bands upon observing the initiation of a beat gesture Bosker & Peeters 6 to anticipate processing an assumedly important upcoming word 44 .…”

“…Bosker & Peeters 5 [19][20][21] . We manually point at things to guide the attention of our addressee to relevant aspects of our immediate environment [22][23][24][25] , enrich our speech with iconic gestures that in handshape and movement resemble what we are talking about [26][27][28] , and produce simple flicks of the hand aligned to the acoustic prominence of our spoken utterance to highlight relevant points in speech [29][30][31][32][33][34] . Over the past decades, it has become clear that such co-speech manual gestures are semantically aligned and temporally synchronized with the speech we produce 35-40 . It is an open question, however, whether the temporal synchrony between these hand gestures and the speech signal can actually influence which speech sounds we hear.…”