Gesture-speech physics in fluent speech and rhythmic upper limb movements

Abstract

Communicative hand gestures are often coordinated with prosodic aspects of speech, and salient moments of gestural movement (e.g., quick changes in speed) often co-occur with salient moments in speech (e.g., near peaks in fundamental frequency and intensity). A common understanding is that such gesture and speech coordination is culturally and cognitively acquired, rather than having a biological basis. Recently, however, the biomechanical physical coupling of arm movements to speech movements has been identified as a potentially important factor in understanding the emergence of gesture-speech coordination. Specifically, in the case of steady-state vocalization and mono-syllable utterances, forces produced during gesturing are transferred onto the tensioned body, leading to changes in respiratory-related activity and thereby affecting vocalization F0 and intensity. In the current experiment (N = 37), we extend this previous line of work to show that gesture-speech physics impacts fluent speech, too. Compared with non-movement, participants who are producing fluent self-formulated speech, while rhythmically moving their limbs, demonstrate heightened F0 and amplitude envelope, and such effects are more pronounced for higher-impulse arm versus lower-impulse wrist movement. We replicate that acoustic peaks arise especially during moments of peak-impulse (i.e., the beat) of the movement, namely around deceleration phases of the movement. Finally, higher deceleration rates of higher-mass arm movements were related to higher peaks in acoustics. These results confirm a role for physical-impulses of gesture affecting the speech system. We discuss the implications of gesture-speech physics for understanding of the emergence of communicative gesture, both ontogenetically and phylogenetically.

Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

“…That such DNN's can reconstruct gesture kinematics based on speech acoustics alone, suggests that there must be a tight link between acoustics and manual movement that are still to be fully appreciated and understood by gesture-speech researchers. However, we acknowledge that current research provides no evidence that current effects can generalize to fluent speech (but see Pouw, de Jonge-Hoekstra, Paxton, Harrison, Dixon, 2020).…”

“…For ontogeny, research could focus on the role of physical impulses during vocal-motor babbling in infants (e.g., Ejiri, 1998). In extension of the current line of work, much more is needed to understand how gesture-speech physics plays out in more naturalistic speech (Cravotta et al, 2019;Pouw, de Jonge-Hoekstra et al, 2020). In gesture studies, identification of possible different functions that gestures may take in discourse can be improved by taking into account whether such gestures impart a physical impulse on the body (Cooperrider, 2018).…”

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab

“…Indeed, in a study by Perlman, Dale, and Lupyan (2015), it is shown how dynamic aspects of vocalization signaling systems become more efficient, similar to our current reduction in kinematic complexity. These findings, together with work showing the tight connection between speech and gesture (Bosker & Peeters, 2020; Pouw, de Jonge‐Hoekstra, Harrison, Paxton, & Dixon, 2020; Pouw, Harrison, Esteve‐Gibert, & Dixon, 2020), make it a natural next step to look at multimodal iterated learning experiments. Furthermore, our approach can inform work on communicative alignment in conversations (Rasenberg, Özyürek, & Dingemanse, 2020) or the ways in which people can repeat aspects of each other's communicative behavior.…”

The importance of visual control and biomechanics in the regulation of gesture-speech synchrony for an individual deprived of proprioceptive feedback of body position

“…Given that moments of peak impulse in a gesture have recently been shown to be associated with higher acoustic outputs [30], [31], [33], and that the physical impulse of a gesture increases with limb deceleration and the mass of the effector(s), we predict that prosodic markers of speech (F0 and amplitude envelope) will depend upon: 1) gesture deceleration magnitude, and 2) whether gesturing was unimanual or bimanual.…”

“…We therefore explore an alternative explanation of IW's gesture-speech abilities, similarly grounded in the idea that haptic perception and action exploits dynamic perturbations of movements (p. 342, [27]). Recent research in this vein has shown that physical impulses of moving one's hand or arm during a gesture is mechanically transferred onto the respiratory system [28], [29] and can thereby directly affect the Fundamental Frequency (F0; perceived as pitch) and the amplitude envelope of the concurrent vocalization [30]- [33]. F0 and amplitude are key acoustic markers of prosodic stress on a particular syllable.…”