No abstract

Communicative hand gesticulations are tightly coupled to prosodic aspects of speech of speech. Psychologists have characterized this multimodal synchrony as a preplanning process governed by a cognitive timing mechanism acquired only later in development. However, it has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impetus on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) uttered consonant-vowel CV (/pa/) mono-syllables in rhythmic fashion while moving the upper limbs (or not). We show that respiration-related movement is affected by (especially high-impetus) gesturing when vocalizations occur near peaks in physical impetus. We further show that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, we find tight relations between respiration-related movement and vocalization, even in the absence of movement, and even more strong respiration-acoustic relations are found when upper-limb movement is present. The current findings expand a developing line of research showing that speech acoustics is modulated by functional biomechanical linkages between hand gesture and the respiratory system.

No abstract

Hand gestures and speech move in a common rhythm, as exemplified by the synchrony between prosodic contrasts in gesture movement (e.g., peak velocity; maximum effort) and speech (e.g., peaks in fundamental frequency; F0). This joined rhythmic activity is hypothesized to have a variable set of functions, ranging from self-serving cognitive benefits for the gesturer, to communicational advantages that support listeners’ understanding. However, gesture-speech synchrony has been invariably understood as a “neural-cognitive” achievement; i.e., gesture and speech are coupled through neural-cognitive mediation. Yet, it is possible that gesture-speech synchrony emerges out of resonating forces that travel through a common physical medium – the body. In the current pre-registered study, we show that movements with relatively high physical impact affect phonation in a way that accommodates gesture-speech synchrony. Beating with one arm or two arms at a rhythmic pace led to acoustic peaks in the fundamental frequency (F0) and the amplitude envelope of phonation that were entrained with the rhythm of movement. Such effects were not found for upper limb movements with lower physical impetus (wrist movements), nor when participants were phonating without movement. We further provide evidence that postural stability is contributing to the effect of movement on phonation, as entrainment of movement and phonation was more pronounced when participants were standing as opposed to sitting. The current findings suggest that gesture-speech synchrony emerges from biomechanical constraints, potentially obviating the need for a cognitive predictive mechanism that ties gesture and speech in synchrony.

No abstract