Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

Abstract

Communicative hand gesticulations are tightly coupled to prosodic aspects of speech of speech. Psychologists have characterized this multimodal synchrony as a preplanning process governed by a cognitive timing mechanism acquired only later in development. However, it has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impetus on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) uttered consonant-vowel CV (/pa/) mono-syllables in rhythmic fashion while moving the upper limbs (or not). We show that respiration-related movement is affected by (especially high-impetus) gesturing when vocalizations occur near peaks in physical impetus. We further show that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, we find tight relations between respiration-related movement and vocalization, even in the absence of movement, and even more strong respiration-acoustic relations are found when upper-limb movement is present. The current findings expand a developing line of research showing that speech acoustics is modulated by functional biomechanical linkages between hand gesture and the respiratory system.

Acoustic information about upper limb movement in voicing

“…(D) Here we assessed how the fundamental frequency of voicing (in the human range: 75 to 450 Hz) was modulated around the maximum extension for each vocalizer and combined for all vocalizers (red line). D shows that smoothed-average-normalized F0 (also linearly detrended and z-scaled per vocalization trial) peaked around the moment of the maximum extension, when a sudden deceleration and acceleration occurred; normalized F0 dipped at steady-state low-physical-impetus moments of the movement phase (when velocity was constant), rising again for a maximum flexion (∼300 to 375 ms before and after the maximum extension), replicating previous work (21,24). Vocalizer wrist movement showed a less pronounced F0 modulation compared to the vocalizer arm movement trials.…”

Gesture-speech physics in fluent speech and rhythmic upper limb movements

“…For F0, the pattern is somewhat less clear, but positive peaks still occur just before the maximum extension. These findings replicate our earlier work on steadystate vocalization and mono-syllabic utterances, showing that moments of peak deceleration also show peaks in acoustics 34,37 .…”

“…Recent work, however, has investigated a potential physical coupling of arm movements with speech via myofascial-tissue biomechanics. This works shows that hand gesturing physically impacts steady-state vocalizations and mono-syllabic consonant-vowel utterances [34][35][36][37] . Specifically, hand and arm movements can transfer a force (a physical impulse) onto the musculoskeletal system, thereby modulating respiration-related muscle activity, leading to changes in vocalization's intensity.…”

“…However, a complete understanding will involve an appreciation of the body as a pre-stressed system 39,40 , forming an interconnected tensioned network of compressive (e.g., bones) and tensile elements (e.g., fascia, muscles) through which forces may reverberate nonlinearly 41,42 . Specifically, the upper limb movements are controlled by stabilizing musculoskeletal actions of the scapula and shoulder joint, which directly implicate accessory expiratory muscles that also stabilize scapula and shoulder joint actions (e.g., the serratus anterior inferior; see 37 for an overview).…”

“…We reasoned that this is because standing involves more forceful anticipatory postural counter adjustments 46 , which reach the respiratory system via accessory expiratory muscles also implicated in keeping postural integrity (see also 44,45 ). Recently, more direct evidence has been found for the gesturerespiratory-speech link: Respiratory-related activity (measured with a respiratory belt) was enhanced during moments of peak-impetus of gesture as opposed to other phases in the gesture movement, and respiratory-related activity itself was predictive of the gesture-related intensity modulations of mono-syllable utterances 37 .…”