Hand-gestures are seamlessly coordinated with speech. Yet, there is only anecdotal support for gestures’ functional role in speech production. Here we explore temporal aspects of speech production when people use hand gesture. We performed exploratory analyses with a naturalistic German-speaking sample from The Bielefeld Speech and Gesture Alignment Corpus (SaGA), which consisted of 67 minutes of narration data and over 500 gesture events (N = 6). We found that the rhythmic timing of speech (defined as the mean and standard deviations of speech onset intervals) is highly correlated with the likelihood of gesturing. Furthermore, we utilized deep learning methods to track gesture motion, and extracted the amplitude envelope of speech, so as to gauge the degree of (continuous) gesture-speech synchrony. We then performed a continuous time-series analysis (recurrence quantification analysis; RQA) to index how temporal properties of speech change when gesture and speech are more or less synchronized. Our analyses revealed that when gesture and speech were more synchronized, the temporal structure of speech was more ordered and less complex, as indexed by classic measures of dynamic temporal stability (e.g., Entropy, Ratio of %Determinism/Recurrence). We suggest that a fundamental gesture-speech relation is rooted in entrainment, which yields stability in the temporal structure of speech.
Hand gestures during speech move in a common rhythm, as exemplified by the synchrony between prosodic contrasts in gesture movement (e.g., peak velocity) and speech (e.g., peaks in Fundamental Frequency). This joined rhythmic activity is hypothesized to have a variable set of functions, ranging from self-serving cognitive benefits for the gesturer, to communicational advantages that support listeners' understanding. However, gesturespeech synchrony has been invariably understood as a "neural-cognitive" achievement; i.e., gesture and speech are coupled through neural-cognitive mediation. Yet, it is possible that gesture-speech synchrony emerges out of resonating forces that travel through a common physical medium -the body. The current paper presents an exploratory study together with a pre-registration of a larger scale confirmatory study. We provide preliminary evidence that upper limb motions with greater momentum (i.e., physical impetus) affects Fundamental Frequency and the Amplitude Envelope of phonation in a way that accommodates research on gesture-speech synchrony. We speculate that anticipatory postural adjustments and related physical effects of upper limb movements on the musculoskeletal connective system could lead to changes in alveolar (lung) pressure that can impart (prosodic) contrasts in phonation. Here we pre-registered a confirmatory study to more comprehensively address this hypothesis.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.