Entrainment and Modulation of Gesture–Speech Synchrony Under Delayed Auditory Feedback

Abstract

Gesture–speech synchrony re‐stabilizes when hand movement or speech is disrupted by a delayed feedback manipulation, suggesting strong bidirectional coupling between gesture and speech. Yet it has also been argued from case studies in perceptual–motor pathology that hand gestures are a special kind of action that does not require closed‐loop re‐afferent feedback to maintain synchrony with speech. In the current pre‐registered within‐subject study, we used motion tracking to conceptually replicate McNeill's ( 1992 ) classic study on gesture–speech synchrony under normal and 150 ms delayed auditory feedback of speech conditions ( NO DAF vs. DAF ). Consistent with, and extending McNeill's original results, we obtain evidence that (a) gesture‐speech synchrony is more stable under DAF versus NO DAF (i.e., increased coupling effect), (b) that gesture and speech variably entrain to the external auditory delay as indicated by a consistent shift in gesture‐speech synchrony offsets (i.e., entrainment effect), and (c) that the coupling effect and the entrainment effect are co‐dependent. We suggest, therefore, that gesture–speech synchrony provides a way for the cognitive system to stabilize rhythmic activity under interfering conditions.

Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles

“…Xiong et al, 2003) have reappeared in the gesture literature. These methods allow for the decomposition of gesture and speech time-series into several dominant time scales so as to further gauge on which temporal scales gesture and speech couple their activity (Danner et al, 2018;Pouw & Dixon, 2019). Such spectral decomposition approaches help quantify and correlate gesture and speech activity that happens on the syllable, clause, and sentence time scales.…”

“…DAF dataset. The first dataset is fully reported in Pouw and Dixon (2019), and concerns a within-subject kinematic study of gesture's coordination with speech when retelling a cartoon to the experimenter (who was not speaking back) during a delayed auditory feedback (DAF) manipulation of 140ms or retelling without such a manipulation (NO DAF). In this study, participants' hand movements were recorded with a Polhemus Liberty wired motion tracker.…”

The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

“…These gestures have more complex and often more fluid movement trajectories as they need to iconically present meaning. These gestures are often also coupled to speech prosody in a similar way as beat gestures (Wagner et al, 2014;Prieto, P., Cravotta, Kushch, Rohrer, & Vilà-Giménez, 2018), yet these gestures do seem to be more variably (less tightly) coupled with prosodic contrast in speech (Pouw & Dixon, 2018a, 2018b. This is possibly because iconic gestures have movement trajectories that are not recruited primarily to impart physical impetus on the body, rather some degree of freedom is reserved for iconic expression.…”

“…Hand gesture and speech are closely synchronized (see for an example https://osf.io/29h8z/; Leonard & Cummins, 2010;Krivokapić, Tiede, Tyrone, & Goldenberg, 2016;Krivokapić, Tiede, Tyrone, 2017;Pouw & Dixon, 2018a, b; Parrel, Goldstein, Lee, & Byrd, 2014;Rochet-Capellan, Shaiman, Iverson, & Szumisky, 2014;Treffner & Peter, 2002;Zelic, Kim, & Davis, 2015). Specifically, speech' prosodic contrasts, captured by contrasts in the fundamental frequency of speech (F0; perceived as the 'pitch' of speech), structurally aligns with energetic contrasts in gesture (e.g., peak velocity or point of maximum effort; Krivokapić, Tiede, & Tyrone, Goldenberg, 2016;Loehr, 2004;Pouw & Dixon, 2018a, 2018b.…”

“…They include arguments relating to communicative functions, such that the meaning of gesture and speech is less ambiguous (and optimally effective) when performed in synchrony (e.g., Krauss, 2000). Others suggest self-serving (cognitive) functions of gesture for the gesturer, such that gesturing allows for stabilizing the rhythm of speech, and vice versa (Pouw & Dixon, 2018b;Rusiewicz, 2011;Rusiewicz & Esteve-Gibert, 2018), or that gestures can stabilize imagination through extraneural bodily imaginings recruited in talking (Morsella & Krauss, 2004;Pouw & Hostetter, 2016). Another focus has been put on developmental origins of gesture-speech coupling, such that hand and mouth are solicited to interact from birth on (e.g., bringing food to the mouth and opening the mouth), which readies opportunities for increased entrainment of the manual and speech system during social development (Gentilucci & Corballis, 2006;Iverson & Thelen, 1999; see also Esteve-Gibert & Guellai, 2018).…”