The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

Abstract

There is increasing evidence that hand gestures and speech synchronize their activity on multiple dimensions and timescales. For example, gesture's kinematic peaks (e.g., maximum speed) are coupled with prosodic markers in speech. Such coupling operates on very short timescales at the level of syllables (200 ms), and therefore requires high-resolution measurement of gesture kinematics and speech acoustics. High-resolution speech analysis is common for gesture studies, given that field's classic ties with (psycho)linguistics. However, the field has lagged behind in the objective study of gesture kinematics (e.g., as compared to research on instrumental action). Often kinematic peaks in gesture are measured by eye, where a "moment of maximum effort" is determined by several raters. In the present article, we provide a tutorial on more efficient methods to quantify the temporal properties of gesture kinematics, in which we focus on common challenges and possible solutions that come with the complexities of studying multimodal language. We further introduce and compare, using an actual gesture dataset (392 gesture events), the performance of two video-based motion-tracking methods (deep learning vs. pixel change) against a high-performance wired motion-tracking system (Polhemus Liberty). We show that the videography methods perform well in the temporal estimation of kinematic peaks, and thus provide a cheap alternative to expensive motion-tracking systems. We hope that the present article incites gesture researchers to embark on the widespread objective study of gesture kinematics and their relation to speech. Keywords Motion tracking. Video recording. Deep learning. Gesture and speech analysis. Multimodal language There is an increasing interest in the ways that cooccurring hand gestures and speech coordinate their activity (Esteve-Gibert & Guellaï, 2018; Wagner, Malisz, & Kopp, 2014). The nature of this coordination is exquisitely complex, as it operates on multiple levels and timescales. For example, on the semantic level, referential

'Tiny numbers' are actually tiny: Evidence from gestures in the TV News Archive

“…The TV News Archive also offers the advantage that it enables the collection of far more data, especially a higher number of speakers, than would usually be feasible in an experimental setting. Moreover, if time-consuming stages of the analysis procedure could be automated, such as the initial exclusion of unusable videos, still larger databases could be constructed (see [66]), and large portions of the analysis itself could be automated using the Python-operated OpenPose software [67,68] (see [69,70]). Finally, the data in the Archive are openly accessible, allowing completely reproducible analyses to be conducted.…”

The importance of visual control and biomechanics in the regulation of gesture-speech synchrony for an individual deprived of proprioceptive feedback of body position

“…Active exploratory movements generated forces that affected both the insensate peripheral tissues of the hands and arms, as well as sensorily intact tissues of the trunk and neck. The patterns of tissue deformation generated in the trunk and neck are structured by the physical act of the limb wielding the object, therefore carrying information about the properties of the object being wielded [19,21]. This suggests that dynamic perturbations of movements can be exploited in haptic perception, and similarly dynamic resources could be available to IW in orchestrating gesture and speech.…”

“…In the current study, we reanalyze data from the 1998 and 2002 experiments with modern quantitative analyses of gesture kinematics and speech acoustics that were not available at the time of the original experiments. We extract the 2D body-movement trajectories from the video materials using OpenPose [34] in conjunction with acoustic analyses [19], see Figure 1.…”

“…We then used the time-series from the motion tracking as a visual aid to determine the start of the gesture and the end of a gesture. Similar to other motion-tracking studies on gesture [16], [19], we annotated the start of gesture from the point at which the gesture initiated its main meaningful stroke [39] to the point wherein it reaches a halt. Thus, our gesture annotations do not include a pre-stroke initiation that transports the hand from a resting position into a gesture space, nor a post-stroke hold which keeps the gesture motionless while the relevant speech is completing, nor a retraction phase which transports the hand back to a resting position after the main stroke.…”

“…However, it is possible that patterns difficult to catch by eye [1]- [4], [11] might be revealed by analysis of the micro-scale structure of movement and acoustics. A more micro-scale approach is especially appropriate for the phenomena of gesture-speech synchrony because they couple their activity on the relatively fast time-scales of stressed syllables (200 milliseconds) [16], [19].…”