Quantitative analysis of multimodal speech data

Abstract

This study presents techniques for quantitatively analyzing coordination and kinematics in multimodal speech using video, audio and electromagnetic articulography (EMA) data. Multimodal speech research has flourished due to recent improvements in technology, yet gesture detection/annotation strategies vary widely, leading to difficulty in generalizing across studies and in advancing this field of research. We describe how FlowAnalyzer software can be used to extract kinematic signals from basic video recordings; and we apply a technique, derived from speech kinematic research, to detect bodily gestures in these kinematic signals. We investigate whether kinematic characteristics of multimodal speech differ dependent on communicative context, and we find that these contexts can be distinguished quantitatively, suggesting a way to improve and standardize existing gesture identification/annotation strategy. We also discuss a method, Correlation Map Analysis (CMA), for quantifying the relationship between speech and bodily gesture kinematics over time. We describe potential applications of CMA to multimodal speech research, such as describing characteristics of speech-gesture coordination in different communicative contexts. The use of the techniques presented here can improve and advance multimodal speech and gesture research by applying quantitative methods in the detection and description of multimodal speech.

Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles

“…Xiong et al, 2003) have reappeared in the gesture literature. These methods allow for the decomposition of gesture and speech time-series into several dominant time scales so as to further gauge on which temporal scales gesture and speech couple their activity (Danner et al, 2018;Pouw & Dixon, 2019). Such spectral decomposition approaches help quantify and correlate gesture and speech activity that happens on the syllable, clause, and sentence time scales.…”

“…In the current paper, we aim to innovate the quantitative study of gesture ensembles. We hope therefore to fortify ongoing innovations in the quantitative study of gesture kinematics together with speech dynamics (Alviar, Dale, & Galati, 2019;Danner, Barbosa, & Goldstein, 2018;Pouw, Trujillo, & Dixon, in press) by enriching the investigation into how gestural low-level events feed into higher-level linguistic structures (Krivokapić, 2014;Ravignani et al, 2018;Shattuck-Hufnagel & Ren, 2018).…”

Entrainment and Modulation of Gesture-Speech Synchrony under Delayed Auditory Feedback

“…This coordination is also observed on a prosody level, such that the intensity changes of gesture and speech co-vary. Indeed research combining motion-tracking methodology and speech analysis is beginning to confirm that the energetic patterns of gesture and speech are flexibly and tightly synchronized (Chu & Hagoort, 2014, under review;Danner, 2017;Danner, Barbosa, Goldstein, 2018;Leonard & Cummins, 2010;Krivokapić, Tiede, Tyrone, & Goldenberg, 2016;Krivokapic, Tiede, Tyrone, 2017;Pouw & Dixon, 2018a;Parrel, Goldstein, Lee, & Byrd, 2014;Rochet-Capellan, Laboissier, Galvan, & Schwartz, 2008;Rusiewicz, Shaiman, Iverson, & Szumisky, 2014;Treffner & Peter, 2002;Zelic, Kim, & Davis, 2015). The general finding from these studies is that energetic contrasts in gesture (e.g., gesture end-point; peak velocity) are structurally related to prosodic contrasts (e.g., peak pitch; stressed syllable) which are the energetic contrasts in speech.…”

“…The general finding from these studies is that energetic contrasts in gesture (e.g., gesture end-point; peak velocity) are structurally related to prosodic contrasts (e.g., peak pitch; stressed syllable) which are the energetic contrasts in speech. Although evidence for entrained gesture-speech rhythms are largely based on repetitive pointing-, beat-or finger tapping gestures, where speech and gesture are scripted and produced on command (Danner et al, 2018;Pouw & Dixon, 2018a), the findings appear generalizable to more spontaneous and semantically rich gestures on the basis of careful (but subjective) analysis of video recordings (e.g., McClave, 1994;McNeill, 1992McNeill, , 2005Loehr, 2004Loehr, , 2012; for an overview see Wagner, Malisz, Kopp, 2014). Given this growing evidence for the entrainment of gesture-speech rhythms, the question arises of how and why gestures are so closely controlled with respect to the rhythm of speech (Esteve-Gibert & Guellai, 2018;Iverson & Thelen, 1999;Rusciewicz, 2011;Rusiewicz & Esteve-Gibert, 2018;Wagner et al, 2014).…”

Energy flows in gesture-speech physics: Exploratory findings and pre-registration of confirmatory analysis

“…When quantifying gesture-speech synchrony on the side of speech, researchers have particularly looked at peaks in Fundamental Frequency (F0; perceived as pitch) and intensity contours (Krivokapic, Tiede, Tyrone, & Goldenberg, 2016;Parrell, Goldstein, Lee, & Byrd, 2014;. With respect to gesture, researchers have defined salient peaks in kinematics of gestural motions in terms of, for example, peak velocity or onsets of a gestural stroke (Danner, Barbosa, & Goldstein, 2018;Danner et al, 2018;Loehr, 2012;Rochet-Capellan, Laboissière, Galván, & Jean-Luc, 2008;Wagner et al, 2014). This line of research has shown that gesture and speech are tightly and flexibly coupled with one another, raising the question how such strong coupling is established, developmentally as well as cognitively.…”