Complex Communication Dynamics: Exploring the Structure of an Academic Talk

Abstract

Communication is a multimodal phenomenon. The cognitive mechanisms supporting it are still understudied. We explored a natural dataset of academic lectures to determine how communication modalities are used and coordinated during the presentation of complex information. Using automated and semi-automated techniques, we extracted and analyzed, from the videos of 30 speakers, measures capturing the dynamics of their body movement, their slide change rate, and various aspects of their speech (speech rate, articulation rate, fundamental frequency, and intensity). There were consistent but statistically subtle patterns in the use of speech rate, articulation rate, intensity, and body motion across the presentation. Principal component analysis also revealed patterns of system-like covariation among modalities. These findings, although tentative, do suggest that the cognitive system is integrating body, slides, and speech in a coordinated manner during natural language use. Further research is needed to clarify the specific coordination patterns that occur between the different modalities.

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

“…Overall, pixel differentiation has been shown to provide a reliable measure of movement (Paxton & Dale, 2013;Romero et al, 2017) and can be used to capture movement in specific areas of the visual scene (Alviar et al, 2019;Danner et al, 2018). Note, however, that this method is particularly vulnerable to changes in background, such as movement or changes in lighting, and may not be able to capture smaller movements or movements toward the camera.…”

“…The study of the temporal dynamics of gesture-speech coordination has relatively lagged behind in use of kinematic measurement methods, especially as compared to the degree to which state-of-the-art (psycho)linguistic methods are employed for the study of speech (e.g., Loehr, 2012;Shattuck-Hufnagel & Ren, 2018). This manifests itself in the relative scarcity (as compared to other research on instrumental action) of published studies that have applied motion tracking in gesture-speech research (Alexanderson, House, & Beskow, 2013;Alviar, Dale, & Galati, 2019;Chu & Hagoort, 2014;Danner, Barbosa, & Goldstein, 2018;Ishi, Ishiguro, & Hagita, 2014;Leonard & Cummins, 2010;Krivokapic, Tiede, & Tyrone, 2017;Krivokapić, Tiede, Tyrone, & Goldenberg, 2016;Parrell, Goldstein, Lee, & Byrd, 2014;Pouw & Dixon, 2019a;Quek et al, 2002;Rochet-Capellan et al, 2008;Rusiewicz et al, 2014;Treffner & Peter, 2002;Zelic, Kim, & Davis, 2015). It can be argued that the absence of motion tracking in the standard methodological toolkit of the multimodal language researcher has further led to imprecisions and conceptual confusions.…”

“…The further positive implication of using quantitative methods of gesture-speech coordination is that new structured behaviors can be uncovered with multimodal tracking techniques that are otherwise simply hidden from view. For example, Alviar et al (2019) have recently found that lecturing speakers organize their speech and movement in relation to the changing environment (e.g., changing slides) as a single system, in a way that allows for statistical dimensional reduction of its overall activity. For their analyses, it was essential to have continuous synchronized data of speech; such data are impossible or otherwise impractical to acquire without quantitative methods.…”

Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles

“…In the current paper, we aim to innovate the quantitative study of gesture ensembles. We hope therefore to fortify ongoing innovations in the quantitative study of gesture kinematics together with speech dynamics (Alviar, Dale, & Galati, 2019;Danner, Barbosa, & Goldstein, 2018;Pouw, Trujillo, & Dixon, in press) by enriching the investigation into how gestural low-level events feed into higher-level linguistic structures (Krivokapić, 2014;Ravignani et al, 2018;Shattuck-Hufnagel & Ren, 2018).…”

¿Cómo se comunica la investigación a través de la pantalla? Análisis del discurso multimodal en las comunicaciones en congresos digitales en el área de comunicación

“…Dado que la notación a mano de los gestos consume mucho tiempo(Ienaga et al, 2022), actualmente se están realizando esfuerzos por desarrollar herramientas fáciles de aplicar para realizar lecturas automáticas de los gestos. Estudios como los deAlviar et al (2019), Pouw y Dixon (2019) o Krivokapié et al (2017) se han centrado en la medición del movimiento. Ienaga et al (2022) han realizado automatizaciones parciales en el análisis de los gestos, pero la notación a mano todavía es necesaria.En resumen, la automatización del estudio del gesto está creciendo con fuerza, aunque todavía es necesario la anotación manual.…”