The Bursts and Lulls of Multimodal Interaction: Temporal Distributions of Behavior Reveal Differences Between Verbal and Non‐Verbal Communication

Abstract

Recent studies of naturalistic face-to-face communication have demonstrated coordination patterns such as the temporal matching of verbal and non-verbal behavior, which provides evidence for the proposal that verbal and non-verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non-verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, ), we focus on measuring the temporal patterns of specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non-verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non-verbal channels. We propose a "temporal heterogeneity" hypothesis to explain how the language system adapts to the demands of dialog.

Rhythmic Features of Movement Synchrony for Bonding Individuals in Dyadic Interaction

“…Psycholinguistic research has associated on-off vocal activities in dyadic faceto-face interactions with slow-tempo rhythmic cycles (e.g., 150-300 s per cycle; Warner 1992a), whereby a 200-s cycle typically comprises about 100 s of speaking (long or frequent vocalizations), followed by about 100 s of listening (long or frequent pauses) (Warner 1979). Other studies have focused on the temporal characteristics of speech in examining speech rates (e.g., Apple et al 1979), multi-scale clustering (Abney et al 2015), and interval distribution (Abney et al 2018). Recent studies have examined the ways that rhythmic and temporal structures of vocal communication converge to facilitate pro-social behavior (Manson et al 2013).…”

“…Future research could also investigate the relationship between verbal and nonverbal signals from the perspective of synchrony. A recent study suggested that the temporal heterogeneity hypothesis, that is, the temporal distributions (burstiness) of verbal and nonverbal behaviors are different (Abney et al 2018). Language has a hierarchical nested structure, consisting of phonemes, syllables, words, phrases, and sentences, which presents a certain temporal scale pattern (Abney et al 2015).…”

Finding Structure in Time: Visualizing and Analyzing Behavioral Time Series

“…This limitation suggests that the user should use caution when directly comparing B estimates across datasets. One strategy that has been used in recent research that applied the burstiness analysis to multimodal human interaction (Abney et al, 2018), was to generate bootstrapped confidence intervals to determine categorical boundaries of periodic, random, and bursty temporal structure. For example, a Poisson process is generated by an interevent interval distribution of an exponential distribution.…”

Easier Said Than Done? Task Difficulty's Influence on Temporal Alignment, Semantic Similarity, and Complexity Matching Between Gestures and Speech

“…Bursty processes are typical for complex dynamical systems (Barabási, 2005;Karsai, Kaski, Barabási, & Kertész, 2012), and in this sense, burstiness shares similarities with multifractality (albeit the scope of burstiness analysis is not multi-scaled). The methods used by Abney and colleagues (Abney et al, 2014(Abney et al, , 2018Fusaroli et al, 2013) provide viable directions for investigating complexity matching between gestures and speech in more typical and fluent speaking and gesturing.…”