Intrapersonal dependencies in multimodal behavior

Abstract

Human interlocutors automatically adapt verbal and non-verbal signals so that different behaviors become synchronized over time. Multimodal communication comes naturally to humans, while this is not the case for Embodied Conversational Agents (ECAs). Knowing which behavioral channels synchronize within and across speakers and how they align seems critical in the development of ECAs. Yet, there exists little data-driven research that provides guidelines for the synchronization of different channels within an interlocutor. This study focuses on intrapersonal dependencies of multimodal behavior by using cross-recurrence analysis on a multimodal communication dataset to better understand the temporal relationships between language and gestural behavior channels. By shedding light on the intrapersonal synchronization of communicative channels in humans, we provide an initial manual for modality synchronisation in ECAs. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI; • Computing methodologies → Discourse, dialogue and pragmatics.

Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents

“…The integration of verbal and non-verbal channels is also nontrivial. Previous research has examined the integration of verbal and non-verbal behaviors by using cross-recurrence analysis techniques, both at inter-personal and intra-personal levels [2,10]. Such findings can guide the coupling of verbal and non-verbal behaviors.…”