Understanding conversational interaction in multiparty conversations: the EVA Corpus

Abstract

This paper focuses on gaining new knowledge through observation, qualitative analytics, and cross-modal fusion of rich multi-layered conversational features expressed during multiparty discourse. The outlined research stems from the theory that speech and co-speech gestures originate from the same representation; however, the representation is not solely limited to the speech production process. Thus, the nature of how information is conveyed by synchronously fusing speech and gestures must be investigated in detail. Therefore, this paper introduces an integrated annotation scheme and methodology which opens the opportunity to study verbal (i.e., speech) and non-verbal (i.e., visual cues with a communicative intent) components independently, however, still interconnected over a common timeline. To analyse this interaction between linguistic, paralinguistic, and non-verbal components in multiparty discourse and to help improve natural language generation in embodied conversational agents, a high-quality multimodal corpus, consisting of several annotation layers spanning syntax, POS, dialogue acts, discourse markers, sentiment, emotions, non-verbal behaviour, and gesture units was built and is represented in detail. It is the first of its kind for the Slovenian language. Moreover, detailed case studies show the tendency of metadiscourse to coincide with non-verbal behaviour of non-propositional origin. The case analysis further highlights how the newly created conversational model and the corresponding information-rich consistent corpus can be exploited to deepen the understanding of multiparty discourse.

A Roadmap for Technological Innovation in Multimodal Communication Research

“…As such, multimodal corpora often present a lot of variation. For instance, they may include spontaneous conversational speech or play (e.g., the Signes et Familles corpus, [121], or the EVA corpus [116]), recorded presentations (e.g., the M3D-TED corpus [165]) to structured task-based corpora (e.g., the SAGA corpus [106]) to a combination thereof (DGS [154]). For a discussion of the specificity of multimodal corpora, as well as a general overview of the goals of multimodal corpus linguistics, see [138].…”