Toward an infrastructure for data-driven multimodal communication research

Abstract

Research into the multimodal dimensions of human communication faces a set of distinctive methodological challenges. Collecting the datasets is resource-intensive, analysis often lacks peer validation, and the absence of shared datasets makes it difficult to develop standards. External validity is hampered by small datasets, yet large datasets are intractable. Red Hen Lab spearheads an international infrastructure for data-driven multimodal communication research, facilitating an integrated cross-disciplinary workflow. Linguists, communication scholars, statisticians, and computer scientists work together to develop research questions, annotate training sets, and develop pattern discovery and machine learning tools that handle vast collections of multimodal data, beyond the dreams of previous researchers. This infrastructure makes it possible for researchers at multiple sites to work in real-time in transdisciplinary teams. We review the vision, progress, and prospects of this research consortium.

Temporal Axiological Aspect Of Group Intercultural Communication Discourse

“…The pace of dialogue is further understood as the rate of actualization of dialogue acts, that is, the number of dialogue acts per unit of time. It significantly depends on the properties of dialogue acts (Steen et al, 2018). The concept of dialogue is defined as a set of topics actualized by it in the process of communication, and the concept of a cross-section of the discursive flow reflecting the group properties of dialogues is introduced, which will be discussed below.…”

“…Based on the proposed model dialogue group discourse contains the situation of two-way translation and transport of the episodes of discourse is analyzed. In group discourse each communicant becomes the addresser after a dialogue with the current sender, then communicant transfers and reports the received contents to the following communicants (Steen et al, 2018). The assessment of the dynamic situation of the intercultural communication is considered on the basis of a discrete model of group discourse (Fig.…”

Temporal Expressions in English and Spanish: Influence of Typology and Metaphorical Construal

“…In Example 10, the duration of the temporal event, the film, is expressed by pinpointing the moments at which the event started and finished. This is made explicit by the demarcative temporal linguistic structure from start to finish, which is often accompanied by a co-speech gesture that signals two sections of a timeline (Steen et al, 2018;Valenzuela et al, 2020). The mental representation of this temporal structure might resemble Figure 1.…”

“…Demarcative expressions express temporal duration by indicating the time-span that took place within two points in time. The [from X to Y] structure is often used to express the beginning and the end of temporal events (Steen et al, 2018). The temporal structures that were searched in the corpora are: from start to finish, from beginning to end, from start to end, from genesis to revelation, from inception to completion and less idiomatic cases, such as from beginning to finish.…”

Distant viewing and multimodality theory: Prospects and challenges

“…Bateman et al (2016), in turn, show how computer vision can impose structure on filmic media by establishing visual perceptual units using shot boundary detection algorithms, and how face recognition algorithms allow tracking the appearance of characters across these shots. Similar methodologies are now developed collaboratively for audiovisual media and embodied communication as a part of the Red Hen Lab consortium (Steen et al, 2018). O'Halloran et al (2018: 12) propose to anchor computational methods more strongly to theories of multimodal discourse analysis within a 'multimodal mixed methods research framework', which has guided the subsequent development of the Multimodal Analysis Platform (MAS; O'Halloran et al, 2021).…”