Empirical Multimodality Research: The State of Play

Abstract

Multimodality research has always shown a strong reliance on data. However, the field has primarily developed around more exploratory, descriptive, and interpretative work on smaller data sets -as suggested by results we present from a meta-study of contributions to three multimodality-close international journals (Social Semiotics, Visual Communication, Multimodal Communication). Framed by a discussion of the qualitative-quantitative dichotomy and a deliberately broad working definition of empirical, we argue that it is not sample size or quantitative methods alone that support a more solid empirical grounding of multimodality research, but rather an explicit orientation to just how theory and analysis make contact with data. To this end, we propose five quality criteria of empirical practice, that is, completing the empirical feedback loop: from theory to data and back, ensuring objectivity, reliability, and validity in research, and acknowledging the inherent tentativeness of results. We thereby seek to chart paths for an appropriate and productive application of various empirical methods to novel (and supposedly familiar) forms of meaning-making in order to further strengthen the development of theory and methods in multimodality, and to encourage an even more intense exchange among the diverse communities of multimodalists.

Javanese Silly Gags on Daily Communication on Social Media: Pragmatic Meanings and Functions Approach

“…This research does not require the presence of quantitative computations because the aim is not to test the truth of the hypothesis with statistical calculations. Instead, its qualitative descriptive research focuses on describing research findings to be used as meaningful principles for further research (Pflaeging et al, 2021;Richards & Rodgers, 2006). The data of this research are excerpts of social media utterances that contain silly gags to describe the joke markers and the pragmatic meanings of the silly gags.…”

#socialiseresponsibly. Analyzing the Rhetorical Structure of Heineken TV Commercials During the Pandemic

“…In the following, we will direct this question to the corpus of Heineken commercials by using a multimodal annotation approach and by analyzing the discursive and rhetorical structure of these commercials from a more quantitatively-oriented perspective, i.e., in all 15 commercials of Heineken's initiative. For this, we orientate toward the general trend in multimodality research to achieve a stronger empirical foundation for the analysis of multimodal artifacts (see Bateman, 2013;Pflaeging et al, 2021) by applying the qualitative analysis of rhetorical relations as provided in Bateman's framework to our corpus and by describing patterns and systematic overlaps between the various commercials. We develop an annotation scheme for the description and annotation of both technical details such as camera perspective, for example, as well as the discourse analytical details of finding rhetorical relations between audiovisual units.…”

Introducing ethnohistorical research to multimodal studies

“…Over the past 12 years, there has been a growing call for interdisciplinary approaches to multimodality (Bezemer and Jewitt, 2010; O’Halloran and Smith, 2011; Machin, 2013; Bateman et al, 2017; Wildfeuer et al, 2019; Pflaeging et al, 2021). This call is in response to ideological critiques around text-oriented multimodal analysis, particularly its insufficient attention to the broader social practices, processes and people involved in the production or reception of texts (Machin, 2016; Hiippala, 2018; Aiello and Parry, 2019).…”