Towards next-generation visual archives:image, film and discourse

Systemic Functional Linguistics and Computation

“…As the work of O'Halloran and her colleagues shows, rapidly developing fields of study such as computer vision, natural language processing and machine learning will undoubtedly make a significant contribution to the study of multimodality in the coming years. The work of Bateman et al (2016) exemplifies emerging work in this area, combining manually and automatically generated annotation layers in a corpus describing the multimodality of film. Whereas various visual and aural events in film, such as shot boundaries and background music, are captured automatically by algorithms, filmic cohesion is described manually using the framework set out in (Tseng 2013).…”

“…Whereas various visual and aural events in film, such as shot boundaries and background music, are captured automatically by algorithms, filmic cohesion is described manually using the framework set out in (Tseng 2013). Bateman et al (2016) show that automatically generated annotation not only reduces the time and resources spent on compiling multimodal corpora, but also extends their scope by introducing layers of description which could be otherwise considered too demanding for manual annotation. Moreover, these benefits are not limited to the description of complex dynamic multimodal phenomena in film, but apply to page-based artifacts as well, whose annotation has proven equally time-and resource-intensive.…”

The Communicative Effectiveness of Education Videos: Towards an Empirically-Motivated Multimodal Account

“…Transcription of this kind is thus of necessity an interpretation on the part of the analyst, albeit one that leads directly to empirical questions for its validation due to the close linking maintained with the material forms present in the data. Automatic recognition and segmentation of particular aspects of rich data of this kind is consequently also an active area of research (cf., e.g., [34]).…”

“…This substantially complicates the task of performing effective empirical reception studies. Discussions of multimodally complex interactions and presentations vacillate on such basic issues as to whether modes are to be characterized in terms of sensory channels or in terms of presentational forms and adopted models typically assume some compromise position where both play a role ( [5], [34][35]. Further issues such as limited capacity processing and different kinds of memory and representations certainly require that perceptual properties of the understanding process receive adequate attention; however, the now generally accepted assumption that perception is an active, hypothesis-driven process in its own right demands equally that the sources of such hypotheses be characterized more effectively.…”

Distant viewing and multimodality theory: Prospects and challenges

“…To exemplify, Hiippala (2016) applies computer vision algorithms to digital images of page-based media to generate annotations that conform to the schema defined in the Genre and Multimodality framework (Bateman, 2008), but the output nevertheless requires a considerable degree of human post-processing (see also Thomas et al, 2010). Bateman et al (2016), in turn, show how computer vision can impose structure on filmic media by establishing visual perceptual units using shot boundary detection algorithms, and how face recognition algorithms allow tracking the appearance of characters across these shots. Similar methodologies are now developed collaboratively for audiovisual media and embodied communication as a part of the Red Hen Lab consortium (Steen et al, 2018).…”