The paper discusses the problem-oriented approach to the multimodal communication as it is expounded by John Bateman, Janina Wildfeuer and Tuomo Hiippala. The authors claim for the productivity of the approach to the analysis of the contemporary informational communicational system in terms of developing media, i.e. smart- phones, computer games, 3D cinema, VR and AR technologies, etc., which are multimodal by their nature and use. The basic notions associated with multimodality include mode, communication situation, ergodic text with its features of mutability and temporality. The main characteristics of communication situations as an item of the multimodal analysis is the source of information, recipients, conditions of communication and its duration. While interacting, the communication situations bear linear, micro-ergodic, non-mutable ergodic and mutable ergodic types depending on the prevailing way of perception. The authors claim that the most representative results of research of “vague” and unpredictable multimodal objects could be gained by the mixt of quantitative and qualitative methods, including corpus analysis, eye-tracking technics, and computing methods.

The paper discusses several advantages of multimodal approach to the analysis of nature and specifics of media audience’s reception of texts. The authors base their ideas on the principally multimodal nature of media communication, which reflects in the way the recipients interpret texts. Thus, various factors impact on the interpretation of the initialized text. Moreover, such factors predominately include other semiotic resources that are not formally affiliated with the basic text. In the research, the hypothesis was about the impact of verbal comments on interpretation of visual objects by recipients. According to this goal, the authors selected two groups of respondents — a control and an experimental one, which were offered a photograph and were asked to describe it verbally in a free manner. The experimental group of respondents was offered the same photograph with motivating verbal comment, including the general information about the origins and topics of the photograph. The authors compared the results of the both groups descriptions by matching verbalized categories of representation, individual evaluations of the photograph and “the syntax” of verbal description of the visual media text. The research results proved the general supposition: the multimodal approach, being aimed at finding, describing and explicating the meaning effects of semiotic ensembles, contributes to understanding the features of interpretation of visual objects under the influence of other semiotic (here — verbal) resources. Thus, the results show that the interpretation of visual objects is motivated by the verbal comment: it topicalizes and contextualizes the visual reception and interpretation of the basic message. In the conclusion, the authors define the perspectives both of the research and the multimodal approach to media texts study.