Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human-driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science.
KEYWORDSlearning analytics, machine learning, multimodal data, multimodality, sensors, social signal processing
| INTRODUCTIONWith the rise of data-driven techniques to discover insights and generate predictions from the learning process such as learning analytics, the need for 360°data about learners has grown consistently. Combining data coming from multiple sources has become a prominent necessity in learning research and has led to an increased interest in multimodality and consequently into multimodal data analysis. To clarify the concept of multimodality, we use the definition provided by Nigay and Coutaz. The term "multi" refers to "more than one", whereas the term "modal" stands both for "modality" and for "mode". The modality is the type of communication channel used by two agents to convey and acquire information that defines the data exchange.The mode is the state that determines the context in which the information is interpreted (Nigay & Coutaz, 1993). The reasons why multimodality in learning is drawing so much attention can be summarized according to four developments.First of all, multimodality is a consolidated theory. It has been subjected of investigation already for two decades in different fields including functional linguistic, conversational analysis, and social semiotics (Jewitt, Bezemer, & O'Halloran, 2016). Research in multimodal interaction investigated how different modalities interact andThis is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.