Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

Abstract

Students' on‐task engagement during adaptive learning activities has a significant effect on their performance, and at the same time, how these activities influence students' behavior is reflected in their effort exertion. Capturing and explaining effortful (or effortless) behavior and aligning it with learning performance within contemporary adaptive learning environments, holds the promise to timely provide proactive and actionable feedback to students. Using sophisticated machine learning (ML) algorithms and rich learner data, facilitates inference‐making about several behavioral aspects (including effortful behavior) and about predicting learning performance, in any learning context. Researchers have been using ML methods in a “black‐box” approach, ie, as a tool where the input data is the learner data and the output is a given class from the chosen construct. This work proposes a methodological shift from the “black‐box” approach to a “grey‐box” approach that bridges the hypothesis/literature‐driven (feature extraction) “white‐box” approach with the computation/data‐driven (feature fusion) “black‐box” approach. This will allow us to utilize data features that are educationally and contextually meaningful. This paper aims to extend current methodological paradigms, and puts into practice the proposed approach in an adaptive self‐assessment case study taking advantage of new, cutting‐edge, interdisciplinary work on building pipelines for educational data, using innovative tools and techniques. What is already known about this topic Capturing and measuring learners' engagement and behavior using physiological data has been explored during the last years and exhibits great potential. Effortless behavioral patterns commonly exhibited by learners, such as “cheating,” “guessing” or “gaming the system” counterfeit the learning outcome. Multimodal data can accurately predict learning engagement, performance and processes. What this paper adds Generalizes a methodology for building machine learning pipelines for multimodal educational data, using a modularized approach, namely the “grey‐box” approach. Showcases that fusion of eye‐tracking, facial expressions and arousal data provide the best prediction of effort and performance in adaptive learning settings. Highlights the importance of fusing data from different channels to obtain the most suited combinations from the different multimodal data streams, to predict and explain effort and performance in terms of pervasiveness, mobility and ubiquity. Implications for practice and/or policy Learning analytics researchers shall be able to use an innovative methodological approach, namely the “grey‐box,” to build machine learning pipelines from multimodal data, taking advantage of artificial intelligence capabilities in any educational context. Learning design professionals shall have the opportunity to fuse specific features of the multimodal data to drive the interpretation of learning outcomes in terms of physiological learner states. The constraints from th...

Multimodal Learning Analytics research with young children: A systematic review

“…The term “Performance Analytics” includes data on the child’s performance and the countable product of their learning. Some examples are completion time, achievement or otherwise (1 or 0) of the assignment, speed of task resolution, the number of attempts or failures and the level of complexity of the problem‐solving process (Diago, González‐Calero & Arnau, 2019; Kosmas, Ioannou & Zaphiris, 2019; Sharma, Papamitsiou & Giannakos, 2019). Moreover, the study by Roberts, Chung and Parks (2016) measures “accuracy” as the ratio between correct and incorrect answers, and each game is evaluated according to a proficiency level.…”

“…The availability and minimal intrusiveness of video cameras (compact cameras that may be smaller than a coin or incorporated in any mobile device), in conjunction with the potential of machine learning, makes them a device that is commonly used for studying aspects of understanding, achievement, emotions, attention and speech in MMLA research with children. Children’s actions can be recorded in a wide‐angle shot (Pérez, Martínez, Avila & Espinosa, 2018; Ramakrishnan et al ., 2019), with a 360° point‐of‐view multi‐camera system (Malmberg et al ., 2019) or with a webcam on top of a computer screen (Chen et al ., 2016; Pereira et al ., 2018; Sharma et al ., 2019) to analyse the emotions generated during the learning process automatically. Spikol and colleagues (2018) calculated the distance between the participants (recorded with a front camera), which they treated as a proxy for collaboration.…”

“…Rudovic, Lee, Dai, Schuller and Picard, (2018) used cameras in a robot that interacted with children to conduct real‐time extrapolation and processing of the LA from facial expressions, head movements, body movements, posture and gestures. Although facial recognition is widely used in the reviewed literature (Andrade & Worsley, 2017; Chen et al ., 2016; Ouherrou et al ., 2019; Pereira et al ., 2018; Ramakrishnan et al , 2019; Sharma et al ., 2019; Spikol et al ., 2018), Ouherrou and colleagues (2019) highlight the constraints of using it alone compared to several integrated systems (such as head movement or tone of voice), which could be more effective.…”

“… 1 1 Other authors use different abbreviations to refer to multimodal learning analytics, such as ML (Banik, Bhuiyan & Jahan, 2015), MLA (Ochoa and Worsley, 2016) or LAS by multimodal learning analytics system (Spikol, Ruffaldi, Dabisias & Cukurova, 2018). …”

Multimodal learning analytics for game‐based learning

“…Prior research has investigated student models of knowledge (Min et al ., 2017; Shute, Wang, Greiff, Zhao, & Moore, 2016; Spires, Rowe, Mott, & Lester, 2011) affect (Baker, D'Mello, Rodrigo, & Graesser, 2010; Botelho, Baker, & Heffernan, 2017; Henderson, Rowe, Mott, et al ., 2019). Several studies have demonstrated the value of incorporating multiple modalities of data generated from sensor‐based technologies (eg, facial expressions of emotions, eye gaze) to model students’ knowledge of complex topics (Blikstein & Worsley, 2016; Sharma, Papamitsiou, & Giannakos, 2019; Taub, Sawyer, Smith, et al ., 2020).…”