Using touchscreen interaction data to predict cognitive workload

Mock, Philipp; Gerjets, Peter; Tibus, Maike; Trautwein, Ulrich; Möller, Korbinian

doi:10.1145/2993148.2993202

Cited by 17 publications

(21 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further investigation into how these different modalities were used revealed that logs and videos were mostly used to evaluate the performance of the participants, either in a quantitative way using logs (Liu, Stamper, & Davenport, 2018 ;Liu et al ., 2019 ;Mock et al ., 2016 ;Sharma, Papamitsiou, et al ., 2019 ) or environment variables , or in a qualitative manner using videos (Worsley & Blikstein, 2015, 2018. The rest of the multimodal sources were used to quantify behavioral trajectories, such as interaction behavior using touch gestures (Mock et al ., 2016 ), engagement with problem space using EDA and audio (flow, stress; Worsley & Blikstein, 2015, 2018, understanding and misconceptions using physiological data (Liu et al ., 2018(Liu et al ., , 2019, and problem-solving behavior using faces and EEG (Sharma, Papamitsiou, et al ., 2019 ) and eye-tracking .…”

Section: Data Collection Sample Size and Methodologymentioning

confidence: 99%

“…2018used skeleton positions and kinematics features to predict participants' recall; Andrade, Danish, and Maltese ( 2017 ) used hand positions and head poses to predict students' understanding in a predator-prey simulation; Spikol et al . ( 2018 ) used distance between students' faces, hand motion speed and the distance between hands to predict quality of students' projects; and Mock et al . ( 2016 ) used interaction logs from a touchscreen and the hand movements to predict the cognitive workload of the students.…”

Section: For Learning Behavior and Performancementioning

confidence: 99%

“…Therefore, the studies report that MMD can be successfully used to explain students' trajectories while the students engage in the learning task. From the aforementioned studies, we can see that interaction logs, gestures and posture were related to memory (Junokas et al ., 2018 ;Kosmas et al ., 2018 ), conceptual understanding (Andrade et al ., 2017 ), the artifact quality (Spikol et al ., 2018 ) and cognitive workload and motivation (Mock et al ., 2016 ).…”

Section: For Learning Behavior and Performancementioning

confidence: 99%

“…Second, for informal situations in collaborative learning, audio, system logs, gesture and motion data were preferred (Spikol et al ., 2016(Spikol et al ., , 2018. Third, in the case of formal instruction to individuals, audio, system logs, gesture and video data were used (Blikstein, 2013 ;Mock et al ., 2016 ). Finally, for informal and individual cases, audio, gestures and log data were utilized by most contributions (Andrade et al ., 2017 ;Ochoa et al ., 2018 ).…”

Section: Opportunitiesmentioning

confidence: 99%

See 3 more Smart Citations

Multimodal data capabilities for learning: What can multimodal data tell us about learning?

Sharma¹,

Giannakos²

2020

Brit J Educational Tech

119

View full text Add to dashboard Cite

Most research on learning technology uses clickstreams and questionnaires as their primary source of quantitative data. This study presents the outcomes of a systematic literature review of empirical evidence on the capabilities of multimodal data (MMD) for human learning. This paper provides an overview of what and how MMD have been used to inform learning and in what contexts. A search resulted in 42 papers that were included in the analysis. The results of the review depict the capabilities of MMD for learning and the ongoing advances and implications that emerge from the employment of MMD to capture and improve learning. In particular, we identified the six main objectives (ie, behavioral trajectories, learning outcome, learning‐task performance, teacher support, engagement and student feedback) that the MMLA research has been focusing on. We also summarize the implications derived from the reviewed articles and frame them within six thematic areas. Finally, this review stresses that future research should consider developing a framework that would enable MMD capacities to be aligned with the research and learning design (LD). These MMD capacities could also be utilized on furthering theory and practice. Our findings set a baseline to support the adoption and democratization of MMD within future learning technology research and development. What is already known about this topic Capturing and measuring learners’ engagement and behavior using MMD has been explored in recent years and exhibits great potential. There are documented challenges and opportunities associated with capturing, processing, analyzing and interpreting MMD to support human learning. MMD can provide insights into predicting learning engagement and performance as well as into supporting the process. What this paper adds Provides a systematic literature review (SLR) of empirical evidence on MMD for human learning. Summarizes the insights MMD can give us about the learning outcomes and process. Identifies challenges and opportunities of MMD to support human learning. Implications for practice and/or policy Learning analytics researchers will be able to use the SLR as a guide for future research. Learning analytics practitioners will be able to use the SLR as a summary of the current state of the field.

show abstract

Section: Data Collection Sample Size and Methodologymentioning

confidence: 99%

Section: For Learning Behavior and Performancementioning

confidence: 99%

Section: For Learning Behavior and Performancementioning

confidence: 99%

Section: Opportunitiesmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal data capabilities for learning: What can multimodal data tell us about learning?

Sharma¹,

Giannakos²

2020

Brit J Educational Tech

119

View full text Add to dashboard Cite

show abstract

“…MMD has been used to predict performance and engagement in educational contexts in previous research [3], [18], [42]. These contexts vary from games [18], [25], to assessment systems [40], to adaptive systems [32], to collaborative systems [24]. However, one common factor in these studies is the use of multiple data streams (e.g., gaze, facial expressions, Electroencephalography (EEG), heart rate, log data) to predict and explain learning performance [18], [40], behaviour [3], [42] or experience [37], [38].…”

Section: B Multi-modal Data-based Predictions In Educationmentioning

confidence: 99%

Motion-Based Educational Games: Using Multi-Modal Data to Predict Player’s Performance

Lee-Cultura

Sharma

Papavlasopoulou

et al. 2020

2020 IEEE Conference on Games (CoG)

View full text Add to dashboard Cite

Multi-Modal Data (MMD) can help educational games researchers understand the synergistic relationship between player's movement and their learning experiences, and consequently uncover insights that may lead to improved design of movement-based game technologies for learning. Predicting player performance fosters opportunities to cultivate heightened educational experiences and outcomes. However, predicting player's performance utilising player-generated MMD during their interactions with educational Motion-Based Touchless Games (MBTG) is challenging. To bridge this gap, we implemented an in-situ study where 26 users, age 11, played 2 maths MBTGs in a single 20-30 minute session. We collected player's MMD (i.e., gaze data from eye-tracking glasses, physiological data from wristbands, and skeleton data from Kinect) produced during game-play. To investigate the potential of MMD for predicting player's academic performance, we used machine learning techniques and MMD derived from player's game-play. This allowed us to identify the MMD features that drive rapid highly accurate predictions of players' academic performance in educational MBTGs. This might allow us to provide realtime proactive feedback to the player to support them through their educational gaming experience. Our analysis compared two data lengths corresponding to half and full duration of the player's question solving time. We showed that all combinations of extracted features associated with gaze, physiological, and skeleton data, predicted student performance more accurately than the majority baseline. Additionally, the most accurate prediction of player's performance derived from the combination of gaze and physiological data for both full and half data lengths. Our findings emphasise the significance of using MMD for realtime performance prediction in educational MBTG and offer implications for practice.

show abstract

Learning and Problem-Solving with Hypermedia in the Twenty-First Century: From Hypertext to Multiple Web Sources and Multimodal Adaptivity

Gerjets

2017

The Psychology of Digital Learning

View full text Add to dashboard Cite

Using touchscreen interaction data to predict cognitive workload

Cited by 17 publications

References 14 publications

Multimodal data capabilities for learning: What can multimodal data tell us about learning?

Multimodal data capabilities for learning: What can multimodal data tell us about learning?

Motion-Based Educational Games: Using Multi-Modal Data to Predict Player’s Performance

Learning and Problem-Solving with Hypermedia in the Twenty-First Century: From Hypertext to Multiple Web Sources and Multimodal Adaptivity

Contact Info

Product

Resources

About