Capturing self‐regulated learning processes in virtual reality: Causal sequencing of multimodal data

Abstract

This study examines the embodied ways in which learners monitor their cognition while learning about exponential functions in an immersive virtual reality (VR) based game, Pandemic by Prisms of Reality. Traditionally, metacognitive monitoring has been assessed through behavioural traces and verbalised instances. When learning in VR, learners are fully immersed in the learning environment, actively manipulating it based on affordances designed to support learning, offering insights into the relationship between physical interaction and metacognition. The study collected multimodal data from 15 participants, including think‐aloud audio, bird's‐eye view video recordings and physiological data. Metacognitive monitoring was analysed through qualitative coding of the think‐aloud protocol, while movement was measured via optical flow analysis and cognitive load was assessed through heart rate variability analysis. The results revealed embodied metacognition by aligning the data to identify learners' physical states alongside their verbalised metacognition. The findings demonstrated a temporal interplay among cognitive load, metacognitive monitoring, and motion during VR‐based learning. Specifically, cognitive load, indicated by the low‐ and high‐frequency heart rate variability index, predicted instances of metacognitive monitoring, and monitoring predicted learners' motion while interacting with the VR environment. This study further provides future directions in understanding self‐regulated learning processes during VR learning by utilizing multimodal data to inform real‐time adaptive personalised support within these environments. Practitioner notesWhat is already known about this topic Immersive virtual reality (VR) environments have the potential to offer personalised support based on users' individual needs and characteristics. Self‐regulated learning (SRL) involves learners monitoring their progress and strategically regulating their learning when needed. Multimodal data captured during VR learning, such as birds‐eye‐view video, screen recordings, physiological changes and verbalisations, can provide insights into learners' SRL processes and support needs. What this paper adds Provides insights into the embodied aspects of learners' metacognitive monitoring during learning in an immersive VR environment. Demonstrates how SRL processes can be captured via the collection and analysis of multimodal data, including think‐aloud audio, bird's‐eye view video recordings and physiological data, to capture metacognitive monitoring and movement during VR‐based learning. Contributes to the understanding of the interplay between cognitive load, metacognitive monitoring, and motion in immersive VR learning. Implications for practice and/or policy Researchers and practitioners can use the causal relationships identified in this study to identify instances of SRL in an immersive VR setting. Educational technology developers can consider the integration of online measures, such as cognitive load and physiological arousal, into adaptive VR environments to enable real‐time personalised support for learners based on their self‐regulatory needs.

Multimodal and immersive systems for skills development and education

“…On the other hand, it also identifies the gaps and challenges that need to be addressed in future work. The main identified challenge points out the intrinsic complexity of conducting studies on multimodal and immersive systems, which deal with the design of proper tasks and the reliable collection and analysis of multimodal data (Buseyne et al, 2023;Chejara et al, 2023;Kaplan-Rakowski et al, 2023;Sobocinski et al, 2023). In terms of gaps to be addressed for future studies, this special section identifies the need for longitudinal studies to look at long-term effects and transferability of results (Aslan et al, 2023;Yang et al, 2023) and to address ethical concerns due to the private nature of multimodal data (D'Angelo & Rajarathinam, 2024).…”

“…The contributions included in this special section showcase research on multimodal and immersive systems and demonstrate support in six different domains, namely face-to-face collaboration (Buseyne et al, 2023;D'Angelo & Rajarathinam, 2024), mathematics (Aslan et al, 2023;Sobocinski et al, 2023), reading comprehension (Ye & Kaplan-Rakowski, 2024), art appreciation (Kaplan-Rakowski et al, 2023), drilling skills (Yang et al, 2023) and running skills (Cardenas Hernandez et al, 2024).…”

“…The research presented in this special section also shows how it is possible to analyse and hence evaluate collaboration through verbal and non-verbal aspects such as pitch, word count and entrainment (Buseyne et al, 2023); turn-taking, speech duration patterns and teaching-assistant versus student interactions (D'Angelo & Rajarathinam, 2024); head movement and inner brow raiser (Chejara et al, 2023); and spatial and speech data (Zhao et al, 2024). Moreover, the study by Sobocinski et al (2023) identified, through the analysis of multimodal data, a link between heart rate and metacognition while solving mathematical problems in VR. Similarly, the work of Cardenas Hernandez et al (2024) proposed a framework where the analysis of multimodal data can be used to personalise running programs.…”

“…The study focuses on natural spoken communication in a realistic setting. Sobocinski et al (2023) investigate how learners use their bodies and environment to monitor their thinking while learning maths in a VR game. The study collected multimodal data from 15 participants: audio of the participant talking, video of the participant from above and physiological data.…”

“…Moreover, the study by Sobocinski et al. (2023) identified, through the analysis of multimodal data, a link between heart rate and metacognition while solving mathematical problems in VR. Similarly, the work of Cardenas Hernandez et al.…”