A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

Abstract

Due to the COVID-19 crisis, the education sector has been shifted to a virtual environment. Monitoring the engagement level and providing regular feedback during e-classes is one of the major concerns, as this facility lacks in the e-learning environment due to no physical observation of the teacher. According to present study, an engagement detection system to ensure that the students get immediate feedback during e-Learning. Our proposed engagement system analyses the student's behaviour throughout the e-Learning session. The proposed novel approach evaluates three modalities based on the student's behaviour, such as facial expression, eye blink count, and head movement, from the live video streams to predict student engagement in e-learning. The proposed system is implemented based on deep-learning approaches such as VGG-19 and ResNet-50 for facial emotion recognition and the facial landmark approach for eye-blinking and head movement detection. The results from different modalities (for which the algorithms are proposed) are combined to determine the EI (engagement index). Based on EI value, an engaged or disengaged state is predicted. The present study suggests that the proposed facial cues-based multimodal system accurately determines student engagement in real time. The experimental research achieved an accuracy of 92.58% and showed that the proposed engagement detection approach significantly outperforms the existing approaches.

Research on Learning Concentration Recognition with Multi-Modal Features in Virtual Reality Environments

“…As shown in Figure 5, vision data had better a F1 score compared to interaction data. This finding aligns with previous research conducted in online learning environments [55,56], STEM environments [57], and traditional classroom environments [58], where vision data captured through cameras have shown high effectiveness in recognizing concentration. Facial expressions, eye gaze, and other related data serve as important sources for recognizing concentration levels [55][56][57][58].…”

“…This finding aligns with previous research conducted in online learning environments [55,56], STEM environments [57], and traditional classroom environments [58], where vision data captured through cameras have shown high effectiveness in recognizing concentration. Facial expressions, eye gaze, and other related data serve as important sources for recognizing concentration levels [55][56][57][58]. These results emphasized the continued significance of vision data as a valuable source for recognizing concentration in VR environments.…”

Emotion Detection Using Machine Learning: An Analytical Review

“…Their research's primary goal is to aid online learning [8]. They mostly concentrate on six facial expressions [9], [10]. A system able to automatically detect human emotions was developed by Lim et al in 2020.…”

Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model

“…In [30], the authors introduced a sophisticated system designed to detect engagement levels in e-learning environments. Leveraging multimodal facial cues and employing deep learning techniques, the proposed system offers a comprehensive solution for accurately assessing student engagement.…”