Students’ emotion extraction and visualization for engagement detection in online learning

Hasnine, Mohammad Nehal; Bui, Huyen T. T.; Tran, Thuy Thi Thu; Nguyen, Ho Tran; Akçapınar, Gökhan; Ueda, Hiroshi

doi:10.1016/j.procs.2021.09.115

Cited by 45 publications

(20 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The proposed system outperformed other existing engagement detection systems in terms of performance accuracy, as given in Table 7. Our proposed system provided better results than the other approaches used by [28,47,55,69]. Our system used the information from three modalities: facial emotion, eye-blinking, and head movement.…”

Section: Discussionmentioning

confidence: 89%

See 1 more Smart Citation

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

Gupta

Kumar

Tekchandani

2023

Multimed Tools Appl

View full text Add to dashboard Cite

Due to the COVID-19 crisis, the education sector has been shifted to a virtual environment. Monitoring the engagement level and providing regular feedback during e-classes is one of the major concerns, as this facility lacks in the e-learning environment due to no physical observation of the teacher. According to present study, an engagement detection system to ensure that the students get immediate feedback during e-Learning. Our proposed engagement system analyses the student's behaviour throughout the e-Learning session. The proposed novel approach evaluates three modalities based on the student's behaviour, such as facial expression, eye blink count, and head movement, from the live video streams to predict student engagement in e-learning. The proposed system is implemented based on deep-learning approaches such as VGG-19 and ResNet-50 for facial emotion recognition and the facial landmark approach for eye-blinking and head movement detection. The results from different modalities (for which the algorithms are proposed) are combined to determine the EI (engagement index). Based on EI value, an engaged or disengaged state is predicted. The present study suggests that the proposed facial cues-based multimodal system accurately determines student engagement in real time. The experimental research achieved an accuracy of 92.58% and showed that the proposed engagement detection approach significantly outperforms the existing approaches.

show abstract

Section: Discussionmentioning

confidence: 89%

“…In [55], authors used emotions for automatic engaged detection with an accuracy value of 70% on the FER-2013 dataset. Authors in [28] used eye-gaze and emotions to visualise emotion detection on the CK+ dataset with an accuracy of 73.4%. Conclusively, we achieved an accuracy rate of 92.58% for engagement state prediction.…”

Section: Discussionmentioning

confidence: 99%

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

Gupta

Kumar

Tekchandani

2023

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…They have also been used to determine the concentration levels of students. Hasnine et al [ 10 ] utilized CNNs to detect the concentration level of students. Six types of basic emotions were extracted using a pre-trained CNN, and those were used to detect the concentration level of students in a virtual classroom.…”

Section: Literature Reviewmentioning

confidence: 99%

Evaluation of e-learners’ concentration using recurrent neural networks

Jeong

Cho

2022

J Supercomput

View full text Add to dashboard Cite

Recently, interest in e-learning has increased rapidly owing to the lockdowns imposed by COVID-19. A major disadvantage of e-learning is the difficulty in maintaining concentration because of the limited interaction between teachers and students. The objective of this paper is to develop a methodology to predict e-learners’ concentration by applying recurrent neural network models to eye gaze and facial landmark data extracted from e-learners’ video data. One hundred eighty-four video data of ninety-two e-learners were obtained, and their frame data were extracted using the OpenFace 2.0 toolkit. Recurrent neural networks, long short-term memory, and gated recurrent units were utilized to predict the concentration of e-learners. A set of comparative experiments was conducted. As a result, gated recurrent units exhibited the best performance. The main contribution of this paper is to present a methodology to predict e-learners’ concentration in a natural e-learning environment.

show abstract

“…Computer vision-based approaches are used to evaluate both online and offline lecture films to extract the emotions of the students, as emotions are crucial to the learning process. Using a pre-trained Convolutional Neural Network, the six types of basic emotions-angry, disgusted, fearful, happy, sad, surprised, and neutral-are extracted [9]. Natural user interaction in e-learning is the most ongoing task for researchers.…”

Section: Related Workmentioning

confidence: 99%

Cognitive Perception for Scholastic Purposes using Innovative Teaching Strategies

Aruna

Swarna

2022

SCPE

View full text Add to dashboard Cite

The influence of emotion on attention is particularly strong, changing its selectivity in particular and motivating behavior and action. The degree to which a student participates in class determines their level of conceptual knowledge. Various teaching techniques have been developed over time to improve not only the attention of a student but also their engagement of a student. The level of engagement of a student can help us decide the amount of understanding a student can attain throughout the session. Though these techniques have been developed over time, the basic tests to determine the authenticity of these activities have been done mainly by the use of assessment-based methods. According to research in the field of neuroscience, a person's emotions can assist us to determine a student's level of participation. We also have the affective circumplex model to show us the correlation between emotions and the level of engagement of a person. Taking this into account, we developed an attentivity model with the help of an emotion recognition model (made with the help of VGG-16 architecture in CNN) and the eye tracking system to analyze the amount of engagement being displayed by the student in the class. This model applied to the students on the various teaching models helps us in deciding the effectiveness of various teaching methodologies for the primitive methods of teaching.

show abstract

Students’ emotion extraction and visualization for engagement detection in online learning

Cited by 45 publications

References 10 publications

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

A multimodal facial cues based engagement detection system in e-learning context using deep learning approach

Evaluation of e-learners’ concentration using recurrent neural networks

Cognitive Perception for Scholastic Purposes using Innovative Teaching Strategies

Contact Info

Product

Resources

About