Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning

Sharma, Prabin; Joshi, Shubham; Gautam, Subash; Filipe, Vítor; Reis, Manuel J. C. S.

doi:10.1007/978-3-031-22918-3_5

Cited by 49 publications

(27 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this research, students' engagements are detected using the mapping method described in the related literature [7]. The calculation of engagement detection is based on the concentration index calculation method proposed by Sharma et al [17]. The mapping method uses simple assumptions and mathematical grounds to identify student engagement from emotional data.…”

Section: Engagement Detection Methods Using Emotion Datamentioning

confidence: 99%

“…In the same line, Whitehill et al found that students' engagement levels correlate with their task performance [16]. Sharma et al also found a correlation between the emotions expressed during the lesson and the concentration level of the students [17]. However, many potentials and aspects of emotional data are yet to come to light.…”

Section: Literature Reviewmentioning

confidence: 95%

“…The second type of study is noninvasive, and automatic methods based on biometric measurements are used. For example, Sharma et al proposed a prototype computer vision-based system to measure students' levels of concentration (e.g., low, medium, and high) in real-time from the expressed facial emotions during a lesson [17]. In another study, Krithika and Lakshmi Priya proposed a system for identifying and monitoring students' emotions in an e-learning environment by eye and head movements [21].…”

Section: Literature Reviewmentioning

confidence: 99%

See 2 more Smart Citations

A Real-Time Learning Analytics Dashboard for Automatic Detection of Online Learners’ Affective States

Hasnine

Nguyen

Tran

et al. 2023

Sensors

View full text Add to dashboard Cite

Students’ affective states describe their engagement, concentration, attitude, motivation, happiness, sadness, frustration, off-task behavior, and confusion level in learning. In online learning, students’ affective states are determinative of the learning quality. However, measuring various affective states and what influences them is exceedingly challenging for the lecturer without having real interaction with the students. Existing studies primarily use self-reported data to understand students’ affective states, while this paper presents a novel learning analytics system called MOEMO (Motion and Emotion) that could measure online learners’ affective states of engagement and concentration using emotion data. Therefore, the novelty of this research is to visualize online learners’ affective states on lecturers’ screens in real-time using an automated emotion detection process. In real-time and offline, the system extracts emotion data by analyzing facial features from the lecture videos captured by the typical built-in web camera of a laptop computer. The system determines online learners’ five types of engagement (“strong engagement”, “high engagement”, “medium engagement”, “low engagement”, and “disengagement”) and two types of concentration levels (“focused” and “distracted”). Furthermore, the dashboard is designed to provide insight into students’ emotional states, the clusters of engaged and disengaged students’, assistance with intervention, create an after-class summary report, and configure the automation parameters to adapt to the study environment.

show abstract

Section: Engagement Detection Methods Using Emotion Datamentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 95%

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

A Real-Time Learning Analytics Dashboard for Automatic Detection of Online Learners’ Affective States

Hasnine

Nguyen

Tran

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…The DAiSEE dataset is one of the most popular datasets used in previous studies [5], [9], [15]- [19], [22]. Other datasets that have been used for this purpose include the EmotiW 2018 dataset [23], the EmotiW 2020 dataset [24], the Engagement Recognition (ER) database [25], the UPNA head pose dataset [26], and other video recordings [3], [7], [8], [20], [27].…”

Section: Related Workmentioning

confidence: 99%

“…This study's results revealed a positive correlation between students' scores (student learning) and students' predicted engagement levels. Meanwhile, Sharma et al [8] detected students' engagement using video recordings of students' learning through emotional analysis and tracking of eye gaze and head movements based on two machine learning algorithms, namely the Haar Cascade algorithm (for face and eye detection) and the Convolutional Neural Network algorithm (CNN) (for emotion classification). Based on these studies, CNN is a powerful deep learning model that has been successfully used in various studies to detect students' engagement levels in online learning.…”

Section: Introductionmentioning

confidence: 99%

Convolutional Neural Network Model based Students’ Engagement Detection in Imbalanced DAiSEE Dataset

Santoni¹,

Basaruddin²,

Junus³

2023

IJACSA

View full text Add to dashboard Cite

The COVID-19 pandemic has significantly changed learning processes. Learning, which had generally been carried out face-to-face, has now turned online. This learning strategy has both advantages and challenges. On the bright side, online learning is unbound by space and time, allowing it to take place anywhere and anytime. On the other side, it faces a common challenge in the lack of direct interaction between educators and students, making it difficult to assess students' engagement during an online learning process. Therefore, it is necessary to conduct research with the aim of automatically detecting students' engagement during online learning. The data used in this research were derived from the DAiSEE dataset (Dataset for Affective States in E-Environments), which comprises ten-second video recordings of students. This dataset classifies engagement levels into four categories: low, very low, high, and very high. However, the issue of imbalanced data found in the DAiSEE dataset has yet to be addressed in previous research. This data imbalance can cause errors in the classification model, resulting in overfitting and underfitting of the model. In this study, Convolutional Neural Network, a deep learning model, was utilized for feature extraction on the DAiSEE dataset. The OpenFace library was used to perform facial landmark detection, head pose estimation, facial expression unit recognition, and eye gaze estimation. The pre-processing stages included data selection, dimensional reduction, and normalization. The PCA and SVD techniques were used for dimensional reduction. The data were later oversampled using the SMOTE algorithm. The training and testing data were distributed at an 80:20 ratio. The results obtained from this experiment exceeded the benchmark evaluation values on the DAiSEE dataset, achieving the best accuracy of 77.97% using the SVD dimensional reduction technique.

show abstract

Data-Driven Approach for Student Engagement Modelling Based on Learning Behaviour

Orji

Fatahi

Vassileva

2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning

Cited by 49 publications

References 20 publications

A Real-Time Learning Analytics Dashboard for Automatic Detection of Online Learners’ Affective States

A Real-Time Learning Analytics Dashboard for Automatic Detection of Online Learners’ Affective States

Convolutional Neural Network Model based Students’ Engagement Detection in Imbalanced DAiSEE Dataset

Data-Driven Approach for Student Engagement Modelling Based on Learning Behaviour

Contact Info

Product

Resources

About