BNU-LSVED 2.0: Spontaneous multimodal student affect database with multi-dimensional labels

Wei, Qinglan; Sun, Bo; He, Jun; Yu, Lejun

doi:10.1016/j.image.2017.08.012

Cited by 31 publications

(16 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In developing the FER model for the learning environment, it is essential to identify meaningful academic affective states. Academic affective states are part of specific affective states that cannot be represented by basic affective states (Wei, Q., et al 2017). Some related works in the literature have reported the relevant students' facial expressions, which help in determining their academic affective states.…”

Section: Proposed Methodologymentioning

confidence: 99%

“…This paper addresses this research gap by exploring an automatic real‐time system to monitor student group engagement by analysing their facial expressions and recognizing academic affective states. Academic affective states are part of specific affective states that cannot be represented by basic emotions (Wei, Q., et al 2017). For this study six meaningful academic affective states are used, namely: ‘boredom,’ ‘confuse,’ ‘focus,’ ‘frustrated,’ ‘yawning,’ and ‘sleepy,’ pertinent in the learning environment (D'Mello, S. 2013; Tonguç & Ozkara, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition

Pabba

Kumar

2021

Expert Systems

View full text Add to dashboard Cite

Students' disengagement problem has become critical in the modern scenario due to various distractions and lack of student-teacher interactions. This problem is exacerbated with large offline classrooms, where it becomes challenging for teachers to monitor students' engagement and maintain the right-level of interactions. Traditional ways of monitoring students' engagement rely on self-reporting or using physical devices, which have limitations for offline classroom use. Student's academic affective states (e.g., moods and emotions) analysis has potential for creating intelligent classrooms, which can autonomously monitor and analyse students' engagement and behaviours in real-time. In recent literature, a few computer vision based methods have been proposed, but they either work only in the e-learning domain or have limitations in real-time processing and scalability for large offline classes. This paper presents a real-time system for student group engagement monitoring by analysing their facial expressions and recognizing academic affective states: 'boredom,' 'confuse,' 'focus,' 'frustrated,' 'yawning,' and 'sleepy,' which are pertinent in the learning environment. The methodology includes certain pre-processing steps like face detection, a convolutional neural network (CNN) based facial expression recognition model, and post-processing steps like frame-wise group engagement estimation. For training the CNN model, we created a dataset of the aforementioned facial expressions from classroom lecture videos and added related samples from three publicly available datasets, BAUM-1, DAiSEE, and YawDD, to generalize the model predictions. The trained model has achieved train and test accuracy of 78.70% and 76.90%, respectively. The proposed methodology gave promising results when compared with self-reported engagement levels by students.

show abstract

Section: Proposed Methodologymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition

Pabba

Kumar

2021

Expert Systems

View full text Add to dashboard Cite

show abstract

“…S1∼S12 are the 12 items of the scale, after normalization, the value range of each dimension of PAD is [−1, +1], with +P, +A, +D and −P, −A, −D to express various combinations of pleasure, excitement, and dominance. In this study, 112 groups of image sequences with clear and obvious emotional communication were selected from the BNU_LSVED dataset [24], and six researchers were invited to score according to the above emotional scale. After the scoring is completed, the score of each image sequence is normalized according to formula (1) to obtain the PAD value, and the PAD values marked by different emotions are counted according to the emotion category, and the statistical results are summarized and averaged.…”

Section: Data Quantification For Emotion Detectionmentioning

confidence: 99%

An Analysis Model of Learners’ Online Learning Status Based on Deep Neural Network and Multi-Dimensional Information Fusion

Ma¹,

Tang²,

Ma³

et al. 2023

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

The learning status of learners directly affects the quality of learning. Compared with offline teachers, it is difficult for online teachers to capture the learning status of students in the whole class, and it is even more difficult to continue to pay attention to students while teaching. Therefore, this paper proposes an online learning state analysis model based on a convolutional neural network and multi-dimensional information fusion. Specifically, a facial expression recognition model and an eye state recognition model are constructed to detect students' emotions and fatigue, respectively. By integrating the detected data with the homework test score data after online learning, an analysis model of students' online learning status is constructed. According to the PAD model, the learning state is expressed as three dimensions of students' understanding, engagement and interest, and then analyzed from multiple perspectives. Finally, the proposed model is applied to actual teaching, and procedural analysis of 5 different types of online classroom learners is carried out, and the validity of the model is verified by comparing with the results of the manual analysis.

show abstract

“…The lack of accurate assessment of emotional experiences in the participants can jeopardise experiments such as this [22]. To better identify the academic emotions, the participants and external coders are invited to undertake the task in this experiment.…”

Section: Creation Of the Databasementioning

confidence: 99%

Spontaneous facial expression database for academic emotion inference in online learning

Bian

Zhang

Yang

et al. 2019

IET Computer Vision

View full text Add to dashboard Cite

Academic emotions can produce a great impact on the learning effect. Normally, emotions are expressed externally in the students' facial expressions, speech and behaviour. In this paper, the focus is on automatic academic emotion inference based on facial expressions in online learning. Considering the lack of training samples for the inference algorithm, a spontaneous facial expression database is established. It includes the facial expressions of five common academic emotions and consists of two subsets: a video clip database and an image database. A total of 1,274 video clips and 30,184 images from 82 students are included in the database. The samples are labelled by both the participants and external coders. An extensive analysis is carried out on the image database using a convolutional neural network (CNN)-based algorithm to infer selfannotation. Some data augmentation algorithms are applied to improve the algorithm performance. Additionally, an adaptive data augmentation algorithm based on spatial transformer network is introduced, which can remove some confounding factors in the original images. The algorithm can obviously improve the inference performance, which has been proven by comparing some evaluation indicators before and after adoption. Such a database will certainly accelerate the application of affective computing in the educational field.

show abstract

BNU-LSVED 2.0: Spontaneous multimodal student affect database with multi-dimensional labels

Cited by 31 publications

References 38 publications

An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition

An intelligent system for monitoring students' engagement in large classroom teaching through facial expression recognition

An Analysis Model of Learners’ Online Learning Status Based on Deep Neural Network and Multi-Dimensional Information Fusion

Spontaneous facial expression database for academic emotion inference in online learning

Contact Info

Product

Resources

About