Facial Expression Recognition of Instructor Using Deep Features and Extreme Learning Machine

Abstract

Classroom communication involves teacher’s behavior and student’s responses. Extensive research has been done on the analysis of student’s facial expressions, but the impact of instructor’s facial expressions is yet an unexplored area of research. Facial expression recognition has the potential to predict the impact of teacher’s emotions in a classroom environment. Intelligent assessment of instructor behavior during lecture delivery not only might improve the learning environment but also could save time and resources utilized in manual assessment strategies. To address the issue of manual assessment, we propose an instructor’s facial expression recognition approach within a classroom using a feedforward learning model. First, the face is detected from the acquired lecture videos and key frames are selected, discarding all the redundant frames for effective high-level feature extraction. Then, deep features are extracted using multiple convolution neural networks along with parameter tuning which are then fed to a classifier. For fast learning and good generalization of the algorithm, a regularized extreme learning machine (RELM) classifier is employed which classifies five different expressions of the instructor within the classroom. Experiments are conducted on a newly created instructor’s facial expression dataset in classroom environments plus three benchmark facial datasets, i.e., Cohn–Kanade, the Japanese Female Facial Expression (JAFFE) dataset, and the Facial Expression Recognition 2013 (FER2013) dataset. Furthermore, the proposed method is compared with state-of-the-art techniques, traditional classifiers, and convolutional neural models. Experimentation results indicate significant performance gain on parameters such as accuracy, F1-score, and recall.

A Hybrid Model for Driver Emotion Detection Using Feature Fusion Approach

“…We have compared our proposed hybrid model with recent state-of-the-art FER methods, including Inception-Resnet and LSTM [ 35 ], DCMA-CNN [ 27 ], WRF [ 22 ], LMRF [ 24 ], VGG11+SVM [ 40 ], DNN+RELM [ 43 ], LBP+ORB+SVM [ 25 ], and MDNETWORK [ 33 ] on the CK+ dataset. These works adopted machine learning algorithms and deep neural networks in combined manner or individually used.…”

“…Among them, WRF [ 22 ], LMRF [ 24 ], and LBP+ORB+SVM [ 25 ] have created their FER models using machine-learning-based feature extraction methods and classification mechanisms. In the remaining works, Inception-Resnet and LSTM [ 35 ], DNN+RELM [ 43 ], VGG11+SVM [ 40 ] have used both deep neural networks and classifiers along with handcrafted features extracted by machine learning. DCMA-CNN [ 27 ] and MDNETWORK [ 33 ] were implemented using multi branch convolutional neural networks.…”

“…Y.K. Bhatti et al, 2021, [ 43 ] proposed a FER approach using feed-forward learning model, in which deep features are extracted using multiple CNNs; the regularized extreme learning machine was used for classification, achieving an accuracy of 86.5% on the CK+ dataset.…”

A Review on Emotion and Sentiment analysis Using Learning Techniques

“…An automated strategy is used by deep learning algorithms to get attributes that are exclusive to them. [19].…”

A Survey of AI-Based Facial Emotion Recognition: Features, ML & DL Techniques, Age-Wise Datasets and Future Directions

“…Today the need for emotion classification has surpassed the barrier of age because of the global shift towards online platforms such as online education to teach or gain knowledge virtually globally to all remote areas, IoT enabled health monitoring systems and temperature setters in cars and households, robotics, psychiatric evaluation based on violent behaviors of criminals or those mentally disturbed, mood swings study on adolescents to help guide them mentally, deepfake detection, gaming, and many such applications are currently being innovated using state of the art technologies. Also, numerous studies have been conducted on Facial Emotion Recognition by using Computer vision because of its practicality in intelligent robotics, health-related treatment, IoT, Security surveillance, criminal psychological analysis, observation of driver exhaustion, and other human-computer interfaces mechanisms [7][8][9]. With more virtual connectivity through videos and images, the need to adopt the latest technology based on people's emotions is now a critical factor in driving user-friendliness and maximum user satisfaction.…”