Affect-driven Ordinal Engagement Measurement from Video

Abedi, Ali; Khan, Shehroz S.

doi:10.48550/arxiv.2106.10882

Cited by 6 publications

(15 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 2 shows that some of the previous methods use conventional computer-vision features such as box filters, Gabor [18], LBP-TOP [25], and C3D [26]. In some other methods, [23,37], a CNN with convolutional layers followed by fully-connected layers is trained on a facial expression recognition dataset to extract facial embedding features. In most of the previous methods, facial Action Units (AUs), eye movement, gaze direction, and head pose features are extracted using Open-Face [40], or body pose features are extracted using OpenPose [41].…”

Section: Related Workmentioning

confidence: 99%

Section: Feature Extractionmentioning

confidence: 99%

“…Various affect and behavioral features are extracted from the video segments [37,38]. Abedi and Khan [37] showed that sequences of continuous values of valence and arousal extracted from consecutive video frames are important indicators of affective engagement. The pre-trained EmoFAN [53] on the AffectNet dataset [54], a deep neural network to analyze facial affects in face images, is used to extract valence and arousal from consecutive frames of video segments (refer to [37] for more detail).…”

Section: Feature Extractionmentioning

confidence: 99%

See 2 more Smart Citations

Bag of States: A Non-sequential Approach to Video-based Engagement Measurement

Abedi

Thomas

Jayagopi

et al. 2023

Preprint

View full text Add to dashboard Cite

Automatic measurement of student engagement provides helpful information for instructors to meet learning program objectives and individualize program delivery. Students' behavioral and emotional states need to be analyzed at fine-grained time scales in order to measure their level of engagement. Many existing approaches have developed sequential and spatiotemporal models, such as recurrent neural networks, temporal convolutional networks, and three-dimensional convolutional neural networks, for measuring student engagement from videos. These models are trained to incorporate the order of behavioral and emotional states of students into video analysis and output their level of engagement. In this paper, backed by educational psychology, we question the necessity of modeling the order of behavioral and emotional states of students in measuring their engagement. We develop bag-of-words-based models in which only the occurrence of behavioral and emotional states of students is modeled and analyzed and not the order in which they occur. Behavioral and affective features are extracted from videos and analyzed by the proposed models to determine the level of engagement in an ordinal-output classification setting. Compared to the existing sequential and spatiotemporal approaches for engagement measurement, the proposed non-sequential approach improves the state-of-the-art results. According to experimental results, our method significantly improved engagement level classification accuracy on the IIITB Online SE dataset by 26% compared to sequential models and achieved engagement level classification accuracy as high as 66.58% on the DAiSEE student engagement dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Feature Extractionmentioning

confidence: 99%

Section: Feature Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Bag of States: A Non-sequential Approach to Video-based Engagement Measurement

Abedi

Thomas

Jayagopi

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In our experimental work, we first analyze the importance of feature sets to select the best set of features for the resulting trained ED-MTT system. Then, we compare the performance of ED-MTT with 9 different works [1,5,15,20,24,25,27,31,32] from the state-of-the-art which will be reviewed in the next section. Our results show that ED-MTT outperforms these state-of-the-art methods with at least 6% improvement on MSE.…”

Section: Introductionmentioning

confidence: 99%

Engagement Detection with Multi-Task Training in E-Learning Environments

Çopur¹,

Nakıp²,

Scardapane³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recognition of user interaction, in particular engagement detection, became highly crucial for online working and learning environments, especially during the COVID-19 outbreak. Such recognition and detection systems significantly improve the user experience and efficiency by providing valuable feedback. In this paper, we propose a novel Engagement Detection with Multi-Task Training (ED-MTT) system which minimizes mean squared error and triplet loss together to determine the engagement level of students in an e-learning environment. The performance of this system is evaluated and compared against the state-ofthe-art on a publicly available dataset as well as videos collected from real-life scenarios. The results show that ED-MTT achieves 6% lower MSE than the best state-of-the-art performance with highly acceptable training time and lightweight feature extraction.

show abstract

“…5,6 Engagement estimation (EE) is kind of affective computing and behavior recognition, and it goes further to probe the inner intention behind the apparent behavior and emotion. Many methods have been developed to estimate engagement in various scenarios such as general HRI, [7][8][9][10] museum tour guide, 11 classroom or distance learning, [12][13][14][15][16][17][18] and healthcare. 5,[19][20][21] Conventional approaches use nonverbal cues such as proxemics, body pose, gaze patterns, facial expressions, and context information to build classifiers.…”

Section: Introductionmentioning

confidence: 99%

Engagement estimation of the elderly from wild multiparty human–robot interaction

Zhang

Zheng

Thalmann

2022

Computer Animation & Virtual

View full text Add to dashboard Cite

The use of social robots in healthcare systems or nursing homes to assist the elderly and their caregivers will be becoming common, where robots' understanding of engagement of the elderly is important. Traditional engagement estimation (EE) often requires expert involvement in a controlled dyadic interaction environment. In this article, we propose a supervised machine learning method to estimate the engagement state of the elderly in a multiparty human-robot interaction (HRI) scenario from the real-world video recording as input. The method is built upon the basic concept of engagement in geriatric psychiatry and HRI video representations. It adapts pretrained models to extract behavior, affective, and visual signals to form the multi-modal features. These features are then fed into a neural network made of a self-attention mechanism and average pooling for individual learning, a graph attention network for group learning and a fully connected layer to estimate the engagement. We tested the proposed method using 43 wild multiparty elderly robot interaction (ERI) videos. The experimental results show that our method is capable of detecting the key participants and estimating the engagement state of the elderly effectively. Also our study demonstrates the signals from side-participants in the main interaction group considerably contribute to the EE of the elderly in the multiparty ERI.

show abstract

Affect-driven Ordinal Engagement Measurement from Video

Cited by 6 publications

References 50 publications

Bag of States: A Non-sequential Approach to Video-based Engagement Measurement

Bag of States: A Non-sequential Approach to Video-based Engagement Measurement

Engagement Detection with Multi-Task Training in E-Learning Environments

Engagement estimation of the elderly from wild multiparty human–robot interaction

Contact Info

Product

Resources

About