EmotiW 2020: Driver Gaze, Group Emotion, Student Engagement and Physiological Signal based Challenges

Gonçalves

2020 IEEE 4th International Conference on Image Processing, Applications and Systems (IPAS)

et al. 2020

Despite recent efforts, accuracy in group emotion recognition is still generally low. One of the reasons for these underwhelming performance levels is the scarcity of available labeled data which, like the literature approaches, is mainly focused on still images. In this work, we address this problem by adapting an inflated ResNet-50 pretrained for a similar task, activity recognition, where large labeled video datasets are available. Audio information is processed using a Bidirectional Long Short-Term Memory (Bi-LSTM) network receiving extracted features. A multimodal approach fuses audio and video information at the score level using a support vector machine classifier. Evaluation with data from the EmotiW 2020 AV Group-Level Emotion sub-challenge shows a final test accuracy of 65.74% for the multimodal approach, approximately 18% higher than the official baseline. The results show that using activity recognition pretraining offers performance advantages for groupemotion recognition and that audio is essential to improve the accuracy and robustness of video-based recognition.

Section: A Datamentioning

confidence: 99%

“…The authors wish to acknowledge the EmotiW 2020 Grand Challenge [4] organizers and the authors of the MMIT dataset and pretrained models used in this work.…”

Section: Acknowledgmentsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Audiovisual Classification of Group Emotion Valence Using Activity Recognition Networks

Gonçalves

2020 IEEE 4th International Conference on Image Processing, Applications and Systems (IPAS)

et al. 2020

Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge

“…With this in mind, research into the fusion of physiological signals for use with perceived emotional signals is limited, and within this contribution we suggest that there are potentially two benefits to this (1) where agreement between raters is lower, replacing less reliable raters with a physiological signal may improve agreement (2) where only a small number of raters are available, adding a physiological signal to the gold standard may also be fruitful . Physiological signals have been utilised as features [12], or extracted during particular tasks, to better target arousal [13], however there has been minimal research on a combined physiological and perceived arousal gold standard. Recently, in the 2021 edition of the Multimodal Sentiment in-the-wild (MuSe) challenge, the signal of arousal was fused with 𝐸𝐷𝐴 and used as a prediction target for the MuSe-Physio sub-challenge [14].…”

Section: Introductionmentioning

confidence: 99%

A Physiologically-Adapted Gold Standard for Arousal during Stress

Baird

Stappen

Christ

et al. 2021

Emotion is an inherently subjective psycho-physiological human state and to produce an agreed-upon representation (gold standard) for continuously perceived emotion requires time-consuming and costly training of multiple human annotators. With this in mind, there is strong evidence in the literature that physiological signals are an objective marker for states of emotion, particularly arousal. In this contribution, we utilise a multimodal dataset captured during a Trier Social Stress Test to explore the benefit of fusing physiological signals -Heartbeats per Minute (𝐵𝑃𝑀), Electrodermal Activity (𝐸𝐷𝐴), and Respiration-rate -for recognition of continuously perceived arousal utilising a Long Short-Term Memory, Recurrent Neural Network architecture, and various audio, video, and textual based features. We use the MuSe-Toolbox to create a gold standard that considers annotator delay and agreement weighting. An improvement in Concordance Correlation Coefficient (CCC) is seen across features sets when fusing 𝐸𝐷𝐴 with arousal, compared to the arousal only gold standard results. Additionally, BERT -based textual features' results improved for arousal plus all physiological signals, obtaining up to .3344 CCC (.2118 CCC for arousal only). Multimodal fusion also improves CCC. Audio plus video features obtain up to .6157 CCC for arousal plus 𝐸𝐷𝐴, 𝐵𝑃𝑀. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval; • Computing methodologies → Artificial intelligence.

Proceedings of the 2020 International Conference on Multimodal Interaction

“…In this paper, we focus on driver gaze estimation of nine coarse regions in an 'in the wild' setting. We work with the Driver Gaze in the Wild (DGW) dataset [11] released as part of the eighth Emotion Recognition in the Wild Challenge (EmotiW) [1]. It exhibits several modelling challenges, such as diverse ethnic background among the subjects, varying illumination, and potential presence of reflections in the face and environment.…”

Section: Introductionmentioning

confidence: 99%

X-AWARE: ConteXt-AWARE Human-Environment Attention Fusion for Driver Gaze Prediction in the Wild

Stappen

Rizos

Schuller

2020

Reliable systems for automatic estimation of the driver's gaze are crucial for reducing the number of traffic fatalities and for many emerging research areas aimed at developing intelligent vehiclepassenger systems. Gaze estimation is a challenging task, especially in environments with varying illumination and reflection properties. Furthermore, there is wide diversity with respect to the appearance of drivers' faces, both in terms of occlusions (e. g., vision aids) and cultural/ethnic backgrounds. For this reason, analysing the face along with contextual information ś for example, the vehicle cabin environment ś adds another, less subjective signal towards the design of robust systems for passenger gaze estimation. In this paper, we present an integrated approach to jointly model different features for this task. In particular, to improve the fusion of the visually captured environment with the driver's face, we have developed a contextual attention mechanism, X-AWARE , attached directly to the output convolutional layers of InceptionResNetV2 networks. In order to showcase the effectiveness of our approach, we use the Driver Gaze in the Wild dataset, recently released as part of the Eighth Emotion Recognition in the Wild Challenge (EmotiW) challenge. Our best model outperforms the baseline by an absolute of 15.03 % in accuracy on the validation set, and improves the previously best reported result by an absolute of 8.72 % on the test set. CCS CONCEPTS• Computing methodologies → Computer vision; Scene understanding; • Human-centered computing → Human computer interaction (HCI).