Multitask Emotion Recognition with Incomplete Labels

Deng, Didan; Chen, Zhaokang; Shi, Bertram E.

doi:10.1109/fg47880.2020.00131

Cited by 75 publications

(50 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deng et al [2] proposed a multi-task learning method to learn from missing labels. They used a data balancing technique to the dataset.…”

Section: Related Workmentioning

confidence: 99%

A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition

Jin¹,

Zheng

Gao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Analyzing human affect is vital for human-computer interaction systems. Most methods are developed in restricted scenarios which are not practical for in-the-wild settings. The Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest provides a benchmark for this in-the-wild problem. In this paper, we introduce a multi-modal and multi-task learning method by using both visual and audio information. We use both AU and expression annotations to train the model and apply a sequence model to further extract associations between video frames.We achieve an AU score of 0.712 and an expression score of 0.477 on the validation set. These results demonstrate the effectiveness of our approach in improving model performance.

show abstract

“…Deng et al [2] proposed a multi-task learning method to learn from missing labels. They used a data balancing technique to the dataset.…”

Section: Related Workmentioning

confidence: 99%

A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition

Jin¹,

Zheng

Gao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…ABAW consists of three challenges on the same dataset, Aff-Wild2 [10]: dimensional affect recognition (in terms of valence and arousal), categorical affect classification (in terms of the seven basic emotions), and 12 facial action unit detection. Most of the top-ranked teams in ABAW1, which was held in conjunction with FG2020, proposed deep learning based multitask models that output the three challenges at once [1,14]. For the input data, the corresponding image is basically used, and additional (previous or post) images are used to further leverage temporal information [1,14,16].…”

Section: Related Work 21 Abawmentioning

confidence: 99%

“…Most of the top-ranked teams in the first challenge of ABAW (ABAW1) [6], held in conjunction with the 15 th IEEE Conference on Face and Gesture Recognition (FG2020), used convolutional neural networks (CNNs) with single facial images or sequences of such images. In cases where a single image was used, the captured image was inputted to be recognized, and even for teams that used image sequences, past or future images were used along with the image captured at that point [1,14,16]. Although these methods perform well with large-scale data in the wild, they encounter limitations when used in real time.…”

Section: Introductionmentioning

confidence: 99%

Causal affect prediction model using a facial image sequence

Oh,

Jeong,

Lim

2021

Preprint

View full text Add to dashboard Cite

Among human affective behavior research, facial expression recognition research is improving in performance along with the development of deep learning. However, for improved performance, not only past images but also future images should be used along with corresponding facial images, but there are obstacles to the application of this technique to real-time environments. In this paper, we propose the causal affect prediction network (CAPNet), which uses only past facial images to predict corresponding affective valence and arousal. We train CAPNet to learn causal inference between past images and corresponding affective valence and arousal through supervised learning by pairing the sequence of past images with the current label using the Aff-Wild2 dataset. We show through experiments that the well-trained CAPNet outperforms the baseline of the second challenge of the Affective Behavior Analysis in-the-wild (ABAW2) Competition by predicting affective valence and arousal only with past facial images one-third of a second earlier. Therefore, in real-time application, CAPNet can reliably predict affective valence and arousal only with past data.The code is publicly available. 1

show abstract

“…Few approaches have been proposed in the literature to address the issue of incomplete/missing labels in multi-task settings. They usually work by generating missing task labels using different methods, including Bayesian networks [33], rule-based approach [35], knowledge distillation from another model [18]. In our experiments, we opt for a simpler alternative.…”

Section: Multi-task Learningmentioning

confidence: 99%

Robust Training of Social Media Image Classification Models for Rapid Disaster Response

Alam,

Imran

et al. 2021

Preprint

View full text Add to dashboard Cite

Images shared on social media help crisis managers in terms of gaining situational awareness and assessing incurred damages, among other response tasks. As the volume and velocity of such content are really high, therefore, real-time image classification became an urgent need in order to take a faster response. Recent advances in computer vision and deep neural networks have enabled the development of models for real-time image classification for a number of tasks, including detecting crisis incidents, filtering irrelevant images, classifying images into specific humanitarian categories, and assessing the severity of the damage. For developing real-time robust models, it is necessary to understand the capability of the publicly available pretrained models for these tasks. In the current state-of-art of crisis informatics, it is under-explored. In this study, we address such limitations. We investigate ten different architectures for four different tasks using the largest publicly available datasets for these tasks. We also explore the data augmentation, semi-supervised techniques, and a multitask setup. In our extensive experiments, we achieve promising results.

show abstract

Multitask Emotion Recognition with Incomplete Labels

Cited by 75 publications

References 20 publications

A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition

A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition

Causal affect prediction model using a facial image sequence

Robust Training of Social Media Image Classification Models for Rapid Disaster Response

Contact Info

Product

Resources

About