Facial Expression Recognition and Ordinal Intensity Estimation: A Multilabel Learning Approach

Ekundayo, Olufisayo; Viriri, Serestina

doi:10.1007/978-3-030-64559-5_46

Cited by 3 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This process has been discredited [257] [258] because human intuition does not assign numeric value as a measure of emotional intensity. The only reported ordinal intensity estimation is our model [69], we considered FER and intensity estimation as a multilabel learning task and present a deep multilabel model, which adequately predicts the emotion and its intensity concurrently, using ordinal metrics.…”

Section: Discussionmentioning

confidence: 99%

“…The result showed that the proposed method performed better than any existing probabilistic graphical methods in accuracy and computation time. Our approach to FER and intensity estimation is presented in [69]. We considered FER and intensity estimation a multilabel task with the motivation that an instance of a facial expression image contains information about emotion displays and the corresponding intensity.…”

Section: B Facial Expression Recognition and Intensity Estimationmentioning

confidence: 99%

See 1 more Smart Citation

Facial Expression Recognition: A Review of Trends and Techniques

Ekundayo

Viriri

2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

Facial Expression Recognition (FER) is presently the aspect of cognitive and affective computing with the most attention and popularity, aided by its vast application areas. Several studies have been conducted on FER, and many review works are also available. The existing FER review works only give an account of FER models capable of predicting the basic expressions. None of the works considers intensity estimation of an emotion; neither do they include studies that address data annotation inconsistencies and correlation among labels in their works. This work first introduces some identified FER application areas and provides a discussion on recognised FER challenges. We proceed to provide a comprehensive FER review in three different machine learning problem definitions: Single Label Learning (SLL)-which presents FER as a multiclass problem, Multilabel Learning (MLL)-that resolves the ambiguity nature of FER, and Label Distribution Learning-that recovers the distribution of emotion in FER data annotation. We also include studies on expression intensity estimation from the face. Furthermore, popularly employed FER models are thoroughly and carefully discussed in handcrafted, conventional machine learning and deep learning models. We finally itemise some recognise unresolved issues and also suggest future research areas in the field.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: B Facial Expression Recognition and Intensity Estimationmentioning

confidence: 99%

Facial Expression Recognition: A Review of Trends and Techniques

Ekundayo

Viriri

2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…The CNN network with sigmoid function in the output layer serves as the binary classifier. Because of our dataset population, we use the pre-trained network (VGG-16) to avoid model overfitting, which was a challenge in Ekundayo & Viriris (2020) . To reduce intraclass variation and increase interclass variation an aggregation loss (combination of island loss and BCE loss) is proposed which is another additional feature to our work in ( Ekundayo & Viriris, 2020 ).…”

Section: Introductionmentioning

confidence: 99%

Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation

Ekundayo¹,

Viriri²

2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Facial Expression Recognition (FER) has gained considerable attention in affective computing due to its vast area of applications. Diverse approaches and methods have been considered for a robust FER in the field, but only a few works considered the intensity of emotion embedded in the expression. Even the available studies on expression intensity estimation successfully assigned a nominal/regression value or classified emotion in a range of intervals. Most of the available works on facial expression intensity estimation successfully present only the emotion intensity estimation. At the same time, others proposed methods that predict emotion and its intensity in different channels. These multiclass approaches and extensions do not conform to man heuristic manner of recognising emotion and its intensity estimation. This work presents a Multilabel Convolution Neural Network (ML-CNN)-based model, which could simultaneously recognise emotion and provide ordinal metrics as the intensity estimation of the emotion. The proposed ML-CNN is enhanced with the aggregation of Binary Cross-Entropy (BCE) loss and Island Loss (IL) functions to minimise intraclass and interclass variations. Also, ML-CNN model is pre-trained with Visual Geometric Group (VGG-16) to control overfitting. In the experiments conducted on Binghampton University 3D Facial Expression (BU-3DFE) and Cohn Kanade extension (CK+) datasets, we evaluate ML-CNN’s performance based on accuracy and loss. We also carried out a comparative study of our model with some popularly used multilabel algorithms using standard multilabel metrics. ML-CNN model simultaneously predicts emotion and intensity estimation using ordinal metrics. The model also shows appreciable and superior performance over four standard multilabel algorithms: Chain Classifier (CC), distinct Random K label set (RAKEL), Multilabel K Nearest Neighbour (MLKNN) and Multilabel ARAM (MLARAM).

show abstract

3DEmo: For Portrait Emotion Recognition with New Dataset

Liu,

Agaian

2024

J. Comput. Cult. Herit.

View full text Add to dashboard Cite

Emotional Expression Recognition (EER) and Facial Expression Recognition (FER) are active research areas in the affective computing field, which involves studying human emotion, recognition, and sentiment analysis. The main objective of this research is to develop algorithms that can accurately interpret and estimate human emotions from portrait images. The emotions depicted in a portrait can reflect various factors such as psychological and physiological states, the artist’s emotional responses, social and environmental aspects, and the period in which the painting was created. This task is challenging because (i) the portraits are often depicted in an artistic or stylized manner rather than realistically or naturally, (ii) the texture and color features obtained from natural faces and paintings differ, affecting the success rate of emotion recognition algorithms, and (iii) it is a new research area, where practically we don’t have visual arts portrait facial emotion estimation models or datasets. To address these challenges, we need a new class of tools and a database specifically tailored to analyze portrait images. This study aims to develop art portrait emotion recognition methods and create a new digital portrait dataset containing 927 images. The proposed model is based on (i) a 3-dimensional estimation of emotions learned by a deep neural network and (ii) a novel deep learning module (3DEmo) that could be easily integrated into existing FER models. To evaluate the effectiveness of the developed models, we also tested their robustness on a facial emotion recognition dataset. The extensive simulation results show that the presented approach outperforms established methods. We expect that this dataset and the developed new tools will encourage further research in recognizing emotions in portrait paintings and predicting artists’ emotions in the painting period based on their artwork.

show abstract

Facial Expression Recognition and Ordinal Intensity Estimation: A Multilabel Learning Approach

Cited by 3 publications

References 23 publications

Facial Expression Recognition: A Review of Trends and Techniques

Facial Expression Recognition: A Review of Trends and Techniques

Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation

3DEmo: For Portrait Emotion Recognition with New Dataset

Contact Info

Product

Resources

About