Facial Emotion Prediction through Action Units and Deep Learning

Nadeeshani, Madhuka; Jayaweera, Akash; Samarasinghe, Pradeepa

doi:10.1109/icac51239.2020.9357138

Cited by 9 publications

(5 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As executed for CBB', we verified the agreement of sentiments of GT image with the predicted face on the CNN model and AUs. Apparently, as happened in CBB', the results for the two methods are very similar, as it has been indicated in studies like proposed by Nadeeshani et al [2020]. The methods used suggest that the faces, being one of the elements that compose the image, in the CGB' dataset, are in accordance with the GT in 40% of images (for AUs) and 48% (for CNN).…”

Section: Cnn Model In the Cgb' Datasetsupporting

confidence: 70%

“…In the field of elements that make up an image, in the present work, we researched the faces to assess the emotions that are transmitted and verify whether the facial expression could contribute to alleviating contradictions between the image and text domains. The studies of Nadeeshani et al [2020] shows two techniques for predicting facial emotion with Machine Learning and Deep Learning. They conclude that both are possible to model with predictions above 80%, which is above the state of the art in this area.…”

Section: Related Workmentioning

confidence: 99%

“…Two methods were used to detect facial emotion, following the line of work of Nadeeshani et al [2020] that used Action Units (AUs) and a CNN model to compare the prediction of them. For both methods, at the beginning, the OpenFace Baltrusaitis et al [2018] toolkit was used for face detection.…”

Section: Face Detection In Cbb and Cgb Dataset Imagesmentioning

confidence: 99%

See 2 more Smart Citations

Investigating sentiments in Brazilian and German Blogs

Zahn

Molin

Musse

2022

JBCS

View full text Add to dashboard Cite

Social interactions have changed in recent years. People post their thoughts, opinions and sentiments on social media platforms more often, through images and videos, providing a very rich source of data about population of different countries, communities, etc. Due to the increase in the amount of data on the internet, it becomes impossible to perform any analysis in a manual manner, requiring the automation of the process. In this work, we use two blog corpora that contain images and texts. Cross-Media German Blog (CGB) corpus consists of German blog posts, while Cross-Media Brazilian Blog (CBB) contains Brazilian blog posts. Both blogs have the Ground Truth (GT) of images and texts feelings (sentiments), classified according to human perceptions. In previous work, Machine Learning and lexicons technologies were applied to both corpora to detect the sentiments (negative, neutral or positive) of images and texts and compare the results with ground truth (based on subjects perception). In this work, we investigated a new hypothesis, by detecting faces and their emotions, to improve the sentiment classification accuracy in both CBB and CGB datasets. We use two methodologies to detect polarity on the faces and evaluated the results with the images GT and the multimodal GT (the complete blog using text and image). Our results indicate that the facial emotion can be a relevant feature in the classification of blogs sentiment.

show abstract

Section: Cnn Model In the Cgb' Datasetsupporting

confidence: 70%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Investigating sentiments in Brazilian and German Blogs

Zahn

Molin

Musse

2022

JBCS

View full text Add to dashboard Cite

show abstract

“…With the development of more machine learning methods, including deep learning, automatic AU detection became more and more precise [53][54][55]. This makes it possible to apply models learned to detect AU to real-world problems (including real-time detection).…”

Section: Action Unitsmentioning

confidence: 99%

Processing Real-Life Recordings of Facial Expressions of Polish Sign Language Using Action Units

Irasiak

Kozák

Piasecki

et al. 2023

Entropy

View full text Add to dashboard Cite

Automatic translation between the national language and sign language is a complex process similar to translation between two different foreign languages. A very important aspect is the precision of not only manual gestures but also facial expressions, which are extremely important in the overall context of a sentence. In this article, we present the problem of including facial expressions in the automation of Polish-to-Polish Sign Language (PJM) translation—this is part of an ongoing project related to a comprehensive solution allowing for the animation of manual gestures, body movements and facial expressions. Our approach explores the possibility of using action unit (AU) recognition in the automatic annotation of recordings, which in the subsequent steps will be used to train machine learning models. This paper aims to evaluate entropy in real-life translation recordings and analyze the data associated with the detected action units. Our approach has been subjected to evaluation by experts related to Polish Sign Language, and the results obtained allow for the development of further work related to automatic translation into Polish Sign Language.

show abstract

“…Facial expressions are expressed with combinations of these action units [8]. Today, AUs are still used in research on many different subjects [9][10][11][12][13].…”

Section: Introductionmentioning

confidence: 99%