In search of a robust facial expressions recognition model: A large-scale visual cross-corpus study

Ryumina, Elena; Dresvyanskiy, Denis; Karpov, Alexey

doi:10.1016/j.neucom.2022.10.013

Cited by 42 publications

(22 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The gesture recognition model consists of two BiLSTM networks of 64 and 32 units with an attention layer between them. Attention was proposed in [ 139 ] and tested on other CV problems [ 140 ]. The FCNN completes the gesture recognition model and predicts 226 gestures.…”

Section: Methodsmentioning

confidence: 99%

“…The point of the third technique is to add variation to the training data. The first two techniques are used to make trainable models less confident in their predictions [ 140 ], therefore, such models make fewer gross errors, which leads to an increase in the accuracy of SR. The results of experiments on the use of data augmentation techniques are presented in Table 5 .…”

Section: Evaluation Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Ryumin

Ivanko

Ryumina

2023

Sensors

Self Cite

View full text Add to dashboard Cite

Audio-visual speech recognition (AVSR) is one of the most promising solutions for reliable speech recognition, particularly when audio is corrupted by noise. Additional visual information can be used for both automatic lip-reading and gesture recognition. Hand gestures are a form of non-verbal communication and can be used as a very important part of modern human–computer interaction systems. Currently, audio and video modalities are easily accessible by sensors of mobile devices. However, there is no out-of-the-box solution for automatic audio-visual speech and gesture recognition. This study introduces two deep neural network-based model architectures: one for AVSR and one for gesture recognition. The main novelty regarding audio-visual speech recognition lies in fine-tuning strategies for both visual and acoustic features and in the proposed end-to-end model, which considers three modality fusion approaches: prediction-level, feature-level, and model-level. The main novelty in gesture recognition lies in a unique set of spatio-temporal features, including those that consider lip articulation information. As there are no available datasets for the combined task, we evaluated our methods on two different large-scale corpora—LRW and AUTSL—and outperformed existing methods on both audio-visual speech recognition and gesture recognition tasks. We achieved AVSR accuracy for the LRW dataset equal to 98.76% and gesture recognition rate for the AUTSL dataset equal to 98.56%. The results obtained demonstrate not only the high performance of the proposed methodology, but also the fundamental possibility of recognizing audio-visual speech and gestures by sensors of mobile devices.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Evaluation Experimentsmentioning

confidence: 99%

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Ryumin

Ivanko

Ryumina

2023

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Large models can easily overfit to small datasets and struggle to generalize to new data (Salman & Liu, 2019;Zhang et al, 2017). Potential mitigation approaches include data augmentation, cross-corpora training, and self-supervised training (Parry et al, 2019;Ryumina et al, 2022).…”

Section: Limitations Of Deep Learning For Cognitive Modelingmentioning

confidence: 99%

Advancing naturalistic affective science with deep learning

Lin¹,

Bulls²,

Tepfer³

et al. 2023

Preprint

View full text Add to dashboard Cite

People express emotions via a variety of behaviors, including facial muscle movements, body poses and gestures, vocal prosody, and speech. To understand how people experience and perceive emotion, it is crucial to quantify and model these behaviors. However, existing methods are insufficient to address this need. Manually annotating behavior is very time-consuming, making it infeasible to do at scale. Moreover, common linear models cannot fully capture the complex, nonlinear, and interactive affective processes embodied by these behaviors. In this methodology review, we describe how deep learning addresses these challenges and thereby promises to advance naturalistic affective science. First, deep learning provides accessible and efficient tools to annotate dynamic, complex, multi-modal behaviors. These automated annotation tools can scale up behavioral quantification to a degree impossible with human coders, enabling many new, more naturalistic approaches to affective science. Second, deep learning offers innovative paradigms for optimizing and manipulating naturalistic stimuli. This application makes it possible to generate experiment designs with greater generalizability, statistical power, and external validity. Third, deep learning can support flexible, powerful cognitive models of naturalistic affective processing. These novel cognitive models make it possible to explain how the mind and brain engage in the emotional world in ways that are both broader and more precise. However, deep learning is not without its limitations, so we also explore important failure cases, practical issues, and ethical concerns. By detailing the promise and the peril of deep learning, this review paves the way for a more naturalistic and generalizable affective science.

show abstract

“…Reference [60] 75.97% Reference [61] 64.46% Reference [62] 69.57% Reference [63] 70.04% Reference [64] 74.59% Reference [6] 72.03% Reference [52] 73.28% Reference [65] 74.14% Reference [66] 72.16% Reference [67] 75.42% Reference [47] 76.82% Reference [68] 66.46% Reference [69] 66.37% Reference [70] 65.20% Reference [6] 63.36% Reference [71] 63.03% 66.29% Reference [72] 63.00% Reference [73] 62.09% 65.69% Reference [74] 61.60% 65.40% Reference [75] 61.32% 65.74% Reference [76] 53.93% Reference [77] 59.30%…”

Section: Fer2013 Extra Training Datamentioning

confidence: 99%

Robust Human Face Emotion Classification Using Triplet-Loss-Based Deep CNN Features and SVM

Haider

Yang

Lee

et al. 2023

Sensors

View full text Add to dashboard Cite

Human facial emotion detection is one of the challenging tasks in computer vision. Owing to high inter-class variance, it is hard for machine learning models to predict facial emotions accurately. Moreover, a person with several facial emotions increases the diversity and complexity of classification problems. In this paper, we have proposed a novel and intelligent approach for the classification of human facial emotions. The proposed approach comprises customized ResNet18 by employing transfer learning with the integration of triplet loss function (TLF), followed by SVM classification model. Using deep features from a customized ResNet18 trained with triplet loss, the proposed pipeline consists of a face detector used to locate and refine the face bounding box and a classifier to identify the facial expression class of discovered faces. RetinaFace is used to extract the identified face areas from the source image, and a ResNet18 model is trained on cropped face images with triplet loss to retrieve those features. An SVM classifier is used to categorize the facial expression based on the acquired deep characteristics. In this paper, we have proposed a method that can achieve better performance than state-of-the-art (SoTA) methods on JAFFE and MMI datasets. The technique is based on the triplet loss function to generate deep input image features. The proposed method performed well on the JAFFE and MMI datasets with an accuracy of 98.44% and 99.02%, respectively, on seven emotions; meanwhile, the performance of the method needs to be fine-tuned for the FER2013 and AFFECTNET datasets.

show abstract

In search of a robust facial expressions recognition model: A large-scale visual cross-corpus study

Cited by 42 publications

References 33 publications

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Audio-Visual Speech and Gesture Recognition by Sensors of Mobile Devices

Advancing naturalistic affective science with deep learning

Robust Human Face Emotion Classification Using Triplet-Loss-Based Deep CNN Features and SVM

Contact Info

Product

Resources

About