Enhancing Mouth-Based Emotion Recognition Using Transfer Learning

Abstract

This work concludes the first study on mouth-based emotion recognition while adopting a transfer learning approach. Transfer learning results are paramount for mouth-based emotion emotion recognition, because few datasets are available, and most of them include emotional expressions simulated by actors, instead of adopting real-world categorisation. Using transfer learning, we can use fewer training data than training a whole network from scratch, and thus more efficiently fine-tune the network with emotional data and improve the convolutional neural network’s performance accuracy in the desired domain. The proposed approach aims at improving emotion recognition dynamically, taking into account not only new scenarios but also modified situations to the initial training phase, because the image of the mouth can be available even when the whole face is visible only in an unfavourable perspective. Typical applications include automated supervision of bedridden critical patients in a healthcare management environment, and portable applications supporting disabled users having difficulties in seeing or recognising facial emotions. This achievement takes advantage of previous preliminary works on mouth-based emotion recognition using deep-learning, and has the further benefit of having been tested and compared to a set of other networks using an extensive dataset for face-based emotion recognition, well known in the literature. The accuracy of mouth-based emotion recognition was also compared to the corresponding full-face emotion recognition; we found that the loss in accuracy is mostly compensated by consistent performance in the visual emotion recognition domain. We can, therefore, state that our method proves the importance of mouth detection in the complex process of emotion recognition.

Reducing Videoconferencing Fatigue through Facial Emotion Recognition

“…In our study, we use a deep learning based method, a CNN, to recognize a subset of the Ekman model's emotions which are the following: anger, fear, happiness, sadness, and surprise. Furthermore, we augmented the emotions with the neutral expression as a state of control for the recognition results on emotions, similar to Franzoni et al [35]. Moreoever, as suggested by Kim et al [36] and Kuo et al [37], we trained our CNN on a merged data set that consisted of various original FER data sets such as CK+ [12], JAFFE [13], and BU-3DFE [14] to alleviate the problem of overfitting and to further improve its robustness.…”

An Overview of Machine Learning and 5G for People with Disabilities

“…Mouth-based emotion recognition using deep learning [27] helps people with disabilities who have difficulties seeing or recognizing facial emotions. The position and curve of the lips has been analyzed through CNNs.…”

Rapid Prototyping of Virtual Reality Cognitive Exercises in a Tele-Rehabilitation Context

“…NU!Reha Service allows to train cognitive functions such as memory, attention and executive function, changing the difficulty of activities according to the patient's abilities. This characteristic determines a flexibility service that allows a customization of rehabilitation path [4,[18][19][20][21][22][23][24][25][26]. This flexibility implies a variety of proposed activities, and therapists can improve the platform by making new personalized exercises.…”