Outlier Processing in Multimodal Emotion Recognition

Zhang, Ge; Luo, Tianxiang; Pedrycz, Witold; El-Meligy, Mohammed A.; Sharaf, Mohamed A.; Li, Zhiwu

doi:10.1109/access.2020.2981760

Cited by 13 publications

(5 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, RECOLA hired 6 assistants to annotate each frame, and our proposed system can achieve UAR (Arousal)=61.9% and UAR (Valence)=59.7%. LIRIS-ACCEDE, by which Yi et al [56] achieved UAR=45.63/38.20(valence/arousal) and Zhang et al [54] achieved CCC=91.8/94.6 (valence/arousal), takes pairwise comparisons, rather than rating approaches. From each pair of video excerpts, three annotators have to identify the one which can convey most strongly the given emotion in terms of valence or arousal.…”

Section: Discussionmentioning

confidence: 99%

Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things

Xie

Lv³

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The Internet of Things (IoT) technologies such as interconnection and edge computing help emotion recognition to be applied in healthcare, smart education, etc. However, the acquisition and transmission processes may have some situations, such as lost signals and serious interference noise caused by motion, which affect the quality of the received data and limit the performance of IoT emotion detection. We collectively refer to these as invalid data. A multi-step deep (MSD) system is proposed to reliably detect multimodal emotion by the collected records containing invalid data. Semantic compatibility and continuity are utilized to filter out the invalid data. The feature from invalid modal data is replaced through the imputation method to compensate for the impact of invalid data on emotion detection. In this way, the proposed system can automatically process invalid data and improve the recognition performance. Furthermore, considering the spatiotemporal information, the features of video and physiological signals are extracted by specific deep neural networks in the MSD system. The simulation experiments are conducted on a public multimodal database, and the performance of the MSD system measured by the unweighted average recall is better than that of the traditional system. The promising results observed in the experiments verify the potential influence of the proposed system in practical IoT applications. INDEX TERMS Internet of Things, multimodal emotion detection, invalid data, multi-step deep (MSD) system, deep neural networks

show abstract

Section: Discussionmentioning

confidence: 99%

Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things

Xie

Lv³

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…To date, there are few works in the literature investigating the intersection between smart manufacturing and emotion recognition, and these are mainly related to improving human-machine interaction [14,74]. On the other hand, it is not possible to identify work that exploits modelling and emotion recognition as a design technique for customizable and innovative objects.…”

Section: Related Workmentioning

confidence: 99%

Modeling and recognition of emotions in manufacturing

Bertacchini

Bilotta

Pietro

et al. 2022

Int J Interact Des Manuf

View full text Add to dashboard Cite

New consumer needs have led industries to the possibility of creating virtual platforms where users can customize products by creating infinite combinations of different results. This made it possible to expand sales by guaranteeing a wide choice that would satisfy all requests. The dynamic and flexible evolution of factories is guaranteed by the introduction of new technologies such as robotization and 3D printers, recognized as two of the pillars of Industry 4.0. The main aim of this paper is to achieve a workflow for the creation and implementation of personalised jewellery based on faces with different emotional expressions. To date, there are few works in the literature investigating the intersection between smart manufacturing and emotion recognition, and these are mainly related to improving human–machine interaction. The authors’ aim is to research for innovation in the intersection of three different fields of study such as parametric modelling, smart manufacturing and emotion recognition in order to create personalized and innovative manufacturable models. To this purpose, an application has been generated that exploits both visual scripting, typical of parametric modelling, and scripting, in the Python programming language. The generated algorithm implements a machine learning for emotion recognition that identifies the label of each user-generated face, validating the effectiveness of the method.

show abstract

“…The introduction of affective factors to HCIs resulted in developing an interdisciplinary research field, often called affective computing, which attempts to develop human-aware AI that can perceive, understand, and regulate emotions [ 112 ]. Once computers understand humans’ emotions, AI will rise to a new level [ 113 ]. In the HCSII research field, there is an increasing focus on developing emotional AI in HCI since emotion recognition using AI is a fundamental prerequisite to improve HCI [ 106 ].…”

Section: Backgrounds and Related Workmentioning

confidence: 99%

Sensors and Artificial Intelligence Methods and Algorithms for Human–Computer Intelligent Interaction: A Systematic Mapping Study

Šumak

Brdnik

Pušnik

2021

Sensors

View full text Add to dashboard Cite

To equip computers with human communication skills and to enable natural interaction between the computer and a human, intelligent solutions are required based on artificial intelligence (AI) methods, algorithms, and sensor technology. This study aimed at identifying and analyzing the state-of-the-art AI methods and algorithms and sensors technology in existing human–computer intelligent interaction (HCII) research to explore trends in HCII research, categorize existing evidence, and identify potential directions for future research. We conduct a systematic mapping study of the HCII body of research. Four hundred fifty-four studies published in various journals and conferences between 2010 and 2021 were identified and analyzed. Studies in the HCII and IUI fields have primarily been focused on intelligent recognition of emotion, gestures, and facial expressions using sensors technology, such as the camera, EEG, Kinect, wearable sensors, eye tracker, gyroscope, and others. Researchers most often apply deep-learning and instance-based AI methods and algorithms. The support sector machine (SVM) is the most widely used algorithm for various kinds of recognition, primarily an emotion, facial expression, and gesture. The convolutional neural network (CNN) is the often-used deep-learning algorithm for emotion recognition, facial recognition, and gesture recognition solutions.

show abstract

Outlier Processing in Multimodal Emotion Recognition

Cited by 13 publications

References 49 publications

Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things

Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things

Modeling and recognition of emotions in manufacturing

Sensors and Artificial Intelligence Methods and Algorithms for Human–Computer Intelligent Interaction: A Systematic Mapping Study

Contact Info

Product

Resources

About