Correspondence Learning for Deep Multi-Modal Recognition and Fraud Detection

Park, Jongchan; Kim, Min Hyun; Choi, Dong‐Geol

doi:10.3390/electronics10070800

Cited by 4 publications

(1 citation statement)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The CMU MOSEI dataset is one of the most popular datasets used for multimodal emotion recognition. It has been heavily referenced with 129 citations on Scopus from which 35 papers 25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59 directly utilize CMU MOSEI features in their application, most of which utilize deep-learning architectures for their analysis. This existing research on the CMU MOSEI dataset however does not explore the explainability of the CMU MOSEI features.…”

Section: Problem With Pre-extracted Featuresmentioning

confidence: 99%

Evaluating Significant Features in Context-Aware Multimodal Emotion Recognition with XAI Methods

Shaikh

Khalane

Makwana

et al. 2023

Preprint

View full text Add to dashboard Cite

Analysis of human emotions from multimodal data for making critical decisions is an emerging area of research. The evolution of deep learning algorithms has improved the potential for extracting value from multimodal data. However, these algorithms do not often explain how certain outputs from the data are produced. This study focuses on the risks of using black-box deep learning models for critical tasks, such as emotion recognition, and describes how human understandable interpretations of these models are extremely important. This study utilizes one of the largest multimodal datasets available - CMU-MOSEI. Many researchers have used the pre-extracted features provided by the CMU Multimodal SDK with black-box deep learning models making it difficult to interpret the contribution of individual features. This study describes the implications of individual features from various modalities (audio, video, text) in Context-Aware Multimodal Emotion Recognition. It describes the process of curating reduced feature models by using the GradientSHAP XAI method. These reduced models with highly contributing features achieve comparable and even better results compared to their corresponding all feature models as well as the baseline model GraphMFN proving that carefully selecting significant features can help improve the model robustness and performance and in turn make it trustworthy.

show abstract

Section: Problem With Pre-extracted Featuresmentioning

confidence: 99%

Evaluating Significant Features in Context-Aware Multimodal Emotion Recognition with XAI Methods

Shaikh

Khalane

Makwana

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions

et al. 2023

View full text Add to dashboard Cite

Classifying Gas Data Measured Under Multiple Conditions Using Deep Learning

et al. 2022

View full text Add to dashboard Cite

Gas classification is a machine learning problem that is important for various applications including monitoring systems, health care, public security, etc. Since measuring the characteristic of gas molecules is greatly affected by external factors such as wind speed and the internal setting of detecting sensors, classification should be done by taking into account the combination of these individual factors, which we call a condition in this paper. In particular, when classifying gas data measured under multiple conditions, the data from each condition need to be integrated, which we call multi-conditioned gas classification. While there have been some studies on gas classification for a single condition, no previous approach deals with the multi-conditioned gas classification problem to the best of our knowledge. In this paper, we propose a novel multi-conditioned gas classification method for the first time. We present a new deep learning network structure that can efficiently extract features from the data of multiple conditions and effectively integrate them, which is referred to as a multi-conditioned gas classification network (MCGCN). We also propose a new training loss function to guarantee good performance reliably for the varying number of given conditions. Experimental results demonstrate the superiority of the proposed method, which achieves accuracies of 99.15% ± 0.41 regardless of the number of conditions with 15 times fewer model parameters in comparison to the existing method.

show abstract

Correspondence Learning for Deep Multi-Modal Recognition and Fraud Detection

Cited by 4 publications

References 25 publications

Evaluating Significant Features in Context-Aware Multimodal Emotion Recognition with XAI Methods

Evaluating Significant Features in Context-Aware Multimodal Emotion Recognition with XAI Methods

Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions

Classifying Gas Data Measured Under Multiple Conditions Using Deep Learning

Contact Info

Product

Resources

About