VAE-Based Adversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis

Wang, Yanan; Wu, Jianming; Furumai, Kazuaki; Wada, Shinya; Kurihara, Satoshi

doi:10.1109/access.2022.3174215

Cited by 10 publications

(6 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1. MOSEI drops the data lacking modalities to fairly evaluate recent modality fusion-based methods [20]. We compared the video segment IDs of each data point for each modality and saved only the data points associated with a common segment ID.…”

Section: Methodsmentioning

confidence: 99%

“…Existing works on multimodal transfer learning unify adversarial learning to regularize the embedding distributions between different modalities, leading to effective multimodal fusion [14], [17], [18], [19], [20]. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

et al. 2023

Self Cite

View full text Add to dashboard Cite

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). With an intuition that the best learning performance comes with professional advisers and smart students, we use a CLIP-based teacher model to provide expressive multimodal knowledge supervision signals to a RoBERTa-based student model via optimizing a step-distillation objective loss-first step: the teacher distills multimodal knowledge of video-enhanced prompts from classification logits to a regression logit-second step: the multimodal knowledge is distilled from the regression logit of the teacher to the student. We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the stateof-the-art method by 3.4% mAP score for VEGAS without additional computations for inference. These results suggest the strengths of our method for achieving high efficiency-performance multimodal transfer learning. INDEX TERMSMultimodal transfer learning, knowledge distillation, fundamental model. YANAN WANG (Student Member, IEEE) received the B.S. degree in engineering from Aoyama Gakuin University, in 2015, and the M.S. degree in engineering from The University of Electro-Communications, Japan, in 2017. He is currently pursuing the Ph.D. degree in engineering with Keio University, Japan. He is an Associate Research Engineer in multimodal modeling topics with KDDI Research Inc. His research interests include multimodal representation learning, emotion recognition, knowledge graph, and graph representation learning. He is a Student Member of JSAI. He is a regular member of IEICE and the Editorial Committee of IEICE Human Communication Group.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…It has been suggested to use a variational autoencoder (VAE) enabled adversarial multimodal domain transfer (VAE-AMDT) to acquire more discriminating multimodal representation that also can enhance performance of the system, and it has been jointly trained with a multiattention mechanism to decrease the distance difference amongst the unimodal representation [37]. The sentiment feature set was employed to train and predict sentiment classification labels in a convolution neural network (CNN) and a bidirectional long short-term memory (BiLSTM) network [38].…”

Section: Literature Reviewmentioning

confidence: 99%

Automatic Sentimental Analysis by Firefly with Levy and Multilayer Perceptron

Elangovan¹,

Subedha²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

The field of sentiment analysis (SA) has grown in tandem with the aid of social networking platforms to exchange opinions and ideas. Many people share their views and ideas around the world through social media like Facebook and Twitter. The goal of opinion mining, commonly referred to as sentiment analysis, is to categorise and forecast a target's opinion. Depending on if they provide a positive or negative perspective on a given topic, text documents or sentences can be classified. When compared to sentiment analysis, text categorization may appear to be a simple process, but number of challenges have prompted numerous studies in this area. A feature selection-based classification algorithm in conjunction with the firefly with levy and multilayer perceptron (MLP) techniques has been proposed as a way to automate sentiment analysis (SA). In this study, online product reviews can be enhanced by integrating classification and feature election. The firefly (FF) algorithm was used to extract features from online product reviews, and a multi-layer perceptron was used to classify sentiment (MLP). The experiment employs two datasets, and the results are assessed using a variety of criteria. On account of these tests, it is possible to conclude that the FFL-MLP algorithm has the better classification performance for Canon (98% accuracy) and iPod (99% accuracy).

show abstract

“…Different from previous studies [22], [23], our work focuses on learning joint representations and multimodal interaction. Joint representations can correlate information from different modalities, thus learning joint representations facilitates the process of fusion.…”

Section: B Multimodal Representation Learningmentioning

confidence: 99%

CMJRT: Cross-Modal Joint Representation Transformer for Multimodal Sentiment Analysis

Liang

Fang

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Multimodal Sentiment Analysis (MSA) is an emerging research field that aims to identify the sentiment of speakers through Audio (A), Video (V), and Text (T) modalities. The major challenge is to capture joint representations that can associate and integrate information from various modalities. Most of the existing methods are prone to acquiring joint representation through the concatenation of input features. However, these methods are short of exploiting interactions fully to ensure consistency and complementarity among modalities. To solve this problem, we design a novel multimodal sentiment analysis framework named Cross-Modal Joint Representation Transformer (CMJRT), which exploits hierarchical interactions among modalities by passing joint representations from bimodality to unimodality. Specifically, we adopt cyclic translation to obtain joint representations of bimodality, where one modality is translated to the other modality forward and backward by encoder-decoders. The translation process ensures consistency between modalities. In addition, to explore complementarity among modalities, the cross-modal transformer is used to reinforce each unimodality with common information from bimodality. Extensive experiments on CMU-MOSI and CMU-MOSEI datasets demonstrate that our proposed method outperforms existing approaches.INDEX TERMS Joint representation, multimodal sentiment analysis, transformer.

show abstract

VAE-Based Adversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis

Cited by 10 publications

References 29 publications

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

Automatic Sentimental Analysis by Firefly with Levy and Multilayer Perceptron

CMJRT: Cross-Modal Joint Representation Transformer for Multimodal Sentiment Analysis

Contact Info

Product

Resources

About