Memory Fusion Network for Multi-view Sequential Learning

Zadeh, Amir; Liang, Paul Pu; Mazumder, Navonil; Poria, Soujanya; Cambria, Erik; Morency, Louis–Philippe

doi:10.1609/aaai.v32i1.12021

Cited by 415 publications

(122 citation statements)

References 35 publications

Supporting

Mentioning

121

Contrasting

Order By: Relevance

“…Consistent with the previous works [ 12 , 52 ], we adopt the metrics of 7-class accuracy (from strongly negative to strongly positive), binary accuracy (positive/negative sentiments) and F1 score (harmonic mean of the binary precision and recall). Specifically, the predicted digit will be rounded first.…”

Section: Resultsmentioning

confidence: 99%

LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition

Liu

Shen

et al. 2022

Entropy

View full text Add to dashboard Cite

Semantic-rich speech emotion recognition has a high degree of popularity in a range of areas. Speech emotion recognition aims to recognize human emotional states from utterances containing both acoustic and linguistic information. Since both textual and audio patterns play essential roles in speech emotion recognition (SER) tasks, various works have proposed novel modality fusing methods to exploit text and audio signals effectively. However, most of the high performance of existing models is dependent on a great number of learnable parameters, and they can only work well on data with fixed length. Therefore, minimizing computational overhead and improving generalization to unseen data with various lengths while maintaining a certain level of recognition accuracy is an urgent application problem. In this paper, we propose LGCCT, a light gated and crossed complementation transformer for multimodal speech emotion recognition. First, our model is capable of fusing modality information efficiently. Specifically, the acoustic features are extracted by CNN-BiLSTM while the textual features are extracted by BiLSTM. The modality-fused representation is then generated by the cross-attention module. We apply the gate-control mechanism to achieve the balanced integration of the original modality representation and the modality-fused representation. Second, the degree of attention focus can be considered, as the uncertainty and the entropy of the same token should converge to the same value independent of the length. To improve the generalization of the model to various testing-sequence lengths, we adopt the length-scaled dot product to calculate the attention score, which can be interpreted from a theoretical view of entropy. The operation of the length-scaled dot product is cheap but effective. Experiments are conducted on the benchmark dataset CMU-MOSEI. Compared to the baseline models, our model achieves an 81.0% F1 score with only 0.432 M parameters, showing an improvement in the balance between performance and the number of parameters. Moreover, the ablation study signifies the effectiveness of our model and its scalability to various input-sequence lengths, wherein the relative improvement is almost 20% of the baseline without a length-scaled dot product.

show abstract

Section: Resultsmentioning

confidence: 99%

LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition

Liu

Shen

et al. 2022

Entropy

View full text Add to dashboard Cite

show abstract

“…The heat map of the results on the multi-modal fine-grained emotion analysis structure based on feature layer fusion is shown in Figure 9 and Figure 10 . As shown in Table 3 , the benchmark models of this experiment are HiGRU [ 29 ], HiGRU-sf [ 29 ], mement [ 30 ], cLSTM [ 17 ], TFN [ 31 ], MFN [ 32 ], CMU [ 33 ], and ICON [ 10 ], all of which are the optimal results for the multi-modal fine-grained analysis of the IEMOCAP data set. Other models, including SVC, LR, etc., are all models used after over-sampling the samples.…”

Section: Parameter Setting and Results Analysismentioning

confidence: 99%

Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion

Zhao

Dong

Shi

et al. 2022

Sensors

View full text Add to dashboard Cite

With the wide application of social media, public opinion analysis in social networks has been unable to be met through text alone because the existing public opinion information includes data information of various modalities, such as voice, text, and facial expressions. Therefore multi-modal emotion analysis is the current focus of public opinion analysis. In addition, multi-modal emotion recognition of speech is an important factor restricting the multi-modal emotion analysis. In this paper, the emotion feature retrieval method for speech is firstly explored and the processing method of sample disequilibrium data is then analyzed. By comparing and studying the different feature fusion methods of text and speech, respectively, the multi-modal feature fusion method for sample disequilibrium data is proposed to realize multi-modal emotion recognition. Experiments are performed using two publicly available datasets (IEMOCAP and MELD), which shows that processing multi-modality data through this method can obtain good fine-grained emotion recognition results, laying a foundation for subsequent social public opinion analysis.

show abstract

“…The most straightforward way is to directly concatenating feature maps from each modality (Ngiam et al 2011). To leverage complementary information across different modalities, tensor fusion (Zadeh et al 2017;Liu et al 2018), memory fusion (Zadeh et al 2018a), factorization fusion (Valada, Mohan, and Burgard 2020) explicitly account for intra-modal and inter-modal dynamics. The above mentioned methods are aggregation-based fusion and the modality gap heavily affects cross-modal fusion.…”

Section: Related Workmentioning

confidence: 99%

Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition

Zhang

Chen

Shen

et al. 2022

AAAI

View full text Add to dashboard Cite

Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), aiming to refine multi-modal representations and enhance discriminative capacity of each label. Specifically, we design an adversarial multi-modal refinement module to sufficiently explore the commonality among different modalities and strengthen the diversity of each modality. To further exploit label-modal dependence, we devise a BERT-like cross-modal encoder to gradually fuse private and common modality representations in a granularity descent way, as well as a label-guided decoder to adaptively generate a tailored representation for each label with the guidance of label semantics. In addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in both aligned and unaligned settings, which demonstrate the superiority of TAILOR over the state-of-the-arts.

show abstract

Memory Fusion Network for Multi-view Sequential Learning

Cited by 415 publications

References 35 publications

LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition

LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition

Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion

Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition

Contact Info

Product

Resources

About