Multimodal sentiment analysis of heterogeneous bimodal interactions

Abstract

Aiming at the problems of unsatisfactory cross-modal feature interaction in multimodal sentiment analysis and the influence of each modality on the final sentiment classification results. Through the research on multimodal sentiment analysis methods, a multimodal sentiment analysis model based on heterogeneous extracted feature network and bimodal interaction is constructed. The model firstly uses DPCNN network to capture the deep semantics of text, BiLSTM network to obtain the upper and lower dependencies of speech, and TCN network to extract the relevant features for facial images; in order to learn the fusion features between different modalities, the interaction information within and between modalities is captured by the bimodal interaction network layer; and then the attention mechanism is used to determine the attentional weights of the modalities in the final sentiment features, and to classify the influence of modalities on the final sentiment classification results. attentional weights and fuses the modalities; finally, the final sentiment classification is performed through the fully connected layer and multi-head attention. The model is experimented and evaluated on the datasets CMU-MOSI and CMU-MOSEI, and the experimental results show that the model outperforms other models in the sentiment analysis problem compared to the existing models.