Cm-Bert

Abstract

Multimodal sentiment analysis is an emerging research field that aims to enable machines to recognize, interpret, and express emotion. Through the cross-modal interaction, we can get more comprehensive emotional characteristics of the speaker. Bidirectional Encoder Representations from Transformers (BERT) is an efficient pre-trained language representation model. Fine-tuning it has obtained new state-of-the-art results on eleven natural language processing tasks like question answering and natural language inference. However, most previous works fine-tune BERT only base on text data, how to learn a better representation by introducing the multimodal information is still worth exploring. In this paper, we propose the Cross-Modal BERT (CM-BERT), which relies on the interaction of text and audio modality to fine-tune the pre-trained BERT model. As the core unit of the CM-BERT, masked multimodal attention is designed to dynamically adjust the weight of words by combining the information of text and audio modality. We evaluate our method on the public multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results show that it has significantly improved the performance on all the metrics over previous baselines and text-only finetuning of BERT. Besides, we visualize the masked multimodal attention and proves that it can reasonably adjust the weight of words by introducing audio modality information. CCS CONCEPTS • Information systems → Sentiment analysis; Multimedia and multimodal retrieval; • Computing methodologies → Natural language processing;

Text‐based sentiment analysis in finance: Synthesising the existing literature and exploring future directions

“…In line with similar conclusions from psychology literature surrounding the importance of paralinguistic cues in the communication process, there are a number of studies employing sentiment analysis techniques that suggest a combination of text and audio data may improve classification accuracy, and consequently create a more robust representation of sentiment (Bhaskar et al, 2014; Dair et al, 2021; Houjeij et al, 2012; Yang et al, 2020). Hence, given that prior literature suggests both textual and vocal characteristics of earnings calls to be informative, and that Natural Language Processing literature finds a combination of text and audio to significantly increase classification accuracy, the adhesion of both measures represents a natural future direction for the literature.…”

“…The gold standard for multimodal sentiment analysis is considered to be the combination of all three communication modalities-text, audio and visual. Various studies have used the combination of all three modalities to define sentiment, showing that the use of a tri-modality model is more robust at classifying sentiment over bi-modal and singular modality models (Bhaskar et al, 2014;Dair et al, 2021;Houjeij et al, 2012;Morency et al, 2011;Poria et al, 2015;Yang et al, 2020). 44 The main advantage of using multimodal classifiers for sentiment classification is the additional behavioural cues provided by the visual and audio data.…”

“…They note that there is a scarcity of advanced NLP techniques being applied in the financial domain 32 . While the ML techniques discussed in the previous section have been shown to classify financial sentiment better than more rudimentary approaches, alternative approaches such as transformer architecture (Alamoudi & Alghamdi, 2021; Munikar et al, 2019; Sun et al, 2019) and multimodal analysis (Bhaskar et al, 2014; Dair et al, 2021; Houjeij et al, 2012; Yang et al, 2020) have been demonstrated as having greater abilities in accurately capturing sentiment.…”

Integrative Sentiment Analysis: Leveraging Audio, Visual, and Textual Data

“…Researchers have been working on different combinations of text, audio, and image modalities that can enhance prediction accuracy [10]. This area of study focuses on various methods of integrating multimodal information, primarily through feature fusion and decision fusion, as outlined in several studies [22][12] [5]. The choice of method often depends on the specific application and the nature of the data being analyzed.…”

cross-modal fusion techniques for utterance-level emotion recognition from text and speech

“…Emotion recognition in conversations (ERC) is vital and very challenging in the natural human machine interaction [1], intelligent education tutoring [2], and mental health analysis applications [3]. In daily life, humans utter a multi-turn conversation in a natural way which conveys emotion state through language and nonverbal content (e.g., facial expression and body language) [4].…”