Sentiment-aware multimodal pre-training for multimodal sentiment analysis

Ye, Junjie; Zhou, Jie; Tian, Junfeng; Wang, Rui; Zhou, Jingyi; Gui, Tao; Zhang, Qi; Huang, Xuanjing

doi:10.1016/j.knosys.2022.110021

Cited by 37 publications

(8 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ViT. Following SMP (Ye et al, 2022), we adopt ViT to model the image by dividing it into 16 by 16 patches. A CLS token is added at the beginning and fed into the Transformer (Vaswani et al, 2017) encoder to obtain the image representation I " rv cls , v 1 , v 2 , ..., v 196 s, where v i P R 768 .…”

Section: Unimodal Encodermentioning

confidence: 99%

“…The following research on the TMSC task can be divided into two directions. On the one hand, there is the continuous exploration of how to enhance the interactions between modalities (Khan and Fu, 2021), and on the other hand, there is the application of pre-trained models to this task (Ye et al, 2022;Ling et al, 2022). Despite these efforts, the current models have not yet achieved significant performance gains relative to the text-only models.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification

Ye,

Zhou,

Tian

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Recently, Target-oriented Multimodal Sentiment Classification (TMSC) has gained significant attention among scholars. However, current multimodal models have reached a performance bottleneck. To investigate the causes of this problem, we perform extensive empirical evaluation and in-depth analysis of the datasets to answer the following questions: Q1: Are the modalities equally important for TMSC? Q2: Which multimodal fusion modules are more effective? Q3: Do existing datasets adequately support the research? Our experiments and analyses reveal that the current TMSC systems primarily rely on the textual modality, as most of targets' sentiments can be determined solely by text. Consequently, we point out several directions to work on for the TMSC task in terms of model design and dataset construction. The code and data can be found in https://github.com/Junjie-Ye/ RethinkingTMSC.

show abstract

Section: Unimodal Encodermentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification

Ye,

Zhou,

Tian

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…TF-IDF is a word weighting algorithm that multiplies two-term weighting concepts, namely the frequency of occurrence of terms in a document (term frequency) and the distribution of terms in document collections (inverse document frequency). Term Frequency is defined in Equation (1). Term Frequency -Inverse Document Frequency defined in Equation (3)…”

Section: 1 Text Preprocessingmentioning

confidence: 99%

“…Who has complaints while staying at the hotel so they can be handled directly to prevent losing customers? To be able to handle complaints submitted by customers immediately, an artificial intelligence program is needed that can detect negative sentiments on the digital platform owned by the company so that hoteliers can immediately follow up on complaints submitted by their customers so that no customers are reluctant to visit because of an incident [1], [2].…”

Section: Introductionmentioning

confidence: 99%

Sentiment Analysis Using Backpropagation Method to Recognize the Public Opinion

Wiguna

Sugiartawan

Sudipa

et al. 2022

Indonesian J. Comput. Cybern. Syst.

View full text Add to dashboard Cite

Improve the service quality of tourism actors by conducting sentiment analysis on digital platforms owned by tourism businesses and collecting negative sentiments to improve the quality of services from companies owned by tourism businesses. The growth of the hospitality industry in Indonesia is experiencing rapid growth every year. The tourism industry, part of the hospitality industry, also does not escape the influence of positive and negative sentiments. One method to perform accurate sentiment analysis is Backpropagation Neural Network. Based on the results of tests on the neural network, the best accuracy is obtained when using one hidden layer with the first layer of 10 neurons. The learning rate is 0.000002, where the accuracy is 71.630%. More epochs do not guarantee better accuracy. Based on the results of the research that has been done, suggestions for further researchers are to analyze the review dataset processing method so that it gets a cleaner dataset and is expected to improve better accuracy.

show abstract

“…We consider the polarity of each aspect to provide personalized recommendations based on the user's preferences. To determine polarity, we employed a sentiment analysis method that has been demonstrated to classify reviews as positive, negative or neutral (Punetha and Jain, 2023; Ye et al , 2022). The multiple attention network (Qiang et al , 2020) incorporates two attention mechanisms, i.e.…”

Section: Introductionmentioning

confidence: 99%

AsCDPR: a novel framework for ratings and personalized preference hotel recommendation using cross-domain and aspect-based features

Wang,

Justitia,

Wang

2023

DTA

View full text Add to dashboard Cite

PurposeThe explosion of data due to the sophistication of information and communication technology makes it simple for prospective tourists to learn about previous hotel guests' experiences. They prioritize the rating score when selecting a hotel. However, rating scores are less reliable for suggesting a personalized preference for each aspect, especially when they are in a limited number. This study aims to recommend ratings and personalized preference hotels using cross-domain and aspect-based features.Design/methodology/approachWe propose an aspect-based cross-domain personalized recommendation (AsCDPR), a novel framework for rating prediction and personalized customer preference recommendations. We incorporate a cross-domain personalized approach and aspect-based features of items from the review text. We extracted aspect-based feature vectors from two domains using bidirectional long short-term memory and then mapped them by a multilayer perceptron (MLP). The cross-domain recommendation module trains MLP to analyze sentiment and predict item ratings and the polarities of the aspect based on user preferences.FindingsExpanded by its synonyms, aspect-based features significantly improve the performance of sentiment analysis on accuracy and the F1-score matrix. With relatively low mean absolute error and root mean square error values, AsCDPR outperforms matrix factorization, collaborative matrix factorization, EMCDPR and Personalized transfer of user preferences for cross-domain recommendation. These values are 1.3657 and 1.6682, respectively.Research limitation/implicationsThis study assists users in recommending hotels based on their priority preferences. Users do not need to read other people's reviews to capture the key aspects of items. This model could enhance system reliability in the hospitality industry by providing personalized recommendations.Originality/valueThis study introduces a new approach that embeds aspect-based features of items in a cross-domain personalized recommendation. AsCDPR predicts ratings and provides recommendations based on priority aspects of each user's preferences.

show abstract

Sentiment-aware multimodal pre-training for multimodal sentiment analysis

Cited by 37 publications

References 33 publications

RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification

RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification

Sentiment Analysis Using Backpropagation Method to Recognize the Public Opinion

AsCDPR: a novel framework for ratings and personalized preference hotel recommendation using cross-domain and aspect-based features

Contact Info

Product

Resources

About