Modality adaptation in multimodal data

Razzaghi, Parvin; Abbasi, Karim; Shirazi, Mahmoud; Shabani, Niloofar

doi:10.1016/j.eswa.2021.115126

Cited by 18 publications

(4 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to single-modal, multimodal AI anthropomorphic interaction is capable of visual interaction alongside voice interaction, and the information of voice and visual can be consistent and complementary to each other, thus achieving better interaction results. The dual coding theory shows [65] that the human brain can enhance information recall and recognition by processing information visually and verbally simultaneously, as information between different modalities is corroborated and complemented by each other [66]. Similarly, emotional cues are also mutually corroborated and complemented, eliminating emotional differences and enabling more accurate emotional transmission during interaction [67].…”

Section: Moderating Effect Of Modalities Of Ai Anthropomorphic Intera...mentioning

confidence: 99%

Enablers or Inhibitors? Unpacking the Emotional Power Behind In-Vehicle AI Anthropomorphic Interaction: A Dual-Factor Approach by Text Mining

Bai,

Yu,

Han

et al. 2024

IEEE Trans. Eng. Manage.

View full text Add to dashboard Cite

The intelligent strategy of the new energy vehicle (NEV) industry has triggered the rapid prevalence of in-vehicle anthropomorphic artificial intelligence (AI) assistants. There is still a lack of clarity regarding NEV users' attitudes toward this cuttingedge technology and whether they receive a satisfactory intelligent service experience. To circumvent potential emerging technology resistance, in this article, we utilize text analysis techniques for the identification of AI interaction emotions, love and disgust (enablers and inhibitors) with significant influence on user satisfaction, and validates the improving role of multimodality on the effectiveness of anthropomorphic interaction. In addition, this study innovatively constructs a multidimensional corpus of modality × emotion, using structural topic modeling to uncover the constituent elements and real-time changes of love and disgust emotions in different modalities, from which development opportunities and improvement directions for AI anthropomorphic interaction technologies are identified. The findings provide new insights into the application of emotion analysis methods to improve users' intelligent service experience and provide a realistic reference for mitigating emerging technology resistance in the NEV industry.

show abstract

Section: Moderating Effect Of Modalities Of Ai Anthropomorphic Intera...mentioning

confidence: 99%

Enablers or Inhibitors? Unpacking the Emotional Power Behind In-Vehicle AI Anthropomorphic Interaction: A Dual-Factor Approach by Text Mining

Bai,

Yu,

Han

et al. 2024

IEEE Trans. Eng. Manage.

View full text Add to dashboard Cite

show abstract

“…For example, Karim et al [ 34 ] migrated relevant knowledge from the source data set to the target data set, which solved the problem of the small amount of available data for small molecule compounds. Parvin et al [ 35 ] proposed the modality adaptation technique to effectively transfer and fuse the domain knowledge of multimodality and improve the performance of multimodal applications. Besides, the pre-trained convolutional neural network has been successfully used as a preprocessing tool and is widely used in medical image classification [ 36 , 37 ].…”

Section: Related Workmentioning

confidence: 99%

NSCGCN: A novel deep GCN model to diagnosis COVID-19

Tang

Sun

et al. 2022

Computers in Biology and Medicine

View full text Add to dashboard Cite

“…People’s information exchange and emotional expression usually use various data modalities ( Ma et al, 2020 ), which refer to the different channels of information that are semantically correlated and provide complementary information to each other ( Razzaghi et al, 2021 ), such as text, audio, pictures and videos. According to the number of data modalities included, it can be divided into single-modal data and multimodal data.…”

Section: Literature Review and Hypotheses Developmentmentioning

confidence: 99%

Is negative e-WOM more powerful? Multimodal data analysis on air passengers’ perception of COVID-19 safety measures

Bai

Yang

et al. 2022

Front. Psychol.

View full text Add to dashboard Cite

During the normalization stage of the COVID-19 epidemic prevention and control, the safety threats caused by improper epidemic prevention measures of airlines have become the primary concern for air passengers. Negative e-WOM related to safety perception obtained based on online multimodal reviews of travel websites has become an important decision-making basis for potential air passengers when making airline choices. This study aims to examine the relationship between potential air passengers’ negative safety perception and the usefulness of online reviews, as well as to test the moderating effect of review modality and airline type. It also further explores the effectiveness and feasibility of applying big data sentiment analysis to e-WOM management. To this end, the theoretical model of negative safety perception, review modality, and airline type affecting review usefulness was constructed. Then we select 10 low-cost airlines and 10 full-service airlines, respectively, according to the number of reviews sorted by the TripAdvisor website, and use crawling techniques to obtain 10,485 reviews related to COVID-19 safety of the above companies from December 2019 to date, and conduct safety perception sentiment analysis based on Python’s Textblob library. Finally, to avoid data overdispersion, the model is empirically analyzed by negative binomial regression using R software. The results indicate that (1) Negative safety perception significantly and negatively affects review usefulness, that is, extreme negative safety perception can provide higher review usefulness for potential air passengers. (2) Review modality and airline type have a significant moderating effect on the relationship between negative safety perception and review usefulness, in which multimodal reviews and full-service airlines both weakened the negative impact of negative safety perception on review usefulness. The theoretical model in this paper is both an extension of the application of big data sentiment analysis techniques and a beneficial supplement to current research findings of e-WOM, providing an important reference for potential air passengers to identify useful reviews accurately and thus reduce safety risks in online decision-making.

show abstract

Modality adaptation in multimodal data

Cited by 18 publications

References 29 publications

Enablers or Inhibitors? Unpacking the Emotional Power Behind In-Vehicle AI Anthropomorphic Interaction: A Dual-Factor Approach by Text Mining

Enablers or Inhibitors? Unpacking the Emotional Power Behind In-Vehicle AI Anthropomorphic Interaction: A Dual-Factor Approach by Text Mining

NSCGCN: A novel deep GCN model to diagnosis COVID-19

Is negative e-WOM more powerful? Multimodal data analysis on air passengers’ perception of COVID-19 safety measures

Contact Info

Product

Resources

About