Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions

Gandhi, Ankita; Adhvaryu, Kinjal; Poria, Soujanya; Cambria, Erik; Hussain, Amir

doi:10.1016/j.inffus.2022.09.025

Cited by 255 publications

(64 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…27 NLP, along with other deep learning algorithms, has already been recognized as an effective sentiment analysis technique required to understand human cognitive behavior. 28 2. Incident alerts: The generation of false positives and false negatives is a crucial problem in IoT security.…”

Section: Possible Future Directions In Iot System Security Using Nlpmentioning

confidence: 99%

Security in IoT systems using natural language processing: Future challenges and directions

Kumar

2023

Internet Technology Letters

View full text Add to dashboard Cite

The internet of things (IoT) is advancing human lives by providing various intelligent services. The advancement of cutting-edge technologies necessitates an increase in the need for cyber security. In this light, an extensive research has been conducted to address IoT security concerns. However, several security challenges must be addressed before IoT can be utilized to its full potential in real-time. Motivated by this fact, this study presents a novel integration approach of Natural Language Processing (NLP) with IoT systems to enhance its security. It investigates the current state of NLP and its applications. The key contribution of this paper is the development of a system for identifying malicious behaviors in an IoT environment using n-gram and word-embedding techniques. Several security challenges and possible future directions to enhance IoT security using NLP are also addressed in this study.

show abstract

Section: Possible Future Directions In Iot System Security Using Nlpmentioning

confidence: 99%

Security in IoT systems using natural language processing: Future challenges and directions

Kumar

2023

Internet Technology Letters

View full text Add to dashboard Cite

show abstract

“…Sentiment analysis is a classification task that classifies a text into a positive or negative orientation. Sentiment analysis is a reasonably complex research, sentiment analysis is the process of using text analytics to obtain various data sources from the internet and social media platforms to determine the emotional tone or opinion of users on the platform [7], [8]. Sentiment analysis is also often referred to as opinion mining because it is often used to explore the emotions behind every user's words when using the internet or in conversations on social media [8], [9].…”

Section: Sentiment Analysismentioning

confidence: 99%

Sentiment Analysis Using Backpropagation Method to Recognize the Public Opinion

Wiguna

Sugiartawan

Sudipa

et al. 2022

Indonesian J. Comput. Cybern. Syst.

View full text Add to dashboard Cite

Improve the service quality of tourism actors by conducting sentiment analysis on digital platforms owned by tourism businesses and collecting negative sentiments to improve the quality of services from companies owned by tourism businesses. The growth of the hospitality industry in Indonesia is experiencing rapid growth every year. The tourism industry, part of the hospitality industry, also does not escape the influence of positive and negative sentiments. One method to perform accurate sentiment analysis is Backpropagation Neural Network. Based on the results of tests on the neural network, the best accuracy is obtained when using one hidden layer with the first layer of 10 neurons. The learning rate is 0.000002, where the accuracy is 71.630%. More epochs do not guarantee better accuracy. Based on the results of the research that has been done, suggestions for further researchers are to analyze the review dataset processing method so that it gets a cleaner dataset and is expected to improve better accuracy.

show abstract

“…In the past decade, machine learning and deep learning methods have started to play an important role in the development of theoretical methods. From machine learning with chemical property descriptors to deep learning with primitives, models are becoming more accurate and generalized than ever before. − ,− Recently, multimodal deep learnings have began to flourish, because of the advancement of diversified information acquisition algorithms . Since the representation of a target can be extracted from various expression forms, such as images, semantic sequence, spatial network, etc., multimodal models making full use of multiform inputs exhibit superiority in disease diagnosis, attack detection, semantic interpretability analysis, etc.…”

Section: Introductionmentioning

confidence: 99%

“…32−35,37−39 Recently, multimodal deep learnings have began to flourish, because of the advancement of diversified information acquisition algorithms. 40 Since the representation of a target can be extracted from various expression forms, such as images, semantic sequence, spatial network, etc., multimodal models making full use of multiform inputs exhibit superiority in disease diagnosis, 41 attack detection, 42 semantic interpretability analysis, 43 etc. All these types of machine learning models have been used for predicting NCIs as well.…”

Section: Introductionmentioning

confidence: 99%

TFRegNCI: Interpretable Noncovalent Interaction Correction Multimodal Based on Transformer Encoder Fusion

Wang

et al. 2023

J. Chem. Inf. Model.

View full text Add to dashboard Cite

The interpretability is an important issue for end-to-end learning models. Motivated by computer vision algorithms, an interpretable noncovalent interaction (NCI) correction multimodal (TFRegNCI) is proposed for NCI prediction. TFRegNCI is based on RegNet feature extraction and a transformer encoder fusion strategy. RegNet is a network design paradigm that mainly focuses on local features. Meanwhile, the Vision Transformer is also leveraged for feature extraction, because it can capture global features better than RegNet while lowering the computational cost. Using a transformer encoder as the fusion strategy rather than multilayer perceptron can enhance model performance, due to its emphasis on important features with less parameters. Therefore, the proposed TFRegNCI achieved high accurate prediction (mean absolute error of ∼0.1 kcal/mol) comparing with the coupled cluster single double (triple) (CCSD(T)) benchmark. To further improve the model efficiency, TFRegNCI applies two-dimensional (2D) inputs transformed from three-dimensional (3D) electron density cubes, which saves time (30%), while the model accuracy remains. To improve model interpretability, a visualization module, Gradient-weighted Regression Activation Mapping (Grad-RAM) has been embedded. Grad-RAM is promoted from the classification algorithm, Gradient-weighted Class Activation Mapping, to perform feature visualization for the regression task. With Grad-RAM, the visual location map for features in deep learning models can be displayed. The feature map visualizations suggest that the 2D model has the similar performance as the 3D model, because of equally effective feature extractions from electron density. Moreover, the valid feature region on the location map by the 3D model is consistent with the NCIPLOT NCI isosurface. It is confirmed that the model does extract significant features related to the NCI interaction. The interpretable analyses are carried out through molecular orbital contribution on effective features. Thereby, the proposed model is likely to be a promising tool to reveal some essential information on NCIs, with regard to the level of electronic theory.

show abstract

Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions

Cited by 255 publications

References 67 publications

Security in IoT systems using natural language processing: Future challenges and directions

Security in IoT systems using natural language processing: Future challenges and directions

Sentiment Analysis Using Backpropagation Method to Recognize the Public Opinion

TFRegNCI: Interpretable Noncovalent Interaction Correction Multimodal Based on Transformer Encoder Fusion

Contact Info

Product

Resources

About