Fusing Audio, Textual, and Visual Features for Sentiment Analysis of News Videos

Singh

et al. 2023

Multimed Tools Appl

Semantics and Sentiments are parts of our daily speech and expressions that helps to convey the message in the tone intended. The accurate interpretation of emotions and actions is prudent as it expresses the true meaning of the message. This interpretation has been studied extensively in the past two decades, where professionals from various disciplines have pondered this question. Every action and expression—whether it’s in a speech, in a video or through some written material—helps the recipient understand the intent behind the message. The primary motive in these studies has been to automate the analysis of these sentiments by teaching the computers to do so, using the audio, video and text-based data that has been collected so far. Machine Learning (ML) and Deep Learning (DL) is the discipline that can help us tackle such a problem which requires analysis and recognition of copious amounts of data. Classification based on these multi-media inputs has seen the application of several common and uncommon ML techniques such as Support Vector Machines (SVMs), Bayesian Networks (BNs), Decision Trees (DTs), Convolutional Neural Networks (CNNs) and K-Means Clustering. These techniques, to a certain level of accuracy, can classify a certain part of a message into a different emotion. Through this research, firstly, a comparison is represented between the previously conducted studies and secondly, a system is developed of our own that enables Real Time Sentiment Analysis and helps a user assess his/her day-to-day attitude and get appropriate recommendations for the same.

Section: Literature Reviewmentioning

confidence: 99%

Real time sentiment analysis of natural language using multimedia input

Singh

et al. 2023

Multimed Tools Appl

“…Videos and images are analyzed using different suitable model to provide better prediction. Multi modal based solution using suitable model according to the information type is a novel one [11] [16]. The entire application will be in single framework from the collection of data with the help of social media or chatbot analyzing the data using suitable model and the sending the relevant data to the rescue team [12].…”

Section: International Journal On Recent and Innovation Trends In Com...mentioning

confidence: 99%

An Ensemble Learning Approach for Fast Disaster Response using Social Media Analytics

Ayyadurai,

Sujatha,

Guru

et al. 2023

IJRITCC

Natural disaster happens, as a result of natural hazards that cause financial, environmental or human losses. Natural disasters strike unexpectedly, affecting the lives of tens of thousands of people. During the flood, social media sites were also heavily used to disseminate information about flooded areas, rescue agencies, food and relief centres. This work proposes an ensemble learning strategy for combining and analysing social media data in order to close the gap and progress in catastrophic situation. To enable scalability and broad accessibility of the dynamic streaming of multimodal data namely text, image, audio and video, this work is designed around social media data. A fusion technique was employed at the decision level, based on a database of 15 characteristics for more than 300 disasters around the world (Trained with MNIST dataset 60000 training images and 10000 testing images). This work allows the collected multimodal social media data to share a common semantic space, making individual variable prediction easier. Each merged numerical vector(tensors) of text and audio is sent into the K-CNN algorithm, which is an unsupervised learning algorithm (K-CNN), and the image and video data is given to a deep learning based Progressive Neural Artificial Search (PNAS). The trained data acts as a predictor for future incidents, allowing for the estimation of total deaths, total individuals impacted, and total damage, as well as specific suggestions for food, shelter and housing inspections. To make such a prediction, the trained model is presented a satellite image from before the accident as well as the geographic and demographic conditions, which is expected to result in a prediction accuracy of more than 85%.

“…The benefits arising from a combined analysis of different features extracted from different modalities were demonstrated in [73], where a combination of the modulations in speech, textual clues, and facial expressions extracted from videos improved the identification of the level of tension from newscasts.…”

Section: Video Data and Emotion Classificationmentioning

confidence: 99%

Emotion Classification from Speech and Text in Videos Using a Multimodal Approach

Caschera

Grifoni

Ferri

2022

MTI

Emotion classification is a research area in which there has been very intensive literature production concerning natural language processing, multimedia data, semantic knowledge discovery, social network mining, and text and multimedia data mining. This paper addresses the issue of emotion classification and proposes a method for classifying the emotions expressed in multimodal data extracted from videos. The proposed method models multimodal data as a sequence of features extracted from facial expressions, speech, gestures, and text, using a linguistic approach. Each sequence of multimodal data is correctly associated with the emotion by a method that models each emotion using a hidden Markov model. The trained model is evaluated on samples of multimodal sentences associated with seven basic emotions. The experimental results demonstrate a good classification rate for emotions.