Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network

Abstract

Audio Sentiment Analysis is a popular research area which extends the text-based sentiment analysis to depend on effectiveness of acoustic features extracted from speech. However, current progress on audio sentiment analysis mainly focuses on extracting homogeneous acoustic features or doesn't fuse heterogeneous features effectively. In this paper, we propose an utterance-based deep neural network model, which has a parallel combination of CNN and LSTM based network, to obtain representative features termed Audio Sentiment Vector (ASV), that can maximally reflect sentiment information in an audio. Specifically, our model is trained by utterance-level labels and ASV can be extracted and fused creatively from two branches. In the CNN model branch, spectrum graphs produced by signals are fed as inputs while in the LSTM model branch, inputs include spectral centroid, MFCC and other recognized traditional acoustic features extracted from dependent utterances in an audio. Besides, BiLSTM with attention mechanism is used for feature fusion. Extensive experiments have been conducted to show our model can recognize audio sentiment precisely, and demonstrate our ASV are better than traditional acoustic features or vectors extracted from other deep learning models. Furthermore, experimental results indicate that the proposed model outperforms state-of-the-art approaches by 9.33% on MOSI.

Real time sentiment analysis of natural language using multimedia input

“…Sentiment analysis of News Videos was conducted by Pereira et al [19] based on the audio, visual and textual features of these videos, using a myriad of ML techniques, achieving an accuracy of 75%. Luo et al [15] used a parallel combination of an LSTM and CNN based network to conduct audio-based sentiment detection on the MOSI dataset. Naïve Bayes algorithm was used on the Twitter dataset by Parveen et al [18] for sentiment analysis, which yielded an accuracy of 57%.…”

DeepSafety:Multi-level Audio-Text Feature Extraction and Fusion Approach for Violence Detection in Conversations

“…The research field of emotion recognition using audio-based technologies is gaining increasing attention in recent years, leading to an increasing number of research studies in the field [42,43,44,45,46]. For instance, the work in [6] proposes a Bidirectional Long Short-Term Memory (Bi-LSTM) with attention model to classify four different emotions including "anger", "Excitement", "Neutral" and "Sadness" from the IEMOCAP dataset, which includes a range of dyadic sessions where actors perform improvisations or scripted scenarios specifically selected to elicit emotional expressions.…”

A review on sentiment discovery and analysis of educational big‐data

“…This is because sentiment recognition depends on detecting a very focused vocabulary in the spoken comments (Kaushik, Sangwan, & Hansen, ). Furthermore, during the voice is transferred into the text, some sentiment‐related signal characteristics are also lost, resulting in a decrease in the classification accuracy of sentiments (Luo et al, ).…”

“…Based on a large number of experiments, a BiLSTM with the attention mechanism is employed for sentiment information of each utterance in audio. Specifically, spectrum graphs, spectral features and cepstrum coefficient produced by signals are fed into CNN and LSTM for feature fusion (Luo, Xu, & Chen, ). Compared with the acoustic signal, texture analysis of audio spectral image can effectively characterize sentiment states (Özseven, ).…”