Speech Emotion Recognition with Heterogeneous Feature Unification of Deep Neural Network

Abstract

Automatic speech emotion recognition is a challenging task due to the gap between acoustic features and human emotions, which rely strongly on the discriminative acoustic features extracted for a given recognition task. We propose a novel deep neural architecture to extract the informative feature representations from the heterogeneous acoustic feature groups which may contain redundant and unrelated information leading to low emotion recognition performance in this work. After obtaining the informative features, a fusion network is trained to jointly learn the discriminative acoustic feature representation and a Support Vector Machine (SVM) is used as the final classifier for recognition task. Experimental results on the IEMOCAP dataset demonstrate that the proposed architecture improved the recognition performance, achieving accuracy of 64% compared to existing state-of-the-art approaches.

Ensemble Learning of Hybrid Acoustic Features for Speech Emotion Recognition

“…Pérez-Espinosa et al [35] analyzed 6920 acoustic features from the interactive emotion dyadic motion capture (IEMOCAP) database to discover that features based on groups of MFCC, LPC and cochleagrams are important for estimating valence, activation and dominance emotions in speech respectively. The authors in [36] proposed a DNN architecture for extracting informative feature representatives from heterogeneous acoustic feature groups that may contain redundant and unrelated information. The architecture was tested by training the fusion network to jointly learn highly discriminating acoustic feature representations from the IEMOCAP database for speech emotion recognition using SVM to obtain an overall accuracy of 64.0%.…”

“…Emotion recognition is still a great challenge because of several reasons as previously alluded in the introductory message. Further reasons include the existence of a gap between acoustic features and human emotions [31,36] and the non-existence of a solid theoretical foundation relating the characteristics of voice to the emotions of a speaker [20]. These intrinsic challenges have led to the disagreement in the literature on which features are best for speech emotion recognition [20,36].…”

Speech Emotion Recognition based on Voiced Emotion Unit

“…The algorithm used for this segmentation is based only on acoustic information that make it easy to be re-implemented in real-time. Suppose the utterance is segmented into its voiced segments using this algorithm, the output of segmentation process are the waveforms of all voiced segments which can be written as follows: (1) where (i) is the utterance index, represents the sequence of all voiced segments for utterance , is the jth voiced segment, and is the number of voiced segments in this utterance. For example, the result of segmentation of an utterance into its voiced segments is shown in Figure 1.…”

“…Automatic speech emotion recognition (ASER) systems becoming a very important technology for human-computer interaction [1]. This technology can be embedded in many computer applications as well as in robots to make them sensitive to the user's emotional voice [2].…”

“…Linguistically motivated units such as utterance has many limitations for continuous emotion recognition for the following reasons: (1) it requires preprocessing by an automatic speech recognition (ASR) system to obtain unit boundaries, (2) time constrictions make it obligatory not to wait with ASR till the speaker has complete his/her whole turn, (3) emotion is dynamic and may changes during the utterance, especially in spontaneous speech, utterances' duration are much varied from very short to very long utterance. Therefore, long utterances may include different emotional states.…”