Comparative Study on Feature Selection and Fusion Schemes for Emotion Recognition from Speech

Abstract

The automatic analysis of speech to detect affective states may improve the way users interact with electronic devices. However, the analysis only at the acoustic level could be not enough to determine the emotion of a user in a realistic scenario. In this paper we analyzed the spontaneous speech recordings of the FAU Aibo Corpus at the acoustic and linguistic levels to extract two sets of features. The acoustic set was reduced by a greedy procedure selecting the most relevant features to optimize the learning stage. We compared two versions of this greedy selection algorithm by performing the search of the relevant features forwards and backwards. We experimented with three classification approaches: Naïve-Bayes, a support vector machine and a logistic model tree, and two fusion schemes: decision-level fusion, merging the hard-decisions of the acoustic and linguistic classifiers by means of a decision tree; and featurelevel fusion, concatenating both sets of features before the learning stage. Despite the low performance achieved by the linguistic data, a dramatic improvement was achieved after its combination with the acoustic information, improving the results achieved by this second modality on its own. The results achieved by the classifiers using the parameters merged at feature level outperformed the classification results of the decision-level fusion scheme, despite the simplicity of the scheme. Moreover, the extremely reduced set of acoustic features obtained by the greedy forward search selection algorithm improved the results provided by the full set.

Minimal cross-correlation criterion for speech emotion multi-level feature selection

“…Some predefined speech emotion feature sets are extracted during speech analysis step. These sets can include various temporal, spectral, cepstral features, their mathematical derivatives, statistical estimates and should represent the emotion state of the speaker [2,3,4,8]. During the classification step the extracted feature sets are classified trying to define the emotional class of the analyzed speech patterns.…”

“…For example, signal energy features would define low energy and high energy emotion classes. During the second stage every of these classes are classified into lower level classes (or particular emotions) using different second-level feature sets F m (2) . In general case there can be L classification stages with different feature sets F M (L) .…”

“…Speech emotion recognition can be applied for assessment of caller's emotional state in call centers, evaluation of listener's emotional reaction, automated analysis of audio content in dialogue systems, tutoring / e-learning process, entertainment and serious games [2]. Regardless of intensive researches and numerous experimental results the main problem of the speech emotion recognition task is still unsolved.…”