Classification of Complex Information: Inference of Co-Occurring Affective States from Their Expressions in Speech

Abstract

We present a classification algorithm for inferring affective states (emotions, mental states, attitudes, and the like) from their nonverbal expressions in speech. It is based on the observations that affective states can occur simultaneously and different sets of vocal features, such as intonation and speech rate, distinguish between nonverbal expressions of different affective states. The input to the inference system was a large set of vocal features and metrics that were extracted from each utterance. The classification algorithm conducted independent pairwise comparisons between nine affective-state groups. The classifier used various subsets of metrics of the vocal features and various classification algorithms for different pairs of affective-state groups. Average classification accuracy of the 36 pairwise machines was 75 percent, using 10-fold cross validation. The comparison results were consolidated into a single ranked list of the nine affective-state groups. This list was the output of the system and represented the inferred combination of co-occurring affective states for the analyzed utterance. The inference accuracy of the combined machine was 83 percent. The system automatically characterized over 500 affective state concepts from the Mind Reading database. The inference of co-occurring affective states was validated by comparing the inferred combinations to the lexical definitions of the labels of the analyzed sentences. The distinguishing capabilities of the system were comparable to human performance.

Speech emotion recognition in emotional feedback for Human-Robot Interaction

“…Emotions are visualized through various indicators in humans, many of these indicators have been previously analyzed to provide affective knowledge to machines, focusing on facial expressions [5], [6], vocal features [7], [8], [9], body movements and postures [10], [11], [12], [13] and the integration of all of them in emotion analysis systems [14], [15], [16]. But human beings cannot always hope that robots may be able to react in a timely and sensible manner, especially if they haven't be able to recover all the affective information through their sensors.…”

Real-Time Recognition of Affective States from Nonverbal Features of Speech and Its Application for Public Speaking Skill Analysis

“…Popular classification models used include, among others, different decision trees [25], support vector machines [5], [26], [27], [17], [6], neural networks [12], [10] and Hidden Markov Models. Again, which is the best classifier often depends on the application and corpus.…”

“…This subset consisted of 548 samples spoken by 10 different actors. Each category contains samples from the groups as [5], [25], allowing direct comparison of results.…”

Multimodal Affect Recognition in Intelligent Tutoring Systems

“…The audio analysis component used in the research is an adaptation of the framework introduced by Sobol-Shikler [17] and enhanced by Pfister [18]. The OpenSMILE library extracts 6555 features which represent pitch, spectral envelope, and energy feature groups (amongst others); delta and acceleration information; and their functionals (e.g.…”