Bimodal HCI-related affect recognition

Abstract

Perhaps the most fundamental application of affective computing would be Human-Computer Interaction (HCI) in which the computer is able to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.

Bimodal Emotion Recognition using Speech and Physiological Changes

“…They could improve the performance of decision-level fusion by considering the dominant modality, determined by empirical studies, in case significant discrepancy between the outputs of each unimodal classifier has been observed. Recently, a large-scale audio-visual dataset was collected by (Zeng et al, 2004), which contains five HCI-related affective responses (confusion, interest, boredom, and frustration) in addition to seven affects (the six basic emotions + neutral). To classify the 11 emotions subject-dependently, they used the SNoW (Sparse Network of Winnow) classifier with Naive Bayes as the update rule and achieved a recognition accuracy of almost 90% through bimodal fusion while the unimodal classifiers yielded only 45-56%.…”

Audio-Visual Affect Recognition in Activation-Evaluation Space

“…The number of 30 features seems to be good point where the training and testing errors are about 0.14 and 0.16 on evaluation axis, and 0.11 and 0.13 on activation axis. These coarse-emotion-category-based performances are greatly above our previous fineemotion-category-based classification at the frame level [11]. …”

“…In addition, [11] considered four HCI-related cognitive states besides six basic emotions. Contrary to the fine categories of emotion representation, this paper explores the coarse categories for automatic affect recognition.…”

Steps in Identifying Interaction Design Patterns for Multimodal Systems

“…This variant is manifested in very different application areas including among others data input (audio-visual speech recognition), person identification [39], emotion recognition [44].…”