Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions

Abstract

Depression is one of the most common mood disorders. Technology has the potential to assist in screening and treating people with depression by robustly modeling and tracking the complex behavioral cues associated with the disorder (e.g., speech, language, facial expressions, head movement, body language). Similarly, robust affect recognition is another challenge which stands to benefit from modeling such cues. The Audio/Visual Emotion Challenge (AVEC) aims toward understanding the two phenomena and modeling their correlation with observable cues across several modalities. In this paper, we use multimodal signal processing methodologies to address the two problems using data from human-computer interactions. We develop separate systems for predicting depression levels and affective dimensions, experimenting with several methods for combining the multimodal information. The proposed depression prediction system uses a feature selection approach based on audio, visual, and linguistic cues to predict depression scores for each session. Similarly, we use multiple systems trained on audio and visual cues to predict the affective dimensions in continuous-time. Our affect recognition system accounts for context during the frame-wise inference and performs a linear fusion of outcomes from the audio-visual systems. For both problems, our proposed systems outperform the videofeature based baseline systems. As part of this work, we analyze the role played by each modality in predicting the target variable and provide analytical insights.

Automatic affective dimension recognition from naturalistic facial expressions based on wavelet filtering and PLS regression

“…It was also compared with all the stateof-the-art methods in the AVEC2014 affect recognition subchallenge with fairly good performance. NLPR [4], SAIL [9], BU-CMPE [11] and our method achieve better performance than baseline method, while Ulm [10] achieves best performance. However, it utilized extra information on subjects and annotation process that is not comparable with other methods.…”

“…The single modality regressors are then combined using particle filtering, by treating these independent regression outputs as measurements of the affective states in a Bayesian filtering framework, where previous observations provide prediction about the current state by means of learned affect dynamics. At AVEC2014 affect recognition sub-challenge, the temporal relations in naturalistic expressions was used to boost the performance in decision level filtering [10] [9]. Kachele et al [10] proposed an approach based on abstract meta information about individual subjects and also prototypical task and label dependent templates to infer the respective emotional states and achieved the best performance.…”

“…Kachele et al [10] proposed an approach based on abstract meta information about individual subjects and also prototypical task and label dependent templates to infer the respective emotional states and achieved the best performance. Gupta et al [9] proposed a 4-stage system including individual training, prediction processing, system fusion and temporal regression. Chao et.…”

Survey of Approaches for Building Emotion Detection Applications using a Multi-modal Approach

“…Liyanage C. De Silva, Pei Chi Ng, of Singapore used statistical techniques and Hidden Markov Models (HMM) [8] for the recognition of emotions. The method classifies six fundamental emotions namely anger, dislike, fear, happiness, sadness and surprise from facial expressions and emotional speech.…”

The Influence of Annotation, Corpus Design, and Evaluation on the Outcome of Automatic Classification of Human Emotions

“…Thus, it was possible to outperform all other submitted approaches that used elaborate learning techniques such as deep neural networks (Chao et al, 2014) and support vector machines (Gupta et al, 2014) on complex audio and video features.…”