Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text

Abstract

International audienceDepression is a major cause of disability world-wide. The present paper reports on the results of our participation to the depression sub-challenge of the sixth Audio/Visual Emotion Challenge (AVEC 2016), which was designed to compare feature modalities ( audio, visual, interview transcript-based) in gender-based and gender-independent modes using a variety of classification algorithms. In our approach, both high and low level features were assessed in each modality. Audio features were extracted from the low-level descriptors provided by the challenge organizers. Several visual features were extracted and assessed including dynamic characteristics of facial elements (using Landmark Motion History Histograms and Landmark Motion Magnitude), global head motion, and eye blinks. These features were combined with statistically derived features from pre-extracted features ( emotions, action units, gaze, and pose). Both speech rate and word-level semantic content were also evaluated. Classification results are reported using four different classification schemes: i) gender-based models for each individual modality, ii) the feature fusion model, ii) the decision fusion model, and iv) the posterior probability classification model. Proposed approaches outperforming the reference classification accuracy include the one utilizing statistical descriptors of low-level audio features. This approach achieved f1-scores of 0.59 for identifying depressed and 0.87 for identifying notdepressed individuals on the development set and 0.52/0.81, respectively for the test set

Optimizing Speech-Input Length for Speaker-Independent Depression Classification

“…Speech technology offers promise because speaking is natural, can be used at a distance, requires no special training, and carries information about a speaker's state. A growing line of AI research has shown that depression can be detected from speech signals using natural language processing (NLP), acoustic models, and multimodal models [3], [4], [5], [6], [7], [8], [9], [10]. Common evaluations with shared data sets, features, and tools have recently led to progress, especially in modeling methods [11], [12], [13], [14], [15].…”

Quantitative comparison of motion history image variants for video-based depression assessment

“…The present work introduced the GMHI, a novel variant of MHI and reported on the first application of LMHI [23] on the AVEC dataset. Another novelty of the proposed work is that categorical assessment of depressive symptomatology was performed using deep learning methods, for the first time on this dataset.…”

“…In the DSC of AVEC'14, Pérez Espinoza et al [21] employed MHI, and for the same challenge, Jan et al [22] proposed the 1-D MHH, an extension of MHI, which is computed on the feature vector sequence instead of the intensity image. As part of their DSC-AVEC'16 participation, Pampouchidou et al [23] introduced Landmark Motion History Images (LMHI), which instead of considering intensities from image sequences, considers sequences of facial landmarks.…”

“…The LMHI originally proposed in Pampouchidou et al [23] considers the landmarks derived from OpenFace. The landmarks considered are the ones which correspond to the facial features (eyes, eyebrows, nose-tip, and mouth), while the face outline is excluded.…”

Topic Modeling Based Multi-modal Depression Detection

“…Further, we add the presence of each topic to the feature vector, because each interview only covers a few topics and the topic presence might be correlated to the subject's status. Finally, gender is also a ached to the feature vector similar to the work in [22] and [14], where the authors report that gender information can greatly improve the classification performance. Figure 3 illustrates the structure of the feature vector and Table 2 shows the dimension of each feature category in the feature vector.…”