Depression Level Prediction Using Deep Spatiotemporal Features and Multilayer Bi-LTSM

Uddin, Azher; Joolee, Joolekha Bibi; Lee, Young Koo

doi:10.1109/taffc.2020.2970418

Cited by 49 publications

(41 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most existing video-based automatic depression analysis approaches are single-stage methods, i.e., extracting depression feature from a single frame/thin slice or the entire video. In particular, the frame/thin slice-level methods attempted to model depression status based on individuals' facial appearance [13], [16], [17] (e.g., frame-level modelling) or short-term facial behaviours (thin slice-level modelling) [14], [15], [19], [36], [37], [38], [39]. The frame-level modelling approaches usually focus on learning the depressionrelated salient facial appearance information.…”

Section: Video-based Automatic Depression Analysismentioning

confidence: 99%

“…1, these methods fail to consider the important long-term facial behaviours/dynamics for depression recognition. Although some of the methods [15], [15], [16], [19] uses RNNs/LSTMs to model long-term temporal dependencies from the video, the CNNs of such methods are trained by pairing a frame/thin slice with the video-level label are problematic.…”

Section: Video-based Automatic Depression Analysismentioning

confidence: 99%

“…Existing video-based automatic depression analysis approaches can be categorized into two groups: frame/thin slice-level modelling methods and video-level modelling methods. The frame/thin slice-level modelling methods [12], [13], [14], [15], [16], [17] individually infer depression status for each frame or thin slice of the video, primarily focusing on the depression-related cues from subjects' facial appearance or the short-term facial dynamics. Most of these approaches either disregard the temporal information or only consider single-scale short-term facial dynamics exhibited within a pre-defined time-window.…”

Section: Introductionmentioning

confidence: 99%

“…Moreover, as discussed in [18], only using short-term facial behaviours to infer depression is not reliable as similar short-term facial behaviours may be exhibited by subjects with different depression severity levels. Although some of these approaches [15], [16], [19] employ RNNs/LSTMs to learn long-term dependencies from the learned frame/thin slice-level features, regressors that are trained by pairing a frame/thin slice with the video-level depression label cannot learn a good hypothesis. This is because such a training strategy may lead the regressors to focus on learning nondepression related facial attributes that are invariant for the subject in each video, e.g., identity, rather than depressionrelated facial actions.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

Xu¹,

Song²,

Kusumam³

et al. 2021

Preprint

View full text Add to dashboard Cite

Video-based automatic depression analysis provides a fast, objective and repeatable self-assessment solution, which has been widely developed in recent years. While depression clues may be reflected by human facial behaviours of various temporal scales, most existing approaches either focused on modelling depression from short-term or video-level facial behaviours. In this sense, we propose a two-stage framework that models depression severity from multi-scale short-term and video-level facial behaviours. The short-term depressive behaviour modelling stage first deep learns depression-related facial behavioural features from multiple short temporal scales, where a Depression Feature Enhancement (DFE) module is proposed to enhance the depression-related clues for all temporal scales and remove non-depression noises. Then, the video-level depressive behaviour modelling stage proposes two novel graph encoding strategies, i.e., Sequential Graph Representation (SEG) and Spectral Graph Representation (SPG), to re-encode all short-term features of the target video into a video-level graph representation, summarizing depression-related multi-scale video-level temporal information. As a result, the produced graph representations predict depression severity using both short-term and long-term facial beahviour patterns. The experimental results on AVEC 2013 and AVEC 2014 datasets show that the proposed DFE module constantly enhanced the depression severity estimation performance for various CNN models while the SPG is superior than other video-level modelling methods. More importantly, the result achieved for the proposed two-stage framework shows its promising and solid performance compared to widely-used one-stage modelling approaches.

show abstract

Section: Video-based Automatic Depression Analysismentioning

confidence: 99%

Section: Video-based Automatic Depression Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

Xu¹,

Song²,

Kusumam³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Similarly, VGG-LSTM and C3D-LSTM are employed in [11]. However, these approaches exhibit relatively weak performance regarding capturing facial dynamics [12]. More-over, on large-scale video data, deep learning-based methods experience computational efficiency issues.…”

Section: Introductionmentioning

confidence: 99%

Dynamic Facial Expression Understanding Using Deep Spatiotemporal LDSP On Spark

2021

Self Cite

View full text Add to dashboard Cite

Facial expressions are the most common medium for expressing human emotions. Due to the wide range of real-world applications, facial expression understanding has received extensive attention from researchers. One of the most vital issues of facial expression recognition is the extraction and modeling of the temporal dynamics of facial emotions from videos. Additionally, the rapid growth of video data from various multimedia sources is becoming a serious concern. Therefore, to address these issues, in this paper, we introduce a novel approach on top of Spark for facial expression understanding from videos. First, we propose a new dynamic feature descriptor, namely, the local directional structural pattern from three orthogonal planes (LDSP-TOP), which analyzes the structural aspects of the local dynamic texture. Second, we design a 1-D convolutional neural network (CNN) to capture additional discriminative features. Third, a long short-term memory (LSTM) autoencoder is employed to learn the spatiotemporal features. Finally, an extensive experimental investigation is carried out to demonstrate the performance and scalability of the proposed framework.

show abstract

Intelligent system for depression scale estimation with facial expressions and case study in industrial intelligence

Guo

Tiwari

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

As a mental disorder, depression has affected people's lives, works, and so on. Researchers have proposed various industrial intelligent systems in the pattern recognition field for audiovisual depression detection. This paper presents an end‐to‐end trainable intelligent system to generate high‐level representations over the entire video clip. Specifically, a three‐dimensional (3D) convolutional neural network equipped with a module spatiotemporal feature aggregation module (STFAM) is trained from scratch on audio/visual emotion challenge (AVEC)2013 and AVEC2014 data, which can model the discriminative patterns closely related to depression. In the STFAM, channel and spatial attention mechanism and an aggregation method, namely 3D DEP‐NetVLAD, are integrated to learn the compact characteristic based on the feature maps. Extensive experiments on the two databases (i.e., AVEC2013 and AVEC2014) are illustrated that the proposed intelligent system can efficiently model the underlying depression patterns and obtain better performances over the most video‐based depression recognition approaches. Case studies are presented to describes the applicability of the proposed intelligent system for industrial intelligence.

show abstract

Depression Level Prediction Using Deep Spatiotemporal Features and Multilayer Bi-LTSM

Cited by 49 publications

References 36 publications

Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

Dynamic Facial Expression Understanding Using Deep Spatiotemporal LDSP On Spark

Intelligent system for depression scale estimation with facial expressions and case study in industrial intelligence

Contact Info

Product

Resources

About