Multi-level Attention network using text, audio and video for Depression Prediction

Ray, Anupama; Kumar, Siddharth; Reddy, Rutvik; Mukherjee, Prerana; Garg, Ritu

doi:10.48550/arxiv.1909.01417

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies in depression prediction using speech [15,16] have shown the superiority of MFCCs over other audio based features like extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [7] and DEEP SPECTRUM features [1]. Huang et al [9] showed with their depression classification study that coordination features computed from MFCCs perform better with respect to formants and eGeMAPS features.…”

Section: Mel-frequency Cepstral Coefficients (Mfccs)mentioning

confidence: 99%

Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks

Siriwardena

Kitchen

Kelly

et al. 2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

This study investigates the speech articulatory coordination in schizophrenia subjects exhibiting strong positive symptoms (e.g. hallucinations and delusions), using two distinct channel-delay correlation methods. We show that the schizophrenic subjects with strong positive symptoms and who are markedly ill pose complex articulatory coordination pattern in facial and speech gestures than what is observed in healthy subjects. This distinction in speech coordination pattern is used to train a multimodal convolutional neural network (CNN) which uses video and audio data during speech to distinguish schizophrenic patients with strong positive symptoms from healthy subjects. We also show that the vocal tract variables (TVs) which correspond to place of articulation and glottal source outperform the Mel-frequency Cepstral Coefficients (MFCCs) when fused with Facial Action Units (FAUs) in the proposed multimodal network. For the clinical dataset we collected, our best performing multimodal network improves the mean F1 score for detecting schizophrenia by around 18% with respect to the full vocal tract coordination (FVTC) baseline method implemented with fusing FAUs and MFCCs. CCS CONCEPTS• Computing methodologies → Neural networks; • Social and professional topics → People with disabilities.

show abstract

Section: Mel-frequency Cepstral Coefficients (Mfccs)mentioning

confidence: 99%

Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks

Siriwardena

Kitchen

Kelly

et al. 2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

show abstract

“…14 A study used the Extended Distress Analysis Interview Corpus Database to predict depression. 11 The combination of text, audio, and video examinations resulted in the best predictive accuracy.…”

Section: Introductionmentioning

confidence: 98%

“…4,10 These changes eventually alter vocal expression, facial gestures, and speech content. 11 Among these signals, voice and facial expressions are relatively difficult to hide because emotions are often expressed subconsciously. 12 Additionally, they can be easily recorded using various digital imaging and voice recording devices.…”

Section: Introductionmentioning

confidence: 99%

Distinguishing apathy and depression in older adults with mild cognitive impairment using text, audio, and video based on multiclass classification and shapely additive explanations

Zhou

Yao

Han

et al. 2022

Int J Geriat Psychiatry

View full text Add to dashboard Cite

Objectives This study aimed to develop a classification model to detect and distinguish apathy and depression based on text, audio, and video features and to make use of the shapely additive explanations (SHAP) toolkit to increase the model interpretability. Methods Subjective scales and objective experiments were conducted on 319 mild cognitive impairment (MCI) patients to measure apathy and depression. The MCI patients were classified into four groups, depression only, apathy only, depressed‐apathetic, and the normal group. Speech, facial and text features were extracted using the open‐source data analysis toolkits. Multiclass classification and SHAP toolkits were used to develop a classification model and explain the contribution of specific features. Results The macro‐averaged f1 score and accuracy for overall model were 0.91 and 0.90, respectively. The accuracy for the apathetic, depressed, depressed‐apathetic, and normal groups were 0.98, 0.88, 0.93, and 0.82, respectively. The SHAP toolkit identified speech features (Mel‐frequency cepstral coefficient (MFCC) 4, spectral slopes, F0, F1), facial features (action unit (AU) 14, 26, 28, 45), and text feature (text 6 semantic) associated with apathy. Meanwhile, speech features (spectral slopes, shimmer, F0) and facial expression (AU 2, 6, 7, 10, 14, 26, 45) were associated with depression. Apart from the shared features mentioned above, new speech (MFCC 2, loudness) and facial (AU 9) features were observed in the depressive‐apathetic group. Conclusions Apathy and depression shared some verbal and facial features while also exhibited distinct features. A combination of text, audio, and video could be used to improve the early detection and differential diagnosis of apathy and depression in MCI patients.

show abstract

Issues and Challenges in Detecting Mental Stress from Multimodal Data Using Machine Intelligence

Sadruddin,

Khairnar,

Vora

2024

SN COMPUT. SCI.

View full text Add to dashboard Cite

Multi-level Attention network using text, audio and video for Depression Prediction

Cited by 3 publications

References 0 publications

Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks

Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks

Distinguishing apathy and depression in older adults with mild cognitive impairment using text, audio, and video based on multiclass classification and shapely additive explanations

Issues and Challenges in Detecting Mental Stress from Multimodal Data Using Machine Intelligence

Contact Info

Product

Resources

About