Reading Between the Frames: Multi-modal Depression Detection in Videos from Non-verbal Cues

Gimeno-Gómez, David; Bucur, Ana-Maria; Cosma, Adrian; Martínez-Hinarejos, Carlos-David; Rosso, Paolo

doi:10.1007/978-3-031-56027-9_12

Lecture Notes in Computer Science

2024

DOI: 10.1007/978-3-031-56027-9_12

|View full text |Cite

Reading Between the Frames: Multi-modal Depression Detection in Videos from Non-verbal Cues

David Gimeno-Gómez,

Ana-Maria Bucur,

Adrian Cosma

et al.

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Article1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Harnessing multimodal approaches for depression detection using large language models and facial expressions

Sadeghi,

Richer,

Egger

et al. 2024

npj Mental Health Res

View full text Add to dashboard Cite

Detecting depression is a critical component of mental health diagnosis, and accurate assessment is essential for effective treatment. This study introduces a novel, fully automated approach to predicting depression severity using the E-DAIC dataset. We employ Large Language Models (LLMs) to extract depression-related indicators from interview transcripts, utilizing the Patient Health Questionnaire-8 (PHQ-8) score to train the prediction model. Additionally, facial data extracted from video frames is integrated with textual data to create a multimodal model for depression severity prediction. We evaluate three approaches: text-based features, facial features, and a combination of both. Our findings show the best results are achieved by enhancing text data with speech quality assessment, with a mean absolute error of 2.85 and root mean square error of 4.02. This study underscores the potential of automated depression detection, showing text-only models as robust and effective while paving the way for multimodal analysis.

show abstract

Harnessing multimodal approaches for depression detection using large language models and facial expressions

Sadeghi,

Richer,

Egger

et al. 2024

npj Mental Health Res

View full text Add to dashboard Cite

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Reading Between the Frames: Multi-modal Depression Detection in Videos from Non-verbal Cues

Cited by 1 publication

References 73 publications

Harnessing multimodal approaches for depression detection using large language models and facial expressions

Harnessing multimodal approaches for depression detection using large language models and facial expressions

Contact Info

Product

Resources

About