Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

Abstract

ObjectiveThe current clinical practice of psychiatric evaluation suffers from subjectivity and bias, and requires highly skilled professionals that are often unavailable or unaffordable. Objective digital biomarkers have shown the potential to address these issues. In this work, we investigated whether behavioral and physiological signals, extracted from remote interviews, provided complimentary information for assessing psychiatric disorders.MethodsTime series of multimodal features were derived from four conceptual modes: facial expression, vocal expression, linguistic expression, and cardiovascular modulation. The features were extracted from simultaneously recorded audio and video of remote interviews using task-specific and foundation models. Averages, standard deviations, and hidden Markov model-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination.ResultsStatistically significant feature differences were found between controls and subjects with mental health conditions. Correlations were found between features and self-rated depression and anxiety scores. Visual heart rate dynamics achieved the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities achieved AUROCs of 0.72-0.82. Features from task-specific models outperformed features from foundation models.ConclusionMultimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status.SignificanceThe proposed multimodal approach has the potential to facilitate objective, remote, and low-cost assessment for low-burden automated mental health services.

Evaluating and mitigating unfairness in multimodal remote mental health assessments

“…The same dataset described in our previous study [19] was used in this study. The Emory University Institutional Review Board and the Grady Research Oversight Committee granted approval for this study (IRB# 00105142).…”

“…The control group included the remaining subjects. The categorization details can be found in our previous study [19].…”

“…A multimodal analysis framework was proposed and evaluated in our previous study [19]. For each audiovisual interview recording, It extracted visual, vocal, language, and remote photoplethysmography (rPPG) time series signals at the frame or segment level, summarizes those time series with statistical and temporal dynamic features at the subject level (except for text embedding from the large language model, where the model directly generated subject-level embedding), and evaluates the performance of these features in clinical diagnoses (control vs. MHC) or self-rated depression/anxiety classification tasks.…”

“…The probability of the presence of seven basic emotions (neutral, happiness, sadness, surprise, fear, disgust, and anger) and 12 facial action units [45] (AU1, 2, 4, 6, 7, 10, 12, 14, 15, 17, 23, 24) was estimated using convolutional neural network (CNN) based models [46, 47]. The fairness of the visual foundation model described in our previous study [19] was not evaluated due to its poor performance.…”

AI Therapist for Daily Functioning Assessment and Intervention Using Smart Home Devices

“…Researchers leveraged LLMs for mental health prediction via online text data and evaluated the capabilities of multiple LLMs on various mental health prediction tasks via online text data [71,97]. In addition, [43] leverages RoBERTa [49] and Llama-65b [90] in the system for classifying psychiatric disorder, major depressive disorder, self-rated depression, and self-rated anxiety based on time-series multimodal features.…”