Scoping Review on the Multimodal Classification of Depression and Experimental Study on Existing Multimodal Models

Abstract

Depression is a prevalent comorbidity in patients with severe physical disorders, such as cancer, stroke, and coronary diseases. Although it can significantly impact the course of the primary disease, the signs of depression are often underestimated and overlooked. The aim of this paper was to review algorithms for the automatic, uniform, and multimodal classification of signs of depression from human conversations and to evaluate their accuracy. For the scoping review, the PRISMA guidelines for scoping reviews were followed. In the scoping review, the search yielded 1095 papers, out of which 20 papers (8.26%) included more than two modalities, and 3 of those papers provided codes. Within the scope of this review, supported vector machine (SVM), random forest (RF), and long short-term memory network (LSTM; with gated and non-gated recurrent units) models, as well as different combinations of features, were identified as the most widely researched techniques. We tested the models using the DAIC-WOZ dataset (original training dataset) and using the SymptomMedia dataset to further assess their reliability and dependency on the nature of the training datasets. The best performance was obtained by the LSTM with gated recurrent units (F1-score of 0.64 for the DAIC-WOZ dataset). However, with a drop to an F1-score of 0.56 for the SymptomMedia dataset, the method also appears to be the most data-dependent.

Multimodal prediction of obsessive-compulsive disorder, comorbid depression, and energy of deep brain stimulation

“…Audio and video as well as text may include personally identifiable information that is difficult to eliminate without impairing value of the data. In part for this reason, DAIC-WOZ has been one of the few depression databases made available to qualified researchers [7]. We will seek IRB permission to distribute de-identified and anonymized feature data from this clinical study for use by other researchers.…”

“…Multimodal communication encompasses facial expression, head and body motion, gaze, voice, speech behavior, and speech or language. Yet, as a recent scoping review [7] found, the vast majority of studies on depression include only one or two modalities. This is the case even though large numbers of studies have collected the necessary audio-visual data [8].…”

Taking All the Factors We Need: A Multimodal Depression Classification With Uncertainty Approximation

“…Although the multimodal approach is common in depression detection, the majority of the existing research focuses on bi-modality or tri-modality. A review study by Arioz et al [35] shows that in the existing 1095 research, only 20 papers devised their methodology on more than two modalities. The prevalent modalities comprise of acoustic characteristics and visual cues, primarily obtained from video recordings.…”

“…The approaches discussed suffer from a loss of multimodality due to the absence of feature-merging techniques and reliance on a single classifier for feature mixing. Arroz et al [35] compared algorithms for unimodal, automatic, and multimodal classification conversations, with LSTM and gated recurrent units (GRU). Alternative approaches to multimodal depression detection encompass the examination of various indicators such as the dynamics of acoustic, facial, head movement [27], [39], behavioural and physiological signals [40], brain functional abnormalities, heart rate variability, hemodynamic parameters [41], and partially convergent structural features [23].…”