Multimodal Machine Learning Framework to Detect the Bipolar Disorder

Abstract

The current approaches for diagnosing mental disorders rely heavily on self-reported and clinical interview ratings. The development of an automatic recognition system assists in the early detection and discovery of biological markers for diagnostic purposes. In this paper, develops a multimodal machine learning model, where it processes multiple modalities like visual, acoustic and textual features using a cross-modality correlation. The study uses a Denoising Autoencoder that finds the multimodal representations and then it adopts Fisher Vector encoding to form session-level descriptors. Paragraph Vector (PV) are used for textual modality that embeds the interview sessions transcripts into document representations that tends to capture the mental disorder cues. Finally, the textual and audio-visual features are fused before training the 3-layered denoising autoencoder (DAE) with Residual Neural Network classifier. The proposed model is validated using two bipolar disorders that includes depression and bipolar disorder. The study uses two different datasets including Extended Distress Analysis Interview Corpus (E-DAIC) and Bipolar Disorder Corpus (BDC) to analyse the depression and bipolar disorders. The experimental evaluation is conducted to show the performance improvement using the proposed multimodal model than other state-of-the art methods in detecting the depression and bipolar disorders. The results of simulation show that the proposed method obtains an improved detection rate than existing models.

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

“… Features Tools Studies Feature Category Low-level descriptors: jitter, shimmer, amplitude, pitch perturbation quotients, Mel-frequency cepstral coefficients (MFCCs), Teager-energy cepstrum coefficients (TECCs) [ 320 ], Discrete Cosine Transform (DCT) coefficients OpenSmile [ 267 ], COVAREP [ 321 ], YAAFE [ 322 ], Praat [ 323 ], Python libraries (pyAudioAnalysis [ 324 ], DisVoice [ 325 ]), My-Voice Analysis [ 326 ], Surfboard [ 327 ], librosa [ 328 ] [ 12 , 48 , 51 , 72 , 74 , 78 , 81 , 87 , 88 , 90 , 91 , 92 , 94 , 97 , 99 , 101 , 104 , 107 , 108 , 184 , 192 , 195 , 196 , 197 , 198 , 199 , 211 , 214 ] …”

“…Depression AV [43, SM [20,25,98, SS [99,100,104,105, WS [149][150][151][155][156][157][158]164,169,[171][172][173][178][179][180][181][182] Suicidal intent AV [100,183,184] SM [185][186][187][188][189] SS [100,147,190] WS [181,182,191] Bipolar disorder AV [101][102][103][192][193][194][195][196][197][198][199][200] SM…”

“…OpenSmile [267], COVAREP [321], YAAFE [322], Praat [323], Python libraries (pyAudioAnalysis [324], DisVoice [325]), My-Voice Analysis [326], Surfboard [327], librosa [328] [ 12,48,51,72,74,78,81,87,88,[90][91][92]94,97,99,101,104,107,108,184,192,[195][196][197][198][199]211,214] Voice Existing acoustic feature sets: Interspeech 2010 Paralinguistics [329],…”

“…SoundNet [259], SincNet [335], Wav2Vec [336], sentence embedding model [337], HuBERT [338], convolutional neural network (CNN), bidirectional LSTM (BiLSTM), ResNet [308] [192,199,214] Representations, Derived Facial/body appearance, landmarks, eye gaze, head pose OpenFace [269], OpenCV [270], Viola Jones' face detector [342], CascadeObjectDetector function in MATLAB's vision toolbox, Haar classifier [270], Gauss-Newton Deformable Part Model (GN-DPM) [343], OpenPose [271],…”

“…com/affdex-for-market-research (accessed on 10 December 2023), DBFace https: //github.com/dlunion/DBFace (accessed on 10 December 2023), FaceMesh https://developers. google.com/android/reference/ com/google/mlkit/vision/ facemesh/FaceMesh (accessed on 10 December 2023), dlib [347] [ 25,53,55,56,68,79,80,83,91,92,94,98,99,101,108,174,193,199,201,203,220 OpenFace [269], Face++ [352],…”