A Multimodal Approach for Predicting Changes in PTSD Symptom Severity

Abstract

The rising prevalence of mental illnesses is increasing the demand for new digital tools to support mental wellbeing. Numerous collaborations spanning the fields of psychology, machine learning and health are building such tools. Machine-learning models that estimate effects of mental health interventions currently rely on either user self-reports or measurements of user physiology. In this paper, we present a multimodal approach that combines selfreports from questionnaires and skin conductance physiology in a web-based trauma-recovery regime. We evaluate our models on the EASE multimodal dataset and create PTSD symptom severity change estimators at both total and cluster-level. We demonstrate that modeling the PTSD symptom severity change at the total-level with self-reports can be statistically significantly improved by the combination of physiology and self-reports or just skin conductance measurements. Our experiments show that PTSD symptom cluster severity changes using our novel multimodal approach are significantly better modeled than using self-reports and skin conductance alone when extracting skin conductance features from triggers modules for avoidance, negative alterations in cognition & mood and alterations in arousal & reactivity symptoms, while it performs statistically similar for intrusion symptom.

Machine Learning in Mental Health

“…, for audio: robust speaker detection; distinguish personal speaking style from symptoms) Chang et al [ 31 ]; Mallol-Ragolta et al [ 112 ]; Rabbi et al [ 153 ]; Salekin et al [ 168 ]; Spathis et al [ 176 , 177 ]; Zhou et al [ 222 ] Managing irrelevant, redundant information Ojeme and Mbogho [ 136 ] Dataset limitations Restrictions due to data subjects/scale/study context Too small or restricted study sample/need for larger (more diverse) datasets Adamou et al [ 2 ]; Diederich et al [ 45 ]; Feng et al [ 61 ], Kavuluru et al [ 89 ]; Morshed et al [ 128 ], Nobles et al [ 134 ]; Ojeme and Mbogho [ 136 ]; Parades et al [ 140 ]; Park et al [ 141 ], Pestian et al [ 145 ]; Quisel et al [ 152 ]; Ray et al [ 155 ]; Salekin et al [ 168 ]; Spathis et al [ 177 ], Yazdavar et al [ 211 ]; Zhou et al [ 222 ] Unknown confounding variables + limitations of study context Fatima et al [ 57 ]; Saha and De Choudhury [ 165 ]; Salekin et al [ 168 ] Reference dataset not explicitly designed for mental health-related analysis Alam et al [ 6 ] Biased, missing, incomplete data General acknowledgement of biases inherent to model design and datasset used for training Ernala et al [ 53 ]; Hirsch et al [ 78 ]; Park et al [ 141 ] Difficulties due to missing data values/sparse data Alam et al [ 6 ]; Spathis et al [ 176 , 177 ] Need for inclusion of other information (e.g . , biological and genetic data, fMRI, video, facial expressions, social media data) Diedrich et al [ 45 ]; Pestian et al [ 145 ]; Mallol-Ragolta et al [ 112 ]<...>…”

“…Other examples include the AVEC 2013 audiovisual dataset for studies of depression [ 122 , 168 ], and the EASE dataset of people undergoing trauma treatment, e.g . , for PTSD [ 112 ]. The BiAffect mobile phone and Depresjon dataset were used to access acceleration data of people with depression [ 62 ] and bipolar conditions [ 27 ], while the English Longitudinal Study of Ageing (ELSA) provided psychological and mental health data on older adults as indicators of depression [ 209 ].…”

“…To further demonstrate how newly developed models "outperformed existing ones" (with few exceptions [ 44 , 135 , 136 ]), the performance metrics are often compared with default or baseline models, and other state-of-the art approaches [ 2 , 27 , 33 , 62 , 122 , 134 , 135 , 152 , 165 , 168 , 176 , 177 , 211 , 218 ]. In addition to performance reports, a number of these papers foregrounded methodological contributions such as: new approaches to data labelling [ 168 , 211 ] and feature extraction [ 112 ]; the inclusion of timedependent features [ 33 , 165 , 192 , 211 , 218 ]; improvements to data representations [ 177 ] and data integration [ 27 ]; and strategies to optimize data collection (periods) [ 152 , 128 , 218 ]. Building on these results, authors often concluded how their work presented a "proof-of-concept" that showed "the potential" of using a particular technology [ 27 , 50 , 153 ], data source [ 57 , 62 , 64 , 112 , 128 , 134 , 165 , 176 , 218 ], or algorithm [ 44 ] for understanding, detecting or inferring a relationship with mental health.…”

“…Nevertheless, there may be a need for more explicit dialogue and efforts to nurture a clearer understanding for those from whom data is being collected and analyzed as to what constitutes the purpose of the data analysis, and what risks and benefits sharing their data may entail for the [ 4 ]; Alam et al [ 6 ]; Chang et al [ 31 ]; Chen et al [ 33 ]; DeMasi and Recht [ 44 ]; Diederich et al [ 45 ]; Fatima et al [ 57 ]; Frogner et al [ 62 ]; Galiatsatos et al [ 63 ]; Gjoreski et al [ 67 ]; Joshi et al [ 86 ]; Kavuluru et al [ 89 ]; Mallol-Ragolta et al [ 112 ]; Nguyen et al [ 133 ]; Nosakhare and Picard [ 135 ]; Panagiotakopoulos et al [ 139 ]; Patterson and Cloud [ 144 ]; Pestian et al [ 145 ]; Rastogi et al [ 154 ]; Ray et al [ 155 ]; Spathis et al [ 177 ]; Tavabi [ 184 ]; Tran et al [ 192 ]; Tsiakas et al [ 193 ]; Wilbourne et al [ 205 ] Reports of ethical approval/review exemption Institutional/Regional IRB Doryab et al [ 50 ]; Ernala et al [ 53 ]; Nobles et al [ 134 ]; Paredes et al [ 140 ]; Salekin et al [ 168 ]; Wang et al [ 201 ]; Yazdevar et al [ 211 ]; Zakaria et al [ 217 ] Re-use of data (e.g . , that previously received or was exempt from ethical approval)…”

Learning based visual engagement and self-efficacy

“…In our recent work [145], we explored the feasibility of replacing CSE t questionnaires with skin conductance responses and also the fusion of both modalities, in order to estimate changes in trauma severity.…”