Vocal markers of schizophrenia: assessing the generalizability of machine learning models and their clinical applicability

Abstract

Background and Hypothesis: Machine Learning (ML) models have been argued to reliably predict diagnosis and symptoms of schizophrenia based on voice data only. However, it is unclear to what extent such ML markers would generalize to different clinical samples and different languages, a crucial assessment to move towards clinical applicability. In this study, we systematically assessed the generalizability of ML models of vocal markers of schizophrenia across contexts and languages. Study Design: We trained models relying on a large cross-linguistic dataset (Danish, German, Chinese) of 217 patients with schizophrenia and 221 controls, and used a conservative pipeline to minimize overfitting. We tested the models' generalizability on: (i) new participants, speaking the same language; (ii) new participants, speaking a different language; (iii) further, we assessed whether training on data with multiple languages would improve generalizability using Mixture of Expert (MoE) and multilingual models. Results: Model performance was comparable to state-of-the-art findings (F1-score ~ 0.75) within the same language; however, models did not generalize well - showing a substantial decrease - when tested on new languages. The performance of MoE and multilingual models was also generally low (F1-score ~ 0.50). Conclusions: Overall, the cross-linguistic generalizability of vocal markers of schizophrenia is limited. We argue that more emphasis should be placed on collecting large open cross-linguistic datasets to systematically test the generalizability of voice-based ML models, and on identifying more precise mechanisms of how the clinical features of schizophrenia are expressed in language and voice, and how different languages vary in that expression.