How accurately can supervised machine learning model predict a targeted psychiatric disorder?

Abstract

Background Hoarding disorder (HD) is characterized by a compulsion to collect belongings, and to experience significant distress when parting from them. HD is often misdiagnosed for several reasons. These include patient and family lack of recognition that it is a psychiatric disorder and professionals’ lack of relevant expertise with it. This study evaluates the ability of a supervised machine learning (ML) model to match the diagnostic skills of psychiatrists when presented with equivalent information pertinent to symptoms of HD. Methods Five hundred online participants were randomly recruited and completed the Hoarding Rating Scale-Self Report (HRS-SR) and the Generalized Anxiety Disorder 7-item (GAD-7) scale. Responses to the questionnaires were read by an ML model. Responses to the HRS-SR were then converted into anonymized, random-equivalent texts. Each of these individual texts was presented in random order to two experienced psychiatrists who were independently asked for a provisional diagnosis - e.g.; the presence or absence of HD. In case of disagreement between the two assessors, a third psychiatrist broke the tie. A decision tree classification model was employed to predict clinical HD using self-report data from two psychological tests, the HRS-SR and GAD-7. The target variable was whether a participant had clinical HD, while the predictive variables were the continuous scores from the HRS-SR and GAD-7 tests. The model’s performance was evaluated using a confusion matrix, which compared the observed diagnoses with the predicted diagnoses to assess accuracy. Results According to the psychiatrists, approximately 10% of the participants fulfilled DSM-5 diagnostic criteria for HD. 93% of the clinician-identified cases were identified by the ML model based on HRS-SR and GAD-7 scores. A decision tree plot model demonstrated that about 60% of the cases could be detected by the HRS-SR alone while the rest required a combination of HRS-SR and GAD-7 scores. ML evaluation metrics showed satisfactory performance, with a Matthews Correlation Coefficient of 55%; Area Under Curve (AUC), 79%; a Negative Predictive Value of 76%; and a False Negative Rate of 24%. Conclusions Study findings strongly suggest that ML can, in the future, play a significant role in the risk assessment of psychiatric disorders prior to face-to-face consultation. By using AI to scan big data questionnaire responses, wait time for seriously ill patients can be substantially cut, and prognoses substantially improved.