Natural Language Processing and Psychosis: On the Need for Comprehensive Psychometric Evaluation

Abstract

Background and Hypothesis Despite decades of “proof of concept” findings supporting the use of Natural Language Processing (NLP) in psychosis research, clinical implementation has been slow. One obstacle reflects the lack of comprehensive psychometric evaluation of these measures. There is overwhelming evidence that criterion and content validity can be achieved for many purposes, particularly using machine learning procedures. However, there has been very little evaluation of test-retest reliability, divergent validity (sufficient to address concerns of a “generalized deficit”), and potential biases from demographics and other individual differences. Study Design This article highlights these concerns in development of an NLP measure for tracking clinically rated paranoia from video “selfies” recorded from smartphone devices. Patients with schizophrenia or bipolar disorder were recruited and tracked over a week-long epoch. A small NLP-based feature set from 499 language samples were modeled on clinically rated paranoia using regularized regression. Study Results While test–retest reliability was high, criterion, and convergent/divergent validity were only achieved when considering moderating variables, notably whether a patient was away from home, around strangers, or alone at the time of the recording. Moreover, there were systematic racial and sex biases in the model, in part, reflecting whether patients submitted videos when they were away from home, around strangers, or alone. Conclusions Advancing NLP measures for psychosis will require deliberate consideration of test-retest reliability, divergent validity, systematic biases and the potential role of moderators. In our example, a comprehensive psychometric evaluation revealed clear strengths and weaknesses that can be systematically addressed in future research.

Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies

“…While diversity sampling can alert humans to discrepancies in the model’s representation of demographic groups, it will never solve the issue of missing data. Furthermore, the implementation of evaluation metrics such as test-retest reliability and divergent validity 41 remain critical for garnering trust in these approaches.…”

Language Network Dysfunction and Formal Thought Disorder in Schizophrenia

“…Increasingly, computational linguistic approaches are being used for NLP, providing an objective means to quantify subtle deviations and idiosyncrasies in speech using recorded readouts [107][108][109][110][111]. These measures can be derived from speech samples across various contexts -social, written texts, media posts, video interviews, and descriptive speech inside a scanner thus obviating the need for a one-to-one clinical interviewing for rating [112,113]. Hypothesis driven studies utilizing such automated measures in conjunction with brain imaging have already shown promising results, providing leads for readouts that can be employed in focal perturbation studies [114][115][116][117].…”

Language Network Dysfunction and Formal Thought Disorder in schizophrenia

“…Increasingly, computational linguistic approaches are being used for NLP, providing an objective means to quantify subtle deviations and idiosyncrasies in speech using recorded readouts [109][110][111]. These measures can be derived from speech samples across various contexts -social, written texts, media posts, video interviews, and descriptive speech inside a scanner thus obviating the need for a one-to-one clinical interviewing for rating [112,113]. Hypothesis driven studies utilizing such automated measures in conjunction with brain imaging have already shown promising results, providing leads for readouts that can be employed in focal perturbation studies [114][115][116][117].…”