A Feasibility Study Using a Machine Learning Suicide Risk Prediction Model Based on Open-Ended Interview Language in Adolescent Therapy Sessions

Abstract

Background: As adolescent suicide rates continue to rise, innovation in risk identification is warranted. Machine learning can identify suicidal individuals based on their language samples. This feasibility pilot was conducted to explore this technology’s use in adolescent therapy sessions and assess machine learning model performance. Method: Natural language processing machine learning models to identify level of suicide risk using a smartphone app were tested in outpatient therapy sessions. Data collection included language samples, depression and suicidality standardized scale scores, and therapist impression of the client’s mental state. Previously developed models were used to predict suicidal risk. Results: 267 interviews were collected from 60 students in eight schools by ten therapists, with 29 students indicating suicide or self-harm risk. During external validation, models were trained on suicidal speech samples collected from two separate studies. We found that support vector machines (AUC: 0.75; 95% CI: 0.69–0.81) and logistic regression (AUC: 0.76; 95% CI: 0.70–0.82) lead to good discriminative ability, with an extreme gradient boosting model performing the best (AUC: 0.78; 95% CI: 0.72–0.84). Conclusion: Voice collection technology and associated procedures can be integrated into mental health therapists’ workflow. Collected language samples could be classified with good discrimination using machine learning methods.

Integration and Validation of a Natural Language Processing Machine Learning Suicide Risk Prediction Model Based on Open-Ended Interview Language in the Emergency Department

“…The NLP/ML pipeline used in this study followed similar techniques used by Pestian et al, focused on the term frequency of n-grams (contiguous sequence of n number of words) and SVMs (21,22,(40)(41)(42). The Porter Stemmer algorithm was applied to participant language to normalize morphologically related terms (43).…”

“…During SVM tuning, hyperparameters considered include: the regularization parameter (C), the kernel (radial basis function and linear kernels), the kernel coefficient (gamma, if applicable), and the class weight. Additional details on NLP/ML methods may be found in previous work (42). We have evaluated the performance of Logistic Regression and XGBoost models in previous work and found comparable performance across models (42), and therefore decided to continue with the SVM used previously (21).…”

“…Additional details on NLP/ML methods may be found in previous work (42). We have evaluated the performance of Logistic Regression and XGBoost models in previous work and found comparable performance across models (42), and therefore decided to continue with the SVM used previously (21). A comparison of different models is beyond the scope of this paper.…”

Use of Artificial Intelligence-Based Strategies for Assessing Suicidal Behavior and Mental Illness: A Literature Review

“…A wide spectrum of AI tools was used in these 13 studies. A combination of NLP and ML was used in four studies [11][12][13][14]. ML as the exclusive AI technique was used in four studies [15][16][17][18], artificial neural network (ANN) was used in three studies [19][20][21], and NLP in two studies [9,22].…”

Improving the Field’s Understanding of Suicide Protective Factors and Translational Suicide Prevention Initiatives

“…Thus, although suicide prediction appears to be a difficult pursuit [16], suicide risk identification is a necessary step to determine the level of suicide-specific care that may be warranted. Cohen and colleagues [17] conducted a feasibility investigation of whether machine learning techniques with natural language processing of adolescent discussions with their therapists may improve suicide risk assessment practices. The investigation demonstrated that this practice focused on augmenting standardized clinical risk assessment efforts may provide important information for assessing risk of STBs and thus the cueing of suicide-specific care practices.…”