Detecting Recovery Problems Just in Time: Application of Automated Linguistic Analysis and Supervised Machine Learning to an Online Substance Abuse Forum

Abstract

BackgroundOnline discussion forums allow those in addiction recovery to seek help through text-based messages, including when facing triggers to drink or use drugs. Trained staff (or “moderators”) may participate within these forums to offer guidance and support when participants are struggling but must expend considerable effort to continually review new content. Demands on moderators limit the scalability of evidence-based digital health interventions.ObjectiveAutomated identification of recovery problems could allow moderators to engage in more timely and efficient ways with participants who are struggling. This paper aimed to investigate whether computational linguistics and supervised machine learning can be applied to successfully flag, in real time, those discussion forum messages that moderators find most concerning.MethodsTraining data came from a trial of a mobile phone-based health intervention for individuals in recovery from alcohol use disorder, with human coders labeling discussion forum messages according to whether or not authors mentioned problems in their recovery process. Linguistic features of these messages were extracted via several computational techniques: (1) a Bag-of-Words approach, (2) the dictionary-based Linguistic Inquiry and Word Count program, and (3) a hybrid approach combining the most important features from both Bag-of-Words and Linguistic Inquiry and Word Count. These features were applied within binary classifiers leveraging several methods of supervised machine learning: support vector machines, decision trees, and boosted decision trees. Classifiers were evaluated in data from a later deployment of the recovery support intervention.ResultsTo distinguish recovery problem disclosures, the Bag-of-Words approach relied on domain-specific language, including words explicitly linked to substance use and mental health (“drink,” “relapse,” “depression,” and so on), whereas the Linguistic Inquiry and Word Count approach relied on language characteristics such as tone, affect, insight, and presence of quantifiers and time references, as well as pronouns. A boosted decision tree classifier, utilizing features from both Bag-of-Words and Linguistic Inquiry and Word Count performed best in identifying problems disclosed within the discussion forum, achieving 88% sensitivity and 82% specificity in a separate cohort of patients in recovery.ConclusionsDifferences in language use can distinguish messages disclosing recovery problems from other message types. Incorporating machine learning models based on language use allows real-time flagging of concerning content such that trained staff may engage more efficiently and focus their attention on time-sensitive issues.

Transfer Learning for Risk Classification of Social Media Posts: Model Evaluation Study

“…However, in a systematic review of social networking sites for mental health interventions, [5] identify the use of moderators as a key component of successful interventions on these online platforms [5] . The development of automated triage systems in these contexts can facilitate professional intervention by prioritizing users for specialized care [6,7] or improving the rate of response when a risk for self-harm is identified [8] .…”

Associations Between Substance Use and Instagram Participation to Inform Social Network–Based Screening Models: Multimodal Cross-Sectional Study

“…As detailed below, emerging research suggests that individuals’ SNS posts can be leveraged to detect substance use [ 10 , 13 , 14 ]. However, these computational strategies function more efficiently and reliably with increasing amounts of data [ 15 ].…”

“…Studies showing that online forum content can be used to detect risk in clinical SUD samples are also instructive. Kornfield et al [ 13 , 14 ] used natural language processing in 2 studies, one of which also employed machine learning, to predict risk for negative substance use outcomes among individuals with both alcohol and other drug use disorders participating in a smartphone app online recovery forum. In the first study [ 14 ], they examined the utility of natural language processing with a Linguistic Inquiry and Word Count approach to predict binge drinking (5+ drinks for men and 4+ for women in 2 hours) in participants with alcohol use disorder.…”

Data Science Support at the Academic Library

“…Data science, which applies concepts from statistics and computer sciences to questions in numerous domains, is driving novel discoveries across disciplines. Such applications include automated linguistic analysis for "just in time" action in online support forums (Kornfield et al, 2018), neuroimage processing for predicting health outcomes (Lancaster, Lorenz, Leech, & Cole, 2018), and relationships between environmental variation and childhood cognitive development (Stingone, Pandey, Claudio, & Pandey, 2017). However, applying such approaches requires specialized skills and resources, including computer programming and data visualization.…”