Text classification for assisting moderators in online health communities

Abstract

Objectives Patients increasingly visit online health communities to get help on managing health. The large scale of these online communities makes it impossible for the moderators to engage in all conversations; yet, some conversations need their expertise. Our work explores low-cost text classification methods to this new domain of determining whether a thread in an online health forum needs moderators’ help. Methods We employed a binary classifier on WebMD’s online diabetes community data. To train the classifier, we considered three feature types: (1) word unigram, (2) sentiment analysis features, and (3) thread length. We applied feature selection methods based on χ2 statistics and under sampling to account for unbalanced data. We then performed a qualitative error analysis to investigate the appropriateness of the gold standard. Results Using sentiment analysis features, feature selection methods, and balanced training data increased the AUC value up to 0.75 and the F1-score up to 0.54 compared to the baseline of using word unigrams with no feature selection methods on unbalanced data (0.65 AUC and 0.40 F1-score). The error analysis uncovered additional reasons for why moderators respond to patients’ posts. Discussion We showed how feature selection methods and balanced training data can improve the overall classification performance. We present implications of weighing precision versus recall for assisting moderators of online health communities. Our error analysis uncovered social, legal, and ethical issues around addressing community members’ needs. We also note challenges in producing a gold standard, and discuss potential solutions for addressing these challenges. Conclusion Social media environments provide popular venues in which patients gain health-related information. Our work contributes to understanding scalable solutions for providing moderators’ expertise in these large-scale, social media environments.

Detecting Recovery Problems Just in Time: Application of Automated Linguistic Analysis and Supervised Machine Learning to an Online Substance Abuse Forum (Preprint)

“…Notably, in all cases, we achieved good performance relative to Huh and colleagues [12], who also attempted to detect appropriate messages for moderator intervention, and who achieved f-scores up to 0.54. This may in part reflect the difference in machine learning approach; we used Boosted Decision Trees rather than the Naive Bayes technique they report.…”

“…Labeled data can be generated in a number of ways. For instance, naturally occurring response patterns can be used, such as where Huh and colleagues [12] labeled as problematic those messages to which moderators had previously responded in a health support forum, using their linguistic features to classify new messages that moderators would likely be interested in. Alternately, human judgment can be used to generate labeled data, as was implemented by Balani and De Choudhury [47].…”

“…As mentioned earlier, gold standard data in this study substantially differs from that used in prior work using moderators' natural response patterns [12]. Our conversations with moderators revealed that contextual considerations, such as whether participants had already received competent help from peers, or whether moderators had already had offline contact with participants (e.g., by phone call or private message), would influence their likelihood of responding on the discussion forum, thus introducing error in using natural response patterns.…”

“…Through just-in-time support, moderators contribute to efficiency of health services at a systems level, making additional attention and resources available to those who most need them, while maintaining less intensive support for those at a lower level of need. Yet, attending to changing needs of an at-risk online community poses a considerable challenge since participants can produce a massive volume of text exchanges [12]. Demands on staff represent a key hurdle in scaling up digital mental health interventions [13].…”

Predicting Medical Roles in Online Health Fora

“…The Text Mining and Natural Language Processing communities have extensively investigated the huge amount of data on online health fora for different purposes, such as: classifying lay requests to an internal medical expert [1], assisting moderators on online health fora [2], identifying sentiments and emotions [3], identifying the targets of the emotions [4], etc. Indeed, online health fora are increasingly visited by both sick and healthy users to get help and information related to their health [2].…”