Addressing Extreme Imbalance for Detecting Medications Mentioned in Twitter User Timelines

Abstract

Tweets mentioning medications are valuable for efforts in digital epidemiology to supplement traditional methods of monitoring public health. A major obstacle, however, is to differentiate them from the large majority of tweets on other topics posted in a user's timeline: solving the infamous 'needle in a haystack' problem. While deep learning models have significantly improved classification, their performance and inference processing time remain low on extremely imbalanced corpora where the tweets of interest are less than 1% of all tweets. In this study, we empirically evaluate under-sampling, fine-tuning, and filtering heuristics to train such classifiers. Using a corpus of 212 Twitter timelines (181,607 tweets with only 0.2% tweets mentioning a medication), our results show that combining these heuristics is necessary to impact the classifier's performance. In our intrinsic evaluation, a classifier based on a lexicon and a BERT-base neural network achieved a 0.838 F1-score, a score similar to the ones of the best existing classifier, but it processed the corpus 28 times faster -a positive result, since processing speed is still a roadblock to deploying classifiers on large cohorts of Twitter users needed for pharmacovigilance. In our extrinsic evaluation, our classifier helped a labeler to extract the spans of medications more accurately and achieved a 0.76 Strict F1-score. To the best of our knowledge, this is the first evaluation of medications extraction in Twitter timelines and it establishes the first benchmark for future studies.

Automatic Extraction of Medication Mentions from Tweets—Overview of the BioCreative VII Shared Task 3 Competition

“…At the beginning of the competition, code, documentation and trained models of a baseline extractor were released by the organizers to help participants start their development. Its detailed description and evaluation can be found in ( 12 ); we summarize the details of the system in this section.…”

“…The generic references were manually compiled. In ( 12 ), the authors found that the lexical match approach alone for classification achieved good recall (0.756) but low precision (0.253) due to the ambiguity of some entries in the lexicon. Thus, they improved the performance of the classifier by training a (BERT) Bidirectional Encoder Representations from Transformers model to disambiguate the tweets selected by the lexicon match step.…”

Task reformulation and data-centric approach for Twitter medication name extraction

“…Previous research ( 2 ) shows that better results are obtained by adding a classification step before medication name extraction to classify whether a tweet contains medication names. In this paper, in addition to classifying the tweet in the first step, we also reversed the order and checked whether the extraction was correct by classifying it with PLM after the drug name was extracted.…”

“…For the extractive QA component, we only use the Splinter model, which has the highest performance. In addition, we used the medication name dictionary ( 2 ) from the baseline method provided by the task organizer to exclude data for which none of the tokens of selected spans belong to the dictionary. The lexicon of medication names in the dictionary was tokenized by spaces.…”

“…Since tweet data have the above characteristics, it is not easy to train a sequence labeling model directly on the dataset. The previous research ( 2 ) breaks the task down into two subtasks, proposed a two-stage approach and demonstrated that it is effective for tweet NER in general domains. Thus, we developed a two-stage system for medication extraction, as illustrated in Figure 1 (Procedure I).…”