A corpus for mining drug-related knowledge from Twitter chatter: Language models and their utilities

Abstract

In this data article, we present to the data science, natural language processing and public heath communities an unlabeled corpus and a set of language models. We collected the data from Twitter using drug names as keywords, including their common misspelled forms. Using this data, which is rich in drug-related chatter, we developed language models to aid the development of data mining tools and methods in this domain. We generated several models that capture (i) distributed word representations and (ii) probabilities of n-gram sequences. The data set we are releasing consists of 267,215 Twitter posts made during the four-month period—November, 2014 to February, 2015. The posts mention over 250 drug-related keywords. The language models encapsulate semantic and sequential properties of the texts.

Translational Biomedical Informatics and Pharmacometrics Approaches in the Drug Interactions Research

“…In addition, another corpus, also created by the Gonzalez laboratory, consists of 267,215 Twitter posts. In this corpus, two sets of language models were created to encapsulate “semantic properties” by presenting word tokens as dense vectors and “n‐gram sequences” by capturing sequential patterns . Moreover, TwiMed is one of the most recent corpus, which comprises 1,000 tweets and 1,000 PubMed sentences .…”

“…The corpora for social media were annotated differently from those in literature. Two corpora, created by the Gonzalez laboratory, were annotated in different scopes . One focused on entity level and another focused on language models.…”

“…By the same authors, the distribution word representations were generated to capture different types of semantic information and an n‐gram sequential language model was used to capture sequential word occurrence probability. Utilizing both information facilitates for the text classification and text normalization for drug‐related knowledge . Except for the commonly used features extracted from narratives, sentiment analysis features is valuable for improving the performance of detection.…”

“…In this corpus, two sets of language models were created to encapsulate "semantic properties" by presenting word tokens as dense vectors and "n-gram sequences" by capturing sequential patterns. 88 Moreover, TwiMed is one of the most recent corpus, which comprises 1,000 tweets and 1,000 PubMed sentences. 86 The annotations covered entities (drug, symptom, and disease) and their relations (outcome-negative, outcome-positive, and reason-to-use).…”

Capturing the Patient’s Perspective: a Review of Advances in Natural Language Processing of Health-Related Text

“…Pimpalkhute et al [101] proposed a phonetic spelling variant generator that automatically generates common misspellings given a term. While the system has been used for collecting medication-related chatter from Twitter [102] and personal health messages [103], it may be applied to a variety of other data collection tasks. Data collection from OHCs has not faced similar challenges, as posts are usually categorized/structured, and the strategies employed have been simpler.…”