SEED: Symptom Extraction from English Social Media Posts using Deep Learning and Transfer Learning

Abstract

Abstract. The increase of social media usage across the globe has fueled efforts in digital epidemiology for mining valuable information such as medication use, adverse drug effects and reports of viral infections that directly and indirectly affect population health. Such information can, however, be scarce, hard to find and mostly expressed in very colloquial language. In this work, we focus on a fundamental problem that enables social media mining for disease monitoring. We present and make available SEED, a natural language processing approach to detect symptom and disease mentions from social media data obtained from platforms such as Twitter and DailyStrength, and to normalize them into UMLS terminology. Using multi-corpus training and deep learning models, the tool achieves an overall F1 score of 0.86 for extracting mentions of symptoms on a health forum dataset and an F1 score of 0.72 on a balanced Twitter dataset, significantly improving over previous, more narrowly defined, approaches on the same datasets. We apply the tool on Twitter posts that report COVID19 symptoms to quantify whether the SEED system can extract symptoms absent in the training data. The study results also draw attention to the potential of multi-corpus training for performance improvements and the need for continual training on newly obtained data for consistent performance amidst the ever-changing nature of the social media vocabulary.

Automated data analysis of unstructured grey literature in health research: A mapping review

“…Another relaxation method was to decrease the amount of potential labels to only include top-level categories. 47,49,50,56 Even when grouping and comparing classification scores for the three classification tasks separately, it is not straightforward to determine the best-performing algorithm within each category. Algorithm performance reported on a domain-specific dataset labelled by a group of researchers according to their own annotation guidelines may vary when the same algorithm is tested on completely new and therefore independent data labelled by different persons.…”

“…42 Other features of unstructured data that cause challenges are colloquialisms, abbreviations, spelling errors, and other variations that appear in natural language. 41,56,65,66 Chee et al 65 describe heterogeneity not within a data source, but between them. Due to different text lengths and/or languages it becomes hard to achieve knowledgeor domain-transfer between sources such as Twitter and online forum entries.…”

“…68,69 Ambiguity also increases complexity, for example when drug name can have multiple synonyms, trade names, or multiple correct labels. 39,56 Noise is a concept that generally refers to data being unreliable due to their unstructured and naturally expressed form, thus causing errors both while labelling gold-standard data and when processing and predicting on new data. 31,52 Sparsity or imbalance within the data ML or neural networks architectures require annotated data during the learning process.…”

Modeling approaches for early warning and monitoring of pandemic situations as well as decision support

“…They further classified these posts while performing NER to obtain mitigation types (such as distancing, disinfection, personal protective equipment) and detection types (such as symptoms, testing) and analyzed for a certain time period the change of people's sentiments toward masks in these posts. For disease monitoring, Magge et al ( 76 ) built a system to collect symptoms and disease mentions from social media platforms and normalized them to unified medical language system (UMLS) terminology. Using deep learning methods (such as BERT and RoBERTa) that were trained on multiple available corpora (such as TwiMed, MedNorm, DS-NER), they achieved an F 1 -score of 0.86 and 0.75 on DailyStrength and Twitter datasets, respectively.…”

Automatic Symptoms Identification from Massive Volume of Unstructured Medical Consultations Using Deep Neural and BERT Models

“…Roughly, different studies for symptom identification were concerned with general text that is not related directly to the medical or clinical text. For instance, Magge et al [37] developed a framework called "SEED" for symptom extraction from social media posts. The authors implemented deep learning and transfer learning approach that achieved an f1-score of 85%.…”