ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance

Abstract

Introduction The US FDA is interested in a tool that would enable pharmacovigilance safety evaluators to automate the identification of adverse drug events (ADEs) mentioned in FDA prescribing information. The MITRE Corporation (MITRE) and the FDA organized a shared task-Adverse Drug Event Evaluation (ADE Eval)-to determine whether the performance of algorithms currently used for natural language processing (NLP) might be good enough for real-world use. Objective ADE Eval was conducted to evaluate a range of NLP techniques for identifying ADEs mentioned in publicly available FDA-approved drug labels (package inserts). It was designed specifically to reflect pharmacovigilance practices within the FDA and model possible pharmacovigilance use cases. Methods Pharmacovigilance-specific annotation guidelines and annotated corpora were created. Two metrics modeled the experiences of FDA safety evaluators: one measured the ability of an algorithm to identify correct Medical Dictionary for Regulatory Activities (MedDRA ®) terms for the text from the annotated corpora, and the other assessed the quality of evidence extracted from the corpora to support the selected MedDRA ® term by measuring the portion of annotated text an algorithm correctly identified. A third metric assessed the cost of correcting system output for subsequent training (averaged, weighted F1-measure for mention finding). Results In total, 13 teams submitted 23 runs: the top MedDRA ® coding F1-measure was 0.79, the top quality score was 0.96, and the top mention-finding F1-measure was 0.89. Conclusion While NLP techniques do not perform at levels that would allow them to be used without intervention, it is now worthwhile exploring making NLP outputs available in human pharmacovigilance workflows.

Natural Language Processing: from Bedside to Everywhere

“…However, it is not trivial even for professional clinicians to decide if a disorder written in a document was certainly caused by some drugs or not, which may result in difficulty in annotations [107]. In fact, the performance in detecting ADE relations, which distributes around 0.5 F1-score, were substantially low in comparison to drug-attribute relation extraction, most models of which achieved around 0.9 F1-score [103,107,109].…”

“…ADE detection tends to be defined as RE [103,104,107,109] so that the causality information of possible ADEs is directly encoded. In our example (the middle row of Figure 1), the drug entity "nivlumab" should be connected to "liver damage" by an ADE-causing relation ("CAUSED").…”

Exploring opportunities for AI supported medication error categorization: A brief report in human machine collaboration

“…Much work has been done on the development of algorithms and techniques to categorize and identify adverse drug events in FAERS reports (Botsis et al, 2014;Combi et al, 2018;Eskildsen et al, 2020). A recent review of addition to FAERS reports adverse drug events were able to be extracted from drug labels and Vaccine Adverse Event Reporting System (VAERS) reports using NLP and rule-based techniques (Botsis et al, 2013;Ly et al, 2018;Bayer et al, 2021;Du et al, 2021). Vaccine Adverse Event Text Miner (VaeTM) is an example of a text mining system developed to extract safety concepts from VAERS reports (Botsis et al, 2011;Botsis et al, 2012;Baer et al, 2016).…”

Multilabel classification of medical concepts for patient clinical profile identification

“…Such examples of natural language processing for the selection of clinical trial cohorts [9] or pharmacovigilance studies [10] have already been proposed but were task specific.…”