Evaluation of Natural Language Processing (NLP) systems to annotate drug product labeling with MedDRA terminology

Ly, Thomas; Pamer, Carol; Dang, Oanh; Brajovic, Sonja; Haider, Shahrukh; Botsis, Taxiarchis; Milward, David; Winter, Andrew G.; Lu, Susan; Ball, Robert

doi:10.1016/j.jbi.2018.05.019

Cited by 41 publications

(41 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The FDA Center for Drug Evaluation and Research (CDER) has a long-standing interest in the ability to automatically extract ADEs from PIs for the purpose of pharmacovigilance. Ly et al [3] evaluated the performance of the following three NLP systems for their ability to extract ADE terms from PI labels and normalize the terms to MedDRA ® PTs:…”

Section: Us Fda Center For Drug Evaluation and Research National Insmentioning

confidence: 99%

“…https ://daily med.nlm.nih.gov/daily med/. 3 MedDRA® terminology is the international medical terminology developed under the auspices of the International Council on Harmonization (ICH) of Technical Requirements for Registration of Pharmaceuticals for Human Use. The MedDRA® trademark is registered by the International Federation of Pharmaceutical Manufacturers and Associations (IFPMA) on behalf of the ICH.…”

Section: Us Fda Center For Drug Evaluation and Research National Insmentioning

confidence: 99%

“…However, within the current SE workflow, the process of determining and comparing the labeled status of an ADE in a PI with that of the ADEs described in ICSRs is a manual one. This is because the ADEs reported in each ICSR are standardized to the Medical Dictionary for Regulatory Activities (MedDRA ® ; https ://www.meddr a.org) terminology 3 but the ADEs mentioned in a PI are not and may appear as unstructured text in forms that do not exactly match any of the alternative terms listed in MedDRA ® for the relevant ADE. Because a common terminology is crucial for SEs to readily determine and compare the labeled status of ADEs with that of the MedDRA ® -coded ADEs reported in ICSRs, it would be extremely useful for the FDA to have a tool that could summarize, for a set of PIs, the particular ADEs mentioned, using MedDRA ® as the reference vocabulary, and could locate, within the particular PI sections, the evidence for the ADEs mentioned.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance

et al. 2020

Self Cite

View full text Add to dashboard Cite

Introduction The US FDA is interested in a tool that would enable pharmacovigilance safety evaluators to automate the identification of adverse drug events (ADEs) mentioned in FDA prescribing information. The MITRE Corporation (MITRE) and the FDA organized a shared task-Adverse Drug Event Evaluation (ADE Eval)-to determine whether the performance of algorithms currently used for natural language processing (NLP) might be good enough for real-world use. Objective ADE Eval was conducted to evaluate a range of NLP techniques for identifying ADEs mentioned in publicly available FDA-approved drug labels (package inserts). It was designed specifically to reflect pharmacovigilance practices within the FDA and model possible pharmacovigilance use cases. Methods Pharmacovigilance-specific annotation guidelines and annotated corpora were created. Two metrics modeled the experiences of FDA safety evaluators: one measured the ability of an algorithm to identify correct Medical Dictionary for Regulatory Activities (MedDRA ®) terms for the text from the annotated corpora, and the other assessed the quality of evidence extracted from the corpora to support the selected MedDRA ® term by measuring the portion of annotated text an algorithm correctly identified. A third metric assessed the cost of correcting system output for subsequent training (averaged, weighted F1-measure for mention finding). Results In total, 13 teams submitted 23 runs: the top MedDRA ® coding F1-measure was 0.79, the top quality score was 0.96, and the top mention-finding F1-measure was 0.89. Conclusion While NLP techniques do not perform at levels that would allow them to be used without intervention, it is now worthwhile exploring making NLP outputs available in human pharmacovigilance workflows.

show abstract

Section: Us Fda Center For Drug Evaluation and Research National Insmentioning

confidence: 99%

Section: Us Fda Center For Drug Evaluation and Research National Insmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, topic words can be positioned and spatially clustered to realize the quantitative, automated, and spatialized construction of meaning maps and comprehensively sense feedback on various aspects of daily life from people from different regions. Moreover, NLP has made remarkable progress in text-based perspective extraction, topic word extraction, and in the semantic analysis of emotions [33][34][35]. Abundant samples are available for place name extraction and type recognition, which can provide data and a basis for the analysis methods in this study, and substantially promote the feasibility of the analysis.…”

Section: Introductionmentioning

confidence: 99%

Analysis of Spatial Interaction between Different Food Cultures in South and North China: Practices from People’s Daily Life

Zhang

Zhou

Huang

2020

IJGI

View full text Add to dashboard Cite

An important component of research in cultural geography involves the exploration and analysis of the laws of regional cultural differences. This topic has considerable significance in the discovery of distinctive cultures, protection of regional cultures, and in-depth understanding of cultural differences. In recent years, with the “spatial turn” of sociology, scholars have focused increasing attention to implicit spatial information in social media data, as well as the social phenomena and laws they reflect. Grasping sociocultural phenomena and their spatial distribution characteristics through texts is an important aspect. Using machine learning methods, such as the popular natural language processing (NLP) approach, this study extracts hotspot cultural elements from text data and accurately detects the spatial interaction patterns of specific cultures, as well as the characteristics of emotions toward non-native cultures. Through NLP, this study examines cultural differences among people from South and North China by analyzing 6128 answers to the question, “What are the differences between South and North China that you ever know?” posted on the Zhihu Q&A platform. Moreover, this study probes individuals’ emotions and cognition of cultural differences between South and North China in three aspects, namely, spatial interaction patterns of hotspot cultural elements, components of hotspot cultures, and emotional characteristics under the influence of cultural differences between the two regions. Results reveal that: (1) people from North and South China exhibit considerable differences in recognizing each other’s culture; (2) among numerous cultural differences, food culture is the most popular; and (3) people tend to have a negative attitude toward food cultures that differ from their own. These factors can shed light on regional cultural differences and help address cultural conflicts. In addition, this study provides effective solutions from a macro perspective, which has been challenging for new cultural geography.

show abstract

“…More importantly, all these data contain location information, meaning not only the topic words can be extracted from the massive social media data, for emotion analysis, but also the extracted topic words can be positioned and spatially clustered, in a bid to realize the quantitative, automated and spatialized construction of the map of meaning and comprehensively sense the feedback on various aspects of daily life by people from different regions. What's more, NLP has made great progress in text-based perspective extraction, topic word extraction, and semantic analysis of emotions [33][34][35]. There are lots of available samples used for place name extraction and type recognition, which provide data and basis methods for analysis in this paper, greatly promotes the feasibility of analysis.…”

Section: Introductionmentioning

confidence: 99%

Analysis of Spatial Interaction between Different Food Cultures in South and North of China — Practices from People's Daily Life

Zhang¹,

Zhou²,

Huang³

2019

Preprint

View full text Add to dashboard Cite

As an important research content in cultural geography, the exploration and analysis of the laws of regional cultural differences has great significance for the discovery of distinctive cultures, protection of regional cultures and in-depth understanding of cultural differences. In recent years, with the "spatial turn" of sociology, scholars are paying more and more attention to the implicit spatial information in social media data and the various social phenomena and laws they reflect. One important aspect is to grasp the social cultural phenomena and its spatial distribution characteristics through the text. Using machine learning methods such as the popular natural language processing (NLP), this paper can not only extract hotspot cultural elements from text data but also accurately detect the spatial interaction pattern of some specific cultures and the characteristics of emotions towards non-native cultures. Taking the 6,128 answers to the question “what are the differences between South and North China that you never know” on the Zhihu Q&A Platform as an example, with the help of NLP, this paper has explored the cultural differences between South and North China in people’s mind. This paper probes into people’s feeling and cognition of the cultural differences between South and North China from three aspects, including spatial interaction patterns of hotspot cultural elements, components of hotspot culture and emotional characteristics under the influence of cultural differences between North and South. The study reveals that 1) people from North and South China have great differences in recognizing each other’s culture. 2) Food culture is the most popular among many cultural differences. 3) People tend to show negative attitude towards the food cultures different from their own. All these findings shed light upon the understanding of regional cultural differences and addressing cultural conflicts. In addition, this paper also provides an effective solution to the study from a macro perspective, which have been difficult for new cultural geography.

show abstract

Evaluation of Natural Language Processing (NLP) systems to annotate drug product labeling with MedDRA terminology

Cited by 41 publications

References 12 publications

ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance

ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance

Analysis of Spatial Interaction between Different Food Cultures in South and North China: Practices from People’s Daily Life

Analysis of Spatial Interaction between Different Food Cultures in South and North of China — Practices from People's Daily Life

Contact Info

Product

Resources

About

Evaluation of Natural Language Processing (NLP) systems to annotate drug product labeling with MedDRA terminology

Cited by 41 publications

References 12 publications

ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance

ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance

Analysis of Spatial Interaction between Different Food Cultures in South and North China: Practices from People’s Daily Life

Analysis of Spatial Interaction between Different Food Cultures in South and North of China &mdash; Practices from People's Daily Life

Contact Info

Product

Resources

About

Analysis of Spatial Interaction between Different Food Cultures in South and North of China — Practices from People's Daily Life