Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports

Abstract

A significant amount of information about drug-related safety issues such as adverse effects are published in medical case reports that can only be explored by human readers due to their unstructured nature. The work presented here aims at generating a systematically annotated corpus that can support the development and validation of methods for the automatic extraction of drug-related adverse effects from medical case reports. The documents are systematically double annotated in various rounds to ensure consistent annotations. The annotated documents are finally harmonized to generate representative consensus annotations. In order to demonstrate an example use case scenario, the corpus was employed to train and validate models for the classification of informative against the non-informative sentences. A Maximum Entropy classifier trained with simple features and evaluated by 10-fold cross-validation resulted in the F₁ score of 0.70 indicating a potential useful application of the corpus.

On the creation of a clinical gold standard corpus in Spanish: Mining adverse drug reactions

“…The steps followed are documented and reproducible, taking into account the lessons obtained from other similar annotation projects [1,11,16,17,20].…”

“…The ADE corpus [17] is comprised of a subset of 2972 Medline case reports that were manually annotated by three annotators and subsequently harmonized. It contains annotations of 5063 drugs, 5776 conditions (e.g.…”

“…All the aforementioned corpora with adverse drug reactions annotated are gathered from Medline [16,17] or from comments on social media [19,23]. Also, they are all English-language corpora: for Spanish we have only notice of a corpus of user comments annotated with drugs and their effects [23].…”

Adverse Drug Event classification of health records using dictionary based pre-processing and machine learning

“…Gurulingappa et al (2012) developed a manually annotated corpora in English consisting of 3,000 medical case reports (i.e. published scientific reports of specific patients, their drugs and their side effects).…”

Filtering big data from social media – Building an early warning system for adverse drug reactions

“…Based on the annotation guidelines in [33], we recruited domain experts to tag a large sample of posts related to the drugs in our study. The experts read each post assigned to them and assess whether any ADR has been discussed.…”