Pre-annotating Clinical Notes and Clinical Trial Announcements for Gold Standard Corpus Development: Evaluating the Impact on Annotation Speed and Potential Bias

Lingren, Todd; Deléger, Louise; Molnar, Katalin; Zhai, Haijun; Meinzen‐Derr, Jareen; Kaiser, Megan; Stoutenborough, Laura; Li, Qi; Solti, Imre

doi:10.1109/hisb.2012.33

Cited by 6 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to our own experience [27, 36] and others [37], automatic pre-annotation can help accelerate the manual annotation process. First, each PMID document was pre-annotated using the Inference Method developed for disease name normalization [9], which properly handles abbreviation recognition, robust string matching, etc.…”

Section: Methodsmentioning

confidence: 99%

NCBI disease corpus: A resource for disease name recognition and concept normalization

Doğan

Leaman

2014

Journal of Biomedical Informatics

597

310

View full text Add to dashboard Cite

Information encoded in natural language in biomedical literature publications is only useful if efficient and reliable ways of accessing and analyzing that information are available. Natural language processing and text mining tools are therefore essential for extracting valuable information, however, the development of powerful, highly effective tools to automatically detect central biomedical concepts such as diseases is conditional on the availability of annotated corpora. This paper presents the disease name and concept annotations of the NCBI disease corpus, a collection of 793 PubMed abstracts fully annotated at the mention and concept level to serve as a research resource for the biomedical natural language processing community. Each PubMed abstract was manually annotated by two annotators with disease mentions and their corresponding concepts in Medical Subject Headings (MeSH®) or Online Mendelian Inheritance in Man (OMIM®). Manual curation was performed using PubTator, which allowed the use of pre-annotations as a pre-step to manual annotations. Fourteen annotators were randomly paired and differing annotations were discussed for reaching a consensus in two annotation phases. In this setting, a high inter-annotator agreement was observed. Finally, all results were checked against annotations of the rest of the corpus to assure corpus-wide consistency. The public release of the NCBI disease corpus contains 6,892 disease mentions, which are mapped to 790 unique disease concepts. Of these, 88% link to a MeSH identifier, while the rest contain an OMIM identifier. We were able to link 91% of the mentions to a single disease concept, while the rest are described as a combination of concepts. In order to help researchers use the corpus to design and test disease identification methods, we have prepared the corpus as training, testing and development sets. To demonstrate its utility, we conducted a benchmarking experiment where we compared three different knowledge-based disease normalization methods with a best performance in F-measure of 63.7%. These results show that the NCBI disease corpus has the potential to significantly improve the state-of-the-art in disease name recognition and normalization research, by providing a high-quality gold standard thus enabling the development of machine-learning based approaches for such tasks.

show abstract

Section: Methodsmentioning

confidence: 99%

NCBI disease corpus: A resource for disease name recognition and concept normalization

Doğan

Leaman

2014

Journal of Biomedical Informatics

597

310

View full text Add to dashboard Cite

show abstract

“…They recommended the identification of biases early in the study and to appropriately train annotators prior to the annotation task. Another study on assessing the effects of the pre-annotation on annotation time and annotator bias was performed by Lingren, et al [137]. Their study involved extracting disease/disorder and sign/symptom entities with the aim of building a gold standard in clinical trial announcements (CTAs).…”

Section: Pre-annotation Settingsmentioning

confidence: 99%

“…This is achieved by reducing the number of annotations that an annotator must add/modify/remove. Pre-annotated data, which is commonly generated using a dictionary [137] or existing NLP systems [138,139,140,141], have resulted in considerable less annotation time compared to fully manually annotated data. However, the accuracy of pre-annotations is directly correlated with savings in annotation time.…”

Section: Annotation Cost Analysis In Practicementioning

confidence: 99%

“…Although its constitution does not have a significant effect on the final AL annotation rate [176], a highly effective initial model learnt on the seed set can serve to provide high quality pre-annotations to annotators early in the AL process. Indeed, studies in fully supervised learning settings have shown that approaches based on high quality pre-annotations lead to lower manual annotation costs compared to an annotation-from-scratch approach [137,141,143].…”

Section: Analysis Of Seed Selection Strategiesmentioning

confidence: 99%

“…Pre-annotated data, which is commonly generated using a dictionary [137] or existing NLP systems [138,139,140,141,143], have resulted in significantly reduced annotation time compared to manually annotating the full dataset. Lingren, et al [137] used a dictionary-based approach to pre-annotate disease/disorder and sign/symptom entities with the aim of developing a gold standard for clinical named entity recognition in clinical trial announcements. Fort and Sagot [143] proposed another approach to pre-annotate parts of speech in the Penn Treebank corpus using POS taggers with different levels of accuracy.…”

Section: Background and Significancementioning

confidence: 99%

See 2 more Smart Citations

Active Learning for Concept Extraction from Clinical Free Text

Kholghi¹

View full text Add to dashboard Cite

An increasing volume of clinical free-text data, such as discharge summaries and progress reports, has been collected by hospitals and healthcare centres and stored electronically for further processing. Extracting structured clinical information from such unstructured text resources is necessary for enabling secondary usage of reports, such as reporting, reasoning and retrieving, and for further processing in down-stream eHealth workflows. However, this analysis cannot be done manually, due to the high cost incurred by qualified experts to annotate the clinical free text. A significant initial step in extracting information from clinical free text is concept extraction, which involves identifying entities of interest in the clinical domain (such as diseases, medications, and symptoms). Currently, supervised machine learning approaches effectively extract clinical concepts by building powerful statistical models. However, these approaches require a large amount of high quality, annotated train data, which is created manually by domain experts through a costly and timeconsuming process. This results in a robust active learning framework for extracting clinical concepts, using state-of-the-art, active learning approaches. The second step is to leverage clinical information resources (i.e., terminologies and clinical information extraction tools) and other machine learning approaches (i.e., semi-supervised learning, unsupervised learning, and representation learning) to develop domain-specific and generic active learning approaches. This leads to a number of novel, active learning query strategies and a seed selection approach that outperform the state-of-the-art approaches with less manual annotation effort. The last step is to validate the benefits of the developed AL-based framework in reducing the annotation cost (i.e., time) through a comprehensive user study. An AL-assisted pre-annotation scheme is also introduced, in which the learning models built across the AL process generate high quality pre-annotations to be reviewed by human annotators. This further accelerates the annotation process, by significantly reducing the number of manual annotations that must be added or corrected compared to de novo annotation.The results of this study demonstrate that AL plays an important role in reducing the manual annotation cost. The CEAL framework extracts high quality domain concepts from clinical narratives, while significantly reducing the labour cost with up to 35% less annotation time required. Additionally, AL-assisted preannotations accelerate the de novo annotation process with a further 20% less annotation time required. This thesis contributes to information extraction from clinical unstructured text resources by alleviating the burden of manual annotation.The practical significance of this research is three-fold: (1) benefitting the overall patient healthcare by facilitating downstream eHealth workflows such as supporting clinical information processing, reporting, reasoning, and efficient decision m...

show abstract