Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements

Lingren, Todd; Deléger, Louise; Molnar, Katalin; Zhai, Haijun; Meinzen‐Derr, Jareen; Kaiser, Megan; Stoutenborough, Laura; Li, Qi; Solti, Imre

doi:10.1136/amiajnl-2013-001837

Cited by 68 publications

(53 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using our corpus in combination with another corpus means that the other corpus might not need to be as large as it would if used on its own. Also, our corpus can be used to train a system and pre-annotate a new corpus to speed up the manual annotation process [24]. An interesting direction for future work is to investigate domain adaptation methods to improve the machine learning algorithms so that they are less dependent on the underlying dataset to achieve cross-corpora performance closer to in-corpus training experiments.…”

Section: Discussionmentioning

confidence: 99%

Preparing an annotated gold standard corpus to share with extramural investigators for de-identification research

Deléger

Lingren

et al. 2014

Journal of Biomedical Informatics

Self Cite

View full text Add to dashboard Cite

Objective The current study aims to fill the gap in available healthcare de-identification resources by creating a new sharable dataset with realistic Protected Health Information (PHI) without reducing the value of the data for de-identification research. By releasing the annotated gold standard corpus with Data Use Agreement we would like to encourage other Computational Linguists to experiment with our data and develop new machine learning models for de-identification. This paper describes: (1) the modifications required by the Institutional Review Board before sharing the de-identification gold standard corpus; (2) our efforts to keep the PHI as realistic as possible; (3) and the tests to show the effectiveness of these efforts in preserving the value of the modified data set for machine learning model development. Material and Methods In a previous study we built an original de-identification gold standard corpus annotated with true Protected Health Information (PHI) from 3,503 randomly selected clinical notes for the 22 most frequent clinical note types of our institution. In the current study we modified the original gold standard corpus to make it suitable for external sharing by replacing HIPAA-specified PHI with newly generated realistic PHI. Finally, we evaluated the research value of this new dataset by comparing the performance of an existing published in-house de-identification system, when trained on the new de-identification gold standard corpus, with the performance of the same system, when trained on the original corpus. We assessed the potential benefits of using the new de-identification gold standard corpus to identify PHI in the i2b2 and PhysioNet datasets that were released by other groups for de-identification research. We also measured the effectiveness of the i2b2 and PhysioNet de-identification gold standard corpora in identifying PHI in our original clinical notes. Results Performance of the de-identification system using the new gold standard corpus as a training set was very close to training on the original corpus (92.56 vs. 93.48 overall F-measures). Best i2b2/PhysioNet/CCHMC cross-training performances were obtained when training on the new shared CCHMC gold standard corpus, although performances were still lower than corpus-specific trainings. Discussion and conclusion We successfully modified a de-identification dataset for external sharing while preserving the de-identification research value of the modified gold standard corpus with limited drop in machine learning de-identification performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Preparing an annotated gold standard corpus to share with extramural investigators for de-identification research

Deléger

Lingren

et al. 2014

Journal of Biomedical Informatics

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the clinical domain Lingren et al [11] demonstrate that pre-annotation reduces the annotation time significantly for the NE annotation layer. As an additional result they reviled that pre-annotation did not influence the Inter Annotator Agreement (IAA) or annotator performance.…”

Section: Related Workmentioning

confidence: 99%

“…Lingren et al [11], Loftsson et al [12], and Fort and Sagot [13] investigate the annotation process with pre-annotated corpora and they show that annotation time can be reduced. We reuse the idea of pre-annotation in the semi-automatic annotation process of the QPT (c. f. Section 3.2), resulting in higher annotation speed (c. f. Section 4).…”

Section: Related Workmentioning

confidence: 99%

Quick Pad Tagger : An Efficient Graphical User Interface for Building Annotated Corpora with Multiple Annotation Layers

Schreiber

Barkschat

Kraft

et al. 2015

Computer Science &Amp; Information Technology ( CS &Amp; IT )

View full text Add to dashboard Cite

More and more domain specific applications in the internet make use of Natural Language Processing (NLP) tools (e. g. Information Extraction systems). The output quality of these applications relies on the output quality of the used NLP tools. Often, the quality can be increased by annotating a domain specific corpus. However, annotating a corpus is a time consuming and exhaustive task. To reduce the annotation time we present a custom GraphicalUser Interface for different annotation layers.

show abstract

“…For the task of medical named entity labeling, Lingren et al (2013) investigate the impact of automatic suggestions on annotation speed and potential biases using dictionary-based annotations. This technique results in 13.83% to 21.5% time saving and in an inter-annotator agreement (IAA) increase by several percentage points.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations

2014

View full text Add to dashboard Cite

ii Preface Welcome to the proceedings of the system demonstration session. This volume contains the papers of the system demonstrations presented at the 52nd Annual Meeting of the Association for Computational Linguistics, on June 23-24, 2014 in Baltimore, Maryland, USA.The system demonstrations program offers the presentation of early research prototypes as well as interesting mature systems. The system demonstration chairs and the members of the program committee received 39 submissions, of which 21 were selected for inclusion in the program (acceptance rate of 53.8%) after review by three members of the program committee.We would like to thank the members of the program committee for their timely help in reviewing the submissions.iii AbstractWe introduce a method that extracts keywords in a language with the help of the other. In our approach, we bridge and fuse conventionally irrelevant word statistics in languages. The method involves estimating preferences for keywords w.r.t. domain topics and generating cross-lingual bridges for word statistics integration. At run-time, we transform parallel articles into word graphs, build cross-lingual edges, and exploit PageRank with word keyness information for keyword extraction. We present the system, BiKEA, that applies the method to keyword analysis. Experiments show that keyword extraction benefits from PageRank, globally learned keyword preferences, and cross-lingual word statistics interaction which respects language diversity.

show abstract

Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements

Cited by 68 publications

References 15 publications

Preparing an annotated gold standard corpus to share with extramural investigators for de-identification research

Preparing an annotated gold standard corpus to share with extramural investigators for de-identification research

Quick Pad Tagger : An Efficient Graphical User Interface for Building Annotated Corpora with Multiple Annotation Layers

Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations

Contact Info

Product

Resources

About