Corpus annotation for mining biomedical events from literature

Kim, Jin-Dong; Ohta, Tomoko; Tsujii, Jun’ichi

doi:10.1186/1471-2105-9-10

Cited by 270 publications

(303 citation statements)

References 50 publications

Supporting

Mentioning

299

Contrasting

Unclassified

Order By: Relevance

“…Secondly, interpretation of the guideline documents differs from annotator to annotator; as a result, some annotation criteria remain problematic and can cause discrepancies [10]. For example, the event annotation on the Genia corpus by [17] only achieves 56% inter-annotator agreement with strict match [20] even though all annotators have been trained and educated using example annotations and guidelines.…”

Section: How To Annotate Properly: What Have We Learnt?mentioning

confidence: 99%

“…Kim et al [17] show by experiments that high level of discrepancy will form without annotation guidelines even if the task is carried out by well-educated domain experts. Their studies on event annotation on the Genia corpus [24] took 1.5 years of effort of five graduate students and two coordinators.…”

Section: How To Annotate Properly: What Have We Learnt?mentioning

confidence: 99%

“…The second case is referred to as intra-annotator "agreement", "consistency" or "discrepancy". Inter-annotator discrepancies are often caused by the differences in annotators' knowledge and experiences, their understanding and reasoning of the corpora [17]. Intra-annotator discrepancies exist because annotators' level of interest and motivation may drop and level of fatigue rises as the annotation process continues [12], as a result, annotators make mistakes.…”

Section: Annotator Discrepancymentioning

confidence: 99%

“…It has been noted in many relevant fields, such as Word Sense Disambiguation (WSD) [23], Speech Recognition [12], Information Retrieval [1], event recognition [17] and Named Entity Recognition (NER) [10][28] [29]. Depending on the difficulty of the annotation task, the inter-annotator agreement can vary significantly.…”

Section: Annotator Discrepancymentioning

confidence: 99%

See 3 more Smart Citations

A Methodology towards Effective and Efficient Manual Document Annotation: Addressing Annotator Discrepancy and Annotation Quality

Zhang

Chapman

Ciravegna

2010

Knowledge Engineering and Management by the Masses

View full text Add to dashboard Cite

Abstract. Manual document annotation is an essential technique for knowledge acquisition and capture. Creating high-quality annotations is a difficult task due to inter-annotator discrepancy, the problem that annotators can never agree completely on what and exactly how to annotate. To address this, traditional document annotation involves multiple domain experts working on the same annotation task in an iterative and collaborative manner to identify and resolve discrepancies progressively. However, such a detailed process is often ineffective despite taking significant time and effort; unfortunately, discrepancies remain high in many cases. This paper proposes an alternative approach to document annotation. The approach tackles the problem by firstly studying annotators' suitability based on the types of information to be annotated; then identifying and isolating the most inconsistent annotators who tend to cause the majority of discrepancies in a task; finally distributing annotation workload among the most suitable annotators. Tested in a named entity annotation task in the domain of archaeology, we show that compared to the traditional approach to document annotation, it produces larger amounts of better quality annotations that result in higher machine learning accuracy while requires significantly less time and effort.

show abstract

Section: How To Annotate Properly: What Have We Learnt?mentioning

confidence: 99%

Section: How To Annotate Properly: What Have We Learnt?mentioning

confidence: 99%

Section: Annotator Discrepancymentioning

confidence: 99%

Section: Annotator Discrepancymentioning

confidence: 99%

See 2 more Smart Citations

A Methodology towards Effective and Efficient Manual Document Annotation: Addressing Annotator Discrepancy and Annotation Quality

Zhang

Chapman

Ciravegna

2010

Knowledge Engineering and Management by the Masses

View full text Add to dashboard Cite

show abstract

“…The edges define each association (binding, regulation or indirect regulation) between genes in the families extracted from the literature. The definition of 'binding' and 'regulation' was adopted from Gene Ontology (GO) by GENIA corpus (Kim, Ohta, and Tsujii 2008). The event annotations in GENIA corpus were used for training the text-mining system.…”

Section: Text-mining Datamentioning

confidence: 99%

Finding novel relationships with integrated gene-gene association network analysis of Synechocystis sp. PCC 6803 using species-independent text-mining

Kreula

Kaewphan

Ginter

et al. 2017

Preprint

View full text Add to dashboard Cite

The increasing move towards open access full-text scientific literature enhances our ability to utilize advanced text-mining methods to construct information-rich networks that no human will be able to grasp simply from 'reading the literature'. The utility of text-mining for well-studied species is obvious though the utility for less studied species, or those with no prior track-record at all, is not clear. Here we present a concept for how advanced textmining can be used to create information-rich networks even for less well studied species and apply it to generate an open-access gene-gene association network resource for Synechocystis sp. PCC 6803, a representative model organism for cyanobacteria and first case-study for the methodology. By merging the text-mining network with networks generated from species-specific experimental data, network integration was used to enhance the accuracy of predicting novel interactions that are biologically relevant. A rulebased algorithm was constructed in order to automate the search for novel candidate genes with a high degree of likely association to known target genes by (1) ignoring established relationships from the existing literature, as they are already 'known', and (2) demanding multiple independent evidences for every novel and potentially relevant relationship. Using selected case studies, we demonstrate the utility of the network resource and rule-based algorithm to (i) discover novel candidate associations between different genes or proteins in the network, and (ii) rapidly evaluate the potential role of ABSTRACTThe increasing move towards open access full-text scientific literature enhances our ability to utilize advanced text-mining methods to construct information-rich networks that no human will be able to grasp simply from 'reading the literature'. The utility of text-mining for well-studied species is obvious though the utility for less studied species, or those with no prior track-record at all, is not clear. Here we present a concept for how advanced text-mining can be used to create information-rich networks even for less well studied species and apply it to generate an openaccess gene-gene association network resource for Synechocystis sp. PCC 6803, a representative model organism for cyanobacteria and first case-study for the methodology. uy merging the textmining network with networks generated from species-specific experimental data, network integration was used to enhance the accuracy of predicting novel interactions that are biologically relevant. A rule-based algorithm was constructed in order to automate the search for novel candidate genes with a high degree of likely association to known target genes by (1) ignoring established relationships from the existing literature, as they are already 'known', and (2) demanding multiple independent evidences for every novel and potentially relevant relationship. Using selected case studies, we demonstrate the utility of the network resource and rule-based algorithm to (i) discover novel candidate ass...

show abstract

Augmenting Biological Text Mining with Symbolic Inference

Park

Lee

2013

Biological Knowledge Discovery Handbook

View full text Add to dashboard Cite

Corpus annotation for mining biomedical events from literature

Cited by 270 publications

References 50 publications

A Methodology towards Effective and Efficient Manual Document Annotation: Addressing Annotator Discrepancy and Annotation Quality

A Methodology towards Effective and Efficient Manual Document Annotation: Addressing Annotator Discrepancy and Annotation Quality

Finding novel relationships with integrated gene-gene association network analysis of Synechocystis sp. PCC 6803 using species-independent text-mining

Augmenting Biological Text Mining with Symbolic Inference

Contact Info

Product

Resources

About