SemEval-2016 Task 12: Clinical TempEval

Bethard, Steven; Savova, Guergana; Chen, Wei-Te; Derczynski, Leon; Pustejovsky, James; Verhagen, Marc

doi:10.18653/v1/s16-1165

Cited by 166 publications

(129 citation statements)

References 15 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…For example, Bada et al (2012) achieved 90?% annotator-reviser agreement for biomedical concept annotation in the CRAFT corpus. In the THYME corpus, Bethard et al (2016) reported an interannotator agreement of 0.731 (F1) for temporal expressions, and an annotator-adjudicator agreement of 0.830. Tables 6 and 7 report the figures of the IAA values between pairs of annotators, computed as the average F-measure of both sets that were double-annotated.…”

Section: Inter-annotator Agreement (Iaa)mentioning

confidence: 98%

“…Research challenges have also fuelled the annotation of resources or enrichment of available texts. Well-known corpora come from the i2b2 challenges (Uzuner et al 2010(Uzuner et al , 2011Sun et al 2013), SemEval (Bethard et al 2016) and the Shared Annotated Resources (ShARe)/CLEF eHealth labs. 8 Overall, two levels of annotations have been applied in clinical texts.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

Campillos

Deléger

Grouin

et al. 2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

Quality annotated resources are essential for Natural Language Processing. The objective of this work is to present a corpus of clinical narratives in French annotated for linguistic, semantic and structural information, aimed at clinical information extraction. Six annotators contributed to the corpus annotation, using a comprehensive annotation scheme covering 21 entities, 11 attributes and 37 relations. All annotators trained on a small, common portion of the corpus before proceeding independently. An automatic tool was used to produce entity and attribute pre-annotations. About a tenth of the corpus was doubly annotated and annotation differences were resolved in consensus meetings. To ensure annotation consistency throughout the corpus, we devised harmonization tools to automatically identify annotation differences to be addressed to improve the overall corpus quality. The annotation project spanned over 24 months and resulted in a corpus comprising 500 documents (148,476 tokens) annotated with 44,740 entities and 26,478 relations. The average inter-annotator agreement is 0.793 F-measure for entities and 0.789 for relations. The performance of the pre-annotation tool for entities reached 0.814 F-measure when sufficient training data was available. The performance of our entity pre-annotation tool shows the value of the corpus to build and evaluate information extraction methods. In addition, we introduced harmonization methods that further improved the quality of annotations in the corpus.Electronic supplementary material The online version of this article

show abstract

Section: Inter-annotator Agreement (Iaa)mentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

Campillos

Deléger

Grouin

et al. 2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

show abstract

“…We are also interested in extending the method to PPIs beyond the sentence boundary. Finally, we would like to test and generalize this approach to other biomedical relations such as chemical-disease relations (Wei et al, 2016). …”

Section: Resultsmentioning

confidence: 99%

BioNLP 2017

Cohen¹,

Demner‐Fushman²,

Ananiadou³

et al. 2017

View full text Add to dashboard Cite

According to the Association for Computational Linguistics guidelines on special interest groups (SIGs), The function of a SIG is to encourage interest and activity in specific areas within the ACL's field [1]. Is the SIGBioMed special interest group "within the ACL's field"? The titles of this year's papers suggest that it is, in that the current interest in deep learning in its many and varied manifestations is mirrored in those titles. Do those papers cover a specific area? They do, and in doing so, they demonstrate one of the great satisfactions of working in biomedical natural language processing.One of the joys of involvement in the biomedical natural language processing community is seeing the development of research with clinical applications. As examples of such work being presented at BioNLP 2017, we would like to point out the two papers that discuss the application of natural language processing to the diagnosis of neurological disorders. Bhatia et al. [2] describe an approach to using speech processing in the assessment of patients with amyotrophic lateral sclerosis (also known as Lou Gehrig's disease), one of the more horrific motor neuron diseases. Good assessment of amyotrophic lateral sclerosis patients is important for a number of reasons, including the fact that accurate tracking of the inevitable deterioration that is a hallmark of this disease gives patients and their families the possibility of purposeful planning for the attendant disability and death. However, current methodologies for evaluating the status of amyotrophic lateral sclerosis patients necessarily involve expensive equipment and highly trained personnel; when further developed, this methodology could make such evaluation much more, and more frequently, available to ALS patients. The fact that the work reported here involves a speech modality is especially exciting, as speech-related indicators of future ALS can be present long before diagnosis. The paper uses measurements of phonological features of speech and their divergence from a baseline, and demonstrates correlation with physiological measures.Adams et al. [3] describe work on detecting and categorizing word production errors associated with anomia, a particular kind of inability to find words. Screening for anomia is important because anomia is a symptom of stroke, but it is difficult and time-consuming to do, and therefore is not done as often as it should be. Automatic detection of anomia could be a nice enabler of improved care for stroke victims, but it is made difficult due to the subtlety of the phonological and semantic judgments that have to be made when assessing the phenomenon. The paper uses a combination of language modeling and phonologically-based edit distance calculation to approach the task, applying these techniques to data from the AphasiaBank collection of transcribed aphasic and healthy speech.Although we have summarized only these two examples that address neurological disorders, there are several other papers on the use of natural language proc...

show abstract

“…It is a successor to Clinical TempEval 2016 (Bethard et al, 2016), Clinical TempEval 2015 (Bethard et al, 2015), and the i2b2 temporal challenge (Sun et al, 2013).…”

Section: Introductionmentioning

confidence: 99%

GUIR at SemEval-2017 Task 12: A Framework for Cross-Domain Clinical Temporal Information Extraction

MacAvaney¹,

Cohan²,

Goharian³

2017

Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)

View full text Add to dashboard Cite

Clinical TempEval 2017 (SemEval 2017 Task 12) addresses the task of crossdomain temporal extraction from clinical text. We present a system for this task that uses supervised learning for the extraction of temporal expression and event spans with corresponding attributes and narrative container relations. Approaches include conditional random fields and decision tree ensembles, using lexical, syntactic, semantic, distributional, and rulebased features. Our system received best or second best scores in TIMEX3 span, EVENT span, and CONTAINS relation extraction.

show abstract

SemEval-2016 Task 12: Clinical TempEval

Cited by 166 publications

References 15 publications

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

BioNLP 2017

GUIR at SemEval-2017 Task 12: A Framework for Cross-Domain Clinical Temporal Information Extraction

Contact Info

Product

Resources

About