The TempEval challenge: identifying temporal relations in text

Abstract

TempEval is a framework for evaluating systems that automatically annotate texts with temporal relations. It was created in the context of the SemEval 2007 workshop and uses the TimeML annotation language. The evaluation consists of three subtasks of temporal annotation: anchoring an event to a time expression in the same sentence, anchoring an event to the document creation time, and ordering main events in consecutive sentences. In this paper we describe the TempEval task and the systems that participated in the evaluation. In addition, we describe how further task decomposition can bring even more structure to the evaluation of temporal relations.

The State of the Art in Semantic Representation

“…Most temporal semantic work in NLP has focused on temporal relations between events, either by timestamping them according to time expressions found in the text, or by predicting their relative order in time. Important resources include TimeML, a specification language for temporal relations (Pustejovsky et al, 2003), and the TempEval series of shared tasks and annotated corpora (Verhagen et al, 2009(Verhagen et al, , 2010UzZaman et al, 2013). A different line of work explores scripts: schematic, temporally ordered sequences of events associated with a certain scenario Jurafsky, 2008, 2009;Regneri et al, 2010).…”

A corpus of clinical narratives annotated with temporal information

“…The TempEval competition series [10,11] focuses on the extraction of events, temporal expressions and the relations between them for general natural language text, based on the Time Markup Language (TimeML) [6]. Our annotation scheme is based on a subset the TempEval2 challenge.…”

Computational semantic analysis of language: SemEval-2007 and beyond

“…Of the five papers accepted, four are papers that present detailed descriptions of tasks organized at SemEval-2007, while the fifth stands independent of the workshop. Each of the first four papers (Girju et al 2009;McCarthy and Navigli 2009;Verhagen et al 2009;Markert and Nissim 2009) SemEval tasks by outlining the motivation for the task, the guidelines used to create the data and resources, the participant systems from SemEval-2007, and the main contributions and lessons learned from the evaluation. The fifth paper (Chen and Palmer 2009) presents a work on robust verb sense disambiguation, which also includes a post-workshop evaluation using SemEval-2007 data.…”

“…Since there can be no definitive ''truth set'' for this task, the authors spend considerable time discussing the formation of the data set and the post-hoc analysis of the participant systems' results. Verhagen et al (2009) present SemEval-2007 Task 15: TempEval Temporal Relation Identification. The TempEval task encompasses three temporal relation subtasks: specifying the relation between an event and a time expression within a sentence, specifying the relationship between an event and the document creation time, and providing an ordering of events in consecutive sentences.…”