A Multi-Axis Annotation Scheme for Event Temporal Relations

Ning, Qiang; Wu, Hao; Roth, Dan

doi:10.18653/v1/p18-1122

Cited by 108 publications

(164 citation statements)

References 27 publications

Supporting

Mentioning

163

Contrasting

Order By: Relevance

“…Following that, we provide a benchmark evaluation in Sec. 3 on the TempEval3 and the MATRES datasets (UzZaman et al, 2013;Ning et al, 2018c). Finally, we point out directions for future work and conclude this paper.…”

Section: Introductionmentioning

confidence: 79%

“…Specifically, CogCompTime only considers those main-axis events, so event extraction is simply a binary classification problem (i.e., whether or not a token is a main-axis event or not). As defined by the MATRES annotation scheme (Ning et al, 2018c), main-axis events are those events that form the primary timeline of a story and approximately 60%-70% of the verbs are on the main-axis in MATRES. We extract lemmas and POS tags within a fixed window, SRL, and prepositional phrase head, and train a sparse averaged perceptron for event extraction.…”

Section: Event Extraction Componentmentioning

confidence: 99%

See 1 more Smart Citation

CogCompTime: A Tool for Understanding Time in Natural Language

Ning¹,

Zhou²,

Feng³

et al. 2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

Self Cite

View full text Add to dashboard Cite

Automatic extraction of temporal information is important for natural language understanding. It involves two basic tasks: (1) Understanding time expressions that are mentioned explicitly in text (e.g., February 27, 1998 or tomorrow), and (2) Understanding temporal information that is conveyed implicitly via relations. This paper introduces CogCompTime, a system that has these two important functionalities. It incorporates the most recent progress, achieves state-of-the-art performance, and is publicly available. 1 We believe that this demo will provide valuable insight for temporal understanding and be useful for multiple time-aware applications.

show abstract

Section: Introductionmentioning

confidence: 79%

Section: Event Extraction Componentmentioning

confidence: 99%

CogCompTime: A Tool for Understanding Time in Natural Language

Ning¹,

Zhou²,

Feng³

et al. 2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

Self Cite

View full text Add to dashboard Cite

show abstract

“…In terms of "events" commonsense, Rashkin et al (2018) investigated the intent and reaction of participants of an event, and Zellers et al (2018) best of our knowledge, no earlier work has focused on temporal commonsense, although it is critical for event understanding. For instance, Ning et al (2018c) argues that resolving ambiguous and implicit mentions of event durations in text (a specific kind of temporal commonsense) is necessary to construct the timeline of a story.…”

Section: Related Workmentioning

confidence: 99%

“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding

Zhou¹,

Khashabi²,

Ning³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

109

105

View full text Add to dashboard Cite

Understanding time is crucial for understanding events expressed in natural language. Because people rarely say the obvious, it is often necessary to have commonsense knowledge about various temporal aspects of events, such as duration, frequency, and temporal order. However, this important problem has so far received limited attention. This paper systematically studies this temporal commonsense problem. Specifically, we define five classes of temporal commonsense, and use crowdsourcing to develop a new dataset, MCTACO , that serves as a test set for this task. We find that the best current methods used on MCTACO are still far behind human performance, by about 20%, and discuss several directions for improvement. We hope that the new dataset and our study here can foster more future research on this topic. 1 1 The dataset, annotation interfaces, guidelines, and qualification tests are available at:

show abstract

“…To avoid missing relations, annotators are required to exhaustively label every pair of events in a document (i.e., the complete annotation scheme), so it is necessary to study ESPA in this context. Here we adopt the MATRES dataset (Ning et al, 2018b) for its better inter-annotator agreement and relatively large size.…”

Section: Temporal Relation Extractionmentioning

confidence: 99%

Partial Or Complete, That’s The Question

Ning

Fan

et al. 2019

Proceedings of the 2019 Conference of the North

Self Cite

View full text Add to dashboard Cite

For many structured learning tasks, the data annotation process is complex and costly. Existing annotation schemes usually aim at acquiring completely annotated structures, under the common perception that partial structures are of low quality and could hurt the learning process. This paper questions this common perception, motivated by the fact that structures consist of interdependent sets of variables. Thus, given a fixed budget, partly annotating each structure may provide the same level of supervision, while allowing for more structures to be annotated. We provide an information theoretic formulation for this perspective and use it, in the context of three diverse structured learning tasks, to show that learning from partial structures can sometimes outperform learning from complete ones. Our findings may provide important insights into structured data annotation schemes and could support progress in learning protocols for structured tasks.

show abstract

A Multi-Axis Annotation Scheme for Event Temporal Relations

Cited by 108 publications

References 27 publications

CogCompTime: A Tool for Understanding Time in Natural Language

CogCompTime: A Tool for Understanding Time in Natural Language

“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding

Partial Or Complete, That’s The Question

Contact Info

Product

Resources

About