QA-It: Classifying Non-Referential It for Question Answer Pairs

Abstract

This paper introduces a new corpus, QA-It, for the classification of non-referential it. Our dataset is unique in a sense that it is annotated on question answer pairs collected from multiple genres, useful for developing advanced QA systems. Our annotation scheme makes clear distinctions between 4 types of it, providing guidelines for many erroneous cases. Several statistical models are built for the classification of it, showing encouraging results. To the best of our knowledge, this is the first time that such a corpus is created for question answering.

Event versus entity co-reference: Effects of context and form of referring expression

“…If there are event-favoring properties of the context sentence that human participants are sensitive to, it is a tractable task to build automatic classifiers that learn to recognize such properties. This supports the idea that the task of differentiating anaphoric and pleonastic instances of It (Evans, 2001;Boyd et al, 2005;Bergsma and Yarowsky, 2011;Lee et al, 2016;Loáiciga et al, 2017) could potentially improve performance.…”

Classifying Referential and Non-referential It Using Gaze

“…In all three classifiers, the clause-anaphoric class was consistently predicted with lowest accuracy (Table 5), which is not surprising given that it only accounts for 11% of the retained data. In line with the observation of Lee et al (2016) (Section 2), the embeddings do not show a stable contribution. In our case, this is likely related to the small amount of data, to which we add 300 dimensions per word.…”

“…The inter-annotator agreement for the three categories was κ = 0.636, p < 0.0005, indicating substantial agreement between the annotators. This number corresponds to a percentage agreement of 77.47% for the three categories and is comparable to the 81% agreement reported in Lee et al (2016). We perceived adjudication between cases of disagreement (237 instances) to be extremely arbitrary, so those cases were excluded rather than resolved.…”

“…Annotation scheme: A total of 1,052 instances of it were found in the corpus 4 . Each of the instances of it was annotated by two annotators and assigned to one of three categories: Pleonastic, Nominal anaphoric and Clause anaphoric, following the scheme used by Lee et al (2016). The annotators were free to view as much of the previous text as necessary to decide on a label.…”

“…The majority of the machine learning approaches to classifying the pronoun it in different languages are based on linguistic features capturing token, syntactic and semantic context. Different papers report varying majority baseline metrics (between 50% and 75%) depending on the annotated corpora, and an improvement over the majority baselines of between 2% and 15% for classification of more than two classes of it (Loáiciga et al, 2017;Uryupina et al, 2016;Lee et al, 2016;Müller, 2006;Boyd et al, 2005;Hoste et al, 2007;Evans, 2001). For example, Loáiciga et al (2017) train a bidirectional recurrent neural network (RNN) to classify three classes of it in the ParCor corpus (Guillou et al, 2014) and compare its performance to a feature-based maximum entropy classifier.…”