A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation

Poesio, Massimo; Chamberlain, Jon; Paun, Silviu; Yu, Juntao; Uma, Alexandra; Kruschwitz, Udo

doi:10.18653/v1/n19-1176

Cited by 35 publications

(50 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior work examines how writer intentions are often misaligned with reader perceptions (Chang et al, 2020), which further motivates our focus on the reader (our annotator). While our work focuses on subjectivity, ambiguity is studied in many NLP tasks, including Natural Language Inference (Pavlick and Kwiatkowski, 2019;Nie et al, 2020), evaluation of NLG (Schoch et al, 2020), a recent SemEval 2021 shared task, 1 as well as several discourse tasks (Asher and Lascarides, 2003;Versley, 2011;Webber and Joshi, 2012;Das et al, 2017;Poesio et al, 2019;Webber et al, 2019). Only one study strives to understand how these ambiguities are resolved: Scholman (2019) shows different interpretations of ambiguous coherence relations can be attributable to different cognitive biases.…”

Section: Zuckerbergmentioning

confidence: 99%

“…Discourse, like many uses of language, has inherent ambiguity, meaning it can have multiple, valid interpretations. Much work has focused on characterizing these "genuine disagreements" (Asher and Lascarides, 2003;Das et al, 2017;Poesio et al, 2019;Webber et al, 2019) and incorporating their uncertainty through concurrent labels (Rohde et al, 2018) and underspecified structures (Hanneforth et al, 2003). However, prior work does not examine the subjectivity of discourse: how you resolve an ambiguity by applying your personal beliefs and preferences.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Did they answer? Subjective acts and intents in conversational discourse

Ferracane¹,

Durrett²,

Li³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Discourse signals are often implicit, leaving it up to the interpreter to draw the required inferences. At the same time, discourse is embedded in a social context, meaning that interpreters apply their own assumptions and beliefs when resolving these inferences, leading to multiple, valid interpretations. However, current discourse data and frameworks ignore the social aspect, expecting only a single ground truth. We present the first discourse dataset with multiple and subjective interpretations of English conversation in the form of perceived conversation acts and intents. We carefully analyze our dataset and create computational models to (1) confirm our hypothesis that taking into account the bias of the interpreters leads to better predictions of the interpretations, (2) and show disagreements are nuanced and require a deeper understanding of the different contextual factors. We share our dataset and code at http://github.com/ elisaF/subjective_discourse.

show abstract

Section: Zuckerbergmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Did they answer? Subjective acts and intents in conversational discourse

Ferracane¹,

Durrett²,

Li³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…The PD corpus was created using the Phrase Detectives game, whose players are asked to find the antecedent/split-antecedents closest to the mention in question (Poesio et al, 2019). The corpus comes with all raw annotations and silver labels aggregated using the Mention-Pair Annotation model (Paun et al, 2018).…”

Section: Auxiliary Corporamentioning

confidence: 99%

“…We evaluated four different augmentation settings. Two of these involve using additional examples of split antecedent anaphora recoverable from the crowdsourced Phrase Detectives corpus (PD) (Poesio et al, 2019), a corpus of anaphoric annotations, including split-antecedent anaphors, collected using the Phrase Detectives game. 2 The corpus includes both raw annotations and silver labels aggregated using the Mention Pair Annotations model (Paun et al, 2018).…”

mentioning

confidence: 99%

Free the Plural: Unrestricted Split-Antecedent Anaphora Resolution

Moosavi

Paun

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Now that the performance of coreference resolvers on the simpler forms of anaphoric reference has greatly improved, more attention is devoted to more complex aspects of anaphora. One limitation of virtually all coreference resolution models is the focus on single-antecedent anaphors. Plural anaphors with multiple antecedents-so-called split-antecedent anaphors (as in John met Mary. They went to the movies)-have not been widely studied, because they are not annotated in ONTONOTES and are relatively infrequent in other corpora. In this paper, we introduce the first model for unrestricted resolution of split-antecedent anaphors. We start with a strong baseline enhanced by BERT embeddings, and show that we can substantially improve its performance by addressing the sparsity issue. To do this, we experiment with auxiliary corpora where split-antecedent anaphors were annotated by the crowd, and with transfer learning models using element-of bridging references and single-antecedent coreference as auxiliary tasks. Evaluation on the gold annotated ARRAU corpus shows that the out best model uses a combination of three auxiliary corpora achieved F1 scores of 70% and 43.6% when evaluated in a lenient and strict setting, respectively, i.e., 11 and 21 percentage points gain when compared with our baseline. 1(1)The Joneses i went to the park. They i had a good time.(2) John and Mary i went to the park. They i had a good time.(3) John i met Mary j in the park. They i,j had a good chat .(4) John likes green i , Mary likes blue j , but Tom likes both colours i,j .

show abstract

“…Recent annotation studies recognize that ambiguity, vagueness and varying degrees of difficulty are inherent to semantic phenomena Erk et al, 2003;Kairam and Heer, 2016;Poesio et al, 2019;Pavlick and Kwiatkowski, 2019). Pavlick and Kwiatkowski (2019) demonstrate that the fundamental task of Natural Language Inferencing contains large proportions of instances with multiple valid interpretations and argue that this phenomenon is central to the task rather than an aspect which can be disregarded.…”

Section: Related Workmentioning

confidence: 99%

Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement

Sommerauer¹,

Fokkens²,

Vossen³

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Semantic annotation tasks contain ambiguity and vagueness and require varying degrees of world knowledge. Disagreement is an important indication of these phenomena. Most traditional evaluation methods, however, critically hinge upon the notion of inter-annotator agreement. While alternative frameworks have been proposed, they do not move beyond agreement as the most important indicator of quality. Critically, evaluations usually do not distinguish between instances in which agreement is expected and instances in which disagreement is not only valid but desired because it captures the linguistic and cognitive phenomena in the data. We attempt to overcome these limitations using the example of a dataset that provides semantic representations for diagnostic experiments on language models. Ambiguity, vagueness, and difficulty are not only highly relevant for this use-case, but also play an important role in other types of semantic annotation tasks. We establish an additional, agreement-independent quality metric based on answer-coherence and evaluate it in comparison to existing metrics. We compare against a gold standard and evaluate on expected disagreement. Despite generally low agreement, annotations follow expected behavior and have high accuracy when selected based on coherence. We show that combining different quality metrics enables a more comprehensive evaluation than relying exclusively on agreement.

show abstract

A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation

Cited by 35 publications

References 41 publications

Did they answer? Subjective acts and intents in conversational discourse

Did they answer? Subjective acts and intents in conversational discourse

Free the Plural: Unrestricted Split-Antecedent Anaphora Resolution

Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement

Contact Info

Product

Resources

About