Inherent Disagreements in Human Textual Inferences

Abstract

We analyze human’s disagreements about the validity of natural language inferences. We show that, very often, disagreements are not dismissible as annotation “noise”, but rather persist as we collect more ratings and as we vary the amount of context provided to raters. We further show that the type of uncertainty captured by current state-of-the-art models for natural language inference is not reflective of the type of uncertainty present in human disagreements. We discuss implications of our results in relation to the recognizing textual entailment (RTE)/natural language inference (NLI) task. We argue for a refined evaluation objective that requires models to explicitly capture the full distribution of plausible human judgments.

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification

“…In this case, a piece of evidence contradicts a relative clause in the claim but does not refute the entire claim. Similar problems regarding the uncertainty of NLI tasks have been pointed out in previous works (Zaenen et al, 2005;Pavlick and Kwiatkowski, 2019;Chen et al, 2020a).…”

“…However, we find that the decision between RE-FUTED and NOTENOUGHINFO can be ambiguous in many-hop claims and even the high-quality, trained annotators from Appen, instead of Mturk, cannot consistently choose the correct label from these two classes. Recent works (Pavlick and Kwiatkowski, 2019;Chen et al, 2020a) have raised concern over the uncertainty of NLI tasks with categorical labels and proposed to shift to a probabilistic scale. Since this work is mainly targeting the many-hop retrieval, we combine the REFUTED and NOTENOUGHINFO into a single class, namely NOT-SUPPORTED.…”

The Extraordinary Failure of Complement Coercion Crowdsourcing

“…In their study, annotators had to select the degree to which a premise entails a hypothesis, on a scale (Chen et al, 2020) (instead of discrete labels). Pavlick and Kwiatkowski (2019) show that even though these datasets are reported to have high agreement scores, specific examples suffer from inherent disagreements. For instance, in about 20% of the inspected examples, "there is a nontrivial second component" (e.g.…”

“…Our findings are related to theirs, although not identical: while the disagreements they report are due to the individuals' interpretations of a situation, in our case, disagreements are due to the difficulty in imagining a different scenario. While some works propose to collect annotator disagreements and use them as inputs (Plank et al, 2014;Palomaki et al, 2018) (see Pavlick and Kwiatkowski (2019) for an elaborated overview), this will not hold in our case, because only one of the labels is typically correct.…”

Social Media Attributions in the Context of Water Crisis

“…Next, disagreements are resolved through a followup adjudication process. Following (Pavlick and Kwiatkowski, 2019), we surface any inherent ambiguity/disagreement between annotators in the final set of labels. Even after the adjudication process, if raters fail to resolve (say, rater 1 sticks to attribute a and rater 2 and rater 3 stick to attribute b), we propagate {a, b} as the final label (accounting for 2.4% of the non-singleton labels).…”