A Crowdsourced Frame Disambiguation Corpus with Ambiguity

Abstract

We present a resource for the task of FrameNet semantic frame disambiguation of over 5,000 word-sentence pairs from the Wikipedia corpus. The annotations were collected using a novel crowdsourcing approach with multiple workers per sentence to capture interannotator disagreement. In contrast to the typical approach of attributing the best single frame to each word, we provide a list of frames with disagreement-based scores that express the confidence with which each frame applies to the word. This is based on the idea that inter-annotator disagreement is at least partly caused by ambiguity that is inherent to the text and frames. We have found many examples where the semantics of individual frames overlap sufficiently to make them acceptable alternatives for interpreting a sentence. We have argued that ignoring this ambiguity creates an overly arbitrary target for training and evaluating natural language processing systemsif humans cannot agree, why would we expect the correct answer from a machine to be any different? To process this data we also utilized an expanded lemma-set provided by the Framester system, which merges FN with WordNet to enhance coverage. Our dataset includes annotations of 1,000 sentence-word pairs whose lemmas are not part of FN. Finally we present metrics for evaluating frame disambiguation systems that account for ambiguity.

Inherent Disagreements in Human Textual Inferences

“…Similar approaches have been applied in parsing (Martínez Alonso et al, 2015) and supersense tagging (Martínez Alonso et al, 2016). Specifically relevant to this work is past discussion of disagreement on semantic annotation tasks, including anaphora resolution (Poesio and Artstein, 2005), coreference (Versley, 2008;Recasens et al, 2011), word sense disambiguation (Erk and McCarthy, 2009;Passonneau et al, 2012;Jurgens, 2013), veridicality (Geis andZwicky, 1971;Karttunen et al, 2014;de Marneffe et al, 2012), semantic frames (Dumitrache et al, 2019), and grounding (Reidsma and op den Akker, 2008).…”

Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future

“…These areas have developed specialized training and evaluation methods (Papineni et al, 2002;Lin, 2004). More surprisingly, disagreements in interpretation have been found to be frequent in annotation projects concerned with apparently more 'objective' aspects of language, such as coreference (Poesio and Artstein, 2005;Recasens et al, 2011), part-of-speech tagging , word sense disambiguation (Passonneau et al, 2012) and semantic role labelling (Dumitrache et al, 2019), to name a few examples. Even if in these tasks individual instances can be found to be reasonably objective, these findings appear to reflect the existence of extensive and systematic disagreement on what can be concluded from a natural language statement (Pavlick and Kwiatkowski, 2019).…”

A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels

“…A range of previous studies have explored methods of crowdsourcing SRL. Most work has focused on crowd-only workflows, with comparatively low accuracy or extensive worker training (Fossati et al, 2013;Feizabadi and Padó, 2014;Chang et al, 2015;Dumitrache et al, 2019;Hahm et al, 2020). This work guided our user interface designs and our understanding of challenges in SRL annotation.…”