Did they answer? Subjective acts and intents in conversational discourse

Abstract

Discourse signals are often implicit, leaving it up to the interpreter to draw the required inferences. At the same time, discourse is embedded in a social context, meaning that interpreters apply their own assumptions and beliefs when resolving these inferences, leading to multiple, valid interpretations. However, current discourse data and frameworks ignore the social aspect, expecting only a single ground truth. We present the first discourse dataset with multiple and subjective interpretations of English conversation in the form of perceived conversation acts and intents. We carefully analyze our dataset and create computational models to (1) confirm our hypothesis that taking into account the bias of the interpreters leads to better predictions of the interpretations, (2) and show disagreements are nuanced and require a deeper understanding of the different contextual factors. We share our dataset and code at http://github.com/ elisaF/subjective_discourse.

Analyzing Dataset Annotation Quality Management in the Wild

“…The filtering can, for instance, be based on expert judgment or if there is no majority agreement (Bastan et al 2020). Sometimes, measuring the time it takes for annotators to process instances and filter out annotations with improbably high annotation times might also be helpful (Ferracane et al 2021).…”

Learning with Different Amounts of Annotation: From Zero to Many Labels

“…Rich prior work studies ambiguity in language interpretations (Aroyo and Welty, 2015). A few studies (Passonneau et al, 2012;Ferracane et al, 2021) frame diverging, subjective interpretations as a multi label classification, and few studies Zhang et al, 2017;Chen et al, 2020b) introduce graded human responses. Mayhew et al (2020) studies training machine translation system with the goal of generating diverse set of reference translations.…”

“…Even after thorough quality control, it is often infeasible to reach complete annotator agreement, as annotators make mistakes (Freitag et al, 2021) and ambiguity is a key feature of human communication (Asher and Lascarides, 2005). Rich prior works (Passonneau et al, 2012;Pavlick and Kwiatkowski, 2019;Nie et al, 2020;Min et al, 2020;Ferracane et al, 2021) show 1 Code and data split is available at https://github. com/szhang42/Uneven_training_data.…”

“…Despite observing such inherent ambiguity, most work have not embraced ambiguity into the training procedure. Most existing datasets (Wang et al, 2019;Rajpurkar et al, 2016) present a single label per each training example while collecting multiple labels for examples in the evaluation set, with a few notable exceptions on subjective tasks (Passonneau et al, 2012;Ferracane et al, 2021). We challenge this paradigm and re-distribute annotation budget unevenly among training examples, generating small amount of training examples with multiple labels.…”

ALLSH: Active Learning Guided by Local Sensitivity and Hardness

“…Crowdsourcing annotations (Rajpurkar et al, 2016;Bowman et al, 2015) has become a common practice for developing NLP benchmark datasets. Rich prior works (Pavlick and Kwiatkowski, 2019;Nie et al, 2020;Ferracane et al, 2021) show that the time-consuming and expensive manual labeling in crowdsourcing annotations are not an annotation artifact but rather core linguistic phenomena. Active Learning (AL) is introduced to efficiently acquire data for annotation from a (typically large) pool of unlabeled data.…”