Exploring Methodologies for Collecting High-Quality Implicit Reasoning in Arguments

Abstract

Annotation of implicit reasoning (i.e., warrant) in arguments is a critical resource to train models in gaining deeper understanding and correct interpretation of arguments. However, warrants are usually annotated in unstructured form, having no restriction on their lexical structure which sometimes makes it difficult to interpret how warrants relate to any of the information given in claim and premise. Moreover, assessing and determining better warrants from the large variety of reasoning patterns of unstructured warrants becomes a formidable task. Therefore, in order to annotate warrants in a more interpretative and restrictive way, we propose two methodologies to annotate warrants in a semi-structured form. To the best of our knowledge, we are the first to show how such semi-structured warrants can be annotated on a large scale via crowdsourcing. We demonstrate through extensive quality evaluation that our methodologies enable collecting better quality warrants in comparison to unstructured annotations. To further facilitate research towards the task of explicating warrants in arguments, we release our materials publicly (i.e., crowdsourcing guidelines and collected warrants).

Teach Me How to Argue: A Survey on NLP Feedback Systems in Argumentation

“…Large annotated corpora are required to improve implicit reasoning detection for models. To address this need, various studies have proposed methods for annotating implicit knowledge, leading to the development of multiple datasets (Becker et al, 2020;Singh et al, 2021. In Singh et al (2021), semi-structured warrants, i.e.…”

“…To address this need, various studies have proposed methods for annotating implicit knowledge, leading to the development of multiple datasets (Becker et al, 2020;Singh et al, 2021. In Singh et al (2021), semi-structured warrants, i.e. links between a claim and evidence (c.f.…”