The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

Abstract

Reasoning is a crucial part of natural language argumentation. To comprehend an argument, one must analyze its warrant, which explains why its claim follows from its premises. As arguments are highly contextualized, warrants are usually presupposed and left implicit. Thus, the comprehension does not only require language understanding and logic skills, but also depends on common sense. In this paper we develop a methodology for reconstructing warrants systematically. We operationalize it in a scalable crowdsourcing process, resulting in a freely licensed dataset with warrants for 2k authentic arguments from news comments. 1 On this basis, we present a new challenging task, the argument reasoning comprehension task. Given an argument with a claim and a premise, the goal is to choose the correct implicit warrant from two options. Both warrants are plausible and lexically close, but lead to contradicting claims. A solution to this task will define a substantial step towards automatic warrant reconstruction. However, experiments with several neural attention and language models reveal that current approaches do not suffice.

SemEval-2018 Task 12: The Argument Reasoning Comprehension Task

“…We summarize the main points from its construction process, which is described in detail in (Habernal et al, 2018).…”

“…We performed similar quality measures with reasonable agreement for the other crowdsourcing steps too. Details are given in (Habernal et al, 2018).…”

“…In addition, it removes much lexical and topical overlap. The same split has been used by Habernal et al (2018). The shared task had two phases, trial and test.…”

“…What is easy for humans in many cases, however, turns out to be hard for machines, because reasoning usually depends on context and common sense. In (Habernal et al, 2018), we have thus introduced the argument reasoning comprehension task in order to study the construction and identification of implicit warrants for natural language arguments. It forms the basis of the shared task presented here:…”

YNU-HPCC at SemEval-2018 Task 12: The Argument Reasoning Comprehension Task Using a Bi-directional LSTM with Attention Model

“…Because of the implicit relationship between claim and reason, this task is very difficult (Habernal et al, 2018 input, a sufficiently complex model is required. However, too little training data is not sufficient for the model to learn all of the features.…”