Empirical Study on Human Evaluation of Complex Argumentation Frameworks

Abstract

In abstract argumentation, multiple argumentation semantics have been proposed that allow to select sets of jointly acceptable arguments from a given argumentation framework, i.e. based only on the attack relation between arguments. The existence of multiple argumentation semantics raises the question which of these semantics predicts best how humans evaluate arguments. Previous empirical cognitive studies that have tested how humans evaluate sets of arguments depending on the attack relation between them have been limited to a small set of very simple argumentation frameworks, so that some semantics studied in the literature could not be meaningfully distinguished by these studies. In this paper we report on an empirical cognitive study that overcomes these limitations by taking into consideration twelve argumentation frameworks of three to eight arguments each. These argumentation frameworks were mostly more complex than the argumentation frameworks considered in previous studies. All twelve argumentation framework were systematically instantiated with natural language arguments based on a certain fictional scenario, and participants were shown both the natural language arguments and a graphical depiction of the attack relation between them. Our data shows that grounded and CF2 semantics were the best predictors of human argument evaluation. A detailed analysis revealed that part of the participants chose a cognitively simpler strategy that is predicted very well by grounded semantics, while another part of the participants chose a cognitively more demanding strategy that is mostly predicted well by CF2 semantics. Argumentation Framework 11Argument W (48): Islander Walter says that islander Xavier is not trustworthy and that there is a treasure buried between the ponds. So we should not trust what Xavier says, and we should dig up the sand between the ponds. Argument X (49): Islander Xavier says that islander Yanis and islander Zoe are not trustworthy and that there is a treasure buried next to the temple. So we should not trust what Yanis and Zoe say, and we should dig up the sand next to the temple. Argument Y (50):Islander Yanis says that islander Walter and islander Zoe are not trustworthy and that there is a treasure buried to the east of the lake. So we should not trust what Walter and Zoe say, and we should dig up the sand to the east of the lake. Argument Z (51):Islander Zoe says that islander Anna is not trustworthy, and that there is a treasure buried between the two highest mountains. So we should not trust what Anna says, and we should dig up the sand between the two highest mountains.Argument A (52): Islander Anna says that there is a treasure buried to the west of the lake. So we should dig up the sand to the west of the lake. Argumentation Framework 12Argument C (53): Islander Charlie says that islander Dorothy is not trustworthy and that there is a treasure buried to the south of the high mountain. So we should not trust what Dorothy says, and we should dig up the sand to the south of the high m...

SCF2 -- an Argumentation Semantics for Rational Human Judgments on Argument Acceptability: Technical Report

“…Finally, we discuss findings of a recent cognitive study by Cramer and Guillaume [13] whose results suggest that SCF2 is more in line with the judgments of participants than any existing semantics. So our hypothesis that SCF2 corresponds well to what humans consider a rational judgment on the acceptability of arguments is motivated not only by theoretical but also by empirical observations.…”

“…While the argumentation frameworks used in Rahwan et al's studies could not distinguish between preferred semantics and naive-based semantics like CF2, two more recent studies by Cramer and Guillaume [12,13] address this issue. Both of these studies made use of a group discussion methodology that is known to stimulate more rational thinking.…”

“…Two recent empirical cognitive studies on argumentation semantics by Cramer and Guillaume [12,13] showed CF2 to be better predictors of human argument evaluation than admissibility-based semantics like grounded and preferred. This finding sheds some doubt on principles that are only satisfied by admissibilitybased semantics, e.g.…”

Explanations of Non-monotonic Inference in Admissibility-Based Abstract Argumentation

“…Let us highlight that at the current stage, we consider an HCI evaluation premature. The behavior of Dung-style admissible set-based semantics is in many cases counter-intuitive (see [9], and consider, for instance, the argumentation framework AF = ({a, b}, {(a, a), (a, b)}) and preferred semantics); hence, an HCI study is likely to yield rather "noisy" results, considering that technically correct explanation of counterintuitive semantics behavior may confuse study participants.…”

Computational Argumentation and Cognition

“…The formalism would need a high degree of flexibility to allow dynamic changes and the revision of drawn conclusions. A candidate such framework is that of (dialectic) argumentation as advocated in Philosophy and Cognitive Psychology in early work [25,26,27,32,38] and newly supported by many recent studies connecting argumentation with a wide spectrum of different aspects of human reasoning [3,4,5,9,10,11,12,16,19,20,23,24,33,35,36,37,39], including such studies within the context of AI 1 [1,2,6,7,8,13,14,15,17,18,21,22,28,29,30,31,34]. The main premise is that argumentation is native to human reasoning and can provide a foundation for (higher-level) cognition and, in turn, for human-centric AI.…”