Who Has the Last Word? Understanding How to Sample Online Discussions

Abstract

In online debates, as in offline ones, individual utterances or arguments support or attack each other, leading to some subset of arguments (potentially from different sides of the debate) being considered more relevant than others. However, online conversations are much larger in scale than offline ones, with often hundreds of thousands of users weighing in, collaboratively forming large trees of comments by starting from an original post and replying to each other. In large discussions, readers are often forced to sample a subset of the arguments being put forth. Since such sampling is rarely done in a principled manner, users may not read all the relevant arguments to get a full picture of the debate from a sample. This article is interested in answering the question of how users should sample online conversations to selectively favour the currently justified or accepted positions in the debate. We apply techniques from argumentation theory and complex networks to build a model that predicts the probabilities of the normatively justified arguments given their location in idealised online discussions of comments and replies, which we represent as trees. Our model shows that the proportion of replies that are supportive, the distribution of the number of replies that comments receive, and the locations of comments that do not receive replies (i.e., the “leaves” of the reply tree) all determine the probability that a comment is a justified argument given its location. We show that when the distribution of the number of replies is homogeneous along the tree length, for acrimonious discussions (with more attacking comments than supportive ones), the distribution of justified arguments depends on the parity of the tree level, which is the distance from the root expressed as number of edges. In supportive discussions, which have more supportive comments than attacks, the probability of having justified comments increases as one moves away from the root. For discussion trees that have a non-homogeneous in-degree distribution, for supportive discussions we observe the same behaviour as before, while for acrimonious discussions we cannot observe the same parity-based distribution. This is verified with data obtained from the online debating platform Kialo. By predicting the locations of the justified arguments in reply trees, we can therefore suggest which arguments readers should sample, to grasp the currently accepted opinions in such discussions. Our models have important implications for the design of future online debating platforms.

GraphNLI: A Graph-based Natural Language Inference Model for Polarity Prediction in Online Debates

“…Once the polarity of posts is known, it can be used to develop conversation health metrics such as whether a conversation is supporting or acrimonious (e.g., cf. [10]). Hate speech: Hate speech on online forums is a common [15] challenge, including in nationally important conversations between citizens and their elected representatives [1].…”

“…Previous work has also looked at how the conclusions of a logical reader can change depending on which parts of a debate they read, thus underscoring the dangers of sampling only parts of a large online debate [41][42][43]. Other work has shown how the location of justified arguments can be significantly influenced by whether the debate is acrimonious or supporting [10]. BAFs are thus useful representational tools for modelling online debates, allowing the application of both argument-theoretic and graph-theoretic ideas to gain insights about online discussions.…”

“…Kialo is an online debating platform that helps people "engage in thoughtful discussion, understand different points of view, and help with collaborative decision-making". 6 In this study, we use data from discussions hosted on the Kialo debating platform as used by [10]. Figure 1 illustrates an example Kialo discussion.…”

“…The ability to accurately predict the polarity of replies in an online debate can allow us to measure properties of the debate, such as how "controversial" a discussion is, e.g. by counting the number of supporting vs. attacking replies [10]. Perhaps more importantly, if the polarity is known, we can then use techniques from argumentation theory, a branch of artificial intelligence concerned with the formal representation and resolution of disagreements [37], to formally compute which arguments have been attacked and rebutted, and which ones stand unrebutted or are further justified by additional supporting replies.…”

Reason against the machine? Future directions for mass online deliberation

“…Furthermore, an assumption that unrefuted arguments are winning may unduly influence which arguments are of high quality. Even if no refuting reply is present, it does not necessarily mean that the argument is sound (Boschi et al, 2021). These issues suggest that using argument maps may potentially reduce the overall quality of deliberation unless designs are modified to rectify them.…”