Judgment aggregation, discursive dilemma and reflective equilibrium: Neural language models as self-improving doxastic agents

Abstract

Neural language models (NLMs) are susceptible to producing inconsistent output. This paper proposes a new diagnosis as well as a novel remedy for NLMs' incoherence. We train NLMs on synthetic text corpora that are created by simulating text production in a society. For diagnostic purposes, we explicitly model the individual belief systems of artificial agents (authors) who produce corpus texts. NLMs, trained on those texts, can be shown to aggregate the judgments of individual authors during pre-training according to sentence-wise vote ratios (roughly, reporting frequencies), which inevitably leads to so-called discursive dilemmas: aggregate judgments are inconsistent even though all individual belief states are consistent. As a remedy for such inconsistencies, we develop a self-training procedure—inspired by the concept of reflective equilibrium—that effectively reduces the extent of logical incoherence in a model's belief system, corrects global mis-confidence, and eventually allows the model to settle on a new, epistemically superior belief state. Thus, social choice theory helps to understand why NLMs are prone to produce inconsistencies; epistemology suggests how to get rid of them.

Probabilistic coherence, logical consistency, and Bayesian learning: Neural language models as epistemic agents

“…Self-training improves logical alignment of credences. It brings down, consistent with previous findings [ 61 ], the frequency of transitivity violations ( Fig 3 ) and improves further consistency metrics (see also S3 Fig ).…”

“…And in case self-training doesn’t improve a specific metric (such as logical alignment with entailment relation, reflexivity, or complementarity), we can explain the lack of improvement, namely with reference to low initial levels, or by means of data segmentation and in-depth analysis. The observed logical benefits of self-training are consistent with [ 61 , 73 ].…”

“…The artificial language described in the following, which has been used in computational experiments with NLMs before [ 61 ], unites efficient semantics with a sufficiently rich inferential structure. In particular, it has all the logical properties required to give rise to voting paradoxes in judgment aggregation [ 62 ], which, it has been argued [ 61 ], is a potential reason for why pretrained NLMs are prone to suffer from inconsistencies.…”

“…Self-training pre evidence introduction , i.e. phase 2 , consists in 300 training steps and proceeds—with the exception of data augmentation—in close analogy to self-training in [ 61 ]. Self-training is designed so as to mimic a local belief revision procedure that consists in (i) identifying some strongly held beliefs, (ii) spelling out inferential consequences of these beliefs, and (iii) reinforcing one’s beliefs in these consequences.…”

“…Third, the laws and mechanisms of belief formation in NLMs can (so far) best be studied in a fully-controlled experimental set-up, where NLMs are trained from scratch on synthetic corpora in a simple, artificial language (we will resort to a language for expressing linear orders). In these two latter regards, the paper follows [ 61 ].…”