Pragmatic competence of pre-trained language models through the lens of discourse connectives

Abstract

As pre-trained language models (LMs) continue to dominate NLP, it is increasingly important that we understand the depth of language capabilities in these models. In this paper, we target pre-trained LMs' competence in pragmatics, with a focus on pragmatics relating to discourse connectives. We formulate cloze-style tests using a combination of naturally-occurring data and controlled inputs drawn from psycholinguistics. We focus on testing models' ability to use pragmatic cues to predict discourse connectives, models' ability to understand implicatures relating to connectives, and the extent to which models show humanlike preferences regarding temporal dynamics of connectives. We find that although models predict connectives reasonably well in the context of naturally-occurring data, when we control contexts to isolate high-level pragmatic cues, model sensitivity is much lower. Models also do not show substantial humanlike temporal preferences. Overall, the findings suggest that at present, dominant pretraining paradigms do not result in substantial pragmatic competence in our models.

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

“…More tradi- 2021) by taking a closer look at the internal workings of the selfattention component. Looking at prior work analyzing the amount of discourse information in PLMs, structures are solely explored through the use of proxy tasks, such as connective prediction (Pandia et al, 2021), relation classification (Kurfalı and Östling, 2021), and others (Koto et al, 2021a). However, despite the difficulties of encoding arbitrarily long documents, we believe that to systematically explore the relationship between PLMs and discourse, considering complete documents is imperative.…”

“…Besides their strong empirical results on most real-world problems, such as summarization (Zhang et al, 2020;Xiao et al, 2021a), questionanswering (Joshi et al, 2020;Oguz et al, 2021) and sentiment analysis (Adhikari et al, 2019;, uncovering what kind of linguistic knowledge is captured by this new type of pretrained language models (PLMs) has become a prominent question by itself. As part of this line of research, called BERTology (Rogers et al, 2020), researchers explore the amount of linguistic understanding encapsulated in PLMs, exposed through either external probing tasks (Raganato and Tiedemann, 2018;Zhu et al, 2020;Koto et al, 2021a) or unsupervised methods (Wu et al, 2020;Pandia et al, 2021). Previous work thereby either focuses on analyzing the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al (2020)), relations (Papanikolaou et al, 2019), ontologies (Michael et al, 2020) or, to a more limited extend, discourse related behaviour (Zhu et al, 2020;Koto et al, 2021a;Pandia et al, 2021).…”

“…As part of this line of research, called BERTology (Rogers et al, 2020), researchers explore the amount of linguistic understanding encapsulated in PLMs, exposed through either external probing tasks (Raganato and Tiedemann, 2018;Zhu et al, 2020;Koto et al, 2021a) or unsupervised methods (Wu et al, 2020;Pandia et al, 2021). Previous work thereby either focuses on analyzing the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al (2020)), relations (Papanikolaou et al, 2019), ontologies (Michael et al, 2020) or, to a more limited extend, discourse related behaviour (Zhu et al, 2020;Koto et al, 2021a;Pandia et al, 2021).…”

Pragmatic and Logical Inferences in NLI Systems: The Case of Conjunction Buttressing

“…To our knowledge, Pandia et al (2021) is the only NLP study dealing with conjuction buttressing: the authors tested if Transformer-based masked language models can predict the temporal connective corresponding to the correct interpretation of the enriched and, using the stimuli by Politzer-Ahles et al (2017). Unlike their study, we created and used labeled data for the evaluation of NLI systems, testing a pragmatic hypothesis (enriched interpretation of and) vs. a logical one (commutative interpretation).…”

Proceedings of the Second Workshop on Understanding Implicit and Underspecified Language

“…A lot of attention has been paid to increase LMs' general transparency (Ettinger, 2020;Rogers et al, 2020), among which studies on LMs' interpretation of implicitness mostly focus on scalar implicature or presupposition (Schuster et al, 2020;Jeretic et al, 2020;Pandia et al, 2021). To our knowledge, no studies in this line have been done on gradable adjectives' EVAL implicature, although EVAL and gradability are classic topics in context sensitivity.…”